Checks for erroneous data; new data types; reduced the number of photos needed to trigger a shorter timespan

[Tadusko]

Hi! Sorry that this is bit of a mess, since these changes came over time as I kept working on it locally.

This branch has multiple changes:

1. Two checks for input that seemed to crash threads (in my runs, the threads were dying over time until none were left).

• Some titles and descriptions included this null character '\x00', which postgres did not like. Titles and descriptions are now checked for it and it's removed (clean_string)
• Sometimes, the result object had no photos. That's now checked for now and that execution is broken if there's nothing. if "photos" not in results or "photo" not in results["photos"]:. I don't know if this is the best approach.

2. Added three new columns for the photo object that seemed useful (or, that I needed at least.)

• Tags: tags added by the user (string)
• License: the user-defined license, if any (int, 1–10 explanations here)
• Accuracy: geographic scale of the photo (int, 1–16 rough scale here

3. Reduced the number of photos needed to trigger DownloadBatchIsTooLargeError from 4000 to 3000. A more conservative estimate, may return more photos.

[christophfink]

Hi @Tadusko,

this looks kinda good, I have a few suggestions, though:

1. If you change the database schema (adding columns, for instance), add a transition mechanism. I once wrote something like that for twitter history, see here: gitlab.com/christophfink/twitterhistory/.../databaseschemaupdater.py You can copy that file and simply adapt it to flickrhistory (e.g., redefine SCHEMA_UPDATES), and call it from somewhere sensible (for twitterhistory, that‘s here)
2. Why do you have the conditional in the clean_string() def? (... if isinstance(input_string, str) else input_string) - is there any case when the keys exist but something else than a string is returned? If so, would catching an additional TypeError be more pythonic?
3. Could you implement the license and accuracy tags using a lookup-table and foreign keys there? So that it is self-documenting?
4. the 4000 limit was documented somewhere, but now when I google for it, I only find second-hand references that read ‘about 4000 photos‘, so lowering the limit definitely makes sense
5. I ran the black linter to check whether it would be huge changes, but they were tiny formatting things, so I simply pushed them to your repo - hope that‘s ok :)

[christophfink]

See my comments in this reply: #13 (comment)

[Tadusko]

Hi Chris! Thank you for reviewing this and the comments! I'll get back to it soon – I need to dig a bit to check what were the exact errors I got & why I ended up with the current checks (ChatGPT assisted fixes, for full disclosure).

[christophfink]

@Tadusko sorry for making so massive changes, in the end - once you start touching one thing, a million other things pop up

This is now working quite fine, I still would like to add another worker to fetch missing tags, license, and geo-accuracy information for existing photos

Addressed all of my own concerns, requesting @Tadusko‘s review

[christophfink]

@Tadusko I took all your changes in, but restructured a lot of things in general. I‘m going to merge this and release a dev-release. Could you then test it on a ‘real’ installation? (I‘ll write you in Slack about it)