Generalize ingestion status to a string, and mark/skip failed ingestions #162
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changes the schema for IngestedSources, adding a text column 'status' that can describe whether indexing of a source_id succeeded, failed, or resulted in some other status. Well-known values are 'ingested', 'failed', but using
mark_source_ingested()it can be set to any other string value, and the status can be retrieved usingget_source_status(). The existingis_source_ingested()returns True only if the status field has the exact value 'ingested'.The behavior of
add_messages_with_indexing()changes subtly: it now sets the status to 'failed' before attempting any work; it sets it to 'ingested' once it is done indexing.To alter a precious existing database, do this using sqlite3:
That sets the status for every row present to 'ingested', matching the original behavior.