python text pipeline not operational

[heijligers]

Looks like the text_pipeline is not operational.
i'm trying to use scan and replace,
but the replace expects annotations of type AnnotationResults,
But it's getting normal strings or empty lists from the annotator

Error in run_text_pipeline_sync: 'str' object has no attribute 'start'
Traceback (most recent call last):
  File "/Users/datascience/PycharmProjects/FF/Topologix/src/SQLModel_ff_db.py", line 2705, in <module>
    replaced_text = client.run_text_pipeline_sync([content])[0]
  File "/Users/datascience/PycharmProjects/FF/Topologix/.venv/lib/python3.10/site-packages/datafog/main.py", line 200, in run_text_pipeline_sync
    return [
  File "/Users/datascience/PycharmProjects/FF/Topologix/.venv/lib/python3.10/site-packages/datafog/main.py", line 201, in <listcomp>
    self.anonymizer.anonymize(text, annotations).anonymized_text
  File "/Users/datascience/PycharmProjects/FF/Topologix/.venv/lib/python3.10/site-packages/datafog/models/anonymizer.py", line 56, in anonymize
    return self.replace_pii(text, annotations)
  File "/Users/datascience/PycharmProjects/FF/Topologix/.venv/lib/python3.10/site-packages/datafog/models/anonymizer.py", line 67, in replace_pii
    for annotation in sorted(annotations, key=lambda x: x.start, reverse=True):
  File "/Users/datascience/PycharmProjects/FF/Topologix/.venv/lib/python3.10/site-packages/datafog/models/anonymizer.py", line 67, in <lambda>
    for annotation in sorted(annotations, key=lambda x: x.start, reverse=True):
AttributeError: 'str' object has no attribute 'start'

[sidmohan0]

Hey @heijligers - thanks for flagging - I'll check this out myself later today and follow up

[sidmohan0]

do you mind sharing the datafog python version that you're using @heijligers?

[sidmohan0]

Hi @heijligers !

I took a look - there was a mismatch between our README instructions and expected functionality:
so as documented:

client = DataFog(operations=[OperationType.SCAN, OperationType.REDACT])
redacted_text = client.run_text_pipeline_sync([text])[0]

returns your error. I noted that the CLI commands did still work but the issue seems to be with the Python SDK.

With the existing version (4.0.0) you can correctly run it in the python SDK by doing this:

from datafog.models.spacy_nlp import SpacyAnnotator
from datafog.models.anonymizer import Anonymizer, AnonymizerType

annotator = SpacyAnnotator()
anonymizer = Anonymizer(anonymizer_type=AnonymizerType.REDACT)

this is a mismatch in expected functionality so we'll put out a fix for this but update the README in the meantime with a warning.  Thanks for bringing to our attention! 

Edit: let me know if you have any other questions and I'll mark this closed upon next release

[heijligers]

Hi Sidmohan0
thanks for the feedback.
I rewrote some of the functionality to be consistent with what I expected. I could try to make a PR available, but right now Im not actively using my workflow. This might change in the future.
I was very grateful for you having shared this library. It took me only an hour to get it working as I needed.

Keep it up.