Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Fdedup (simpler API) transform does not return a success/error code #957

Open
1 of 2 tasks
sujee opened this issue Jan 21, 2025 · 0 comments
Open
1 of 2 tasks
Assignees
Labels
bug Something isn't working

Comments

@sujee
Copy link
Contributor

sujee commented Jan 21, 2025

Search before asking

  • I searched the issues and found no similar issues.

Component

Transforms/universal/fdedup

What happened + What you expected to happen

From here : https://github.com/IBM/data-prep-kit/blob/dev/transforms/universal/fdedup/fdedup_python.ipynb

result = Fdedup(input_folder='test-data/input',
  output_folder='output',
  contents_column= "contents",
  document_id_column= "int_id_column",
  num_permutations= 112,
  num_bands= 14,
  num_minhashes_per_band= 8,
  operation_mode="filter_duplicates").transform()

transform should return 0 for success or any error code

Reproduction script

https://github.com/IBM/data-prep-kit/blob/dev/transforms/universal/fdedup/fdedup_python.ipynb

Anything else

No response

OS

Ubuntu

Python

3.11.x

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!
@sujee sujee added the bug Something isn't working label Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants