Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup dag #118

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Speedup dag #118

wants to merge 14 commits into from

Conversation

ggmarshall
Copy link
Collaborator

@ggmarshall ggmarshall commented Mar 13, 2025

more dag efficencies

Copy link

codecov bot commented Mar 13, 2025

Codecov Report

Attention: Patch coverage is 60.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 42.01%. Comparing base (b1bbe44) to head (29fae09).

Files with missing lines Patch % Lines
workflow/src/legenddataflow/pars_loading.py 57.14% 3 Missing ⚠️
workflow/src/legenddataflow/execenv.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #118      +/-   ##
==========================================
- Coverage   42.10%   42.01%   -0.09%     
==========================================
  Files          19       18       -1     
  Lines        1209     1202       -7     
==========================================
- Hits          509      505       -4     
+ Misses        700      697       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gipert
Copy link
Member

gipert commented Mar 14, 2025

> snakemake --workflow-profile workflow/profiles/lngs -n 'all-l200-p13-r003-evt.gen'
Using workflow specific profile workflow/profiles/lngs for setting default command line arguments.
ValueError in file /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/Snakefile, line 47:
Can't run Props.read_from on sources-value of type <class 'pathlib.PosixPath'>
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/Snakefile", line 47, in <module>
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/src/legenddataflow/cal_grouping.py", line 21, in __init__
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/.venv/lib/python3.12/site-packages/dbetto/catalog.py", line 269, in read_from
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/.venv/lib/python3.12/site-packages/dbetto/catalog.py", line 267, in read_impl

@gipert
Copy link
Member

gipert commented Mar 14, 2025

can we try to add something to the test suite to catch these problems?

@ggmarshall
Copy link
Collaborator Author

hmm i dont get this error

@ggmarshall
Copy link
Collaborator Author

its a dbetto version issue, we require >= 1.2.3 which is correct

@ggmarshall ggmarshall force-pushed the speedup_dag branch 2 times, most recently from d69a8e6 to efbce87 Compare March 17, 2025 03:02
@ggmarshall
Copy link
Collaborator Author

this needs testing that passing the table_maps as json strings works as intended

@gipert
Copy link
Member

gipert commented Mar 17, 2025

its a dbetto version issue, we require >= 1.2.3 which is correct

you mean i have an old one?

@ggmarshall
Copy link
Collaborator Author

i believe so

@gipert
Copy link
Member

gipert commented Mar 17, 2025

boh

InputFunctionException in rule build_dsp in file /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/rules/dsp.smk, line 37:
Error:
  AttributeError: 'Catalog' object has no attribute 'channelmap'
Wildcards:
  datatype=anp
  period=p13
  run=r003
  experiment=l200
  timestamp=20241217T094846Z
Traceback:
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/rules/dsp.smk", line 45, in <lambda>
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/src/legenddataflow/scripts/table_name.py", line 34, in get_table_mapping (rule build_dsp, line 37, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/rules/dsp.smk)

@ggmarshall
Copy link
Collaborator Author

I refer you to my previous statement of this needs testing

@ggmarshall
Copy link
Collaborator Author

ok should be good now

@gipert
Copy link
Member

gipert commented Mar 18, 2025

still

InputFunctionException in rule build_evt in file /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/rules/evt.smk, line 15:
Error:
  TypeError: 'NoneType' object is not iterable
Wildcards:
  datatype=anp
  period=p13
  run=r003
  experiment=l200
  timestamp=20241217T094846Z
Traceback:
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/rules/evt.smk", line 25, in <lambda>
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/src/legenddataflow/pars_loading.py", line 105, in get_par_file (rule build_evt, line 15, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.2/workflow/rules/evt.smk)

I don't have time now but we should figure out why the ci tests pass...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants