Evaluate subevents #43

i-be-snek · 2024-07-18T06:46:46Z

This PR:

specific instances = subevents

matches subevents together from two lists (one generated by an LLM sys, and one from our gold annotations gold)
creates a "padded" pair with NoneType values for subevents when a match is missing
evaluates subevents separately using the evaluator script
adds tests for this new matcher class

TODO:

instead of looping through each specific instance, calculate all distances and then choose the top matches.

i-be-snek · 2024-07-24T12:39:36Z

README.md

@@ -146,6 +147,37 @@ poetry run python3 Evaluation/evaluator.py --sys-file  Database/output/nlp4clima
 --weights_config nlp4climate
 ```

+#### Evaluate sub events (ie. specific instances)


@MurathanKurfali

Includes subevents extracted from flat excel datatable

…gle None)

handle exceptions better, more informative error messages

…found for an event_id

…ched events)

Co-authored-by: Shorouq <[email protected]>

i-be-snek changed the title ~~DRAFT Evaluate subevents~~ DRAFT: Evaluate subevents Jul 18, 2024

i-be-snek force-pushed the evaluate-subevents branch from eed6722 to 5ba99f2 Compare July 22, 2024 13:52

i-be-snek changed the title ~~DRAFT: Evaluate subevents~~ Evaluate subevents Jul 24, 2024

i-be-snek commented Jul 24, 2024

View reviewed changes

i-be-snek added 26 commits July 30, 2024 14:58

✨ Add Event_ID decimal to UUID mapping

811da81

Includes subevents extracted from flat excel datatable

📦️ Convert flat datatable to specific instances with short uuid Event_ID

a0cefd2

📝 Fix docs + move to parent dir

9cd8321

🐛 Parse columns named "Damage", not ending with "_Damage"

9755dee

🚨 Fix lint warnings + formatting

9974bf4

♻️ Refactor location extraction return (tuple of empty lists, not sin…

9f2cbf7

…gle None)

➕ Add pytest deps + init

f2c196b

✨ Add minimal specific instance matcher

cb04871

✅ Add unit test for matcher

b30fc7c

✅ Add negative example to unit test

71d3901

🍻 Add unfinished code

04e0e57

⚡️ Calculate distance for matching subevents with Comparer

ba283d9

✨ Create matcher for specific instances

8c4b527

🥅 Fail when column name is unsupported

2eeec45

♻️ Fix imports

a88cac5

✅ Update similarity calculator test

e911065

✅ Update error handling for unsupported column names

2ed2ea8

✅ Add test for specific instance matching functionality

443ea01

🔥 Remove debug code

333a8a2

⚰️ Remove unnecessary del of var

5b7dee7

✅ Add example for testing matching and padding

2b3181f

✅ Update similarity calc test with clearer param names

6bc3eb6

🚨 Fix lint warning

43b9ceb

💬 Update column names -> shorter variation

91135e0

🐛 Preserve Event_ID when creating padding for specific instances

0c9cfa1

🔊 Add logging

6006616

i-be-snek added 26 commits July 30, 2024 15:00

⚡️ Fix module imports for Evaluation and Database/scr packages

ad75a53

➕ Add spacy[transformers] dep

7af86c0

🧑‍💻 Set pythonpath for pytest

f3ea9b3

🔧 Fix syntax error in pytest config

1b54090

🚧 WIP: adapt evaluation script to subevent evaluation

337fb3c

📝 Remove outdated todo boxes + lint

f5cb803

🐛 Fix small bugs in matcher

0530a20

handle exceptions better, more informative error messages

🧑‍💻 Improve error message informativeness & readability

a3c4c9d

🐛 Fix bug that creates excess event pairs

9a77711

✨ Specific instances sorted by Evnet_ID + num

71f331d

🐛 Handle bug in scoring similarity between specific instances

5058679

🔧 Reduce null penalty for comparer when matching specific instances

0df2762

✅ Update tests with threshold + null_penalty

34197b8

✅ Update test with None example (to test null penalty)

28e33cb

♻️ Allow passing null_penalty into matcher

5735055

🚨 Fix lint errors

dd937ef

✨ Support empty sys_list or gold_list when no specific instances are …

eca9b6d

…found for an event_id

✅ Update tests to cover cases of empty gold_list/sys_list

ce6769b

✨ Evaluate sub-events (specific instances)

abf1e37

📝 Update docs on how to evaluate subevents

ca3fbb3

✨ Add average error rate for specific instances grouped by event_id

75a0ad9

📝 Update docs with info on eval results for specific instances

026534b

📦️ Add dummy files for sys/gold for specific instance eval

6fa7fcb

🔥 Remove dead files

be57029

⚡️ Match events based on similarity matrix (not in linear order)

b4cd571

✅ Update expected test output (due to change in final indexing of mat…

94b9cd9

…ched events)

i-be-snek force-pushed the evaluate-subevents branch from 777d213 to 94b9cd9 Compare July 30, 2024 13:03

i-be-snek merged commit 6ab138d into main Jul 31, 2024
1 check passed

MurathanKurfali pushed a commit that referenced this pull request Jul 31, 2024

Evaluating subevents (#43)

601feab

Co-authored-by: Shorouq <[email protected]>

i-be-snek deleted the evaluate-subevents branch January 17, 2025 13:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Evaluate subevents #43

Evaluate subevents #43

i-be-snek commented Jul 18, 2024 •

edited

Loading

i-be-snek Jul 24, 2024

Evaluate subevents #43

Evaluate subevents #43

Conversation

i-be-snek commented Jul 18, 2024 • edited Loading

i-be-snek Jul 24, 2024

Choose a reason for hiding this comment

i-be-snek commented Jul 18, 2024 •

edited

Loading