Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

81 level of information evalutation #90

Merged
merged 46 commits into from
Sep 18, 2024
Merged

81 level of information evalutation #90

merged 46 commits into from
Sep 18, 2024

Conversation

i-be-snek
Copy link
Collaborator

@i-be-snek i-be-snek commented Aug 30, 2024 •
edited
Loading

See #81

Definition of done:

  • Commands to evaluate:
    • l1
    • l2 monetary
    • l3 monetary
    • l2 numerical
    • l3 numerical
  • Commands to append the database
    • l1
    • l2
    • l3
  • Bash script to parse, evaluate, and insert data from llm output (full working example)

an additional requirement:

  • The database size is reduced and GeoJson objects are not duplicated, but rather represented by a nid (uid)

@i-be-snek i-be-snek linked an issue Aug 30, 2024 that may be closed by this pull request
Level of information: introducing L1-L3 (3 PRs) #81
Closed
13 tasks
@i-be-snek i-be-snek changed the base branch from main to 81-level-of-information-dev September 5, 2024 06:30
@i-be-snek i-be-snek force-pushed the 81-level-of-information-evalutation branch from a774ab5 to 87486b1 Compare September 8, 2024 13:13
@i-be-snek i-be-snek force-pushed the 81-level-of-information-dev branch from 91e533f to 9f9bcd7 Compare September 8, 2024 13:13
@i-be-snek i-be-snek self-assigned this Sep 8, 2024
@i-be-snek i-be-snek closed this Sep 8, 2024
@i-be-snek i-be-snek reopened this Sep 8, 2024
@i-be-snek
Copy link
Collaborator Author

i-be-snek commented Sep 8, 2024 •
edited
Loading

TODO:

  • adapt db insertion scripts of l2+l3
  • make sure db insertion scripts work for l1
  • float("nan") instead of using numpy
  • test l1 evaluation script
  • bash command to evaluate l2
  • bash command to evaluate l3
  • double check all column names
  • LEFTOVER: evaluate based on available currency
  • get fresh data pull for the dev set
  • README: add bash script for dummies
  • README: bash script for parsing + evaluating + inserting the dev set
  • README: one last sweep of all READMEs in the repo to make sure they are up to date
  • request reviews when the PR is ready
  • update weights
  • "Event_Names" or "Event_Name"? Not evaluated anyway
  • De-duplicate GeoJson blobs in db
  • open new PR for leftovers
  • Admin Area evaluation gives a score of 1: data type mismatch?

PROBLEMS: All commented out categories in weights.py are not properly working.

  • Monetary impact types do not yet have evaluation by the correct currency implemented
  • Lists of areas have problems with data type mismatches -- could be the code or the llm output

@i-be-snek i-be-snek force-pushed the 81-level-of-information-evalutation branch from 977e255 to d5cae0e Compare September 16, 2024 12:45
@i-be-snek i-be-snek force-pushed the 81-level-of-information-dev branch 2 times, most recently from 6e562df to ab1f732 Compare September 16, 2024 12:47
@i-be-snek i-be-snek force-pushed the 81-level-of-information-evalutation branch from d5cae0e to ed1f7f1 Compare September 16, 2024 12:49
i-be-snek and others added 18 commits September 16, 2024 14:51
@i-be-snek
🚧 Make preliminary modifications to the eval script
02ed0be
@i-be-snek
81 level of information parsing (#88)
fdce4a5
@i-be-snek
81 level of information database (#85)
92d2215 
- The new database schema corresponds with the drawio png image

- There is a new empty database

- We can extract L1-3 from the flat excel. Note: we don't have the final data for that, only a working copy since more edits are required in the flat data table, but as long as we have a file that exactly matches the template in the excel sheet, we should have no problems
@i-be-snek
🔧 Update weights config (unified column names)
5f2f661
@i-be-snek
💡 Add comments on weights config
cb0302e
@i-be-snek
🗃️ Rename v0 db
26ac243
@i-be-snek
🗃️ Fix database insertion scr
6205f84
@i-be-snek
♻️ Refactor "type" -> "level/lvl" for event levels
eaba796
@i-be-snek
🚚 Update column names in comparer
3275531 
Choose dynamically, some shared columns hard-coded
@i-be-snek
📝 Add preliminary doc changes -m more to do here
8aa5c56
@i-be-snek
💬 Fix column name for l1 (Area -> Areas)
1ff0fbc
@i-be-snek
🚚 Fix Event_Name (str) -> Event_Names)
926ea0a
@i-be-snek
🚚 Fix *_Units -> *_Unit
1f054d8
@i-be-snek
🚚 Move Units -> Unit and Source -> Sources
0692013
@i-be-snek
🧑‍💻 Only cache ok (200) responses
3c8b29a
@i-be-snek
🔧 Add missing column Main_Event
c0624ca
@i-be-snek
✏️ Fix typo in admin area col name
1b6948c
@i-be-snek
🗃️ Fix validation for db insertion
0678704
@i-be-snek
🔥 Remove bad caterogy name dummy output
ebcbd78
@i-be-snek
⚡️ Insert events with bash script
c049860
@i-be-snek
🦺 Fix *_Min and *_Max >= 0 db validation
adecdf8
@i-be-snek
📝 Add docs on inserting events to the db (l1-l3)
4f45f33
@i-be-snek
💬 Fix wrong output column name
0ef9cfb
@i-be-snek
📦️ Update parsed dummy l1 file
fb737b2
@i-be-snek
🗃️ Update db schema + regenerate d
7047f05
@i-be-snek
🔊 Improve error log filename for failed db inserts
42965cc
@i-be-snek
🔊 Update log levels
0550b5a
@i-be-snek
🐛 Fix leftover _Unit column name inconsistency
a8d14a4
@i-be-snek
📝 Better param docs
f541246
@i-be-snek
✨ Event insertion pipeline
d859397
@i-be-snek
🧑‍💻 Add more options when creating a fresh db
07f424e
@i-be-snek
✨ Store GeoJson files outside db with filename pointers
a6eaeca
@i-be-snek
✨ Slim impactdb database without duplicate location files
aab4281
@i-be-snek
Restore file from lfs to git
2961342
@i-be-snek
📝 Update docs on full run event insertion for db
102649b
@i-be-snek
🔨 Fall back on default error_path
48e7f63
@i-be-snek i-be-snek force-pushed the 81-level-of-information-evalutation branch from ed1f7f1 to 48e7f63 Compare September 16, 2024 12:52
@i-be-snek i-be-snek changed the title WIP: 81 level of information evalutation 81 level of information evalutation Sep 18, 2024
@i-be-snek i-be-snek merged commit ac0b764 into 81-level-of-information-dev Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@i-be-snek i-be-snek

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Level of information: introducing L1-L3 (3 PRs)
1 participant
@i-be-snek