173 null dam error

[i-be-snek]

This PR:

• fixes the bug where some "NULL"s are interpreted as areas and found on OSM as "T C Null Dam" (https://www.openstreetmap.org/way/1095041600). The reason for the bug was that "NULL" values inside lists were not properly replaced by None or removed altogether.
• small fixes for some columns that should return [] in case it's empty instead of None
• infer encoding when storing parquet, which allows l2/l3 locations and areas to be stored as an array instead of bytes, requiring no literal evaluation to work

To test this PR, you can try:

• parsing events of an existing file that used to contain "T C Null Dam entries". Example:

  poetry run python3 Database/parse_events.py \
  --raw_dir Database/raw/ESSD_2024_V3/dev \
  --filename wiki_dev_whole_infobox_20240729_70single_events_all_categories_V3_gpt-4o-2024-05-13_rawoutput_data_time_nums.json \
  --output_dir Database/output/173_null/dev \
  --event_levels l1,l2,l3

• evaluating a file parsed after the changes in this PR

• database insertion after parsing with the changes in this PR

[liniiiiii]

@i-be-snek , big thanks for the update, I found an issue with this is that after the changes, the Adminstrative_Area_Norm becomes None in the records of events where I found the T C Null Dam. The event 8Lbci8N and iAAUPM5 contain T C Null Dam in the original Specific_Instance_Per_Administrative_Area_Affected/0000.parquet file, and then I ran the command you provided in this branch, then I open the same file, and the T C Null Dam records are gone, but for these Adminstrative_Area_Norm/Geojson/GID all became None, not only for these two events, but for all events in this file, similarly for other L3 files, could you have a look? For L2 and L1 it's fine, thanks!

[i-be-snek]

Thanks @liniiiiii. I'm still a bit sick today bit this seems easy to fix. I'll also be pushing another PR soon to filter out things like "country1"/"location2", etc where the model is hallucinating the instructions.

[liniiiiii]

Thanks @liniiiiii. I'm still a bit sick today bit this seems easy to fix. I'll also be pushing another PR soon to filter out things like "country1"/"location2", etc where the model is hallucinating the instructions.

@i-be-snek , get better soon!!! I will help with code review! Thanks!

[i-be-snek]

@liniiiiii I re-ran the parsing and insertion from scratch on the file wiki_dev_whole_infobox_20240729_70single_events_all_categories_V3_gpt-4o-2024-05-13_rawoutput_data_time_nums.json and don't find any dates missing in l3.

Could you share the file you were inspecting on your end, or re-run the pipeline? I did add one commit that improves cleaning out "None" values from lists of locations/areas.

[i-be-snek]

I was able to replicate this, but only for this field.

I'll work on resolving it tomorrow, so no need to review this now.

[i-be-snek]

@liniiiiii it should be okay now :) please try

[liniiiiii]

Solved with both issues in the comments, TC NULL Dam and the None issue of Adminstritive area, approved to merge to main, thanks!