Code

Data Gap #174

Code

Merged

i-be-snek merged 57 commits intomainVUB-HYDR/Wikimpacts:mainfrom data-gapVUB-HYDR/Wikimpacts:data-gapCopy head branch name to clipboardNov 25, 2024

Commits

Commits on Nov 16, 2024

• ✨ Add data gap utils

  i-be-snek

  committed
  823612a

  Copy full SHA for 823612a

• 🚧 Add data gap script

  i-be-snek

  committed
  3ccf223

  Copy full SHA for 3ccf223

• 🔊 Update logs

  i-be-snek

  committed
  166314f

  Copy full SHA for 166314f

• 🚨 Fix mypy typing warnings

  i-be-snek

  committed
  012e458

  Copy full SHA for 012e458

• ➕ Add dev deps for typing hints with vcode mypy extention

  i-be-snek

  committed
  1c6c6a5

  Copy full SHA for 1c6c6a5

• 🚨 Fix more mypy lint warnings

  i-be-snek

  committed
  23c9518

  Copy full SHA for 23c9518

• ✨ Fill YEAR downstream l1->l2/l3

  i-be-snek

  committed
  38f0455

  Copy full SHA for 38f0455

• 🚧 Add data gap filler for areas

  i-be-snek

  committed
  f80be3b

  Copy full SHA for f80be3b

• ✨ Create geojson de-duplicated db in insertion step

  i-be-snek

  committed
  53bbe88

  Copy full SHA for 53bbe88

• 🗃️ Convert to str before insertion

  i-be-snek

  committed
  8d70d06

  Copy full SHA for 8d70d06

• 📝 Update docs on inserting a full run

  i-be-snek

  committed
  2e35c8c

  Copy full SHA for 2e35c8c

• ✏️ Fix typo in docs

  i-be-snek

  committed
  ac3ea98

  Copy full SHA for ac3ea98

• 🔊 Update logs and comments

  i-be-snek

  committed
  313fd53

  Copy full SHA for 313fd53

• 🦺 Improve validation of currencies (None)

  i-be-snek

  committed
  1e27b85

  Copy full SHA for 1e27b85

• 🚧 Fill l3->l2 impact gap

  i-be-snek

  committed
  a1e2865

  Copy full SHA for a1e2865

• 🚧 Append l3 records not found in l2 to l2

  i-be-snek

  committed
  1391147

  Copy full SHA for 1391147

• 🚧 Validate impact values (l3<=l2)

  i-be-snek

  committed
  b22c620

  Copy full SHA for b22c620

• 🐛 Fix repeated list of areas not in l2

  i-be-snek

  committed
  4784ee5

  Copy full SHA for 4784ee5

• 🔊 Update log message for clairty

  i-be-snek

  committed
  da74a09

  Copy full SHA for da74a09

• ♻️ Refactor util code -m add logs and improve flow + add an exception for monetary impacts for now

  i-be-snek

  committed
  c6ff795

  Copy full SHA for c6ff795

• 💡 Remove rogue print statement

  i-be-snek

  committed
  226e235

  Copy full SHA for 226e235

• 🚧 Fill area gap for l2/l3 areas missing in l1

  i-be-snek

  committed
  31d516d

  Copy full SHA for 31d516d

• 🚧 Add various debug fixes

  i-be-snek

  committed
  b73e457

  Copy full SHA for b73e457

• 🐛 Fix countring Nones as 0 values

  i-be-snek

  committed
  570d17d

  Copy full SHA for 570d17d

• 🚧 Fill empty l1 records with l2 aggregates

  i-be-snek

  committed
  97572a4

  Copy full SHA for 97572a4

• 🚧 Fill l2->l1 when l1 is None or < l2

  i-be-snek

  committed
  ae0bdb8

  Copy full SHA for ae0bdb8

• 🚧 Store output files

  i-be-snek

  committed
  ba842e0

  Copy full SHA for ba842e0

Commits on Nov 17, 2024

• 📝 Add docs on data gap filling

  i-be-snek

  committed
  09629bb

  Copy full SHA for 09629bb

• 📦️ Add deduplicated files for full run

  i-be-snek

  committed
  6b332c3

  Copy full SHA for 6b332c3

• 🐛 Fix mystery error when loading parquet from a dir

  i-be-snek

  committed
  c147765

  Copy full SHA for c147765

• 🐛 Add try/except block for show stopper

  i-be-snek

  committed
  40419c6

  Copy full SHA for 40419c6

Commits on Nov 18, 2024

• 🩹 Fix using a subset of impact categories instead of all

  i-be-snek

  committed
  0886f2a

  Copy full SHA for 0886f2a

• 🐛 Convert bytes to str if present

  i-be-snek

  committed
  7101e63

  Copy full SHA for 7101e63

• ♻️ Remove bytes at the start

  i-be-snek

  committed
  ac4ac95

  Copy full SHA for ac4ac95

• ♻️ Chunk at 25 records for each file

  i-be-snek

  committed
  e76955f

  Copy full SHA for e76955f

• 📦️ Add deduplicated files with data gap filled

  i-be-snek

  committed
  c3f6d1e

  Copy full SHA for c3f6d1e

Commits on Nov 19, 2024

• 🐛 Fix L1 not filled with L2 agg

  i-be-snek

  committed
  755cd98

  Copy full SHA for 755cd98

• 🔨 Refactor dev script

  i-be-snek

  committed
  5701407

  Copy full SHA for 5701407

Commits on Nov 20, 2024

• 🐛 Fix L1 NaNs not filled with aggregate L2 values

  i-be-snek

  committed
  e2f45c0

  Copy full SHA for e2f45c0

• 🔊 Add logs for year/date gap filling

  i-be-snek

  committed
  21e7a0f

  Copy full SHA for 21e7a0f

• ♻️ Avoid replacing missing years with None

  i-be-snek

  committed
  7ef4e5c

  Copy full SHA for 7ef4e5c

Commits on Nov 21, 2024

• 🏷️ Fix types

  i-be-snek

  committed
  2399457

  Copy full SHA for 2399457

• ✨ Handle duplicated records

  i-be-snek

  committed
  612fc91

  Copy full SHA for 612fc91

Commits on Nov 22, 2024

• 🐛 Fix duplicate aggregaes

  i-be-snek

  committed
  0baeea4

  Copy full SHA for 0baeea4

• 📦️ Add data gap fixed files

  i-be-snek

  committed
  0ad3004

  Copy full SHA for 0ad3004

Commits on Nov 23, 2024

• 🐛 Fix bytes to str conversion

  i-be-snek

  committed
  b9d8834

  Copy full SHA for b9d8834

• 🗃️ Remove non-climate related events

  i-be-snek

  committed
  66e59b7

  Copy full SHA for 66e59b7

• ♻️ Use preset vars

  i-be-snek

  committed
  c406f05

  Copy full SHA for c406f05

Commits on Nov 24, 2024

• 🐛 Fix nan event_ids

  i-be-snek

  committed
  453ce56

  Copy full SHA for 453ce56

• 🔊 Demote log info->debug for year filling

  i-be-snek

  committed
  1a6b302

  Copy full SHA for 1a6b302

• 🐛 Fix l3->l2 fill bug

  i-be-snek

  committed
  99c919a

  Copy full SHA for 99c919a

• 📦️ Add compiled de-duplicated files with data gap filled (again)

  i-be-snek

  committed
  aa311b4

  Copy full SHA for aa311b4

• 🐛 Fix not null validation in db insert

  i-be-snek

  committed
  197a290

  Copy full SHA for 197a290

• 🗃️ Add impactdb v.1.2 (NOT NULL validation bug fix)

  i-be-snek

  committed
  e695429

  Copy full SHA for e695429

• 🗃️ Data-gap filled database (v1.0)

  i-be-snek

  committed
  7f9c74b

  Copy full SHA for 7f9c74b

• 🚚 Rename v.1 -> v1.0

  i-be-snek

  committed
  b2f843a

  Copy full SHA for b2f843a

• 🦺 Fix dtypes for validation

  i-be-snek

  committed
  ab1eac2

  Copy full SHA for ab1eac2