Skip to content

Commit c310789

Browse files
nmdefriesmelange396
nmdefries
and
melange396
authored
Switch to new flusurv API endpoint (#1278)
* initial switch to new API endpoint; doesn't account for format change * separate fetch fn for whole api obj * get_current_issue to use existing json response * cleanup names and comments - rename input arg to `update` to avoid reassignment later - comment and reuse args_insert - spelling - comment magic constant used in output format - rename location-network/catchmentid map * store extracted rates by strata name rather than position Previously, age strata were numbered sequentially which allowed us to store rate values by position in a list. With the introduction of the new strata, this system is not robust enough to track all the different groups (e.g. ageids are no longer sequential and there are now race and sex groupings with separate numbering systems). * define function to convert json obs to dict grouped by location and epiweek * auto-map from valueids to ordinal and label-based group names * add new strata to sql insert statement by name, not order * pass seasonids around to use in requests for location-specific data * include old and new example API responses * flusurv tests * pass metadata around to reduce API calls * add season label as a descriptive column * move example API responses to make it clear they are not for prod use * add new columns to flusurv endpoint spec * add new age, sex, race strata, and season to schema * list new signals in documentation * move new season column to end of table * add migration script to add new fields to flusurv table * add field descriptions; reformat table so don't need to match line lengths * point from acquisition script to docs and schema def to deduplicate * review cleanup * capitalize constant max_age * move paren to subtract from # of dates Co-authored-by: melange396 <[email protected]> * move n_exected_groups and big groupid comment to global * convert asserts to if-raise * for robust group checking, list all expected groups and do set diff * separate data fetching fns into metadata and location classes * programmatically create location-networkid map; create constants file * switch missing data assert to exception * mock metadata and api objs for tests * stop doing recent season filtering Catchment metadata is not updated with most recent season, so doing this will stop us from fetching data. * warn when no location data returned * test metadata attributes together * correct local module name to load classes from * map some long race groups to abbreviated names * programmatically create sql query to avoid ordering issues * return empty result with right format when no data available * warning prefix * add tests for geo-abbr conversion and location fetching * don't error for missing groups -- some are expected * fn to add empty sex breakdowns to UT * with UT sex subsets getting filled, error on missing groups again * Update flusurv schema and docs with new age, sex, and race groups (#1287) * add new columns to flusurv endpoint spec * add new age, sex, race strata, and season to schema * list new signals in documentation * move new season column to end of table * add migration script to add new fields to flusurv table * add field descriptions; reformat table so don't need to match line lengths * point from acquisition script to docs and schema def to deduplicate * Revert "Update flusurv schema and docs with new age, sex, and race groups" (#1426) This reverts commit 9de7d64. * add new age rate signals * add new columns to flusurv endpoint spec * add schema migration script * move base url to constants * add max age as CLI arg * add flu a/b to DB table * hardcode valueid->signal name; add flu signals * remove ordinal conditional blocks from _groupid_to_name definition * add back female col to sql schema * make error types more specific * add new cols to integration test * remove trailing comma * specify flusurv return fields * warn on unexpected or missing keys and fill in * use unexpected key's ids as suffix; drop UT sex group handling * cast ids to str * drop trailing location * comments and cleanup * cast CLI args to types * note format issue in reference JSONs; rename metadata methods for clarity * filter weeks with user-provided age threshold * add season field to test --------- Co-authored-by: melange396 <[email protected]>
1 parent 3035ca4 commit c310789

File tree

15 files changed

+1575
-338
lines changed

15 files changed

+1575
-338
lines changed

‎docs/api/flusurv.md‎

Lines changed: 39 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -54,22 +54,45 @@ If neither is specified, the current issues are used.
5454

5555
## Response
5656

57-
| Field | Description | Type |
58-
|--------------------------|-----------------------------------------------------------------|------------------|
59-
| `result` | result code: 1 = success, 2 = too many results, -2 = no results | integer |
60-
| `epidata` | list of results | array of objects |
61-
| `epidata[].release_date` | | string |
62-
| `epidata[].location` | | string |
63-
| `epidata[].issue` | | integer |
64-
| `epidata[].epiweek` | | integer |
65-
| `epidata[].lag` | | integer |
66-
| `epidata[].rate_age_0` | | float |
67-
| `epidata[].rate_age_1` | | float |
68-
| `epidata[].rate_age_2` | | float |
69-
| `epidata[].rate_age_3` | | float |
70-
| `epidata[].rate_age_4` | | float |
71-
| `epidata[].rate_overall` | | float |
72-
| `message` | `success` or error message | string |
57+
| Field | Description | Type |
58+
|---|---|---|
59+
| `result` | result code: 1 = success, 2 = too many results, -2 = no results | integer |
60+
| `epidata` | list of results | array of objects |
61+
| `epidata[].release_date` | the date when this record was first received by Delphi | string |
62+
| `epidata[].location` | the name of the catchment (e.g. 'network_all', 'CA', 'NY_albany' | string |
63+
| `epidata[].issue` | the epiweek of receipt by Delphi (e.g. issue 201453 includes epiweeks up to and including 2014w53, but not 2015w01 or following) | integer |
64+
| `epidata[].epiweek` | the epiweek during which the data was collected | integer |
65+
| `epidata[].lag` | number of weeks between `epiweek` and `issue` | integer |
66+
| `epidata[].rate_age_0` | hospitalization rate for ages 0-4 | float |
67+
| `epidata[].rate_age_1` | hospitalization rate for ages 5-17 | float |
68+
| `epidata[].rate_age_2` | hospitalization rate for ages 18-49 | float |
69+
| `epidata[].rate_age_3` | hospitalization rate for ages 50-64 | float |
70+
| `epidata[].rate_age_4` | hospitalization rate for ages 65+ | float |
71+
| `epidata[].rate_overall` | overall hospitalization rate | float |
72+
| `epidata[].rate_age_5` | hospitalization rate for ages 65-74 | float |
73+
| `epidata[].rate_age_6` | hospitalization rate for ages 75-84 | float |
74+
| `epidata[].rate_age_7` | hospitalization rate for ages 85+ | float |
75+
| `epidata[].rate_age_18t29` | hospitalization rate for ages 18 to 29 | float |
76+
| `epidata[].rate_age_30t39` | hospitalization rate for ages 30 to 39 | float |
77+
| `epidata[].rate_age_40t49` | hospitalization rate for ages 40 to 49 | float |
78+
| `epidata[].rate_age_5t11` | hospitalization rate for ages 5 to 11 | float |
79+
| `epidata[].rate_age_12t17` | hospitalization rate for ages 12 to 17 | float |
80+
| `epidata[].rate_age_lt18` | hospitalization rate for ages <18 | float |
81+
| `epidata[].rate_age_gte18` | hospitalization rate for ages >=18 | float |
82+
| `epidata[].rate_age_0tlt1` | hospitalization rate for ages 0-1 | float |
83+
| `epidata[].rate_age_1t4` | hospitalization rate for ages 1-4 | float |
84+
| `epidata[].rate_age_gte75` | hospitalization rate for ages >=75 | float |
85+
| `epidata[].rate_race_white` | hospitalization rate for white people | float |
86+
| `epidata[].rate_race_black` | hospitalization rate for black people | float |
87+
| `epidata[].rate_race_hisp` | hospitalization rate for Hispanic/Latino people | float |
88+
| `epidata[].rate_race_asian` | hospitalization rate for Asian people | float |
89+
| `epidata[].rate_race_natamer` | hospitalization rate for American Indian/Alaskan Native people | float |
90+
| `epidata[].rate_sex_male` | hospitalization rate for males | float |
91+
| `epidata[].rate_sex_female` | hospitalization rate for females | float |
92+
| `epidata[].rate_flu_a` | hospitalization rate for inflenza A | float |
93+
| `epidata[].rate_flu_b` | hospitalization rate for inflenza B | float |
94+
| `epidata[].season` | indicates the start and end years of the winter flu season in the format YYYY-YY (e.g. 2022-23 indicates the flu season running late 2022 through early 2023) | string |
95+
| `message` | `success` or error message | string |
7396

7497
Notes:
7598
* The `flusurv` age groups are, in general, not the same as the ILINet

‎integrations/server/test_flusurv.py‎

Lines changed: 70 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,54 @@ def localSetUp(self):
1010

1111
def test_flusurv(self):
1212
"""Basic integration test for flusurv endpoint"""
13+
# TODO: Should test out the whole pipeline, where we mock the source
14+
# API by pulling more realistic data from a local JSON.
1315
self.cur.execute(
14-
"INSERT INTO `flusurv`(`release_date`, `issue`, `epiweek`, `location`, `lag`, `rate_age_0`, `rate_age_1`, `rate_age_2`, `rate_age_3`, `rate_age_4`, `rate_overall`, `rate_age_5`, `rate_age_6`, `rate_age_7`) VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)",
15-
("2012-11-02", "201243", "201143", "CA", "52", "0", "0", "0", "0.151", "0", "0.029", "0", "0", "0"),
16+
"""
17+
INSERT INTO `flusurv`(
18+
`release_date`,
19+
`issue`,
20+
`epiweek`,
21+
`location`,
22+
`lag`,
23+
`rate_age_0`,
24+
`rate_age_1`,
25+
`rate_age_2`,
26+
`rate_age_3`,
27+
`rate_age_4`,
28+
`rate_overall`,
29+
`rate_age_5`,
30+
`rate_age_6`,
31+
`rate_age_7`,
32+
`rate_age_18t29`,
33+
`rate_age_30t39`,
34+
`rate_age_40t49`,
35+
`rate_age_5t11`,
36+
`rate_age_12t17`,
37+
`rate_age_lt18`,
38+
`rate_age_gte18`,
39+
`rate_age_1t4`,
40+
`rate_age_gte75`,
41+
`rate_age_0tlt1`,
42+
`rate_race_white`,
43+
`rate_race_black`,
44+
`rate_race_hisp`,
45+
`rate_race_asian`,
46+
`rate_race_natamer`,
47+
`rate_sex_male`,
48+
`rate_sex_female`,
49+
`rate_flu_a`,
50+
`rate_flu_b`,
51+
`season`
52+
) VALUES(
53+
%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
54+
%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s
55+
)
56+
""",
57+
(
58+
"2012-11-02", 201243, 201143, "CA", 52, 0, 0, 0, 0.151, 0, 0.029, 0, 0, 0,
59+
2.54, 0, 1, 0, 0, 0, 0, 0, 0.68, 0.46, 1, 1, 0, 0, 0, 99, 0, 0, 22.2, "2012-13"
60+
),
1661
)
1762
self.cnx.commit()
1863
response = self.epidata_client.flusurv(epiweeks=201143, locations="CA")
@@ -25,13 +70,36 @@ def test_flusurv(self):
2570
"location": "CA",
2671
"issue": 201243,
2772
"epiweek": 201143,
73+
"season": "2012-13",
2874
"lag": 52,
2975
"rate_age_0": 0.0,
3076
"rate_age_1": 0.0,
3177
"rate_age_2": 0.0,
3278
"rate_age_3": 0.151,
3379
"rate_age_4": 0.0,
3480
"rate_overall": 0.029,
81+
"rate_age_5": 0.0,
82+
"rate_age_6": 0.0,
83+
"rate_age_7": 0.0,
84+
"rate_age_18t29": 2.54,
85+
"rate_age_30t39": 0.0,
86+
"rate_age_40t49": 1.0,
87+
"rate_age_5t11": 0.0,
88+
"rate_age_12t17": 0.0,
89+
"rate_age_lt18": 0.0,
90+
"rate_age_gte18": 0.0,
91+
"rate_age_1t4": 0.0,
92+
"rate_age_gte75": 0.68,
93+
"rate_age_0tlt1": 0.46,
94+
"rate_race_white": 1.0,
95+
"rate_race_black": 1.0,
96+
"rate_race_hisp": 0.0,
97+
"rate_race_asian": 0.0,
98+
"rate_race_natamer": 0.0,
99+
"rate_sex_male": 99.0,
100+
"rate_sex_female": 0.0,
101+
"rate_flu_a": 0.0,
102+
"rate_flu_b": 22.2,
35103
}
36104
],
37105
"result": 1,

0 commit comments

Comments
 (0)