|
| 1 | +## SQL |
| 2 | + |
| 3 | +SQL should be used only when the data manipulation to be done either can't be done using the web interface or where doing so would be exceedingly tedious. |
| 4 | +For example, some of the steps below can't be done in either the web interface, and the BETYdb-YABA API is not always available. |
| 5 | + |
| 6 | +SQL should primarily be used in the following cases: |
| 7 | + |
| 8 | +* correcting systemic data errors involving large numbers of rows |
| 9 | +* assigning non-point geometries to sites |
| 10 | +* associating experiments with sites and treatments |
| 11 | +* associating sites with cultivars |
| 12 | + |
| 13 | + |
| 14 | +### Metadata Entry Workflow for BETYdb |
| 15 | + |
| 16 | +The following steps have been implemented in the BETYdb-YABA API and python client, but are described below for reference |
| 17 | + |
| 18 | +Here we show SQL code used to add metadata required each season by the TERRA REF database, including experiments, sites, treatments, cultivars, and citations that is required prior to uploading the trait or yield data. Most PEcAn users _will not_ need to add experiments and cultivars, or associate these records with sites. |
| 19 | + |
| 20 | +Also note that in the TERRA REF and other agronomic applications, each record in the 'sites' table will correspond to an experimental plot. |
| 21 | + |
| 22 | +### Step 1: Add new experiments |
| 23 | + |
| 24 | +```sql |
| 25 | +insert into experiments (name, start_date, end_date, user_id) |
| 26 | +values ('MAC Season 6: Sorghum BAP', '2018-04-06', '2018-08-01', 'some text', 'some text', 6000000004); |
| 27 | +``` |
| 28 | + |
| 29 | +### Step 2: Add new sites |
| 30 | + |
| 31 | +To add sites (or many plots at a single site) using the BETYdb-YABA API, you must provide shapefile for a site to have an associated geometry. |
| 32 | + |
| 33 | +Otherwise, you can add sites with points or polygons to the database as follows. |
| 34 | + |
| 35 | +```sql |
| 36 | +insert into sites (city, state, country, sitename) |
| 37 | +values ('Maricopa', 'Arizona', 'USA', 'MAC Field Scanner Season 7 Range 9 Column 15'); |
| 38 | +``` |
| 39 | + |
| 40 | +You can add simple locations with a single point thus: |
| 41 | + |
| 42 | +```sql |
| 43 | +insert into sites (city, state, country, sitename, geometry) |
| 44 | +values ('Urbana', 'Illinois', 'United States', 'My garden plot', ST_SetSRID(ST_makePoint(88, 40, 222), 4326)); |
| 45 | +For the TERRA REF Project, plot definitions may be copied from previous season if same plots are used. |
| 46 | + |
| 47 | +```sql |
| 48 | +with season6 as ( |
| 49 | + select city, state, replace(sitename, 'Season 4', 'Season 6') as sitename, greenhouse, geometry, time_zone from sites where sitename like '%Season 4%' |
| 50 | +) |
| 51 | +insert into sites (city, state, sitename, greenhouse, geometry, time_zone) select * from season6; |
| 52 | +``` |
| 53 | + |
| 54 | + |
| 55 | +### Step 3: Add new treatments |
| 56 | + |
| 57 | +```sql |
| 58 | +insert into treatments (name, definition, control) |
| 59 | +values ('MAC Season 6: Sorghum', 'some text', 't'); |
| 60 | +``` |
| 61 | + |
| 62 | +### Step 4: Add new cultivars |
| 63 | + |
| 64 | +Each cultivar must be associated with a species. If there is no entry for the species in the species table, it must be added before adding the new cultivar. |
| 65 | + |
| 66 | + |
| 67 | +```sql |
| 68 | +insert into cultivars (name, specie_id) |
| 69 | +values ('RIL-CS27_(TX2910/(Macia/R07007)-CS44)-CSF1-PRF2-CS27', 2588); |
| 70 | +``` |
| 71 | + |
| 72 | +### Step 5: Add new citations |
| 73 | + |
| 74 | +```sql |
| 75 | +insert into citations (author, year, title) |
| 76 | +values ('Newcomb, Maria', 2016, 'Maricopa Agricultural Center Field Activities'); |
| 77 | +``` |
| 78 | + |
| 79 | +### Step 6: Associate experiments with sites |
| 80 | + |
| 81 | +As an example, this statement could be used to associate the experiment named "MAC Season 6: Sorghum BAP" with the site named "MAC Field Scanner Season 6 Range 1 Column 1 E". |
| 82 | +```sql |
| 83 | +insert into experiments_sites (experiment_id, site_id) values |
| 84 | + ((select id from experiments where name = 'MAC Season 6: Sorghum BAP'), |
| 85 | + (select id from sites where sitename = 'MAC Field Scanner Season 6 Range 1 Column 1 E')); |
| 86 | +``` |
| 87 | + |
| 88 | +When adding a new season for the TERRA REF project, a statement like the following can be used for associating the new season's experiment [or "an experiment for the new season"] with all of the new season's sites. For example, since MAC Field Center sites are consistently named following the format MAC Field Scanner Season x Range a Column b, we could use the following statement to associate the experment named "MAC Season 6: Sorghum BAP" with all Season 6 sites: |
| 89 | + |
| 90 | +```sql |
| 91 | +insert into experiments_sites (experiment_id, site_id) |
| 92 | + select e.experiment_id, s.site_id |
| 93 | + from (select id as experiment_id from experiments where name = 'MAC Season 6: Sorghum BAP') as e |
| 94 | + cross join |
| 95 | + (select id as site_id from sites where sitename like 'MAC Field Scanner Season 6%') as s; |
| 96 | +``` |
| 97 | + |
| 98 | +### Step 7: Associate experiments with treatments |
| 99 | + |
| 100 | +```sql |
| 101 | +insert into experiments_treatments (experiment_id, treatment_id) |
| 102 | +values ((select id from experiments where name = 'MAC Season 6: Sorghum BAP'), |
| 103 | +(select id from treatments where name = 'MAC Season 6: Sorghum')); |
| 104 | +``` |
| 105 | +When adding a new season for the TERRA REF project, a statement like the following can be used to associate all season 6 experiments with all season 6 treatments assuming the experiment and treatment names follow the format convention `MAC Season x: subexperiment name`. |
| 106 | + |
| 107 | +```sql |
| 108 | +insert into experiments_treatments (experiment_id, treatment_id) |
| 109 | + select e.experiment_id, t.treatment_id |
| 110 | + from (select id as experiment_id from experiments where name like 'MAC Season 6:%') as e |
| 111 | + cross join |
| 112 | + (select id as treatment_id from treatments where name like 'MAC Season 6:%') as s; |
| 113 | +``` |
| 114 | + |
| 115 | +### Step 8: Associate sites with cultivars |
| 116 | + |
| 117 | +As an example, this statement could be used to associate the site name MAC Field Scanner Season 8 Range 1 Column 1 E with the Sorghum bicolor cultivar Tiburon: |
| 118 | + |
| 119 | +```sql |
| 120 | +insert into sites_cultivars (site_id, cultivar_id) |
| 121 | + values ((select id from sites where sitename = 'MAC Field Scanner Season 8 Range 1 Column 1 E'), |
| 122 | + (select id from cultivars where name = 'Tiburon' and specie_id = |
| 123 | + (select id from species where scientificname = 'Sorghum bicolor'))); |
| 124 | +``` |
| 125 | + |
| 126 | +### Step 9: Associate sites with citations |
| 127 | + |
| 128 | +```sql |
| 129 | +insert into citations_sites (citation_id, site_id) |
| 130 | +values ((select id from citations where author = 'Newcomb, Maria' and year = 2016 and title = 'MAC Field Activities'), |
| 131 | +(select id from sites where sitename = 'MAC Field Scanner Season 6 Range 1 Column 1 E')); |
| 132 | +``` |
| 133 | + |
| 134 | +When adding a new season for the TERRA REF project, a statement like the following can be used to associate all citations with author "Newcomb, Maria" with all Season 6 sites since MAC Field Center sites are consistently named following the format `MAC Field Scanner Season x Range a Column b` |
| 135 | + |
| 136 | + |
| 137 | +```sql |
| 138 | +insert into citations_sites (citation_id, site_id) |
| 139 | + select c.id, s.id |
| 140 | + from (select id from citations where author = 'Newcomb, Maria') |
| 141 | + AS c |
| 142 | + cross join |
| 143 | + (select id from sites where sitename like 'MAC Field Scanner Season 6%') AS s; |
| 144 | +``` |
| 145 | + |
| 146 | + |
| 147 | +## Removing duplicate records |
| 148 | +It sometimes happens that multiple rows in a BETYdb table are duplicates, e.g., two species or two sites that were independently added but reference the same entity. A function such as the one that follows could be used to delete a duplicate row and change all references to it to point to a row that we are retaining. (Note that there are serious problems associated with the use of this function, as outlined in [this comment in issue 185](https://github.com/PecanProject/bety/issues/185#issuecomment-530554650){target="_blank"}. We present it here only to outline what might be possible along the lines of partially automating the correction of data errors.) |
| 149 | + |
| 150 | +The function takes three arguments: the name of the table we wish to update (as a string), the id number of the row we wish to remove (the "duplicate"), and the id number of the similar row we wish to retain. For example, if we have two citation rows having essentially the same information having id numbers 286 and 289, we could remove the first and update references to it to point to the second with the statement |
| 151 | + |
| 152 | +```sql |
| 153 | +SELECT update_refs_from_to('citations', 286, 289); |
| 154 | +``` |
| 155 | + |
| 156 | +The following function can be used to combine duplicates or replace an old record with a new one. There are many caveats described in [Issue 185](https://github.com/PecanProject/bety/issues/185){target="_blank"}. |
| 157 | + |
| 158 | +But the basic usage, e.g. to convert all records with `citation_id = 286` to `citation_id = 289`: |
| 159 | + |
| 160 | +```sql |
| 161 | +SELECT update_refs_from_to('citations', 286, 289); |
| 162 | +``` |
| 163 | + |
| 164 | +To make the function available, first run this code: |
| 165 | + |
| 166 | +```sql |
| 167 | +CREATE OR REPLACE FUNCTION update_refs_from_to( |
| 168 | + primary_table_name varchar, |
| 169 | + old_id bigint, |
| 170 | + new_id bigint |
| 171 | +) RETURNS void AS $$ |
| 172 | +DECLARE |
| 173 | + foreign_key_col_name varchar; |
| 174 | + referring_table_name varchar; |
| 175 | + update_stmt varchar; |
| 176 | + delete_stmt varchar; |
| 177 | +BEGIN |
| 178 | + foreign_key_col_name := regexp_replace(primary_table_name, 's$', '') || '_id'; |
| 179 | + FOR referring_table_name IN SELECT table_name FROM information_schema.columns WHERE table_schema = 'public' AND "column_name" = foreign_key_col_name AND is_updatable = 'YES' LOOP |
| 180 | +
|
| 181 | + BEGIN |
| 182 | + update_stmt := 'UPDATE ' || referring_table_name || ' SET ' || foreign_key_col_name || ' = ' || new_id || ' WHERE ' || foreign_key_col_name || ' = ' || old_id; |
| 183 | + RAISE NOTICE 'Attempting to run %', update_stmt; |
| 184 | + EXECUTE update_stmt; |
| 185 | + RAISE NOTICE 'Success!'; |
| 186 | + EXCEPTION |
| 187 | + WHEN unique_violation THEN |
| 188 | + RAISE NOTICE 'UPDATE FAILED!!!'; |
| 189 | + RAISE NOTICE 'Updating table column % in table % would violate uniqueness constraints', foreign_key_col_name, referring_table_name; |
| 190 | + END; |
| 191 | + END LOOP; |
| 192 | + BEGIN |
| 193 | + delete_stmt := 'DELETE FROM ' || primary_table_name || ' WHERE id = ' || old_id; |
| 194 | + RAISE NOTICE 'Attempting to run %', delete_stmt; |
| 195 | + EXECUTE delete_stmt; |
| 196 | + RAISE NOTICE 'Success!'; |
| 197 | + EXCEPTION |
| 198 | + WHEN foreign_key_violation THEN |
| 199 | + RAISE NOTICE 'DELETION FAILED!!!'; |
| 200 | + RAISE NOTICE 'Deletion from table % of the row with id % would cause a foreign-key violation', primary_table_name, old_id; |
| 201 | + END; |
| 202 | +END |
| 203 | +$$ LANGUAGE plpgsql; |
| 204 | +``` |
0 commit comments