Skip to content

Commit a6f240b

Browse files
labkey-alanlabkey-alan
committed
Add experiment.md
1 parent a1378f3 commit a6f240b

1 file changed

Lines changed: 354 additions & 0 deletions

File tree

‎docs/experiment.md‎

Lines changed: 354 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,354 @@
1+
# LabKey Experiment API Support
2+
3+
The Experiment API reads and writes LabKey assay data. Assay data is organized into a three level hierarchy:
4+
5+
- **Batch** — a group of runs imported together. Batch level fields are stored in the assay's batch domain.
6+
- **Run** — a single import of data, e.g. one instrument file. Run level fields are stored in the assay's run domain.
7+
- **Data rows** — the individual result rows of a run, matching the assay's results domain.
8+
9+
Every call is made against a specific assay design, identified by its **assay protocol id** (the `assay_id` argument).
10+
You can find this id in the URL when viewing an assay design in the server UI (e.g.
11+
`.../assay-assayBegin.view?rowId=3315`), or by querying the `assay.AssayList` table with `select_rows`.
12+
13+
The module also exposes the experiment lineage endpoint. See [lineage.md](lineage.md) for that API.
14+
15+
### Additional details from LabKey Documentation:
16+
- [Assay Data](https://www.labkey.org/Documentation/wiki-page.view?name=instrumentData)
17+
18+
## Interfaces
19+
20+
The classes below are imported from `labkey.experiment`. Each constructor accepts keyword arguments, and each
21+
accepts either the Python style name or the server's JSON name (e.g. `data_rows` or `dataRows`), so objects
22+
returned by the server can be modified and passed straight back to a save method.
23+
24+
```python
25+
from labkey.experiment import Batch, Data, Run
26+
```
27+
28+
### `ExpObject`
29+
30+
Base class for all experiment objects. Not used directly.
31+
32+
| Property | Type | Description |
33+
|----------------------------|--------|--------------------------------------------------------------------------|
34+
| `lsid` | `str` | Life Science Identifier. Assigned by the server. |
35+
| `name` | `str` | Display name. |
36+
| `id` / `row_id` | `int` | Primary key. Assigned by the server; set it to update an existing object. |
37+
| `comment` | `str` | Free text comment. |
38+
| `created` / `modified` | `str` | Timestamps. Assigned by the server. |
39+
| `created_by`/`modified_by` | `str` | User display names. Assigned by the server. |
40+
| `properties` | `dict` | Domain field values, keyed by field name. |
41+
42+
### `Batch`
43+
44+
A group of runs. Extends `ExpObject`.
45+
46+
| Property | Type | Description |
47+
|---------------------|-------------|----------------------------------------------------------------------|
48+
| `runs` | `List[Run]` | The runs contained in this batch. |
49+
| `batch_protocol_id` | `int` | Protocol id of the batch. Defaults to `id`. |
50+
| `hidden` | `bool` | Whether the batch is hidden in the UI. Defaults to `False`. |
51+
52+
`properties` on a `Batch` holds the batch domain field values.
53+
54+
### `Run`
55+
56+
A single data import. Extends `ExpObject`.
57+
58+
| Property | Type | Description |
59+
|--------------------|--------------------|-----------------------------------------------------------------------------------------------------|
60+
| `data_rows` | `List[dict]` | Result rows, each keyed by results domain column name. |
61+
| `data_file` | `TextIO` | An open file handle to import results from a file instead of `data_rows`. **`import_run()` only.** |
62+
| `data_inputs` | `List[Data]` | Data objects consumed by the run. |
63+
| `data_outputs` | `List[dict]` | Data objects produced by the run. |
64+
| `material_inputs` | `List[dict]` | Samples/materials consumed by the run. |
65+
| `material_outputs` | `List[dict]` | Samples/materials produced by the run. |
66+
| `experiments` | `List[dict]` | Experiments (run groups) the run belongs to. |
67+
| `file_path_root` | `str` | Server side root path for the run's files. |
68+
| `protocol` | `dict` | The run's protocol. |
69+
| `object_properties`| `List[dict]` | Additional object level properties. |
70+
| `plate_metadata` | `dict` | Well group property values for plate based assays. See [Plate based assays](#plate-based-assays). |
71+
| `workflow_task` | `int` | Row id of a workflow (Sample Manager / LIMS) task to associate the run with. |
72+
73+
`properties` on a `Run` holds the run domain field values. Empty values are dropped from the request payload;
74+
the server supplies its own defaults for keys that are absent.
75+
76+
### `RunItem`
77+
78+
Base class for run inputs and outputs. Extends `ExpObject`.
79+
80+
| Property | Type | Description |
81+
|-----------------------|--------------|---------------------------------------------------|
82+
| `source_protocol` | `dict` | Protocol that produced this item. |
83+
| `run` | `dict` | The run this item belongs to. |
84+
| `target_applications` | `List[dict]` | Protocol applications that consume this item. |
85+
| `successor_runs` | `List[dict]` | Runs derived from this item. |
86+
| `cpas_type` | `str` | LSID of the item's sample type or data class. |
87+
88+
### `Data`
89+
90+
A data object (typically a file) used as a run input or output. Extends `RunItem`.
91+
92+
| Property | Type | Description |
93+
|-----------------|-------|--------------------------------------------------------------------|
94+
| `data_type` | `str` | The data type, e.g. `"Data"`. |
95+
| `data_file_url` | `str` | URL of the underlying file. |
96+
| `pipeline_path` | `str` | Path to the file relative to the container's pipeline root. |
97+
| `role` | `str` | The role this data plays in the run. |
98+
99+
### Plate based assays
100+
101+
Assays configured for plate support add two requirements to each `Run`:
102+
103+
- `properties["PlateTemplate"]` is required and must be the LSID of the plate template the run uses. For standard
104+
assays the available templates and their LSIDs can be read with
105+
`api.query.select_rows("assay.General", "PlateTemplate", columns="Name, Lsid")`.
106+
- `data_rows` must identify the well each result belongs to, using the assay's well location column
107+
(`WellLocation` in the default plate design).
108+
109+
`plate_metadata` optionally supplies property values for the template's well groups. It is a two level dict:
110+
well group type (`"control"`, `"sample"`, ...) → well group name → a dict of property name/value pairs. The
111+
property names must exist on the corresponding well group domain, and the well group names must match those
112+
defined in the plate template.
113+
114+
```python
115+
run.plate_metadata = {
116+
"control": {"positive": {"dilution": 0.005}, "negative": {"dilution": 1.0}},
117+
"sample": {
118+
"SA01": {"dilution": 1.0, "Barcode": "BC_111", "Concentration": 0.0125},
119+
"SA02": {"dilution": 2.0, "Barcode": "BC_222"},
120+
},
121+
}
122+
```
123+
124+
Properties may be omitted per well group; in the example above only `SA01` sets `Concentration`.
125+
126+
## Methods
127+
128+
All methods are available on the `experiment` member of an `APIWrapper` instance.
129+
130+
| Method | Returns | Description |
131+
|---------------------------------------|-------------------------|--------------------------------------------------------------------------------------|
132+
| `load_batch(assay_id, batch_id)` | `Optional[Batch]` | Load a batch, its runs, and its run data from the server. |
133+
| `save_batch(assay_id, batch)` | `Optional[Batch]` | Save one batch and its runs. Returns the saved batch with server assigned ids. |
134+
| `save_batches(assay_id, batches)` | `Optional[List[Batch]]` | Save several batches in one request. Returns the saved batches. |
135+
| `import_run(assay_id, run)` | `dict` | Import a single run without creating or updating a batch explicitly. |
136+
| `lineage(lsids, ...)` | `dict` | Query the experiment lineage graph. See [lineage.md](lineage.md). |
137+
138+
Notes:
139+
140+
- `save_batch()` / `save_batches()` create a batch when `id` is not set, and update the existing batch when it is.
141+
Every run must be supplied on each save; runs omitted from a saved batch are removed from it.
142+
- `import_run()` is the only method that accepts `Run.data_file`. It always stores the imported results as a file
143+
on the server, and it is the method to use when associating a run with a `workflow_task`.
144+
- `save_batches()` raises an exception if any element of `batches` is not a `Batch` instance.
145+
146+
### Examples
147+
148+
Every example below uses an `APIWrapper` instance to make its requests. See [api_wrapper.md](api_wrapper.md) for the
149+
full set of `APIWrapper` arguments, including how to configure the container path, context path, SSL, and
150+
authentication.
151+
152+
#### Save and load an assay batch
153+
154+
```python
155+
from labkey.api_wrapper import APIWrapper
156+
from labkey.experiment import Batch, Run
157+
158+
labkey_server = "www.example.com"
159+
container_path = "Tutorials/HIV Study" # Full project/folder container path
160+
context_path = "labkey"
161+
api = APIWrapper(labkey_server, container_path, context_path)
162+
163+
assay_id = 3315 # provide one from your server
164+
165+
###################
166+
# Save an assay batch
167+
###################
168+
run = Run()
169+
run.name = "python upload"
170+
run.data_rows = [
171+
{
172+
# ColumnName: Value
173+
"SampleId": "Sample 1",
174+
"TimePoint": "2008/11/02 11:22:33",
175+
"DoubleData": 4.5,
176+
"HiddenData": "another data point",
177+
},
178+
{
179+
"SampleId": "Sample 2",
180+
"TimePoint": "2008/11/02 14:00:01",
181+
"DoubleData": 3.1,
182+
"HiddenData": "fozzy bear",
183+
},
184+
]
185+
run.properties["RunFieldName"] = "Run Field Value"
186+
187+
batch = Batch()
188+
batch.name = "python batch"
189+
batch.runs = [run]
190+
batch.properties["PropertyName"] = "Property Value"
191+
192+
saved_batch = api.experiment.save_batch(assay_id, batch)
193+
194+
###################
195+
# Load an assay batch
196+
###################
197+
run_group = api.experiment.load_batch(assay_id, saved_batch.row_id)
198+
199+
if run_group is not None:
200+
print("Batch Id: " + str(run_group.id))
201+
print("Created By: " + run_group.created_by)
202+
for loaded_run in run_group.runs:
203+
print("Run: " + loaded_run.name + ", rows: " + str(len(loaded_run.data_rows)))
204+
else:
205+
print("load_batch: no batch returned")
206+
```
207+
208+
#### Add a run to an existing batch
209+
210+
Load the batch, append a run, and save it back. The existing runs must remain in `batch.runs`.
211+
212+
```python
213+
from labkey.api_wrapper import APIWrapper
214+
from labkey.experiment import Run
215+
216+
api = APIWrapper("www.example.com", "Tutorials/HIV Study", "labkey")
217+
218+
assay_id = 3315
219+
batch_id = 1234
220+
221+
batch = api.experiment.load_batch(assay_id, batch_id)
222+
223+
new_run = Run(
224+
name="second upload",
225+
data_rows=[{"SampleId": "Monkey 4", "DoubleData": 2.7}],
226+
)
227+
batch.runs.append(new_run)
228+
229+
api.experiment.save_batch(assay_id, batch)
230+
```
231+
232+
#### Save multiple batches in one request
233+
234+
```python
235+
from labkey.api_wrapper import APIWrapper
236+
from labkey.experiment import Batch, Run
237+
238+
api = APIWrapper("www.example.com", "Tutorials/HIV Study", "labkey")
239+
240+
assay_id = 3315
241+
242+
batches = [
243+
Batch(
244+
name="plate 1",
245+
runs=[{"name": "plate 1 run", "data_rows": [{"SampleId": "Monkey 1", "DoubleData": 4.5}]}],
246+
),
247+
Batch(
248+
name="plate 2",
249+
runs=[{"name": "plate 2 run", "data_rows": [{"SampleId": "Monkey 2", "DoubleData": 3.1}]}],
250+
),
251+
]
252+
253+
saved_batches = api.experiment.save_batches(assay_id, batches)
254+
255+
for saved in saved_batches:
256+
print(saved.name + " -> rowId " + str(saved.row_id))
257+
```
258+
259+
#### Save a batch for a plate based assay
260+
261+
The run supplies the plate template LSID as a run property, locates each result row in a well, and maps property
262+
values onto the template's well groups. See [Plate based assays](#plate-based-assays) for the
263+
`plate_metadata` structure.
264+
265+
```python
266+
from labkey.api_wrapper import APIWrapper
267+
from labkey.experiment import Batch, Run
268+
269+
api = APIWrapper("www.example.com", "Tutorials/assay", "labkey")
270+
271+
assay_id = 310 # a plate enabled assay design on your server
272+
273+
run = Run()
274+
run.name = "python upload"
275+
run.data_rows = [
276+
{
277+
# ColumnName: Value
278+
"ParticipantId": "1234",
279+
"VisitId": 111,
280+
"WellLocation": "A1",
281+
},
282+
{"ParticipantId": "5678", "VisitId": 222, "WellLocation": "B11"},
283+
{"ParticipantId": "9123", "VisitId": 333, "WellLocation": "F12"},
284+
]
285+
286+
# Required run property for plate enabled assays: the plate template LSID
287+
run.properties["PlateTemplate"] = (
288+
"urn:lsid:labkey.com:PlateTemplate.Folder-6:d8bbec7d-34cd-1038-bd67-b3bd777822f8"
289+
)
290+
291+
# Well group properties, keyed by well group type then well group name
292+
run.plate_metadata = {
293+
"control": {"positive": {"dilution": 0.005}, "negative": {"dilution": 1.0}},
294+
"sample": {
295+
"SA01": {"dilution": 1.0, "Barcode": "BC_111", "Concentration": 0.0125},
296+
"SA02": {"dilution": 2.0, "Barcode": "BC_222"},
297+
"SA03": {"dilution": 3.0, "Barcode": "BC_333"},
298+
"SA04": {"dilution": 4.0, "Barcode": "BC_444"},
299+
},
300+
}
301+
302+
batch = Batch()
303+
batch.name = "python batch"
304+
batch.runs = [run]
305+
batch.properties["PropertyName"] = "Property Value"
306+
307+
saved_batch = api.experiment.save_batch(assay_id, batch)
308+
```
309+
310+
#### Import a run and associate it with a workflow task
311+
312+
Use `import_run()` when you have a single run to import and no batch level properties to set. Passing
313+
`workflow_task` links the resulting run to a Sample Manager or LIMS workflow job task.
314+
315+
```python
316+
from labkey.api_wrapper import APIWrapper
317+
from labkey.experiment import Run
318+
319+
api = APIWrapper("www.example.com", "Biologics")
320+
321+
assay_id = 22858
322+
workflow_task = 50574
323+
324+
rows = [
325+
{"Sample": 404208, "Sample/Name": "S-97", "Value": "1"},
326+
{"Sample": 404207, "Sample/Name": "S-96", "Value": "2"},
327+
{"Sample": 404206, "Sample/Name": "S-95", "Value": "3"},
328+
]
329+
330+
run = Run(name="My Python Run", workflow_task=workflow_task, data_rows=rows)
331+
332+
result = api.experiment.import_run(assay_id, run)
333+
print(result)
334+
```
335+
336+
#### Import a run from a data file
337+
338+
Assign an open file handle to `Run.data_file` instead of supplying `data_rows`. The file must be in a format the
339+
assay design accepts, for example a TSV whose column headers match the results domain.
340+
341+
```python
342+
from labkey.api_wrapper import APIWrapper
343+
from labkey.experiment import Run
344+
345+
api = APIWrapper("www.example.com", "Biologics")
346+
347+
assay_id = 22858
348+
349+
with open("assay_data.tsv", "r") as run_file:
350+
run = Run(name="My Python File Run", data_file=run_file)
351+
result = api.experiment.import_run(assay_id, run)
352+
353+
print(result)
354+
```

0 commit comments

Comments
 (0)