|
1 | 1 | # Logging |
2 | 2 |
|
3 | | -As any other device that is used to record and control an experiment, data streams from Harp devices can also be easily logged in real time. In this case, the decision as to what to log is somewhat easy. Since all the communication between the peripheral and the host is made via a sequence of [`HarpMessage`](xref:Bonsai.Harp.HarpMessage) objects, this is the only piece of information we need to reconstruct the state of the experiment (as seen by the Harp device at least) at any given point in time. |
| 3 | +Data streams from Harp devices can be easily logged in real time. Since all communication between a device and the host is made via a sequence of [`HarpMessage`](xref:Bonsai.Harp.HarpMessage) objects, this is the only piece of information we need to reconstruct the state of the experiment, as seen by the Harp device, at any given point in time. |
4 | 4 |
|
5 | | -Moreover, since all Harp messages follow a simple binary protocol, they can be efficiently (both in time and space) logged into disk using a simple flat binary file. The next sections will cover how to do this and discuss current recommendations for logging data streams from Harp devices. |
| 5 | +Moreover, since all Harp messages follow a simple [binary protocol](../protocol/BinaryProtocol-8bit.md), they can be efficiently logged into disk using a flat binary file. The next sections will cover how to do this and discuss current recommendations for logging data streams from Harp devices. |
6 | 6 |
|
7 | | -## MessageWriter |
| 7 | +## Log to a single binary file |
8 | 8 |
|
9 | | -Since [Harp](https://harp-tech.org/About/How-HARP-works/binary_protocol.html) is a binary protocol, any `HarpMessage` can be logged by simply saving its raw binary representation. The binary representation (as a `byte[]`) can be accessed via the [`MessageBytes`](xref:Bonsai.Harp.HarpMessage.MessageBytes) property. This means we could record the entire raw binary stream by feeding the sequence of message bytes to a binary logger. |
| 9 | +Any `HarpMessage` can be logged by simply saving its raw binary representation. This raw binary representation can be accessed as a `byte[]` via the [`MessageBytes`](xref:Bonsai.Harp.HarpMessage.MessageBytes) property. This means we can record the entire raw binary stream by simply passing the sequence of message bytes to a binary logger. |
10 | 10 |
|
11 | 11 | The `Bonsai.Harp` package provides a dedicated [`MessageWriter`](xref:Bonsai.Harp.MessageWriter) operator that easily encapsulates this functionality: |
12 | 12 |
|
13 | 13 | :::workflow |
14 | 14 |  |
15 | 15 | ::: |
16 | 16 |
|
17 | | -Since all logging takes place on top of any `HarpMessage` stream, the writers can also be used to log multiple devices in parallel, log filtered streams (e.g. after applying [`FilterRegister`](xref:Bonsai.Harp.FilterRegister)) or even save host-generated commands (e.g. messages generated by a [`CreateMessage`](xref:Bonsai.Harp.CreateMessage) operator). |
| 17 | +Since all logging takes place on top of arbitrary `HarpMessage` streams, the `MessageWriter` operator can be used to log multiple devices in parallel, log message streams filtered using the [`FilterRegister`](xref:Bonsai.Harp.FilterRegister) operator, or even save host-generated commands such as messages generated by the [`CreateMessage`](xref:Bonsai.Harp.CreateMessage) operator. |
18 | 18 |
|
19 | | -## GroupByRegister |
| 19 | +> [!Warning] |
| 20 | +> While logging all Harp messages to a single flat binary file is very easy, it is not always the most convenient way to log data for post-processing and analysis, as it requires examining the header of each and every message to determine how to process its contents. |
20 | 21 |
|
21 | | -While logging all Harp messages to a single binary is very easy, it is not always the most convenient way to log data. For instance, if one is interested in logging only a subset of messages (e.g. only the `ADC` messages), then the previous approach would require a post-processing step to filter out the messages of interest. |
| 22 | +## Log to separate files per register |
22 | 23 |
|
23 | | -Furthermore, each address has potentially different data formats (e.g. `U8` vs `U16`) or even different lengths if array registers are involved. This can make it very tedious to parse and analyze a binary file offline, since we will have to examine the header of each and every message in the file to determine how to extract its contents. |
| 24 | +We can use the [`GroupByRegister`](xref:Bonsai.Harp.GroupByRegister) operator to automatically split the single Harp message stream into independent sub-streams for each device register address. When `MessageWriter` receives such a split message sequence, it will automatically generate one independent file for each register. |
24 | 25 |
|
25 | | -This analysis could be entirely eliminated if we knew that all messages in the binary file had the same format. For any Harp device, the payload stored in a specific register will have a fixed type and length. This means that to ensure our simplifying assumption it is enough to save each message from a specific register into a different file. |
26 | | - |
27 | | -The [`GroupByRegister`](xref:Bonsai.Harp.GroupByRegister) operator is designed to automatically split the single Harp message stream into independent sub-streams for each device register address. There are many applications of this advanced operator, but the most common one is to demultiplex register messages before applying the `MessageWriter` operator. If `MessageWriter` receives a grouped message sequence, it will automatically generate one independent file for each register. |
28 | | - |
29 | | -If we do this, we can ensure that all messages on each single file will have the same format and length and can thus be read and parsed in a single bulk operation. A simple implementation of this pattern is shown below: |
| 26 | +Since the payload stored in any single register always has the same format and size, this is enough to ensure we can read and parse the entire single-register binary file in one bulk operation. A simple implementation of this pattern is shown below: |
30 | 27 |
|
31 | 28 | :::workflow |
32 | 29 |  |
33 | 30 | ::: |
34 | 31 |
|
35 | | -The single-register log files can then be loaded using the following Python routine: |
36 | | - |
37 | | -```python |
38 | | -import numpy as np |
39 | | -import pandas as pd |
40 | | - |
41 | | -_SECONDS_PER_TICK = 32e-6 |
42 | | -_payloadtypes = { |
43 | | - 1 : np.dtype(np.uint8), |
44 | | - 2 : np.dtype(np.uint16), |
45 | | - 4 : np.dtype(np.uint32), |
46 | | - 8 : np.dtype(np.uint64), |
47 | | - 129 : np.dtype(np.int8), |
48 | | - 130 : np.dtype(np.int16), |
49 | | - 132 : np.dtype(np.int32), |
50 | | - 136 : np.dtype(np.int64), |
51 | | - 68 : np.dtype(np.float32) |
52 | | - } |
53 | | - |
54 | | -def read_harp_bin(file): |
55 | | - |
56 | | - data = np.fromfile(file, dtype=np.uint8) |
57 | | - |
58 | | - if len(data) == 0: |
59 | | - return None |
60 | | - |
61 | | - stride = data[1] + 2 |
62 | | - length = len(data) // stride |
63 | | - payloadsize = stride - 12 |
64 | | - payloadtype = _payloadtypes[data[4] & ~0x10] |
65 | | - elementsize = payloadtype.itemsize |
66 | | - payloadshape = (length, payloadsize // elementsize) |
67 | | - seconds = np.ndarray(length, dtype=np.uint32, buffer=data, offset=5, strides=stride) |
68 | | - ticks = np.ndarray(length, dtype=np.uint16, buffer=data, offset=9, strides=stride) |
69 | | - seconds = ticks * _SECONDS_PER_TICK + seconds |
70 | | - payload = np.ndarray( |
71 | | - payloadshape, |
72 | | - dtype=payloadtype, |
73 | | - buffer=data, offset=11, |
74 | | - strides=(stride, elementsize)) |
75 | | - |
76 | | - if payload.shape[1] == 1: |
77 | | - ret_pd = pd.DataFrame(payload, index=seconds, columns= ["Value"]) |
78 | | - ret_pd.index.names = ['Seconds'] |
79 | | - |
80 | | - else: |
81 | | - ret_pd = pd.DataFrame(payload, index=seconds) |
82 | | - ret_pd.index.names = ['Seconds'] |
| 32 | +These single-register log files can then be loaded using the [Data Interface](python.md). The `device.yml` metadata file for each device is always available in the root folder of its source repository. |
83 | 33 |
|
84 | | - return ret_pd |
85 | | -``` |
| 34 | +> [!Tip] |
| 35 | +> The `device.yml` metadata file for each device can also be recovered at runtime using the `GetMetadata` operator included with most device packages and saved using the [`WriteAllText`](xref:Bonsai.IO.WriteAllText) operator. |
0 commit comments