Document multi-part ids

docs/developers/real-time.md

@@ -56,6 +56,8 @@ HarperDB supports QoS 0 and 1 for publishing and subscribing.
 
 HarperDB supports multi-level topics, both for subscribing and publishing. HarperDB also supports multi-level wildcards, so you can subscribe to /`my-resource/#` to receive notifications for `my-resource/some-id` as well as `my-resource/nested/id`, or you can subscribe to `my-resource/nested/#` and receive the latter, but not the former, topic messages. HarperDB currently only supports trailing multi-level wildcards (no single-level wildcards with '\*').
 
+When using nested topics, these are translated to multipart ids. The primary key/id for `my-resource/nested/id` would be `["nested","id]` in the database. This allows for a simple and effective way to manage nested topics and subscriptions. See the [Resource API](../technical-details/reference/resource.md) for more information about how define resources and interact with them and reference by primary keys.
+
 ### Ordering
 
 HarperDB is designed to be a distributed database, and an intrinsic characteristic of distributed servers is that messages may take different amounts of time to traverse the network and may arrive in a different order depending on server location and network topology. HarperDB is designed for distributed data with minimal latency, and so messages are delivered to subscribers immediately when they arrive, HarperDB does not delay messages for coordinating confirmation or consensus among other nodes, which would significantly increase latency, messages are delivered as quickly as possible.

docs/developers/rest.md

@@ -101,6 +101,16 @@ GET /my-resource/?property=value&property2=another-value
 
 Note that only one of the attributes needs to be indexed for this query to execute.
 
+REST query parameters are designed so that you can easily and safely construct a set of name-value conditions using standard URL encoding. This allows you to easily construct queries in a URL, and also safely include user-provided values in a query. For example, in JavaScript, you can easily construct a query string from an object of conditions:
+
+```javascript
+let conditions = { category: "software", price: 100 } // these values can come from user input and will be safely encoded
+let url = `/my-resource/?${new URLSearchParams(conditions)}`
+```
+However, if you want to perform queries beyond basic name-value equality conditions, you can use a more advanced query language syntax that allows for comparison operators, unions, and grouping of conditions:
+
+### Comparison Operators
+
 We can also specify different comparators such as less than and greater than queries using [FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompub-fiql-00) syntax. If we want to specify records with an `age` value greater than 20:
 
 ```http
@@ -170,6 +180,27 @@ More complex queries can be created by further nesting groups:
 GET /Product/?price=lt=100|[rating=5&[tag=fast|tag=scalable|tag=efficient]&inStock=true]
 ```
 
+### Multipart primary keys
+
+For tables with multipart primary keys, the primary key values can be specified in the URL path, separated by slashes. This allows us to perform hierarchical queries. For example, if we added a product:
+
+```http
+PUT /Product/electronics/123
+Content-Type: application/json
+
+{ "id": ["electronics", "123"], name": "An electronic product" }
+```
+We can query for products in the electronics category:
+```http
+GET /Product/electronics/
+```
+And this can be combined with other query parameters, for example to find products in the electronics category with a price less than 100:
+```http
+GET /Product/electronics/?price=lt=100
+```
+
+See the [Resource API documentation](../technical-details/reference/resource.md) for how these paths are handled as multipart primary keys.
+
 ## Query Calls
 
 HarperDB has several special query functions that use "call" syntax. These can be included in the query string as its own query entry (separated from other query conditions with an `&`). These include:

docs/technical-details/reference/resource.md

@@ -392,6 +392,14 @@ Table.search({ conditions: [
 ]});
 ```
 
+The `search method` will return an `AsyncIterable` that can be iterated through to access the results of the query. In order to access the elements of the query results, you must use a `for await` loop (it does _not_ return an array, you can not access the results by index).
+```javascript
+for (let record of Table.search({ conditions: [...]})) {
+    // iterate through each record in the query results
+}
+```
+HarperDB queries are performed progressively where possible. This means that the query results are incrementally loaded from the database as you iterate through the query results. This allows for more efficient use of resources and can be more performant, with faster initial response time, than loading all results into memory at once. This is especially useful when dealing with large datasets. If you break out of a query (with `return`, `break`, or an exception), the query will be stopped and the associated read transaction will be released. Note that if you use a sort operation, the query may still need to be loaded entirely into memory to perform the sort operation.
+
 ##### Chained Attributes/Properties
 Chained attribute/property references can be used to search on properties within related records that are referenced by [relationship properties](../../developers/applications/defining-schemas.md) (in addition to the [schema documentation](../../developers/applications/defining-schemas.md), see the [REST documentation](../../developers/rest.md) for more of overview of relationships and querying). Chained property references are specified with an array, with each entry in the array being a property name for successive property references. For example, if a relationship property called `brand` has been defined that references a `Brand` table, we could search products by brand name:
 ```javascript
@@ -524,7 +532,7 @@ Note that you may also need to use `get`/`set` for properties that conflict with
 If you want to save the changes you make, you can call the \`update()\`\` method:
 
 ```javascript
-let product1 = await Product.get(1);
+let product1 = await Product.get('1');
 product1.rating = 3;
 product1.set('newProperty', 'some value');
 product1.update(); // save both of these property changes
@@ -582,7 +590,7 @@ export class CustomProduct extends Product {
 If you need to delete a property, you can do with the `delete` method:
 
 ```javascript
-let product1 = await Product.get(1);
+let product1 = await Product.get('1');
 product1.delete('additionalInformation');
 product1.update();
 ```
@@ -597,6 +605,37 @@ for (let key in plainObject) {
 }
 ```
 
+### Multipart primary keys 
+HarperDB supports multipart primary keys, which can be used to create hierarchical data organization of records within a table. Multipart primary keys are stored and accessed as an array of values, and can be queried and accessed in a similar way to single-part primary keys, but with array values. And with multipart primary keys, records can be queried by partial prefix. To continue with the product example, we could add products with a multipart primary key that includes a category and product id:
+
+```javascript
+await Product.put({ id: ['electronics', '1'], name: 'Alarm Clock', price: 9.99 });
+await Product.put({ id: ['electronics', '2'], name: 'Tablet', price: 299.99 });
+```
+You can retrieve a product by multipart id:
+```javascript
+let product = await Product.get(['electronics', '1']);
+```
+And we can also query products by category:
+```javascript
+for await (let product of Product.search({ conditions: [{ attribute: 'id', value: 'electronics', comparator: 'prefix' }] })) {
+	// iterate through all electronics products
+}
+```
+We can create deeper hierarchy as well:
+```javascript
+Product.put({ id: ['electronics', 'laptops', '1'], name: 'Macbook', price: 1999.99 });
+```
+And query using a longer prefix:
+```javascript
+for await (let product of Product.search({ conditions: [{ attribute: 'id', value: ['electronics', 'laptop'], comparator: 'prefix' }] })) {
+	// iterate through all laptops
+}
+```
+Note that the query for `electronics` above will still return the `laptops` products, as the prefix query will match any records that start with the prefix.
+
+Multipart primary keys work in conjunction with the [REST](../../developers/rest.md) and [MQTT](../../developers/real-time.md) interfaces, where they can be accessed by URL or topic path. For example, if the `Product` table is exported, a product with an id of `['electronics', 1]` would be accessed by a URL like `/Product/electronics/1` or subscribed to through a topic like `Product/electronics/1`. And the URL `/Product/electronics/` will query all products in the electronics category (by using the `prefix` query above). And the MQTT topic `Product/electronics/#` would subscribe to all products in the electronics category. The parsing of REST and MQTT paths is performed by the `parsePath` method, described above, which can be overridden to customize the path parsing behavior.
+
 ### Throwing Errors
 
 You may throw errors (and leave them uncaught) from the response methods and these should be caught and handled by protocol the handler. For REST requests/responses, this will result in an error response. By default the status code will be 500. You can assign a property of `statusCode` to errors to indicate the HTTP status code that should be returned. For example: