|
| 1 | +# IndexedDB: getAllEntries() |
| 2 | + |
| 3 | +## Author: |
| 4 | +- [Steve Becker](https://github.com/SteveBeckerMSFT) |
| 5 | + |
| 6 | +## Participate |
| 7 | +- https://github.com/w3c/IndexedDB/issues/206 |
| 8 | + |
| 9 | +## Introduction |
| 10 | + |
| 11 | +[`IndexedDB`](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) is a transactional database for client-side storage. Each record in the database contains a key-value pair. [`getAll()`](https://developer.mozilla.org/en-US/docs/Web/API/IDBObjectStore/getAll) enumerates database record values sorted by key in ascending order. [`getAllKeys()`](https://developer.mozilla.org/en-US/docs/Web/API/IDBObjectStore/getAllKeys) enumerates database record primary keys sorted by key in ascending order. |
| 12 | + |
| 13 | +This explainer proposes a new operation, `getAllEntries()`, which combines [`getAllKeys()`](https://developer.mozilla.org/en-US/docs/Web/API/IDBObjectStore/getAllKeys) with [`getAll()`](https://developer.mozilla.org/en-US/docs/Web/API/IDBObjectStore/getAll) to enumerate both primary keys and values at the same time. For an [`IDBIndex`](https://developer.mozilla.org/en-US/docs/Web/API/IDBIndex), `getAllEntries()` also provides the record's index key in addition to the primary key and value. Lastly, `getAllEntries()` offers a new option to enumerate records sorted by key in descending order. |
| 14 | + |
| 15 | +## WebIDL |
| 16 | + |
| 17 | +```js |
| 18 | +dictionary IDBGetAllEntriesOptions { |
| 19 | + // A key or an `IDBKeyRange` identifying the records to retrieve. |
| 20 | + any query = null; |
| 21 | + |
| 22 | + // The maximum number of results to retrieve. |
| 23 | + [EnforceRange] unsigned long count; |
| 24 | + |
| 25 | + // Determines how to enumerate and sort results. |
| 26 | + // Use 'prev' to enumerate and sort results by key in descending order. |
| 27 | + IDBCursorDirection direction = 'next'; |
| 28 | +}; |
| 29 | + |
| 30 | +[Exposed=(Window,Worker)] |
| 31 | +partial interface IDBObjectStore { |
| 32 | + // After the `getAllEntries()` request completes, the `IDBRequest::result` property |
| 33 | + // contains an array of entries: |
| 34 | + // `[[primaryKey1, value1], [primaryKey2, value2], ... ]` |
| 35 | + [NewObject, RaisesException] |
| 36 | + IDBRequest getAllEntries(optional IDBGetAllEntriesOptions options = {}); |
| 37 | +} |
| 38 | + |
| 39 | +[Exposed=(Window,Worker)] |
| 40 | +partial interface IDBIndex { |
| 41 | + // Produces the same type of results as `IDBObjectStore::getAllEntries()` above, |
| 42 | + // but each entry also includes the record's index key at array index 2: |
| 43 | + // `[[primaryKey1, value1, indexKey1], [primaryKey2, value2, indexKey2], ... ]` |
| 44 | + [NewObject, RaisesException] |
| 45 | + IDBRequest getAllEntries(optional IDBGetAllEntriesOptions options = {}); |
| 46 | +} |
| 47 | +``` |
| 48 | + |
| 49 | +## Goals |
| 50 | + |
| 51 | +Decrease the latency of database read operations. By retrieving the primary key, value and index key for database records through a single operation, `getAllEntries()` reduces the number of JavaScript events required to read records. Each JavaScript event runs as a task on the main JavaScript thread. These tasks can introduce overhead when reading records requires a sequence of tasks that go back and forth between the main JavaScript thread and the IndexedDB I/O thread. |
| 52 | + |
| 53 | +For batched record iteration, for example, retrieving N records at a time, the primary and index keys provided by `getAllEntries()` can eliminate the need for an [`IDBCursor`](https://developer.mozilla.org/en-US/docs/Web/API/IDBCursor), which further reduces the number of JavaScript events required. To read the next N records, instead of advancing a cursor to determine the range of the next batch, getAllEntries() can use the primary key or the index key retrieved by the results from the previous batch. |
| 54 | + |
| 55 | +## Key scenarios |
| 56 | + |
| 57 | +### Support paginated cursors using batched record iteration |
| 58 | + |
| 59 | +Many scenarios read N database records at a time, waiting to read the next batch of records until needed. For example, a UI may display N records, starting with the last record in descending order. As the user scrolls, the UI will display new content by reading the next N records. |
| 60 | + |
| 61 | +To support this access pattern, the UI calls `getAllEntries()` with the options `direction: 'prev'` and `count: N` to retrieve N records at a time in descending order. After the initial batch, the UI must specify the upper bound of the next batch using the primary key or index key from the `getAllEntries()` results of the previous batch. |
| 62 | + |
| 63 | +```js |
| 64 | +// Define a helper that creates a basic read transaction using `getAllEntries()`. |
| 65 | +// Wraps the transaction in a promise that resolves with the query results or |
| 66 | +// rejects after an error. Queries `object_store_name` unless `optional_index_name` |
| 67 | +// is defined. |
| 68 | +async function get_all_entries_with_promise( |
| 69 | + database, object_store_name, query_options, optional_index_name) { |
| 70 | + return await new Promise((fulfill, reject) => { |
| 71 | + // Create a read-only transaction. |
| 72 | + const read_transaction = database.transaction(object_store_name, 'readonly'); |
| 73 | + const object_store = read_transaction.objectStore(object_store_name); |
| 74 | + |
| 75 | + let query_target = object_store; |
| 76 | + if (optional_index_name) { |
| 77 | + query_target = object_store.index(optional_index_name); |
| 78 | + } |
| 79 | + |
| 80 | + // Start the `getAllEntries()` request. |
| 81 | + const request = query_target.getAllEntries(query_options); |
| 82 | + |
| 83 | + // Resolve the promise with the array of entries after success. |
| 84 | + request.onsuccess = event => { |
| 85 | + fulfill(request.result); |
| 86 | + }; |
| 87 | + |
| 88 | + // Reject promise with an error after failure. |
| 89 | + request.onerror = () => { reject(request.error); }; |
| 90 | + read_transaction.onerror = () => { reject(read_transaction.error); }; |
| 91 | + }); |
| 92 | +} |
| 93 | + |
| 94 | +// Create a simple reverse iterator where each call to `next()` retrieves |
| 95 | +// `batch_size` database records in descending order from an `IDBIndex` with |
| 96 | +// unique keys. |
| 97 | +function reverse_idb_index_iterator( |
| 98 | + database, object_store_name, index_name, batch_size) { |
| 99 | + // Define iterator state. |
| 100 | + let done = false; |
| 101 | + |
| 102 | + // Begin the iteration unbounded to retrieve the last records in the 'IDBIndex'. |
| 103 | + let next_upper_bound = null; |
| 104 | + |
| 105 | + // Gets the next `batch_size` entries. |
| 106 | + this.next = async function () { |
| 107 | + if (done) { |
| 108 | + return []; |
| 109 | + } |
| 110 | + |
| 111 | + let query; |
| 112 | + if (next_upper_bound) { |
| 113 | + query = IDBKeyRange.upperBound(next_upper_bound, /*is_exclusive=*/true); |
| 114 | + } else { |
| 115 | + // The very first query retrieves the last `batch_size` records. |
| 116 | + } |
| 117 | + |
| 118 | + const entries = await get_all_entries_with_promise( |
| 119 | + database, object_store_name, |
| 120 | + /*options=*/{ query, count: batch_size, direction: 'prev' }, index_name); |
| 121 | + |
| 122 | + if (entries.length > 0) { |
| 123 | + // Store the upper bound for the next iteration. |
| 124 | + const last_entry = entries[entries.length-1]; |
| 125 | + next_upper_bound = /*index_key=*/last_entry[2]; |
| 126 | + } else { |
| 127 | + // We've iterated through all the database records! |
| 128 | + done = true; |
| 129 | + } |
| 130 | + return entries; |
| 131 | + }; |
| 132 | +}; |
| 133 | + |
| 134 | +// Get the last 5 records in the `IDBIndex` named `my_index`. |
| 135 | +const reverse_iterator = new reverse_idb_index_iterator( |
| 136 | + database, 'my_object_store', 'my_index', /*batch_size=*/5); |
| 137 | + |
| 138 | +let results = await reverse_iterator.next(); |
| 139 | + |
| 140 | +// Get the next batch of 5 records. |
| 141 | +results = await reverse_iterator.next(); |
| 142 | +``` |
| 143 | + |
| 144 | +### Read query results into a Map or Object |
| 145 | + |
| 146 | +Developers may use the results from `getAllEntries()` to construct a new [`Map`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map) or [`Object`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object) that contains a key-value pair for each database record returned by the query. |
| 147 | + |
| 148 | +```js |
| 149 | +// These examples use the `get_all_entries_with_promise()` helper defined above. |
| 150 | +// |
| 151 | +// Example 1: Read the first 5 database records from the `IDBObjectStore` into a `Map`. |
| 152 | +const result_map = new Map( |
| 153 | + await get_all_entries_with_promise( |
| 154 | + database, 'my_object_store', /*query_options=*/{ count: 5 })); |
| 155 | + |
| 156 | +// Returns the database record value for `key` when the record exists in `result_map`. |
| 157 | +let value = result_map.get(key); |
| 158 | + |
| 159 | +// Use the following to create an iterator for each database record in `result_map`: |
| 160 | +const primary_key_iterator = result_map.keys(); |
| 161 | +const value_iterator = result_map.values(); |
| 162 | +const entry_iterator = result_map.entries(); // Enumerate both primary keys and values. |
| 163 | + |
| 164 | +// Example 2: Read the database records from range `min_key` to `max_key` into an `Object`. |
| 165 | +const result_object = Object.fromEntries( |
| 166 | + await get_all_entries_with_promise( |
| 167 | + database, 'my_object_store', /*query_options=*/{ query: IDBKeyRange.bound(min_key, max_key) })); |
| 168 | + |
| 169 | +// Returns the database record value for `key` when the record exists in `result_object`. |
| 170 | +value = result_object[key]; |
| 171 | + |
| 172 | +// Use the following to create an array containing each database record in `result_object`: |
| 173 | +const keys = Object.keys(result_object); |
| 174 | +const values = Object.values(result_object); |
| 175 | +const entries = Object.entries(result_object); // Produces the same array of key/value pairs |
| 176 | + // as `IDBObjectStore::getAllEntries()`. |
| 177 | +``` |
| 178 | + |
| 179 | +## Stakeholder Feedback / Opposition |
| 180 | + |
| 181 | +- Web Developers: Positive |
| 182 | + - Developers have reported the limitations addressed by `getAllEntries()`. A few examples: |
| 183 | + - ["You cannot build a paginated cursor in descending order."](https://nolanlawson.com/2021/08/22/speeding-up-indexeddb-reads-and-writes/) |
| 184 | + - ["An example where getAll() could help but needs to retrieve the index key and primary key."](https://stackoverflow.com/questions/44349168/speeding-up-indexeddb-search-with-multiple-workers) |
| 185 | +- Chromium: Positive |
| 186 | +- Webkit: No signals |
| 187 | +- Gecko: No signals |
| 188 | + |
| 189 | +## References & acknowledgements |
| 190 | + |
| 191 | +Special thanks to [Joshua Bell](https://github.com/inexorabletash) who proposed `getAllEntries()` in the [W3C IndexedDB issue](https://github.com/w3c/IndexedDB/issues/206). |
| 192 | + |
| 193 | +Many thanks for valuable feedback and advice from: |
| 194 | + |
| 195 | +- [Rahul Singh](https://github.com/rahulsingh-msft) |
| 196 | +- [Foromo Daniel Soromou](https://github.com/fosoromo_microsoft) |
0 commit comments