You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/features/batch.md
+9-49Lines changed: 9 additions & 49 deletions
Original file line number
Diff line number
Diff line change
@@ -49,16 +49,15 @@ journey
49
49
Records expired: 1: Failure
50
50
```
51
51
52
-
This behavior changes when you enable [ReportBatchItemFailures feature](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting) in your Lambda function event source configuration:
52
+
This behavior changes when you enable the [ReportBatchItemFailures feature](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-errorhandling.html#services-sqs-batchfailurereporting) in your Lambda function event source configuration:
53
53
54
-
<!-- markdownlint-disable MD013 -->
55
54
*[**SQS queues**](#sqs-standard). Only messages reported as failure will return to the queue for a retry, while successful ones will be deleted.
56
55
*[**Kinesis data streams**](#kinesis-and-dynamodb-streams) and [**DynamoDB streams**](#kinesis-and-dynamodb-streams). Single reported failure will use its sequence number as the stream checkpoint. Multiple reported failures will use the lowest sequence number as checkpoint.
57
56
58
57
<!-- HTML tags are required in admonition content thus increasing line length beyond our limits -->
59
58
<!-- markdownlint-disable MD013 -->
60
59
???+ warning "Warning: This utility lowers the chance of processing records more than once; it does not guarantee it"
61
-
We recommend implementing processing logic in an [idempotent manner](idempotency.md){target="_blank"} wherever possible.
60
+
We recommend implementing processing logic in an [idempotent manner](./idempotency.md){target="_blank"} whenever possible.
62
61
63
62
You can find more details on how Lambda works with either [SQS](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html){target="_blank"}, [Kinesis](https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html){target="_blank"}, or [DynamoDB](https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html){target="_blank"} in the AWS Documentation.
64
63
@@ -72,7 +71,7 @@ Install the library in your project
72
71
npm i @aws-lambda-powertools/batch
73
72
```
74
73
75
-
For this feature to work, you need to **(1)** configure your Lambda function event source to use `ReportBatchItemFailures`, and **(2)** return [a specific response](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting){target="_blank" rel="nofollow"} to report which records failed to be processed.
74
+
For this feature to work, you need to **(1)** configure your Lambda function event source to use `ReportBatchItemFailures`, so that the response from the Batch Processing utility can inform the service which records failed to be processed.
76
75
77
76
Use your preferred deployment framework to set the correct configuration while this utility handles the correct response to be returned.
78
77
@@ -108,8 +107,8 @@ Processing batches from SQS works in three stages:
108
107
2. Define your function to handle each batch record, and use the `SQSRecord` type annotation for autocompletion
109
108
3. Use **`processPartialResponse`** to kick off processing
110
109
111
-
???+ info
112
-
This code example optionally uses Logger for completion.
110
+
!!! note
111
+
By default, the batch processor will process messages in parallel, which does not guarantee the order of processing. If you need to process messages in order, set the [`processInParallel` option to `false`](#sequential-async-processing), or use [`SqsFifoPartialProcessor`for SQS FIFO queues](#fifo-queues).
113
112
114
113
=== "index.ts"
115
114
@@ -147,30 +146,18 @@ By default, we will stop processing at the first failure and mark unprocessed me
147
146
148
147
Enable the `skipGroupOnError` option for seamless processing of messages from various group IDs. This setup ensures that messages from a failed group ID are sent back to SQS, enabling uninterrupted processing of messages from the subsequent group ID.
Note that `SqsFifoPartialProcessor` is synchronous using `processPartialResponseSync`.
172
-
If you need asynchronous processing while preserving the order of messages in the queue, use `SqsFifoPartialProcessorAsync` with `processPartialResponse`.
173
-
174
161
### Processing messages from Kinesis
175
162
176
163
Processing batches from Kinesis works in three stages:
@@ -179,9 +166,6 @@ Processing batches from Kinesis works in three stages:
179
166
2. Define your function to handle each batch record, and use the `KinesisStreamRecord` type annotation for autocompletion
180
167
3. Use **`processPartialResponse`** to kick off processing
181
168
182
-
???+ info
183
-
This code example optionally uses Logger for completion.
184
-
185
169
=== "index.ts"
186
170
187
171
```typescript hl_lines="1-5 9 12 19-21"
@@ -407,32 +391,6 @@ sequenceDiagram
407
391
<i>Kinesis and DynamoDB streams mechanism with multiple batch item failures</i>
408
392
</center>
409
393
410
-
### Async or sync processing
411
-
412
-
There are two processors you can use with this utility:
413
-
414
-
***`BatchProcessor`** and **`processPartialResponse`** – Processes messages asynchronously
415
-
***`BatchProcessorSync`** and **`processPartialResponseSync`** – Processes messages synchronously
416
-
417
-
In most cases your function will be `async` returning a `Promise`. Therefore, the `BatchProcessor` is the default processor handling your batch records asynchronously.
418
-
There are use cases where you need to process the batch records synchronously. For example, when you need to process multiple records at the same time without conflicting with one another.
419
-
For such cases we recommend to use the `BatchProcessorSync` and `processPartialResponseSync` functions.
420
-
421
-
!!! info "Note that you need match your processing function with the right batch processor"
422
-
*If your function is `async` returning a `Promise`, use `BatchProcessor` and `processPartialResponse`
423
-
* If your function is not `async`, use `BatchProcessorSync` and `processPartialResponseSync`
424
-
425
-
The difference between the two processors is in how they handle record processing:
426
-
427
-
***`BatchProcessor`**: By default, it processes records in parallel using `Promise.all()`. However, it also offers an [option](#sequential-async-processing) to process records sequentially, preserving the order.
428
-
***`BatchProcessorSync`**: Always processes records sequentially, ensuring the order is preserved by looping through each record one by one.
429
-
430
-
???+ question "When is this useful?"
431
-
432
-
For example, imagine you need to process multiple loyalty points and incrementally save in a database. While you await the database to confirm your records are saved, you could start processing another request concurrently.
433
-
434
-
The reason this is not the default behaviour is that not all use cases can handle concurrency safely (e.g., loyalty points must be updated in order).
435
-
436
394
## Advanced
437
395
438
396
### Accessing processed messages
@@ -492,6 +450,8 @@ By default, the `BatchProcessor` processes records in parallel using `Promise.al
492
450
493
451
!!! important "If the `processInParallel` option is not provided, the `BatchProcessor` will process records in parallel."
494
452
453
+
When processing records from SQS FIFO queues, we recommend using the [`SqsFifoPartialProcessor`](#fifo-queues) class, which guarantees ordering of records and implements a short-circuit mechanism to skip processing records from a different message group ID.
0 commit comments