Data Loss Scenario: Consumer metadata causes Limits policy to drop messages on the floor

[vigith]

Describe the bug

Unprocessed == 0 while Stream seq in the stream metadata is moving ahead of the consumer seq number.

Consumer Metadata

Last Delivered Message: Consumer sequence: 221,160,045 Stream sequence: 264,153,054
    Acknowledgment Floor: Consumer sequence: 221,160,045 Stream sequence: 264,153,054
        Outstanding Acks: 0 out of maximum 25,000
    Redelivered Messages: 0
    Unprocessed Messages: 0

Stream Metadata

Messages: 100,000
                 Bytes: 670 MiB
        First Sequence: 226,498,356 @ 2024-07-31 18:05:40 UTC
         Last Sequence: 226,598,355 @ 2024-07-31 18:06:02 UTC

Metadata wrong even without OOM

To Reproduce

It is yet to be figured out. This problem is seen only when we have OOMs on JetStream. The pipeline where we saw this problem was inserting 64MB messages at high TPS, causing OOMs.

Tasks

Preview Give feedback

1. Enable with WorkQueue policy #1885
   area/isb-jetstream +1
   
2. Use better memory limits and run the pipeline to look for OOMs #1886
   area/isb-jetstream opex +2
   
3. setup alert to page if ISB pods are restarting #1887
   area/isb-jetstream opex +2
   
4. emit ISB restart metric via controller #1889
   area/isb opex wontfix +3
   
5. pipeline buffer length cannot be more than ISB maxMsg
6. figure out why Consumer sequence > Stream sequence #1891
   opex +1
   
7. is stream maxBytes the total RAM memory required per stream? #1893
   opex +1
   
8. what should be the MaxBytes for a xMB payload stream (WorkQueue)? #1899
   opex +1
   
9. what should be the the bufferLength for a xMB stream (Limits)? #1900
   opex +1
   
10. setup perf environment using Kafka as the source #1907
    opex +1
    
11. Messages dropped in Sinks when buffer is full and sink is failing #1551
    area/sink bug +2
    
12. Buffer Usage Calculation and ISB Writing Race Condition: Potential Data Loss in High-Throughput Pipelines #1554
    area/isb area/isb-jetstream bug +3
13. Suspected Memory leak on JetStream #2241
    +0