Code

Batched job processing (opt-in)#474

Code

Open

benjie wants to merge 156 commits intomaingraphile/worker:mainfrom pool-centricgraphile/worker:pool-centricCopy head branch name to clipboard

+3,291-591

Commits

Commits on Nov 22, 2024

• Start work on batch fetching jobs

  benjie

  committed
  3b4092b

  Copy full SHA for 3b4092b

• Batched job fetching with watermark

  benjie

  committed
  ecaa062

  Copy full SHA for ecaa062

• Fix getJob call to reflect changes in #469

  benjie

  committed
  9b2ba39

  Copy full SHA for 9b2ba39

• Hoist completeJob and failJob

  benjie

  committed
  484e02c

  Copy full SHA for 484e02c

• Refactor failJob/completeJob in preparation for batching

  benjie

  committed
  862251d

  Copy full SHA for 862251d

• Stub batch function

  benjie

  committed
  2382ed1

  Copy full SHA for 2382ed1

• Implement batching function

  benjie

  committed
  9538247

  Copy full SHA for 9538247

• Lint fix

  benjie

  committed
  3752cdc

  Copy full SHA for 3752cdc

• Tweak graceful/forceful shutdown handover

  benjie

  committed
  113ef71

  Copy full SHA for 113ef71

• Evaluate envvar just once up top

  benjie

  committed
  1821691

  Copy full SHA for 1821691

• Release batches before releasing worker pool

  benjie

  committed
  c542aa3

  Copy full SHA for c542aa3

• Refactor

  benjie

  committed
  93fb462

  Copy full SHA for 93fb462

• Warn about bad batch settings

  benjie

  committed
  1cab674

  Copy full SHA for 1cab674

• Refactor in preparation for new queue

  benjie

  committed
  fde4341

  Copy full SHA for fde4341

• Refactor again and explain the purpose of LocalQueue and how it works

  benjie

  committed
  64f6aab

  Copy full SHA for 64f6aab

• Implement localQueue

  benjie

  committed
  044aa5a

  Copy full SHA for 044aa5a

• Return to POLLING from WAITING; RELEASE mode.

  benjie

  committed
  33faf35

  Copy full SHA for 33faf35

• Fix releasing from POLLING mode

  benjie

  committed
  8ac92e6

  Copy full SHA for 8ac92e6

• Concurrency 24 works better on my new machine

  benjie

  committed
  d7e381e

  Copy full SHA for d7e381e

• Wait for background tasks to complete before releasing

  benjie

  committed
  1f6a4b0

  Copy full SHA for 1f6a4b0

• Default settings

  benjie

  committed
  6b51f26

  Copy full SHA for 6b51f26

• Remove double map

  benjie

  committed
  016ac39

  Copy full SHA for 016ac39

• More efficient implementation

  benjie

  committed
  59f4d99

  Copy full SHA for 59f4d99

• Neater API

  benjie

  committed
  52dd450

  Copy full SHA for 52dd450

• Marginally more efficient

  benjie

  committed
  af7a26b

  Copy full SHA for af7a26b

• Need to test more jobs because we're too fast now

  benjie

  committed
  d05c966

  Copy full SHA for d05c966

• Boolean comparison first

  benjie

  committed
  9da3ed8

  Copy full SHA for 9da3ed8

• Single job different statement, plus disabled alternative approach

  benjie

  committed
  99e9231

  Copy full SHA for 99e9231

• Listen with new LocalQueue too

  benjie

  committed
  f757f0d

  Copy full SHA for f757f0d

• Lint fixes

  benjie

  committed
  37b5055

  Copy full SHA for 37b5055

• Oops

  benjie

  committed
  222c979

  Copy full SHA for 222c979

• 0.17.0-canary.6c2c85c

  benjie

  committed
  96b2126

  Copy full SHA for 96b2126

• 0.17.0-canary.67dbcb6

  benjie

  committed
  333c9c2

  Copy full SHA for 333c9c2

• [ci] bump

  benjie

  committed
  56ca277

  Copy full SHA for 56ca277

• [ci] bump

  benjie

  committed
  afd7059

  Copy full SHA for afd7059

• [ci] bump

  benjie

  committed
  25f1fe8

  Copy full SHA for 25f1fe8

• [ci] bump

  benjie

  committed
  307683e

  Copy full SHA for 307683e

• [ci] bump

  benjie

  committed
  28a173f

  Copy full SHA for 28a173f

• [ci] bump

  benjie

  committed
  32f8299

  Copy full SHA for 32f8299

• [ci] bump

  benjie

  committed
  a119a8b

  Copy full SHA for a119a8b

• [ci] bump

  benjie

  committed
  2b7bd90

  Copy full SHA for 2b7bd90

• [ci] bump

  benjie

  committed
  c43bdf9

  Copy full SHA for c43bdf9

• Update src/localQueue.ts

  benjie

  committed
  6d1cfa4

  Copy full SHA for 6d1cfa4

• Move localQueue options into their own object

  benjie

  committed
  8868d1f

  Copy full SHA for 8868d1f

• Introduce STARTING mode

  benjie

  committed
  9767d48

  Copy full SHA for 9767d48

• Add localQueue refetchDelay feature: be kind on DB when queue is near-empty

  benjie

  committed
  6d19367

  Copy full SHA for 6d19367

• Randomize the abort threshold

  benjie

  committed
  a087440

  Copy full SHA for a087440

• Cleaner branching without setting fetchAgain by accident

  benjie

  committed
  e3ec005

  Copy full SHA for e3ec005

• Lint

  benjie

  committed
  d98598d

  Copy full SHA for d98598d

• Don't trigger refetch once setting mode to released

  benjie

  committed
  8f72ece

  Copy full SHA for 8f72ece

• Fix bug and clarify variable name

  benjie

  committed
  098d203

  Copy full SHA for 098d203

• Comments and variable renames for clarity

  benjie

  committed
  b45b59d

  Copy full SHA for b45b59d

• Clarify and fix behavior of refetch delay

  benjie

  committed
  835502f

  Copy full SHA for 835502f

• Reduce diff

  benjie

  committed
  abc16f8

  Copy full SHA for abc16f8

• 0.17.0-canary.379fb2e

  benjie

  committed
  187c4e0

  Copy full SHA for 187c4e0

• Update performance results

  benjie

  committed
  f044bfe

  Copy full SHA for f044bfe

• Beginnings of tower defence

  benjie

  committed
  da79374

  Copy full SHA for da79374

• More waves and don't wait between waves

  benjie

  committed
  4b8462c

  Copy full SHA for 4b8462c

• Add some events so we can collect stats from tower defence

  benjie

  committed
  6c7d92d

  Copy full SHA for 6c7d92d

• Easier to read results

  benjie

  committed
  24d51c1

  Copy full SHA for 24d51c1

• Extreme values demonstrate problem

  benjie

  committed
  e295c72

  Copy full SHA for e295c72

• Produce the problematic behavior I was worried about

  benjie

  committed
  5749bed

  Copy full SHA for 5749bed

• Clarify doc

  benjie

  committed
  993cf44

  Copy full SHA for 993cf44

• Rename abortThreshold to maxAbortThreshold and update docs and implementation

  benjie

  committed
  c0d7e9b

  Copy full SHA for c0d7e9b

• More waves

  benjie

  committed
  6b9179c

  Copy full SHA for 6b9179c

• Shorter waves

  benjie

  committed
  5126d39

  Copy full SHA for 5126d39

• Ive we've already been running jobs slowly, don't sleep so long

  benjie

  committed
  81bc1bb

  Copy full SHA for 81bc1bb

• Track empty fetches

  benjie

  committed
  1308d84

  Copy full SHA for 1308d84

• Use addJobs API and track latency

  benjie

  committed
  57b411e

  Copy full SHA for 57b411e

• Can't delete DB if workerUtils is connected

  benjie

  committed
  d2e4f12

  Copy full SHA for d2e4f12

• Stupid mistake

  benjie

  committed
  daa2a62

  Copy full SHA for daa2a62

• Fix division by zero error

  benjie

  committed
  ee97cde

  Copy full SHA for ee97cde

• Add another wave, and increase local queue size by 1

  benjie

  committed
  f000ad0

  Copy full SHA for f000ad0

• Use JSON rather than constructing tuples; batch at 1M

  benjie

  committed
  b445bf6

  Copy full SHA for b445bf6

• 0.17.0-canary.6aeb577

  benjie

  committed
  1afae2c

  Copy full SHA for 1afae2c

• 0.17.0-canary.9817f67

  benjie

  committed
  51c5648

  Copy full SHA for 51c5648

• "Breaking" change

  benjie

  committed
  6a7dd03

  Copy full SHA for 6a7dd03

• Fix migration tests

  benjie

  committed
  c141856

  Copy full SHA for c141856

• Fix all the issues from TypeScript strict mode

  benjie

  committed
  e7bd577

  Copy full SHA for e7bd577

• Move release of batching of fail/complete to terminate

  benjie

  committed
  91f1a5e

  Copy full SHA for 91f1a5e

• Test that graceful shutdown works in runOnce

  benjie

  committed
  f4b7c4a

  Copy full SHA for f4b7c4a

• Implement middleware system

  benjie

  committed
  aa43296

  Copy full SHA for aa43296

• gracefulShutdown middleware

  benjie

  committed
  b746b4a

  Copy full SHA for b746b4a

• Shutting down non-continuous worker should go via gracefulShutdown path

  benjie

  committed
  b57766a

  Copy full SHA for b57766a

• f

  benjie

  committed
  4e72225

  Copy full SHA for 4e72225

• Add middleware for forcefulShutdown, guarantee event emitter, rework init middleware

  benjie

  committed
  17ccefd

  Copy full SHA for 17ccefd

• Clarify relationship between CompiledSharedOptions and WorkerPluginContext and share relevant types

  benjie

  committed
  49bcbee

  Copy full SHA for 49bcbee

• Fix lint issues

  benjie

  committed
  8035788

  Copy full SHA for 8035788

• Only shutdown if we're not already doing so

  benjie

  committed
  ddce61f

  Copy full SHA for ddce61f

• Fix onTerminate handling

  benjie

  committed
  18835b1

  Copy full SHA for 18835b1

• Forceful shutdown should error if something went wrong

  benjie

  committed
  c83069a

  Copy full SHA for c83069a

• Forceful shutdown should result in promise rejection

  benjie

  committed
  40afb61

  Copy full SHA for 40afb61

• Refactoring to ensure all cases are handled

  benjie

  committed
  cf50392

  Copy full SHA for cf50392

• We don't release job releasers here any more.

  benjie

  committed
  6afe773

  Copy full SHA for 6afe773

• Ensure that LocalQueue exits with the correct status (e.g. rejects if returning jobs fails)

  benjie

  committed
  4bb7227

  Copy full SHA for 4bb7227

• Fix types in a test

  benjie

  committed
  4b6b5dc

  Copy full SHA for 4b6b5dc

• I threw it myself, I know what it is

  benjie

  committed
  9848f8c

  Copy full SHA for 9848f8c

• Ensure deactivate happens at most once, and yields same errors if called a second time (e.g. from forcefulShutdown)

  benjie

  committed
  b62b8e3

  Copy full SHA for b62b8e3

• Ensure forcefulShutdown and gracefulShutdown yield the same promises when called a second time

  benjie

  committed
  0bdcd9f

  Copy full SHA for 0bdcd9f

• Graceful shutdown should not complete if forceful shutdown has begun

  benjie

  committed
  6bc93fe

  Copy full SHA for 6bc93fe

• Cleanup gracefulShutdown handover to forcefulShutdown

  benjie

  committed
  b3150ac

  Copy full SHA for b3150ac

• More consistently handle errors in forcefulShutdown

  benjie

  committed
  a561feb

  Copy full SHA for a561feb

• Move promise handling outside of the gracefulShutdown/forcefulShutdown hooks

  benjie

  committed
  3c3a237

  Copy full SHA for 3c3a237

• Only finish once

  benjie

  committed
  28f3b0e

  Copy full SHA for 28f3b0e

• The _moment_ that deactivate is called, all future getJob calls should immediately return undefined

  benjie

  committed
  7aa5955

  Copy full SHA for 7aa5955

• Apply timeouts to forceful shutdown

  benjie

  committed
  174c76c

  Copy full SHA for 174c76c

• forcefulShutdown can be successful; channel errors separately

  benjie

  committed
  afcaaed

  Copy full SHA for afcaaed

• On unexpected worker exit, shut down entire pool (gracefully)

  benjie

  committed
  fdd041c

  Copy full SHA for fdd041c

Commits on Dec 6, 2024

• Upgrade graphile-config, simplify middleware handlers

  benjie

  committed
  84fb704

  Copy full SHA for 84fb704

• New event emitter each time

  benjie

  committed
  88dba0f

  Copy full SHA for 88dba0f

• Move to resolvePreset

  benjie

  committed
  fc3f4a5

  Copy full SHA for fc3f4a5

• Rename completeJob -> completeJobs

  benjie

  committed
  d880d2f

  Copy full SHA for d880d2f

• Rename getJob -> getJobs

  benjie

  committed
  3a6195e

  Copy full SHA for 3a6195e

• Rename failJob -> batchFailJobs

  benjie

  committed
  c3946cd

  Copy full SHA for c3946cd

• Rename to batch for consistency

  benjie

  committed
  6b8cbab

  Copy full SHA for 6b8cbab

Commits on Dec 7, 2024

• Add and correct LocalQueue comments

  benjie

  committed
  e922c13

  Copy full SHA for e922c13

• Overhaul error handling in returnJobs

  benjie

  committed
  4149e90

  Copy full SHA for 4149e90

• Minor refactoring and more comments

  benjie

  committed
  0c71374

  Copy full SHA for 0c71374

Commits on Dec 9, 2024

• Tidy

  benjie

  committed
  3a0161c

  Copy full SHA for 3a0161c

• Add to graceful shutdown errors

  benjie

  committed
  31fc5c1

  Copy full SHA for 31fc5c1

• Make code clearer

  benjie

  committed
  16fb4cd

  Copy full SHA for 16fb4cd

• Give 'batch' retry logic with backpressure

  benjie

  committed
  5a75d76

  Copy full SHA for 5a75d76

• Reason for internal shutdown trigger

  benjie

  committed
  eb9c69d

  Copy full SHA for eb9c69d

• Only trigger shutdown if not already shutting down

  benjie

  committed
  d7693bc

  Copy full SHA for d7693bc

• Trigger gracefulShutdown on forcefulShutdown

  benjie

  committed
  53f4580

  Copy full SHA for 53f4580

• Use correct variable

  benjie

  committed
  6d344f0

  Copy full SHA for 6d344f0

• Give users the ability to change how premature worker shutdown is handled

  benjie

  committed
  f43f47f

  Copy full SHA for f43f47f

• Refactor

  benjie

  committed
  9c411dd

  Copy full SHA for 9c411dd

• Clarify message

  benjie

  committed
  37ea013

  Copy full SHA for 37ea013

• Give pool error handlers

  benjie

  committed
  a5ff74d

  Copy full SHA for a5ff74d

Commits on Dec 10, 2024

• Merge branch 'main' into pool-centric

  benjie

  committed
  6e22aff

  Copy full SHA for 6e22aff

• Comments

  benjie

  committed
  01a083f

  Copy full SHA for 01a083f

• Use new retry delay calculator

  benjie

  committed
  0bd0b77

  Copy full SHA for 0bd0b77

• Ensure errors are escalated to batch() so backpressure applies

  benjie

  committed
  c695bcc

  Copy full SHA for c695bcc

• Refactor

  benjie

  committed
  ad686d9

  Copy full SHA for ad686d9

• Use delays from retryables in batch

  benjie

  committed
  e4545e1

  Copy full SHA for e4545e1

• Handle our own errors via onError

  benjie

  committed
  2b250e9

  Copy full SHA for 2b250e9

• Move parallelism into config

  benjie

  committed
  abf2463

  Copy full SHA for abf2463

• Have tasks sleep for 250ms

  benjie

  committed
  79d2b72

  Copy full SHA for 79d2b72

• Centralize configuration of sleepTime

  benjie

  committed
  4905a93

  Copy full SHA for 4905a93

• Add a base sleep duration estimate

  benjie

  committed
  ba43590

  Copy full SHA for ba43590

• Set threshold to one less than the local queue size

  benjie

  committed
  f6a6889

  Copy full SHA for f6a6889

Commits on Dec 11, 2024

• 0.17.0-canary.f6a6889

  benjie

  committed
  698f522

  Copy full SHA for 698f522

• All event handlers should provide ctx

  benjie

  committed
  2ae2f2b

  Copy full SHA for 2ae2f2b

• Kill command

  benjie

  committed
  f676c3d

  Copy full SHA for f676c3d

• Aggregate error messages

  benjie

  committed
  676f6e9

  Copy full SHA for 676f6e9

• .finally() promises still need error handling

  benjie

  committed
  1d9bb10

  Copy full SHA for 1d9bb10

• .finally() promises still need error handling (more)

  benjie

  committed
  852b279

  Copy full SHA for 852b279

• Add migration table data to dump

  benjie

  committed
  2f8d378

  Copy full SHA for 2f8d378

• Upgrade to latest graphile-config

  benjie

  committed
  0e88c16

  Copy full SHA for 0e88c16

• 0.17.0-canary.0e88c16

  benjie

  committed
  e3fa698

  Copy full SHA for e3fa698

• Move tsconfig to devDependencies

  benjie

  committed
  209dbf6

  Copy full SHA for 209dbf6

• Trim grant and stable timestamps

  benjie

  committed
  0103634

  Copy full SHA for 0103634

• Lint

  benjie

  committed
  6f1217a

  Copy full SHA for 6f1217a

• Enable more clash codes

  benjie

  committed
  1fcb2a0

  Copy full SHA for 1fcb2a0

• 0.17.0-canary.1fcb2a0

  benjie

  committed
  c665c80

  Copy full SHA for c665c80