Skip to content

Switch task queue to procrastinate based on psql #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 96 commits into
base: main
Choose a base branch
from
Open

Switch task queue to procrastinate based on psql #11

wants to merge 96 commits into from

Conversation

simonwoerpel
Copy link
Member

@simonwoerpel simonwoerpel commented Jun 19, 2025 •
edited
Loading

this is work in progress

This is a refactor to swap out servicelayer worker (and redis queue) with procrastinate.
It as well separates the analysis code into ftm-analyze

TODO:

catileptic and others added 30 commits June 16, 2025 15:53
@catileptic @simonwoerpel
Add openaleph-procrastinate. Bump versions to satisfy dependencies (p…
190c5ff 
…oetry lock).
@simonwoerpel
🧑‍💻 Add pre-commit, use requirements.txt, upgrade to python3.13
a46a028
@simonwoerpel
🧑‍💻 Add dev requirements only for test build
109b53a
@simonwoerpel
🔥 (github) Drop daily cache job
43a02d1
@simonwoerpel
✅ (tests/test_pdf) Fix whitespace errors from test results
f977f90
@simonwoerpel
🔨 (make) Build before test
d6ad514
@simonwoerpel
👷 Inline base build
8ef339f
@simonwoerpel
🚧 Tweak builds and tags
9e376e4
@simonwoerpel
👷 (github) Skip intermediate arm46 build for tests
39a2385
@simonwoerpel
👷 (github) Skip cache-from [tmp]
03f86fd
@simonwoerpel
Revert "👷 (github) Skip cache-from [tmp]"
ff1a8c4 
This reverts commit 03f86fd.
@simonwoerpel
👷 (github/docker) Try this
4adcfb2
@simonwoerpel
🚨 Apply black
099aef5
@simonwoerpel
👷 (github/docker) Don't use registry cache
4071354
@simonwoerpel
🧪 (test_image) Skip gif test
dd8bb0c
@simonwoerpel
👷 (github/docker) maybe this
9adf413
@simonwoerpel
✨ Boilerplate ingest task for procrastinate
53d0a8a
@simonwoerpel
🔧 Use pydantic_settings
6457ebd
@simonwoerpel
📌 Use openaleph-procrastinate from git
90292be
@simonwoerpel
⚰️ Drop TranscriptionSupport
99046a8
@simonwoerpel
🔥 Remove analysis part
33a33f0
@simonwoerpel
🔥 Remove servicelayer worker
ef84cf7
@simonwoerpel
♻️ Refactor manager and supports to work with procrastinate
b1dadd8
@simonwoerpel
🚧 Make procrastinate task to work with manager
e64425b
@simonwoerpel
➖ languagecodes, pantomime -> rigour
13bbf14
@simonwoerpel
🔊 Tweak global logging
0dacf45
@simonwoerpel
♻️ Refactor cli with typer
bf2a15e
@simonwoerpel
🧪 Make tests work with procrastinate refactor
8873af6
@simonwoerpel
🩹 (ingestors/email) Use relative path
bf26349
@simonwoerpel
✨ (support/timestamp) Fall back to dateparser for unknown formats
bddb556
@simonwoerpel
🔥 Drop unused settings
d0f6f04
@simonwoerpel
⬆️ openaleph-procrastinate 0.0.14
6536961
@simonwoerpel
🚧 (cli) Make foreign_id optional
9058612
@simonwoerpel
✅ Add e2e testing with minio
1501eb1
@simonwoerpel
⬆️ openaleph-procrastinate 0.0.16
1eae3cf
@simonwoerpel
⬆️ openaleph-procrastinate 0.0.16
ee94e1f
@simonwoerpel
💚 (github) Skip e2e
67629db
@simonwoerpel
⬆️ ftmq 4.1.1, openaleph-procrastinate 0.0.18
ce7c91c
@simonwoerpel
🚧 (tests/e2e) Adjustments
3c85e81
@simonwoerpel
🔖 Bump version: 3.24.0 → 5.0.0rc1
27c158a
@simonwoerpel
💚 (github) Enable e2e again
27996cb
@simonwoerpel
⬆️ openaleph-procrastinate 0.0.20
33d6417
@simonwoerpel
🚧 (ingestors/image) Explicitly close PIL obj after processing
999b21d
@simonwoerpel
🚧 (support/shell) Write to subprocess special DEVNULL
6524ffc
@simonwoerpel
🚧 (ingestors/access) Wrap subprocess call in context manager
76e0735
@simonwoerpel
🚧 (ingestors/csv) Properly use context manager for file open
854b1de
@simonwoerpel
🚧 (support/ocr) Clean up OCR engine after use
bd75599
@simonwoerpel
⚗️ memray
48b8dbc
@simonwoerpel
🔧 (settings) Properly configure servicelayer tags
c2a5998
@simonwoerpel
🚧 (tasks) Collect garbage, just in case
fdad9bf
@simonwoerpel
📌 Pin olefile<0.47 as this leaks crazy memory
f06f42c
@simonwoerpel
Merge pull request #14 from openaleph/fix/address-memory-issues
33ebebf 
Fix/address memory issues
@simonwoerpel
📌 Fix RC version string
8a67f36
@simonwoerpel
⬆️ All the things
a9b2b9b
@simonwoerpel
🔖 Bump version: 5.0.0-rc1 → 5.0.0-rc2
b628c80
@simonwoerpel simonwoerpel changed the title [WIP] Feat/procrastinate Switch task queue to procrastinate based on psql Aug 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@catileptic catileptic catileptic requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@simonwoerpel @catileptic