Fulltext refactor, including dupes checking #196

griff-rees · 2023-12-06T14:04:46Z

A refactor to include and check FullText for Newspaper.Items.

updates: - [github.com/pre-commit/mirrors-prettier: v3.0.2 → v3.0.3](pre-commit/mirrors-prettier@v3.0.2...v3.0.3) - [github.com/adamchainz/django-upgrade: 1.14.1 → 1.15.0](adamchainz/django-upgrade@1.14.1...1.15.0) - [github.com/asottile/pyupgrade: v3.10.1 → v3.13.0](asottile/pyupgrade@v3.10.1...v3.13.0) - [github.com/psf/black: 23.7.0 → 23.9.1](psf/black@23.7.0...23.9.1) - [github.com/Riverside-Healthcare/djLint: v1.32.1 → v1.34.0](djlint/djLint@v1.32.1...v1.34.0)

…o pre-commit-ci-update-config

…tions

…dedupe

…_to_rm`

… `QuerySets`

updates: - [github.com/pre-commit/mirrors-mypy: v1.7.0 → v1.7.1](pre-commit/mirrors-mypy@v1.7.0...v1.7.1)

…e-config'

Bumps [jupyter-server](https://github.com/jupyter-server/jupyter_server) from 2.9.1 to 2.11.2. - [Release notes](https://github.com/jupyter-server/jupyter_server/releases) - [Changelog](https://github.com/jupyter-server/jupyter_server/blob/main/CHANGELOG.md) - [Commits](jupyter-server/jupyter_server@v2.9.1...v2.11.2) --- updated-dependencies: - dependency-name: jupyter-server dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]>

updates: - [github.com/pre-commit/mirrors-prettier: v3.1.0 → v4.0.0-alpha.4](pre-commit/mirrors-prettier@v3.1.0...v4.0.0-alpha.4) - [github.com/PyCQA/isort: 5.12.0 → 5.13.0](PyCQA/isort@5.12.0...5.13.0) - [github.com/PyCQA/bandit: 1.7.5 → 1.7.6](PyCQA/bandit@1.7.5...1.7.6)

…11.2' into fulltext-refactor

…o fulltext-refactor

…ath` test `fixture`

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.2...3.1.3) --- updated-dependencies: - dependency-name: jinja2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]>

…-refactor

spool and others added 30 commits August 18, 2023 04:32

feat(test): add fixtures and docs to facilitate adding Fulltext

eb4d977

feat(test): add fixtures and docs to facilitate adding Fulltext

c298557

feat(fulltext): update Fulltext model for alto2txt2fixtures changes

07b97ef

fix(test): correct Fulltext.plaintext_path -> Fulltext.path

00d8141

merge: main with fulltext-hmd

890851e

Merge remote-tracking branch 'origin/pre-commit-ci-update-config' int…

d8d9aa6

…o pre-commit-ci-update-config

merge: poetry.lock fix

dd42c7f

fix: update deps for linux server build

c3a7986

fix: bootstrap.js in vendors.js 5.3.1 -> 5.3.2

ff7d89b

chore: squash gazetteer, mitchells and newspaper database migra…

e621e7e

…tions

merge: fetch from GitHub

3f4d3d6

Merge remote-tracking branch 'origin/fulltext-hmd' into fulltext-hmd-…

257c618

…dedupe

feat: add lwmdb.utils::DupeRemoveConfig and helper functions

2d0bcfc

fix(ci): apply pre-commit to last commit

9adc487

chore: squash newspaper database migrations and fix docstrings

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

55abc50

chore: remove backup newspaper and mitchells migrations

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

ee44fc2

chore: remove residual mitchells migration squash

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

3a9ed15

chore: clean up comments in lwmdb/utils.py

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

ee8e81f

fix(DedupeConfig): add some tests and fixes around DedupeConfig

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

a660141

fix: clean up Mitchells migration

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

83d2b06

feat(utils): improve info dupes_to_rm returns and logs

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

defdade

fix(convert_similar_qs_to_records): aggregate multiple dupe cases

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

637b4d4

feat(test): parameterize test_dupes_to_rm for dupes > 1

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

6cae85d

feat(test): ensure full Newspaper aggregation tested in `test_dupes…

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

c9a70be

…_to_rm`

fix(fulltext): mv previous 0001_initial.py -> 0001_initial.py.bak

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

fe34277

fix(fulltext): fix migrations squash to 0001_initial.py

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

7430c3d

fix(DedupeConfig): attempt to ensure records_to_keep are filterable…

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

eb76ee9

… `QuerySets`

fix(filter_by_not_all_null_fk): qs.Model -> qs.model

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

96ac85e

spool and others added 30 commits October 29, 2023 12:53

feat(FullText): rename Fulltext to FullText and tidy references/tests

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

3040f5c

feat(FullText): replace fulltext references with full_text

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

58eda6e

feat(FullText): add item_code and tests for matching Item records

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

ba52244

feat(test): add test_plaintext.json.zip and fixture loading test

0a835fa

fix(DataProvider): set 'code' fiel to use '-' like 'bl_hmd' -> 'bl-hmd'

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

26651fa

fix(test): add newspapers/tests/__init__.py

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

9f22e87

fix(doc): typo in lwmdb/utils.py docstring

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

dc6cd41

feat(FullText): add means of logging edge cases for Item -> FullText

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

b59b4aa

fix(chore): merge main to fulltext-refactor

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

a97f4f8

feat(ci): update fulltext-refactor from 'origin/pre-commit-ci-updat…

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

4b79b40

…e-config'

refactor: rename DupeRemov -> DupeFix

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

41ccffe

Merge remote-tracking branch 'origin/dependabot/pip/jupyter-server-2.…

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

b0f49e2

…11.2' into fulltext-refactor

Merge remote-tracking branch 'origin/pre-commit-ci-update-config' int…

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

e9cf7be

…o fulltext-refactor

fix: rename DupeFixConfig -> DupeConfig and fix `mitchells_data_p…

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

8fd2fb1

…ath` test `fixture`

fix(doc): docstring typos, improve func names, mailhog -> mailpit

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

b2fc6e8

fix(chore): update jinja2 -> 3.1.3

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

382d419

fix(chore): merge pre-commit updates

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

4bcc27a

fix(ci): add pre-fulltext-backups to .dockerignore

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

607672c

feat(ui): front end layout and branding

3cbe379

fix(ci): add admin_cursor_navigator to mypy pre-commit config

1dc1c02

fix: rm file

0fa1aa2

fix(ci): add migrations used in server config

7199134

Merge remote-tracking branch 'origin/fulltext-refactor' into fulltext…

44bc2c6

…-refactor

fix: fulltext.__str__, mkdocs docker, svg formatting

d63b056

fix: rm mitchells and newspaper migrations

Loading
Loading status checks…

1a3d189

fix(test): add of VITURAL_HOST to ProductionENVGenConfig write test

Unverified

This user has not yet uploaded their public signing key.

Learn about vigilant mode

Loading
Loading status checks…

204f3c9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fulltext refactor, including dupes checking #196

Fulltext refactor, including dupes checking #196

griff-rees commented Dec 6, 2023

Fulltext refactor, including dupes checking #196

Are you sure you want to change the base?

Fulltext refactor, including dupes checking #196

Conversation

griff-rees commented Dec 6, 2023