Skip to content

Archival #538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 46 commits into
base: develop
Choose a base branch
from
Draft

Archival #538

wants to merge 46 commits into from

Conversation

@lukasjuhrich
Copy link
Collaborator

@lukasjuhrich lukasjuhrich commented Feb 21, 2022 •
edited
Loading

Prepares #490

  • Discuss whether we want to do this yearly or regularly && adapt code accordingly
  • Test archival selection
    • test select_user_and_last_mem() select (used as CTE)
    • test cutoff filtering
    • test cutoff calculation from current date
    • test do-not-archive filtering
  • Implement archive_users (cleanup)
  • Add a web endpoint and button archiving all current users
  • Add a celerybeat task executing the archival

@lukasjuhrich lukasjuhrich force-pushed the archival branch 5 times, most recently from b5b4131 to cf0d6b1 Compare August 5, 2022 04:06
@lukasjuhrich lukasjuhrich force-pushed the archival branch 2 times, most recently from 364a437 to bbe334a Compare September 29, 2022 11:18
@lukasjuhrich
Copy link
Collaborator Author

lukasjuhrich commented Jan 15, 2023

This PR, as of now, does some base work that can directly be merged into develop.
All the rest can be coordinated in one or multiple different PRs (see the extended tasklist of #67).
Waiting for the tests to finish.

@lukasjuhrich lukasjuhrich marked this pull request as ready for review January 15, 2023 18:05
@lukasjuhrich lukasjuhrich changed the title Archival Archival preparations Jan 15, 2023
georgeto
georgeto reviewed Jan 16, 2023
View reviewed changes
pycroft/lib/user_deletion.py
def get_invalidated_archive_memberships() -> list[Membership]:
"""Get all memberships in `to_be_archived` of users who have an active `do-not-archive` property.
Selected are those users
- whose last membership in the member_group ended two weeks in the past,
Copy link
Contributor

@georgeto georgeto Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks do me like we are now testing for >= 2 years and not weeks.

@lukasjuhrich lukasjuhrich changed the title Archival preparations Archival Apr 27, 2023
@FestplattenSchnitzel FestplattenSchnitzel marked this pull request as draft June 17, 2025 16:43
@lukasjuhrich lukasjuhrich force-pushed the archival branch 2 times, most recently from 4b76802 to 448ba26 Compare September 18, 2025 12:10
lukasjuhrich and others added 29 commits November 22, 2025 14:07
@lukasjuhrich
Add reference to privacy policy to scrubbing functions
7aee819
@lukasjuhrich
Add stubs to other scrubbers with references to privacy policy
c7a5cbd
@FestplattenSchnitzel @lukasjuhrich
Fix typo
5cc31d0
@FestplattenSchnitzel @lukasjuhrich
Make number of years required for archivability a parameter
7584510
@lukasjuhrich
Update psycopg2-binary, factory-boy and add pytest-freezegun
72c8513
@lukasjuhrich
model: add ScrubLog
4c91fea
@lukasjuhrich
include model.scrubbing in model._all
6aecfa0 
Otherwise it won't be known to our Metadata
@lukasjuhrich
upgrade pytest to v8.4.1
3f132d0
@lukasjuhrich
Upgrade uv to v0.7.11
6be611a
@lukasjuhrich
Add a ScrubLogEntry when scrubbing mail address and test scrubbing
5f06b99
@lukasjuhrich
Fix tests
ea07a25
@lukasjuhrich
Add select statements for other scrubbers
5432d09
@lukasjuhrich
Remove annoying daker args
05cc6e9 
We want to change the file in local development so we get the fix and
not only the complaint. This is what the default behavior does.
@lukasjuhrich
Add stub implementation and test for bulk mail scrubbing
161afdc
@lukasjuhrich
Add bulk mail scrubbing without user log entries
3c241bb
@lukasjuhrich
scrublog: Add server_default and fix migration
f4c268b
@lukasjuhrich
scrublog: fix indices and nullability in new migration
b6ec15f
@lukasjuhrich
Fix usage of _WindowArgs type hints
9d8ee82
@lukasjuhrich
Use type[] instead of typing.Type[]
1442cae
@lukasjuhrich
scrublog: Use server default for execution time
01a3553
@lukasjuhrich
scrubbing: Allow customization of user filter in bulk mail scrubber
f478809
@lukasjuhrich
Create common identity function in helpers.functional
a4af703
@FestplattenSchnitzel @lukasjuhrich
Add users with to-be-deleted name, address statements
6571b25
@lukasjuhrich
First attempt of a bulk deletion query
c437074
@lukasjuhrich
Add row_number()s to intermediate CTEs so we can properly join
44c1f2b 
I now officially hate polymorphism in relational databases when based
on joined tables
@lukasjuhrich
re-lock using uv with --upgrade
bd94f03
@lukasjuhrich
Update alembic to 1.17.1
73e1210
@lukasjuhrich
Replace buggy scrublog migration migrations by proper one
5e22637
@FestplattenSchnitzel @lukasjuhrich
Fix mail deletion query and flash message
9dc8baf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@georgeto georgeto georgeto left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@lukasjuhrich @georgeto @FestplattenSchnitzel