-
Notifications
You must be signed in to change notification settings - Fork 2
Added weekly VACUUM FULL maintenance with cron scheduling for database optimization #4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Miggets7
wants to merge
5
commits into
main
Choose a base branch
from
feature/auto_full_vacuum
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
615b950
Added weekly VACUUM FULL maintenance with cron scheduling for databas…
Miggets7 45506bd
Update Dockerfile
richturner cfbace9
Update Dockerfile.alpine
richturner 98e0202
Made the cron full vacum opt-in and also the schedule configurable
Miggets7 4ec7a80
refactor: simplify VACUUM FULL scheduling with dedicated scripts
Miggets7 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,10 +3,19 @@ ARG PG_MAJOR_PREVIOUS=14 | |
ARG PG_MAJOR=15 | ||
|
||
FROM timescaledev/timescaledb-ha:pg15-multi as trimmed | ||
MAINTAINER [email protected] | ||
LABEL org.opencontainers.image.authors="[email protected]" | ||
|
||
USER root | ||
|
||
# Install cron for scheduled VACUUM FULL operations | ||
RUN apt-get update && apt-get install -y cron && rm -rf /var/lib/apt/lists/* | ||
|
||
# Add scripts for VACUUM FULL operations | ||
RUN mkdir -p /var/lib/postgresql/scripts | ||
ADD scripts/vacuum_full.sh /var/lib/postgresql/scripts/ | ||
ADD scripts/start_cron.sh /var/lib/postgresql/scripts/ | ||
RUN chmod +x /var/lib/postgresql/scripts/*.sh | ||
|
||
# Give postgres user the same UID and GID as the old alpine postgres image to simplify migration of existing DB | ||
RUN usermod -u 70 postgres \ | ||
&& groupmod -g 70 postgres \ | ||
|
@@ -78,7 +87,9 @@ ENV PGROOT=/var/lib/postgresql \ | |
OR_REINDEX_COUNTER=${OR_REINDEX_COUNTER} \ | ||
OR_DISABLE_REINDEX=${OR_DISABLE_REINDEX:-false} \ | ||
POSTGRES_MAX_CONNECTIONS=${POSTGRES_MAX_CONNECTIONS:-50} \ | ||
OR_DISABLE_AUTO_UPGRADE=${OR_DISABLE_AUTO_UPGRADE:-false} | ||
OR_DISABLE_AUTO_UPGRADE=${OR_DISABLE_AUTO_UPGRADE:-false} \ | ||
# VACUUM FULL cron schedule (if not set, VACUUM FULL is disabled) | ||
OR_VACUUM_FULL_CRON_SCHEDULE="${OR_VACUUM_FULL_CRON_SCHEDULE:-}" | ||
|
||
WORKDIR /var/lib/postgresql | ||
EXPOSE 5432 8008 8081 | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,7 +20,7 @@ FROM timescaledev/timescaledb-ha:pg15-multi as timescale-ha | |
# This still doesn't work as timescale code is compiled against glibc and some references don't match with gcompat | ||
|
||
FROM timescale/timescaledb:latest-pg15 | ||
MAINTAINER [email protected] | ||
LABEL org.opencontainers.image.authors="[email protected]" | ||
|
||
ENV GLIBC_VERSION 2.35-r0 | ||
ENV TZ ${TZ:-Europe/Amsterdam} | ||
|
@@ -29,8 +29,17 @@ ENV POSTGRES_DB ${POSTGRES_DB:-openremote} | |
ENV POSTGRES_USER ${POSTGRES_USER:-postgres} | ||
ENV POSTGRES_PASSWORD ${POSTGRES_PASSWORD:-postgres} | ||
|
||
# Add glibc | ||
RUN apk add gcompat | ||
# VACUUM FULL cron schedule (if not set, VACUUM FULL is disabled) | ||
ENV OR_VACUUM_FULL_CRON_SCHEDULE="${OR_VACUUM_FULL_CRON_SCHEDULE:-}" | ||
|
||
# Add glibc and cron for scheduled VACUUM FULL operations | ||
RUN apk add --no-cache gcompat dcron | ||
|
||
# Add scripts for VACUUM FULL operations | ||
RUN mkdir -p /var/lib/postgresql/scripts | ||
ADD scripts/vacuum_full.sh /var/lib/postgresql/scripts/ | ||
ADD scripts/start_cron.sh /var/lib/postgresql/scripts/ | ||
RUN chmod +x /var/lib/postgresql/scripts/*.sh | ||
|
||
|
||
COPY --from=timescale-ha /usr/lib/postgresql/15/lib/bitcode/postgis-3/ /usr/local/lib/postgresql/bitcode/ | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
#!/bin/sh | ||
|
||
# This script sets up and starts the cron service for scheduled database maintenance | ||
# It handles both Alpine Linux (crond) and Debian/Ubuntu (cron) systems | ||
|
||
# Check if VACUUM FULL schedule is set | ||
if [ -n "${OR_VACUUM_FULL_CRON_SCHEDULE}" ]; then | ||
echo "Setting up VACUUM FULL job with schedule: ${OR_VACUUM_FULL_CRON_SCHEDULE}..." | ||
|
||
# Create cron entry | ||
CRON_ENTRY="${OR_VACUUM_FULL_CRON_SCHEDULE} /var/lib/postgresql/scripts/vacuum_full.sh" | ||
echo "Using cron schedule: ${OR_VACUUM_FULL_CRON_SCHEDULE}" | ||
|
||
if [ -x "$(command -v cron)" ]; then | ||
# Debian/Ubuntu (standard cron) | ||
echo "Setting up cron job using standard cron..." | ||
(crontab -u postgres -l 2>/dev/null || echo "") | echo "$CRON_ENTRY" | crontab -u postgres - | ||
echo "VACUUM FULL job has been scheduled using standard cron" | ||
|
||
echo "Starting cron service for scheduled database maintenance..." | ||
service cron start | ||
elif [ -x "$(command -v crond)" ]; then | ||
# Alpine Linux (dcron) | ||
echo "Setting up cron job using Alpine's crond (dcron)..." | ||
mkdir -p /etc/crontabs | ||
echo "$CRON_ENTRY" > /etc/crontabs/postgres | ||
chmod 600 /etc/crontabs/postgres | ||
echo "VACUUM FULL job has been scheduled using dcron" | ||
|
||
echo "Starting crond service for scheduled database maintenance..." | ||
crond -b -L /var/lib/postgresql/cron.log | ||
else | ||
echo "WARNING: No cron system found. VACUUM FULL will not be automatically scheduled." | ||
fi | ||
else | ||
echo "VACUUM FULL is disabled (OR_VACUUM_FULL_CRON_SCHEDULE is not set). Skipping cron service." | ||
fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
#!/bin/sh | ||
|
||
# Log file for vacuum operations | ||
LOG_FILE="/var/lib/postgresql/vacuum_full.log" | ||
Comment on lines
+3
to
+4
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Log to standard out |
||
|
||
echo "$(date): Starting VACUUM FULL operation" >> "$LOG_FILE" | ||
|
||
# Get list of all tables in the database | ||
TABLES=$(psql -t -c "SELECT schemaname || '.' || tablename FROM pg_tables WHERE schemaname NOT IN ('pg_catalog', 'information_schema')" "${POSTGRES_DB:-postgres}") | ||
|
||
# Run VACUUM FULL on each table | ||
for TABLE in $TABLES; do | ||
echo "$(date): Running VACUUM FULL on $TABLE" >> "$LOG_FILE" | ||
psql -c "VACUUM FULL $TABLE" "${POSTGRES_DB:-postgres}" | ||
echo "$(date): Completed VACUUM FULL on $TABLE" >> "$LOG_FILE" | ||
done | ||
|
||
echo "$(date): VACUUM FULL operation completed" >> "$LOG_FILE" |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Keep this outer if in the entrypoint for ease of reading