Backups

SongCollect writes a backup bundle with songcollect-service backup create. A bundle is one tar file named songcollect-<UTC time>.tar, for example songcollect-20261020T020000Z.tar. It contains:

Published files are immutable and never deleted, so a snapshot's files stay readable while the bundle is written. Temporary uploads are not backed up. Bundles are not encrypted yet and secrets are not included. Keep the destination private.

Backups require a local file: database. A Turso Cloud database needs the provider's export instead.

Configure🔗

The S3 user needs ListBucket, GetObject, PutObject, DeleteObject, and the multipart upload actions on the bucket. Uploads are multipart, so a bundle appears only once it is complete. Pruning only touches objects named like a bundle at the top of the prefix, and never the bundle just written.

Production🔗

Production is the Coolify application app (axiwbus6rcnm6gpacfbafzhw, team SongCollect Devs, environment production) on songcollect.schmiednet.de. Bundles go to a different machine: MinIO on beta2, bucket songcollect-backups, prefix production.

• The BACKUP_* variables are production-only runtime variables of app. The MinIO user songcollect-production may only use the songcollect-backups bucket (policy songcollect-backups).
• MinIO is the minio application in Coolify's Private Apps project. Its Coolify domain only serves the console, so the S3 API is published at https://s3.beta2.schmiednet.de by the Traefik file /data/coolify/proxy/dynamic/songcollect-backup-s3.yaml on beta2.
• The nightly job is a Coolify Scheduled Task on app: name nightly-backup, command songcollect-service backup create, frequency 0 2 * * * (02:00 UTC), container app. A non-zero exit marks the run as failed in Scheduled Tasks. Coolify sends its scheduled-task failure notification to the team's configured notification channels.

Check backup age🔗

songcollect-service backup status

The command lists every bundle with its size and upload time, then prints the age of the newest one. It exits non-zero when there is no bundle or the newest is older than BACKUP_MAX_AGE_HOURS. In Coolify, run it from the app terminal, or check the latest nightly-backup run in Scheduled Tasks.

Restore🔗

songcollect-service backup restore <bundle name> restores a bundle into the database at TURSO_DATABASE_URL and the asset store. It refuses to run when that database file exists. Before writing anything it checks:

• the database snapshot and every file against the manifest's SHA-256 and size;
• that the bundle contains exactly the listed files, and that the list matches the snapshot's asset_blobs;
• the schema. A bundle from an older release is migrated to the running release, and an altered migration is refused.

It then writes the files, then the database, and revokes all sessions so everyone signs in again. It prints the number of users, churches, songs, and verified files.

Restore drill in an isolated instance🔗

Restore into a fresh Docker volume on any machine that can reach the bucket, using the production image or a build of the same commit:

export BACKUP_STORE_URL=s3://songcollect-backups/production BACKUP_AWS_ENDPOINT=https://s3.beta2.schmiednet.de BACKUP_AWS_REGION=us-east-1
export BACKUP_AWS_ACCESS_KEY_ID=... BACKUP_AWS_SECRET_ACCESS_KEY=...   # from Coolify
image=songcollect:restore-drill; docker build -t "$image" .
run() { docker run --rm -v songcollect-restore:/data -e TURSO_DATABASE_URL=file:/data/songcollect.db \
  -e BACKUP_STORE_URL -e BACKUP_AWS_ENDPOINT -e BACKUP_AWS_REGION -e BACKUP_AWS_ACCESS_KEY_ID -e BACKUP_AWS_SECRET_ACCESS_KEY "$image" "$@"; }
run songcollect-service backup status
run songcollect-service backup restore songcollect-<time>.tar
docker run --rm -d --name songcollect-restore -p 127.0.0.1:3000:3000 -v songcollect-restore:/data \
  -e TURSO_DATABASE_URL=file:/data/songcollect.db -e APP_ORIGIN=http://localhost:3000 "$image"

Then open http://localhost:3000 and verify:

1. Sign-in works with a production account.
2. The song list and search match production.
3. A song's lyrics and its arrangements show, and a sheet-music PDF opens.

Record the date, bundle, and the time from start to finished checks (the target is four hours). Remove the instance afterwards with docker rm -f songcollect-restore && docker volume rm songcollect-restore, because it holds production data.

Restore production🔗

1. Stop app in Coolify. Keep the old volume until the restored instance is verified.
2. Point TURSO_DATABASE_URL at a new path on the volume, for example file:/data/restored/songcollect.db, and set ASSET_STORE_URL=file:/data/restored/assets.
3. Run songcollect-service backup restore <bundle name> once, for example as a one-off Coolify scheduled task or with docker run against the volume.
4. Start app, verify as in the drill, then remove the old files.