Skip to content

Updates & Upgrades

Need help getting set up?

Bunker Operations provides managed infrastructure and hands-on setup assistance for organizations running Changemaker Lite. We handle domains, tunnels, SMTP, and servers so you can focus on your campaign. Get in touch: bnkops.com | admin@bnkops.ca

Changemaker Lite includes a built-in upgrade system that pulls code updates, rebuilds containers, runs database migrations, and restarts services — all while preserving your customizations.

There are two ways to upgrade:

  1. Admin GUI — Check for updates and run upgrades from Settings > System
  2. CLI — Run ./scripts/upgrade.sh directly from the command line

Both methods execute the same 6-phase upgrade process.

Prerequisites

Upgrade Watcher (Required for GUI Method)

The admin GUI triggers upgrades via a systemd path watcher that monitors for trigger files. This must be installed on the host system.

Install during initial setup:

The config.sh wizard offers to install the watcher automatically (Step 13). If you skipped it, install manually:

# Edit the systemd units to set your project path and user
sed -e "s|__PROJECT_DIR__|$(pwd)|g" scripts/systemd/changemaker-upgrade.path > /tmp/changemaker-upgrade.path
sed -e "s|__PROJECT_DIR__|$(pwd)|g" -e "s|__USER__|$(whoami)|g" scripts/systemd/changemaker-upgrade.service > /tmp/changemaker-upgrade.service

# Install and enable
sudo cp /tmp/changemaker-upgrade.path /tmp/changemaker-upgrade.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now changemaker-upgrade.path

Verify it's running:

sudo systemctl status changemaker-upgrade.path

How the watcher works

The API container writes a trigger.json file to a shared data/upgrade/ volume. The systemd path watcher detects the file and runs scripts/upgrade-watcher.sh on the host, which dispatches to the appropriate script (check or upgrade). Progress and results are communicated back via JSON files that the API reads.

Method 1: Admin GUI

Checking for Updates

  1. Navigate to Settings (/app/settings)
  2. Click the System tab
  3. Click Check for Updates

The System tab shows your current version, last commit message, and auto-upgrade settings:

System tab initial state

The system fetches from the git remote and shows:

  • Current commit hash and message
  • Remote commit hash (if different)
  • Number of commits behind
  • Changelog of incoming changes

When updates are available, the panel highlights how many commits are behind and lists the incoming changes:

Update available notification

Starting an Upgrade

  1. Review the changelog to understand what's changing
  2. Click Start Upgrade
  3. Optionally configure:
    • Skip backup — skip the database backup phase (not recommended)
    • Pull images — also update third-party Docker images (PostgreSQL, Redis, etc.)
    • Use registry images — pull pre-built images from Gitea instead of compiling from source (faster — requires scripts/build-and-push.sh to have been run first)
    • Dry run — preview what would happen without making changes
  4. Monitor the 6-phase progress indicator

Confirm System Upgrade dialog

The GUI polls for progress updates and displays the current phase, percentage, and status message in real time.

Upgrade Results

After the upgrade completes, the System tab shows the result — including the new version, health check status, and any warnings:

Upgrade success result

Tip

If health checks show warnings immediately after an upgrade, wait 1-2 minutes for services to fully start before investigating.

The 6 Upgrade Phases

Both the GUI and CLI methods execute the same 6-phase process:

Phase % Name What Happens
1 5% Pre-flight Checks Verifies Docker, git, disk space (2 GB minimum), remote reachability, and clean working directory
2 15% Backup Runs scripts/backup.sh (pg_dump + archive), backs up user-modifiable content, saves pre-upgrade commit hash
3 30% Code Update Saves user paths, stashes local changes, git pull, pops stash with auto-conflict resolution, detects new .env variables
4 50% Container Rebuild Rebuilds api, admin, media-api from source (default) or pulls pre-built images from the Gitea registry (--use-registry); conditionally rebuilds nginx and code-server if their configs changed; optionally pulls third-party images
5 70% Service Restart Stops app containers, force-recreates LSIO containers, verifies Gancio config, starts infrastructure, waits for PostgreSQL, starts API (runs migrations), starts everything else, restarts Newt tunnel and monitoring if they were running
6 90% Verification Health checks for API, Admin, Media API, Gancio, MkDocs; detects containers in restart loops

What Gets Preserved

The upgrade script automatically preserves user-modifiable paths that you may have customized:

Path What It Contains
mkdocs/docs/ Your documentation content
mkdocs/mkdocs.yml MkDocs configuration
mkdocs/site/ Built documentation site
configs/ Prometheus, Grafana, Alertmanager, Homepage configs
nginx/conf.d/services.conf Custom nginx service proxies

These files are saved before git pull and unconditionally restored afterward, even if the pull introduces changes to them. Your versions always win.

Tip

The .env file is never touched by git pull (it's in .gitignore). However, if new environment variables are added in .env.example, the upgrade script automatically appends them to your .env with their default values and warns you to review them.

Method 2: CLI

Run the upgrade script directly:

./scripts/upgrade.sh

Options

Flag Description
--skip-backup Skip the backup phase (requires --force)
--pull-services Also pull new third-party Docker images
--use-registry Pull pre-built images from Gitea instead of compiling from source
--dry-run Show what would happen without executing
--force Continue past non-critical warnings
--branch BRANCH Git branch to pull (default: current branch)
--rollback Rollback to pre-upgrade commit
--api-mode Write progress/result JSON for admin GUI (used internally)

Examples

# Standard upgrade
./scripts/upgrade.sh

# Preview changes without executing
./scripts/upgrade.sh --dry-run

# Full upgrade including third-party image updates
./scripts/upgrade.sh --pull-services

# Upgrade using pre-built images from Gitea registry (faster, no TypeScript compile)
./scripts/upgrade.sh --use-registry --force --skip-backup

# Rollback to the last pre-upgrade state
./scripts/upgrade.sh --rollback

Registry Mode (Fast Upgrades)

By default, the upgrade script compiles TypeScript from source (npm run build) and rebuilds Docker images on the deployment server. Registry mode skips this by pulling pre-built production images from the Gitea container registry — faster and requires no build tooling on the server.

How It Works

  1. Run scripts/build-and-push.sh on a machine with Docker (usually your dev machine) to build and push production images tagged with the current commit SHA
  2. During the next upgrade, pass --use-registry (CLI) or enable the checkbox (GUI)
  3. The upgrade script pulls gitea.bnkops.com/admin/changemaker-{service}:{sha} instead of rebuilding from source
  4. If a registry image is unavailable (e.g., the SHA wasn't pushed), it automatically falls back to a source build

Building and Pushing Images

# Build and push all core services (api, admin, media-api, nginx)
./scripts/build-and-push.sh

# Skip code-server (9 GB — push only when Dockerfile changes)
./scripts/build-and-push.sh --services api,admin,media-api,nginx

# Build only, no push (verify locally first)
./scripts/build-and-push.sh --no-push

# Also mirror third-party images (postgres, redis, etc.) to Gitea
./scripts/mirror-images.sh

Registry prerequisites

  • Run docker login gitea.bnkops.com once per machine before pushing
  • Set GITEA_REGISTRY_USER and GITEA_REGISTRY_PASS in .env for the admin GUI's Registry status endpoint
  • gitea.bnkops.com must be reachable without proxies that limit upload size (Cloudflare free plan blocks blobs >100 MB)

Release installs upgrade automatically via registry

If you installed from a release tarball (not git clone), the upgrade script automatically uses registry mode. It downloads the latest release package from Gitea instead of running git pull. No additional configuration needed.

Rollback

Automatic Rollback

If the upgrade fails at any phase, the script prints detailed rollback instructions including the pre-upgrade commit hash. Use the --rollback flag:

./scripts/upgrade.sh --rollback

This:

  1. Finds the latest backup archive
  2. Extracts the pre-upgrade commit hash from git-commit.txt inside the archive
  3. Checks out that commit
  4. Rebuilds and restarts all containers

Warning

--rollback restores the code to the pre-upgrade state but does not automatically restore the database. If database migrations were applied during the failed upgrade, you may need to manually restore from the backup archive.

Manual Rollback

# 1. Restore code
cd /path/to/changemaker.lite
git checkout <pre-upgrade-commit-hash>

# 2. Rebuild and restart
docker compose build api admin media-api
docker compose up -d

# 3. Database restore (if needed — destructive!)
ls -lt backups/changemaker-v2-backup-*.tar.gz | head -5
tar xzf backups/<backup>.tar.gz -C /tmp
gunzip -c /tmp/<backup>/v2-postgres.sql.gz | \
  docker exec -i changemaker-v2-postgres psql -U changemaker -d changemaker_v2

New Environment Variables

When upstream code adds new environment variables to .env.example, the upgrade script automatically:

  1. Compares .env.example against your .env
  2. Appends any missing variables with their default values
  3. Warns you to review the new additions
[WARN] New env vars added to .env (review defaults):
    NEW_FEATURE_FLAG
    NEW_API_KEY

Always review new variables after an upgrade — some may need manual configuration.

Update Checker

A separate lightweight script checks for available updates without performing any changes:

./scripts/upgrade-check.sh

This writes data/upgrade/status.json with:

  • Current and remote commit hashes
  • Number of commits behind
  • Changelog (last 30 commits)
  • Timestamp of last check

The admin GUI reads this file to display update availability.

Troubleshooting

Stale Progress Indicator

If the GUI shows an upgrade "in progress" but nothing is happening, the upgrade script may have crashed. The system automatically detects stale progress (no update for 10+ minutes) and treats it as not running.

To manually clear:

rm -f data/upgrade/progress.json

Merge Conflicts

If git pull encounters merge conflicts in user-modifiable paths (docs, configs), the upgrade script auto-resolves by keeping your version. If conflicts occur in project-owned files (api/, admin/), the upgrade fails and asks you to resolve manually.

Lock File

The upgrade script uses .upgrade.lock to prevent concurrent upgrades. If a previous upgrade crashed without cleaning up:

# Verify no upgrade is actually running
ps aux | grep upgrade.sh

# Remove stale lock
rm -f .upgrade.lock

Health Check Failures

If Phase 6 health checks fail, services may still be starting. Wait 1-2 minutes and check manually:

# API health
curl -s http://localhost:4000/api/health

# Container status
docker compose ps

# Recent logs
docker compose logs api --tail 50
docker compose logs admin --tail 50

Systemd Watcher Not Triggering

# Check watcher status
sudo systemctl status changemaker-upgrade.path

# Check service logs
sudo journalctl -u changemaker-upgrade.service --tail 20

# Re-enable if stopped
sudo systemctl enable --now changemaker-upgrade.path