bunker-admin 2fa50b001c Merge changemaker-control-panel into v2 monorepo
Absorbs the separate control-panel git repo as a subdirectory.
Instances and backups directories excluded via .gitignore.

Bunker Admin
2026-02-21 11:51:45 -07:00

27 KiB

Changemaker Control Panel (CCP) — Development Plan

A multi-tenant management system for provisioning, monitoring, and operating multiple Changemaker Lite instances from a single dashboard.


Vision

CCP replaces manual git clone / docker compose up / .env editing with a web UI that can:

  • One-click provision a new Changemaker Lite instance (database, containers, config, tunnel)
  • Monitor instance health across the fleet
  • Start/stop/restart instances and individual services
  • Back up and restore instance data
  • Maintain a full audit trail of operator actions
  • Manage Pangolin tunnels for production exposure

Architecture

                         ┌─────────────────────────┐
                         │   CCP Admin GUI (5100)   │  React + Vite + Ant Design
                         │   Dark theme, SPA        │  Zustand auth store
                         └────────────┬─────────────┘
                                      │ /api/* proxy
                         ┌────────────▼─────────────┐
                         │   CCP API (5000)          │  Express + TypeScript
                         │   JWT auth, RBAC          │  Prisma ORM → PostgreSQL
                         │   Docker socket access    │  Winston logger
                         └────────────┬─────────────┘
                                      │
              ┌───────────┬───────────┼───────────┬──────────┐
              ▼           ▼           ▼           ▼          ▼
        ccp-postgres  ccp-redis  Docker Socket  /opt/ccp/  /var/backups/
        (port 5480)   (port 6399)    (.sock)    instances   ccp-instances

Stack

Layer Technology Notes
API Express 4, TypeScript 5, Node 20 express-async-errors for async route handling
ORM Prisma 6 + PostgreSQL 16 10 models, mapped table names
Auth JWT (jsonwebtoken) + bcryptjs 15min access / 7d refresh, atomic rotation
Encryption AES-256-GCM (Node crypto) Secrets at rest in encrypted_secrets column
Frontend React 19, Vite 6, Ant Design 5 Dark theme, Zustand state, axios interceptors
Docker Docker CLI + socket API compose up/down/ps/exec/logs via shell + HTTP socket
Templates Handlebars .env, docker-compose.yml, nginx configs rendered per-instance
Logging Winston JSON in production, colorized in development
Config Zod schema validation Fails fast on startup with clear error messages

Key Design Decisions

  1. Docker CLI over Docker SDK — Shell out to docker compose commands rather than using dockerode. Simpler, matches what operators would run manually, and docker compose handles all the orchestration logic.

  2. Shared PostgreSQL — All CCP data in one database; each CML instance gets its own isolated PostgreSQL container with unique ports and passwords.

  3. Port Range Allocation — Four non-overlapping port ranges prevent conflicts:

    • API: 14000-14999
    • Admin: 13000-13999
    • PostgreSQL: 15400-15499
    • Nginx: 10000-10999
  4. Async Provisioning — Instance creation returns immediately; provisioning runs fire-and-forget with progress tracked via status/statusMessage fields. Frontend polls for updates.

  5. Template-Based Config — Handlebars templates render per-instance docker-compose.yml, .env, nginx configs, and Pangolin resources. This avoids complex string manipulation and keeps configs readable.


Database Schema (Prisma)

CcpUser ──< CcpRefreshToken
  │
  ├──< AuditLog
  │
Instance ──< PortAllocation
  │
  ├──< HealthCheck
  ├──< Backup
  └──< AuditLog

CcpSetting (key-value)

Models

Model Purpose Key Fields
CcpUser Control panel operators email, password (bcrypt), role (SUPER_ADMIN/OPERATOR/VIEWER)
CcpRefreshToken JWT refresh token storage token (SHA-256 hash), expiresAt
Instance Managed CML instance slug, domain, basePath, composeProject, status, portConfig (JSON), encryptedSecrets, feature flags
PortAllocation Port registry port (unique), service name, instanceId
HealthCheck Periodic health snapshots status (HEALTHY/DEGRADED/UNHEALTHY/UNKNOWN), serviceStatus (JSON), totalServices, healthyServices, responseTimeMs
Backup Backup records status (PENDING→IN_PROGRESS→COMPLETED/FAILED), archivePath, sizeBytes, manifest (JSON)
AuditLog Action trail action (18 types), userId, instanceId, details (JSON), ipAddress
CcpSetting Global key-value config key (PK), value (JSON)

Audit Actions (18 types)

INSTANCE_CREATE, INSTANCE_UPDATE, INSTANCE_DELETE,
INSTANCE_START, INSTANCE_STOP, INSTANCE_RESTART, INSTANCE_UPGRADE,
BACKUP_CREATE, BACKUP_DELETE,
PANGOLIN_SETUP, PANGOLIN_SYNC,
USER_LOGIN, USER_CREATE, USER_UPDATE, USER_DELETE,
SETTINGS_UPDATE

API Endpoints

Authentication

Method Path Auth Description
POST /api/auth/login Public Login, returns JWT pair
POST /api/auth/refresh Public Rotate refresh token
POST /api/auth/logout Public Revoke refresh token
GET /api/auth/me Authenticated Current user profile

Instances — CRUD

Method Path Auth Description
GET /api/instances Authenticated List all instances
GET /api/instances/:id Authenticated Instance detail (includes health + backups)
POST /api/instances SUPER_ADMIN, OPERATOR Create instance (triggers async provisioning)
PUT /api/instances/:id SUPER_ADMIN, OPERATOR Update instance config
DELETE /api/instances/:id SUPER_ADMIN Delete instance (stops containers, removes files)
GET /api/instances/:id/secrets SUPER_ADMIN Decrypt and return instance secrets

Instances — Lifecycle

Method Path Auth Description
POST /api/instances/:id/provision SUPER_ADMIN, OPERATOR Retry provisioning
POST /api/instances/:id/start SUPER_ADMIN, OPERATOR Start all containers
POST /api/instances/:id/stop SUPER_ADMIN, OPERATOR Stop all containers
POST /api/instances/:id/restart SUPER_ADMIN, OPERATOR Restart (optionally ?service=api)

Instances — Services & Logs

Method Path Auth Description
GET /api/instances/:id/services Authenticated Container status via docker compose ps
GET /api/instances/:id/logs Authenticated Logs (?service=api&tail=200&since=1h)

Instances — Health

Method Path Auth Description
POST /api/instances/:id/health-check SUPER_ADMIN, OPERATOR Trigger manual health check
GET /api/instances/:id/health-history Authenticated Paginated health check history

Instances — Backups

Method Path Auth Description
POST /api/instances/:id/backup SUPER_ADMIN, OPERATOR Create backup (async)
GET /api/instances/:id/backups Authenticated List instance backups

Backups (cross-instance)

Method Path Auth Description
GET /api/backups Authenticated List all backups (?instanceId=...&page=1&limit=50)
DELETE /api/backups/:id SUPER_ADMIN Delete backup (file + record)
GET /api/backups/:id/download SUPER_ADMIN Stream backup archive

Health (CCP-level)

Method Path Auth Description
GET /api/health Public CCP API health check
GET /api/health/overview Authenticated All instances with latest health

Audit

Method Path Auth Description
GET /api/audit Authenticated Filtered, paginated audit log (?action=...&instanceId=...&userId=...&from=...&to=...)

Settings

Method Path Auth Description
GET /api/settings Authenticated All settings as key-value map
PUT /api/settings/:key SUPER_ADMIN Upsert a setting value

Frontend Pages

Route Page Description
/login LoginPage Email/password form
/app DashboardPage Stats (total, running, healthy, degraded, stopped, errors) + instance cards
/app/instances InstanceListPage Table view with search/filter
/app/instances/new CreateWizardPage 5-step wizard (info, features, email, tunnel, review)
/app/instances/:id InstanceDetailPage Tabs: Overview, Services, Logs, Backups, Tunnel
/app/backups BackupsPage Cross-instance backup list with stats
/app/audit AuditLogPage Filterable audit log + detail drawer
/app/settings SettingsPage Port ranges, Pangolin config, defaults

Sidebar Navigation

  1. Dashboard (home)
  2. Instances (list)
  3. Backups (cross-instance)
  4. Audit Log (activity trail)
  5. Settings (CCP config)

Provisioning Flow

When POST /api/instances is called, the system:

1.  Validate uniqueness (slug + domain)
2.  Allocate 4 ports from ranges
3.  Generate 14 secrets (passwords, JWT keys, encryption keys)
4.  Create Instance record (status: PROVISIONING)
5.  [async] Create directory: /opt/ccp/instances/{slug}/changemaker.lite/
6.  [async] rsync CML source (excluding node_modules, .git, .env, .claude)
7.  [async] Decrypt secrets → build Handlebars context
8.  [async] Render 7 templates: docker-compose.yml, .env, nginx configs, Pangolin, Prometheus
9.  [async] Copy static files (nginx.conf)
10. [async] docker compose pull (non-fatal if images cached)
11. [async] docker compose build
12. [async] Start infrastructure: v2-postgres + redis-changemaker
13. [async] Wait for healthy (Docker healthcheck polling)
14. [async] Start API → run prisma migrate deploy → prisma db seed
15. [async] docker compose up (all services)
16. [async] Wait for HTTP health (localhost:{api_port}/api/health)
17. [async] Set status: RUNNING

Frontend polls GET /api/instances/:id every 3 seconds during provisioning to show progress.


Health Check System

How It Works

  1. Scheduler starts on API boot (default: every 5 minutes, configurable via HEALTH_CHECK_INTERVAL_MS)
  2. For each RUNNING instance, runs docker compose ps --format json
  3. Parses container states and health check results
  4. Determines overall status:
    • HEALTHY — all containers running, all health checks passing
    • DEGRADED — some containers running but not all, or some health checks failing
    • UNHEALTHY — majority of containers down or failing health checks
    • UNKNOWN — no containers found or compose project doesn't exist
  5. Stores HealthCheck record with per-service status JSON, response time
  6. Updates instance.lastHealthCheck timestamp

Manual Trigger

POST /api/instances/:id/health-check runs an immediate check (SUPER_ADMIN/OPERATOR only).


Backup System

What Gets Backed Up

  1. PostgreSQL dumppg_dump inside the instance's v2-postgres container
  2. Uploads archive — tar.gz of the uploads directory (if it exists)
  3. Manifest — JSON file with file names, sizes, SHA-256 hashes

Backup Flow

1.  Validate instance is RUNNING
2.  Create Backup record (status: PENDING)
3.  [async] Set status: IN_PROGRESS
4.  [async] mkdir /var/backups/ccp-instances/{slug}/{timestamp}/
5.  [async] docker compose exec v2-postgres pg_dump → v2-postgres.sql → gzip
6.  [async] tar -czf uploads.tar.gz (if uploads/ exists)
7.  [async] Write manifest.json with file inventory + SHA-256 hashes
8.  [async] tar -czf final archive → /var/backups/ccp-instances/{slug}/backup-{slug}-{timestamp}.tar.gz
9.  [async] Cleanup temp directory
10. [async] Update Backup record (COMPLETED, archivePath, sizeBytes, manifest)
11. [async] Write audit log

Retention

cleanupOldBackups(retentionDays) deletes archives + records older than the configured retention period (default: 30 days).


Phased Implementation

Phase 1: Foundation (COMPLETE)

Goal: Skeleton that boots — database, auth, project structure.

Delivered:

  • Prisma schema with all 10 models (User, Instance, HealthCheck, Backup, AuditLog, etc.)
  • JWT authentication (access + refresh tokens with atomic rotation)
  • Role-based access control (SUPER_ADMIN, OPERATOR, VIEWER)
  • Zod-validated environment configuration
  • AES-256-GCM encryption for instance secrets
  • React admin shell with dark theme, sidebar navigation, protected routes
  • Zustand auth store with token persistence + refresh interceptor
  • Login page, Dashboard placeholder, Settings page
  • Docker Compose orchestration (PostgreSQL, Redis, API, Admin)
  • Placeholder pages wired for Audit Log, Backups

Phase 2: Docker Lifecycle (COMPLETE)

Goal: Create and manage running CML instances.

Delivered:

  • Instance CRUD with slug/domain uniqueness validation
  • Port allocation across 4 ranges (API, Admin, PostgreSQL, Nginx)
  • Secret generation (14 secrets: postgres, redis, JWT, encryption, admin passwords)
  • Handlebars template engine rendering 7 config files per instance
  • 13-step async provisioning (copy source → render config → pull → build → migrate → seed → start)
  • Lifecycle operations: start, stop, restart (whole stack or individual service)
  • Container status via docker compose ps --format json
  • Log viewing via docker compose logs with service/tail/since filters
  • 5-step Create Instance wizard (basic info → features → email → tunnel → review)
  • Instance detail page with tabs (Overview, Services, Logs, Backups, Tunnel)
  • Service health grid with per-container restart/log-view actions
  • Provisioning progress indicator (polls every 3s)
  • Audit logging on all lifecycle operations (7 action types)

Phase 3: Observability + Backups (COMPLETE)

Goal: Visibility into instance health, operator activity trail, and data protection.

Delivered:

Part A — Audit Log

  • Audit service with filtered queries (action, instance, user, date range) + pagination
  • GET /api/audit endpoint with query parameter filtering
  • IP address capture on all audit log entries (existing + new)
  • USER_LOGIN audit event on successful authentication
  • SETTINGS_UPDATE audit event on settings changes
  • Full AuditLogPage: filterable table, action-colored tags, detail drawer with JSON inspector, server-side pagination, 30s auto-refresh toggle

Part B — Health Checks

  • Health service with checkInstanceHealth() analyzing Docker container states
  • Overall status determination (HEALTHY/DEGRADED/UNHEALTHY/UNKNOWN) from per-container state
  • Scheduled health checker (default 5-minute interval, configurable, 0 to disable)
  • POST /api/instances/:id/health-check for manual checks
  • GET /api/instances/:id/health-history for paginated history
  • Health card in Instance Detail overview with "Check Now" button + history table
  • Dashboard stat cards: Healthy and Degraded instance counts from /api/health/overview

Part C — Backups

  • Backup service: pg_dump via Docker exec, uploads tar.gz, SHA-256 manifest, final archive
  • Async backup creation with PENDING → IN_PROGRESS → COMPLETED/FAILED progression
  • POST /api/instances/:id/backup — create backup
  • GET /api/instances/:id/backups — instance-scoped backup list
  • GET /api/backups — cross-instance backup list with pagination
  • DELETE /api/backups/:id — delete backup (file + DB record)
  • GET /api/backups/:id/download — stream backup archive for download
  • BackupsPage: cross-instance table, instance filter, stats (count/size/last), "Backup All Running"
  • Enhanced Instance Detail backups tab: create/download/delete actions, status polling
  • Backup storage volume mount in docker-compose.yml
  • Old backup cleanup utility (configurable retention days)

Phase 4: Pangolin Integration (PLANNED)

Goal: Automated tunnel setup for exposing instances to the internet.

Scope:

  • Pangolin API client (site creation, resource management, Newt credentials)
  • Automated tunnel setup endpoint (POST /api/instances/:id/setup-tunnel)
  • Per-instance Newt container management (start/stop with stored credentials)
  • Resource sync for all instance subdomains (app, api, media, docs, etc.)
  • Tunnel status monitoring in Instance Detail tunnel tab
  • Bulk tunnel setup for fleet-wide deployment

Phase 5: Upgrades + Git Integration (PLANNED)

Goal: Rolling upgrades and version management.

Scope:

  • Git pull + branch checkout for instance source code
  • Database migration execution (prisma migrate deploy)
  • Docker image rebuild + rolling restart
  • Upgrade progress tracking (similar to provisioning)
  • Rollback capability (pre-upgrade backup + restore)
  • Version display on instance cards and detail pages
  • INSTANCE_UPGRADE audit event

Phase 6: User Management + RBAC (PLANNED)

Goal: Multi-operator support with granular permissions.

Scope:

  • User CRUD pages (create, edit, delete operators)
  • Role assignment and management
  • Per-instance access control (operator can only manage assigned instances)
  • Invitation flow (email invite with temporary password)
  • Password change and reset functionality
  • Session management (view/revoke active sessions)

Phase 7: Monitoring Dashboard (PLANNED)

Goal: Fleet-wide observability with trends and alerting.

Scope:

  • Health trend charts (uptime over time)
  • Resource utilization (CPU, memory, disk via Docker stats)
  • Alert rules (instance down > N minutes, disk usage > threshold)
  • Notification channels (email, webhook)
  • Backup compliance monitoring (last backup age alerts)
  • Fleet summary dashboard with at-a-glance status

Phase 8: Instance Configuration UI (PLANNED)

Goal: Edit instance configuration without SSH.

Scope:

  • Feature flag toggles (media, listmonk, gancio, monitoring)
  • SMTP configuration management
  • .env variable editor (safe subset)
  • Docker Compose service scaling
  • Configuration diff preview before applying
  • Auto-restart on config change

Environment Variables

Variable Default Description
NODE_ENV development Environment mode
PORT 5000 API server port
DATABASE_URL PostgreSQL connection string
REDIS_URL redis://localhost:6399 Redis connection string
JWT_ACCESS_SECRET JWT signing key (min 32 chars)
JWT_REFRESH_SECRET Refresh token signing key (min 32 chars)
JWT_ACCESS_EXPIRES_IN 15m Access token lifetime
JWT_REFRESH_EXPIRES_IN 7d Refresh token lifetime
ENCRYPTION_KEY AES-256 key for secrets at rest (min 32 hex chars)
INITIAL_ADMIN_EMAIL admin@example.com Bootstrap admin email
INITIAL_ADMIN_PASSWORD ChangeMe2025!! Bootstrap admin password (min 12 chars)
CORS_ORIGINS http://localhost:5100 Allowed origins (comma-separated)
INSTANCES_BASE_PATH /opt/ccp/instances Where instance directories live
CML_SOURCE_PATH /home/bunker-admin/changemaker.lite CML source to clone from
CML_GIT_REPO Git repo URL (if cloning remotely)
CML_GIT_BRANCH v2 Default branch for new instances
PORT_RANGE_API_START 14000 API port range start
PORT_RANGE_API_END 14999 API port range end
PORT_RANGE_ADMIN_START 13000 Admin port range start
PORT_RANGE_ADMIN_END 13999 Admin port range end
PORT_RANGE_POSTGRES_START 15400 PostgreSQL port range start
PORT_RANGE_POSTGRES_END 15499 PostgreSQL port range end
PORT_RANGE_NGINX_START 10000 Nginx port range start
PORT_RANGE_NGINX_END 10999 Nginx port range end
PANGOLIN_API_URL Pangolin API base URL
PANGOLIN_API_KEY Pangolin API key
PANGOLIN_ORG_ID Pangolin organization ID
HEALTH_CHECK_INTERVAL_MS 300000 Health check interval (0 to disable)
BACKUP_STORAGE_PATH /var/backups/ccp-instances Backup archive storage directory
BACKUP_RETENTION_DAYS 30 Auto-cleanup threshold

Docker Services

services:
  ccp-postgres:     # PostgreSQL 16 Alpine — CCP database (port 5480)
  ccp-redis:        # Redis 7 Alpine — rate limiting, caching (port 6399)
  ccp-api:          # Node 20 + Docker CLI — API server (port 5000)
  ccp-admin:        # Nginx + React SPA — admin GUI (port 5100)

Volume Mounts (ccp-api)

Host Container Mode Purpose
./api /app rw Source code (dev)
./templates /app/templates ro Handlebars templates
/var/run/docker.sock /var/run/docker.sock Docker CLI access
$INSTANCES_BASE_PATH Same rw Instance directories
$CML_SOURCE_PATH Same ro CML source for provisioning
$BACKUP_STORAGE_PATH Same rw Backup archives

File Inventory

API (api/)

src/
├── server.ts                          # Express app + route mounting + health scheduler start
├── config/
│   ├── env.ts                         # Zod env validation (30+ vars)
│   └── redis.ts                       # Redis client config
├── middleware/
│   ├── auth.ts                        # authenticate + requireRole middleware
│   ├── error-handler.ts               # AppError class + global error handler
│   └── validate.ts                    # Zod request body validation
├── modules/
│   ├── auth/
│   │   ├── auth.routes.ts             # POST login/refresh/logout, GET /me
│   │   ├── auth.schemas.ts            # Zod schemas for auth payloads
│   │   └── auth.service.ts            # JWT generation, bcrypt verify, token rotation
│   ├── instances/
│   │   ├── instances.routes.ts        # CRUD + lifecycle + services + logs + health + backups
│   │   ├── instances.schemas.ts       # Zod create/update validation
│   │   ├── instances.service.ts       # Business logic + audit logging with IP capture
│   │   └── provisioner.ts             # 13-step async provisioning orchestration
│   ├── audit/
│   │   ├── audit.routes.ts            # GET /api/audit with filters + pagination
│   │   └── audit.service.ts           # Prisma query builder for audit logs
│   ├── backups/
│   │   └── backup.routes.ts           # GET list, DELETE, GET download
│   ├── health/
│   │   └── health.routes.ts           # Public health + authenticated overview
│   └── settings/
│       └── settings.routes.ts         # GET all, PUT :key (+ audit logging)
├── services/
│   ├── docker.service.ts              # Docker CLI wrapper (ps, exec, logs, up, down, etc.)
│   ├── health.service.ts              # Health check logic + 5-minute scheduler
│   ├── backup.service.ts              # pg_dump + tar + manifest + cleanup
│   ├── port-allocator.ts              # Port range management
│   ├── secret-generator.ts            # CML-compatible credential generation
│   └── template-engine.ts             # Handlebars rendering for 7 config files
└── utils/
    ├── encryption.ts                  # AES-256-GCM encrypt/decrypt
    └── logger.ts                      # Winston config

Admin (admin/)

src/
├── App.tsx                            # Route definitions + dark theme config
├── main.tsx                           # React entry
├── components/
│   ├── AppLayout.tsx                  # Sidebar nav + header + user dropdown
│   ├── InstanceCard.tsx               # Card with status, health bar, features
│   ├── ServiceHealthGrid.tsx          # Container state table with actions
│   ├── LogViewer.tsx                  # Log display with service filter
│   └── ProtectedRoute.tsx             # Auth guard wrapper
├── pages/
│   ├── LoginPage.tsx                  # Auth form
│   ├── DashboardPage.tsx              # Stats + health overview + instance cards
│   ├── InstanceListPage.tsx           # Instance table
│   ├── CreateWizardPage.tsx           # 5-step provisioning wizard
│   ├── InstanceDetailPage.tsx         # Tabbed detail (overview/services/logs/backups/tunnel)
│   ├── BackupsPage.tsx                # Cross-instance backup manager
│   ├── AuditLogPage.tsx               # Filterable audit trail
│   └── SettingsPage.tsx               # CCP configuration
├── stores/
│   └── auth.store.ts                  # Zustand auth + localStorage persistence
├── lib/
│   └── api.ts                         # Axios + auth interceptors + token refresh
└── types/
    └── api.ts                         # TypeScript interfaces for all API models

Templates (templates/)

docker-compose.yml.hbs                 # Full CML docker-compose with ports/secrets/features
env.hbs                                # Instance .env file
nginx/
├── nginx.conf                         # Global nginx config (static copy)
└── conf.d/
    ├── default.conf.hbs               # Subdomain routing
    ├── api.conf.hbs                   # API reverse proxy
    └── services.conf.hbs              # Service proxies
configs/
├── pangolin/resources.yml.hbs         # Tunnel resource definitions
└── prometheus/prometheus.yml.hbs      # Monitoring scrape targets

Quick Start

# 1. Clone and enter the CCP directory
cd changemaker.lite/changemaker-control-panel

# 2. Copy environment file
cp .env.example .env
# Edit .env: set strong passwords for JWT_ACCESS_SECRET, JWT_REFRESH_SECRET, ENCRYPTION_KEY

# 3. Start CCP
docker compose up -d

# 4. Run migrations + seed
docker compose exec ccp-api npx prisma migrate deploy
docker compose exec ccp-api npx prisma db seed

# 5. Access admin GUI
open http://localhost:5100
# Login with INITIAL_ADMIN_EMAIL / INITIAL_ADMIN_PASSWORD from .env

Development

# API development (hot reload)
cd api && npm install && npx tsx src/server.ts

# Admin development (Vite dev server)
cd admin && npm install && npm run dev

# Type checking
cd api && npx tsc --noEmit
cd admin && npx tsc --noEmit

# Database operations
cd api && npx prisma migrate dev    # Create/apply migrations
cd api && npx prisma studio         # Browse database GUI
cd api && npx prisma db seed        # Re-seed admin user