admin/changemaker.lite
1
0
Fork 0
Code Issues Packages Projects Releases 31 Wiki Activity
changemaker.lite/CLAUDE.md
bunker-admin a71ba20176 Update CLAUDE.md with installer and release workflow documentation 
- Quick Start: add pre-built install section with curl one-liner
- Directory Structure: expand scripts/ with all deployment scripts
- Registry Operations: add build-release, install commands, two-compose note
- V2 Gotchas: document release vs source detection, api/dist gitignore,
  api/.dockerignore purpose
- Key Config Files: add docker-compose.prod.yml and config.sh

Bunker Admin
2026-03-22 20:38:36 -06:00

849 lines
42 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Changemaker Lite is a self-hosted political campaign platform built with Docker Compose. It consolidates advocacy email campaigns, geographic mapping, volunteer management, and administration into a single TypeScript stack. The primary domain is `cmlite.org`.
**Current state:** V2 rebuild substantially complete on the `v2` branch. Core platform operational with Phases 1-14 complete. See `V2_PLAN.md` for the full roadmap.
**Status Summary:**
- ✅ Phases 1-14 Complete (Foundation through Monitoring + DevOps)
- ✅ Security Audit Complete (13 findings addressed, Feb 2026)
- ✅ NAR 2025 Server Import (Canadian electoral data)
- ✅ Media Manager Integration (dual API architecture)
- ✅ Email Templates System
- ✅ Data Quality Dashboard
- ✅ Observability Dashboard
- ✅ **Drizzle to Prisma Migration Complete** (Media API consolidated to single-ORM, Feb 2026)
- ✅ **Automated Pangolin Setup** (One-command tunnel deployment, Feb 2026)
- ✅ **Migration Drift Fixed** (Baseline catch-up migration, 14 migrations cover full schema, Feb 2026)
- 🚧 Phase 15 (Testing + Polish) - Next
---
## V2 Architecture
### Stack
- **Dual API Architecture**
- **Express.js API** (TypeScript, port 4000) — Main V2 features with Prisma ORM + PostgreSQL 16
- **Fastify Media API** (TypeScript, port 4100) — Video library with Prisma ORM (shared DB) ✅ **Migrated from Drizzle (Feb 2026)**
- **React Admin GUI** — Vite + Ant Design + Zustand, port 3000
- **Nginx reverse proxy** — subdomain routing (`*.cmlite.org`)
- **NocoDB v2** — read-only data browser on port 8091
- **Redis** — caching, rate limiting, BullMQ backend, geocoding queue (authenticated)
- **Monitoring Stack** (Docker profile: `monitoring`) — Prometheus, Grafana, Alertmanager, cAdvisor, exporters
### Authentication & Security
- **JWT-based auth:** access tokens (15min) + refresh tokens (7 days, stored in DB)
- **Password policy:** 12+ characters, uppercase, lowercase, digit (enforced at schema level)
- **Initial admin:** Configured via `INITIAL_ADMIN_EMAIL` and `INITIAL_ADMIN_PASSWORD` env vars (auto-created during database seeding)
- **Roles:** `SUPER_ADMIN`, `INFLUENCE_ADMIN`, `MAP_ADMIN`, `BROADCAST_ADMIN`, `CONTENT_ADMIN`, `MEDIA_ADMIN`, `PAYMENTS_ADMIN`, `EVENTS_ADMIN`, `SOCIAL_ADMIN`, `USER`, `TEMP`
- **RBAC:** `requireRole(...roles)`, `requireNonTemp`, `authenticate` middleware; `SUPER_ADMIN` implicitly bypasses all role checks
- **Module-specific role groups** (defined in `api/src/utils/roles.ts`): `INFLUENCE_ROLES`, `MAP_ROLES`, `BROADCAST_ROLES`, `CONTENT_ROLES`, `MEDIA_ROLES`, `PAYMENTS_ROLES`, `EVENTS_ROLES`, `SOCIAL_ROLES`, `SYSTEM_ROLES`, `SCHEDULING_ROLES`
- **User management:** `SUPER_ADMIN` always; other admins need `permissions.canManageUsers: true` for write operations
- **Security features:**
- Refresh token rotation (atomic transaction)
- User enumeration prevention (401 not 404)
- Rate limiting on auth endpoints (10/min)
- Redis authentication required
- XSS/injection prevention (HTML escaping)
- Path traversal protection
- Encryption key for DB secrets (ENCRYPTION_KEY env var)
- Security audit complete (13 findings addressed, see `SECURITY_AUDIT_2025-02-11.md`)
### Email Systems
- **BullMQ** — async advocacy email job queue with SMTP
- **Listmonk** — newsletter/marketing campaigns (opt-in sync via `LISTMONK_SYNC_ENABLED`)
- **MailHog** — dev email capture (`EMAIL_TEST_MODE=true`)
### Directory Structure (Annotated)
```
changemaker.lite/
├── api/ # Dual API servers (Express + Fastify)
│ ├── prisma/
│ │ ├── schema.prisma # 30+ models: User, Campaign, Location, Shift, etc.
│ │ ├── migrations/ # Prisma migration history
│ │ └── seed.ts # Admin user, settings, page blocks
│ ├── drizzle/ # Media tables (Drizzle ORM)
│ ├── Dockerfile.media # Fastify media server container
│ └── src/
│ ├── server.ts # Express API entry point (port 4000)
│ ├── media-server.ts # Fastify media API entry point (port 4100)
│ ├── config/
│ │ └── env.ts # Zod-validated environment config (100+ vars)
│ ├── middleware/ # auth, rbac, rate-limit, validate, error-handler
│ ├── modules/
│ │ ├── auth/ # JWT login, register, refresh, logout
│ │ ├── users/ # User CRUD + pagination + search
│ │ ├── settings/ # Site settings singleton
│ │ ├── services/ # Service health checks
│ │ ├── influence/
│ │ │ ├── campaigns/ # Campaign CRUD + public routes
│ │ │ ├── representatives/ # Represent API integration + cache
│ │ │ ├── responses/ # Response wall + moderation + upvoting
│ │ │ ├── postal-codes/ # Postal code cache service
│ │ │ ├── campaign-emails/ # Email tracking + stats
│ │ │ └── email-queue/ # BullMQ queue admin
│ │ ├── map/
│ │ │ ├── locations/ # Location CRUD + geocoding + NAR import
│ │ │ ├── geocoding/ # Multi-provider geocoding (6 providers)
│ │ │ ├── cuts/ # Polygon CRUD + spatial queries
│ │ │ ├── shifts/ # Shift CRUD + signups
│ │ │ ├── canvass/ # Canvassing sessions + visits + routes
│ │ │ ├── tracking/ # GPS tracking sessions (volunteer + admin routes)
│ │ │ └── settings/ # Map settings singleton
│ │ ├── pages/
│ │ │ ├── pages-admin.routes.ts # Landing page CRUD
│ │ │ ├── pages-public.routes.ts # Public page renderer
│ │ │ └── blocks.routes.ts # Block library API
│ │ ├── email-templates/ # Email template CRUD + rendering
│ │ ├── media/ # Fastify media API (videos, reactions, jobs)
│ │ ├── listmonk/ # Newsletter sync admin routes
│ │ ├── pangolin/ # Tunnel management (Newt integration)
│ │ ├── docs/ # MkDocs + Code Server health checks
│ │ ├── qr/ # QR code PNG generation (public)
│ │ └── observability/ # Prometheus/Grafana/Alertmanager integration
│ ├── services/ # email, email-queue, geocode-queue, listmonk, pangolin, docker
│ ├── types/ # express.d.ts (Request augmentation)
│ └── utils/ # logger (Winston), metrics (prom-client), spatial
│
├── admin/ # React Admin (Vite + Ant Design + Zustand)
│ └── src/
│ ├── App.tsx # Main router + route definitions
│ ├── components/
│ │ ├── AppLayout.tsx # Admin sidebar layout
│ │ ├── PublicLayout.tsx # Public dark theme layout
│ │ ├── VolunteerLayout.tsx # Volunteer portal layout
│ │ ├── MediaPublicLayout.tsx # Public media gallery layout
│ │ ├── GrapesJSEditor.tsx # Landing page editor wrapper (forwardRef, Ctrl+S)
│ │ ├── map/ # Leaflet map components + controls + drawing modes
│ │ ├── canvass/ # GPS tracking, markers, route, visit recording
│ │ ├── media/ # VideoCard, BulkActions, gallery components
│ │ ├── email-templates/ # Email template components
│ │ └── observability/ # Monitoring components
│ ├── pages/
│ │ ├── auth/ # LoginPage
│ │ ├── DashboardPage.tsx # Admin dashboard
│ │ ├── UsersPage.tsx # User CRUD
│ │ ├── SettingsPage.tsx # Global site settings
│ │ ├── influence/
│ │ │ ├── CampaignsPage.tsx # Campaign management
│ │ │ ├── ResponsesPage.tsx # Response moderation
│ │ │ ├── RepresentativesPage.tsx # Rep cache admin
│ │ │ └── EmailQueuePage.tsx # Queue monitoring
│ │ ├── map/
│ │ │ ├── LocationsPage.tsx # Location CRUD + CSV + geocoding
│ │ │ ├── CutsPage.tsx # Cut table + map drawing editor
│ │ │ ├── ShiftsPage.tsx # Shift CRUD + signups drawer
│ │ │ ├── MapSettingsPage.tsx # Map settings
│ │ │ └── DataQualityDashboardPage.tsx # Geocoding quality metrics
│ │ ├── CanvassDashboardPage.tsx # Admin canvass overview
│ │ ├── WalkSheetPage.tsx # Printable walk sheet
│ │ ├── CutExportPage.tsx # Printable location report
│ │ ├── volunteer/
│ │ │ ├── VolunteerMapPage.tsx # Full-screen GPS canvass map
│ │ │ ├── VolunteerShiftsPage.tsx # Assigned shifts
│ │ │ ├── MyActivityPage.tsx # Visit history + outcomes
│ │ │ └── MyRoutesPage.tsx # Route history
│ │ ├── public/
│ │ │ ├── CampaignsListPage.tsx # Public campaign listing
│ │ │ ├── CampaignPage.tsx # Campaign detail + email form
│ │ │ ├── ResponseWallPage.tsx # Public response wall
│ │ │ ├── MapPage.tsx # Public Leaflet map
│ │ │ ├── ShiftsPage.tsx # Public shift signup
│ │ │ ├── LandingPage.tsx # Rendered landing page (/p/:slug)
│ │ │ ├── MediaGalleryPage.tsx # Public video gallery
│ │ │ └── MediaViewerPage.tsx # Video detail page
│ │ ├── media/
│ │ │ ├── LibraryPage.tsx # Video library management
│ │ │ ├── SharedMediaPage.tsx # Public gallery admin
│ │ │ └── MediaJobsPage.tsx # Job queue monitoring
│ │ ├── LandingPagesPage.tsx # Landing page manager
│ │ ├── PageEditorPage.tsx # Full-screen GrapesJS editor
│ │ ├── EmailTemplatesPage.tsx # Email template CRUD
│ │ ├── EmailTemplateEditorPage.tsx # Email template editor
│ │ ├── ListmonkPage.tsx # Newsletter sync management
│ │ ├── PangolinPage.tsx # Tunnel setup wizard
│ │ ├── DocsPage.tsx # MkDocs export management
│ │ ├── MkDocsSettingsPage.tsx # Documentation config
│ │ ├── ObservabilityPage.tsx # Monitoring dashboard
│ │ └── services/
│ │ ├── MiniQRPage.tsx # Mini QR iframe
│ │ ├── MailHogPage.tsx # Email capture UI
│ │ ├── CodeEditorPage.tsx # Code Server management
│ │ ├── N8nPage.tsx # Workflow automation
│ │ ├── GiteaPage.tsx # Git repository hosting
│ │ └── NocoDBPage.tsx # Data browser management
│ ├── stores/ # auth.store.ts, canvass.store.ts (Zustand)
│ ├── lib/ # api.ts, media-api.ts, media-public-api.ts (axios)
│ ├── hooks/ # useDebounce, useLocalStorage
│ └── types/ # api.ts, canvass.ts, media.ts (TypeScript interfaces)
│
├── media-manager/ # Legacy media manager (reference)
├── nginx/ # Reverse proxy config (subdomain routing + CSP)
├── configs/ # Prometheus, Grafana, Alertmanager configs
├── scripts/ # Deployment, backup, upgrade, registry scripts
│ ├── install.sh # Curl-friendly installer (downloads tarball + runs config.sh)
│ ├── build-and-push.sh # Build production images → push to Gitea registry
│ ├── build-release.sh # Package runtime files into release tarball
│ ├── mirror-images.sh # Mirror third-party images to Gitea
│ ├── upgrade.sh # 6-phase upgrade (git or release-tarball mode)
│ ├── upgrade-check.sh # Check for updates (git or Gitea API)
│ ├── upgrade-watcher.sh # Systemd bridge for admin GUI upgrades
│ └── backup.sh # PostgreSQL + Listmonk + uploads backup
├── docker-compose.yml # V2 orchestration (20+ services)
├── docker-compose.v1.yml # V1 backup (reference)
├── .env.example # All required environment variables
└── V2_PLAN.md # Full 14-phase roadmap
```
---
## Quick Start Guide
### Pre-built Install (Production — Recommended)
The fastest way to deploy. No source code, no compilation:
```bash
curl -fsSL https://gitea.bnkops.com/admin/changemaker.lite/raw/branch/v2/scripts/install.sh | bash
```
This downloads a ~9MB release tarball, runs the config wizard, and sets `IMAGE_TAG=latest`. Then:
```bash
cd ~/changemaker.lite && docker compose up -d
```
Pre-built images are pulled from `gitea.bnkops.com/admin` (~2 min). Database migrations and seeding run automatically via the API entrypoint. Access the admin GUI at http://localhost:3000.
### Source Install (Development)
1. **Clone repository and checkout v2 branch:**
```bash
git clone <repo-url> changemaker.lite
cd changemaker.lite
git checkout v2
```
2. **Create environment file:**
```bash
cp .env.example .env
# Edit .env and set:
# - V2_POSTGRES_PASSWORD (strong password)
# - REDIS_PASSWORD (strong password)
# - JWT_ACCESS_SECRET (openssl rand -hex 32)
# - JWT_REFRESH_SECRET (openssl rand -hex 32)
# - ENCRYPTION_KEY (openssl rand -hex 32, must differ from JWT secrets)
```
3. **Start core services:**
```bash
docker compose up -d v2-postgres redis api admin
```
4. **Run database migrations:**
```bash
docker compose exec api npx prisma migrate deploy
docker compose exec api npx prisma db seed
```
5. **Access the application:**
- Admin GUI: http://localhost:3000 (see INITIAL_ADMIN_EMAIL/INITIAL_ADMIN_PASSWORD in .env)
- API: http://localhost:4000
- **Change default password immediately**
### Development Workflow
**Starting services:**
```bash
# Core services
docker compose up -d v2-postgres redis api admin
# Include monitoring stack
docker compose --profile monitoring up -d
# Include media API
docker compose up -d media-api
```
**Local development (without Docker):**
```bash
# Terminal 1: API
cd api && npm install && npm run dev
# Terminal 2: Admin
cd admin && npm install && npm run dev
# Terminal 3 (optional): Media API
cd api && npm run dev:media
```
### Accessing Services
| Service | URL | Default Credentials |
|---------|-----|---------------------|
| Admin GUI | http://localhost:3000 | See INITIAL_ADMIN_EMAIL/INITIAL_ADMIN_PASSWORD in .env |
| API | http://localhost:4000 | - |
| NocoDB | http://localhost:8091 | See `NC_ADMIN_EMAIL`/`NC_ADMIN_PASSWORD` in .env |
| MailHog | http://localhost:8025 | - |
| Grafana | http://localhost:3001 | admin / admin |
| Prometheus | http://localhost:9090 | - |
| Listmonk | http://localhost:9001 | See `LISTMONK_WEB_ADMIN_USER`/`PASSWORD` in .env |
### Feature Flags
Enable optional features in `.env`:
```bash
# Media Manager
ENABLE_MEDIA_FEATURES=true
# Listmonk Newsletter Sync
LISTMONK_SYNC_ENABLED=true
# Email Test Mode (sends to MailHog instead of SMTP)
EMAIL_TEST_MODE=true
```
---
## Development Commands
The user likes to use Docker - recereating services as if in production.
### API Development
```bash
cd api && npm run dev # Express dev server (port 4000)
cd api && npm run dev:media # Fastify media dev server (port 4100)
cd api && npx tsc --noEmit # Type-check
cd api && npx prisma migrate dev # Run/create Prisma migrations
cd api && npx prisma studio # Browse database
cd api && npx drizzle-kit push # Push Drizzle schema changes (media)
```
### Admin Development
```bash
cd admin && npm run dev # Vite dev server (port 3000)
cd admin && npx tsc --noEmit # Type-check
cd admin && npm run build # Production build
```
### Docker Operations
```bash
# Start services
docker compose up -d v2-postgres redis api admin
docker compose up -d media-api
docker compose --profile monitoring up -d
# View logs
docker compose logs -f api
docker compose logs -f media-api
# Database operations
docker compose exec api npx prisma migrate dev
docker compose exec api npx drizzle-kit push
# Stop services
docker compose down
```
### Registry & Release Operations
```bash
# Build production images and push to Gitea registry
./scripts/build-and-push.sh --services api,admin,media-api,nginx
./scripts/build-and-push.sh --no-push # Build only, no push (verify)
# Mirror third-party images to Gitea
./scripts/mirror-images.sh # Core images (postgres, redis, etc.)
./scripts/mirror-images.sh --all # Include heavy images (RC, Jitsi, n8n)
# Build release tarball (for pre-built installs — run AFTER build-and-push)
./scripts/build-release.sh --tag v2.1.0 # Creates releases/changemaker-lite-v2.1.0.tar.gz
./scripts/build-release.sh --tag v2.1.0 --upload # Also upload to Gitea Releases API
./scripts/build-release.sh --dry-run # Preview tarball contents
# Use registry images in upgrade (source installs)
./scripts/upgrade.sh --use-registry --force --skip-backup
# Install from tarball (end-user one-liner)
curl -fsSL https://gitea.bnkops.com/admin/changemaker.lite/raw/branch/v2/scripts/install.sh | bash
```
**Two compose files:**
- `docker-compose.yml` — Development: includes `build:` blocks and `./api:/app` source mounts
- `docker-compose.prod.yml` — Production: `image:` only, no source mounts, `IMAGE_TAG:-latest`
Release tarballs ship `docker-compose.prod.yml` as the compose file. Source installs use `docker-compose.yml`.
**Note:** gitea.bnkops.com must use Pangolin tunnel (not Cloudflare proxy) for large image layers (>100MB). See `docs/REGISTRY_GUIDE.md`.
### Testing & Backup
```bash
# Media API tests
cd api && ./test-media-api.sh
# Backup (PostgreSQL + Listmonk + uploads)
./scripts/backup.sh
# Type-check all projects
cd api && npx tsc --noEmit && cd ../admin && npx tsc --noEmit
```
### API Testing Credentials & Login
**Test admin account:** `admin@bnkops.ca` / `ChangeMe2025!` (SUPER_ADMIN role)
**Reliable login method (avoids shell `!` escaping issues):**
1. Write the JSON body to a file using the **Write tool** (NOT echo/printf — the `!` gets backslash-escaped by bash):
```
Write /tmp/login.json → {"email":"admin@bnkops.ca","password":"ChangeMe2025!"}
```
2. Use `curl -d @/tmp/login.json`:
```bash
curl -s -X POST http://localhost:4002/api/auth/login \
-H "Content-Type: application/json" -d @/tmp/login.json
```
3. Extract token and use for authenticated requests:
```bash
TOKEN=$(curl -s -X POST http://localhost:4002/api/auth/login \
-H "Content-Type: application/json" -d @/tmp/login.json \
| python3 -c "import sys,json; print(json.load(sys.stdin)['accessToken'])")
curl -s http://localhost:4002/api/some-endpoint -H "Authorization: Bearer $TOKEN"
```
**Port mapping:** API container port 4000 → host port **4002**, Admin port 3000 → host port **3002**
**Important:** The `!` character in `ChangeMe2025!` triggers bash history expansion. NEVER pass this password directly in bash command strings. Always use the Write-tool-to-file approach above.
---
## Core Modules Reference
### Auth & Users
**Files:**
- `api/src/modules/auth/` — JWT login, register, refresh, logout
- `api/src/modules/users/` — User CRUD + pagination + search
- `api/src/middleware/auth.ts` — JWT verification + RBAC
- `admin/src/stores/auth.store.ts` — Zustand auth state + token persistence
- `admin/src/lib/api.ts` — Axios with 401 refresh interceptor
**Features:** JWT access/refresh tokens, bcrypt passwords (12+ chars), role-based access control, user enumeration prevention, rate limiting
### Influence Module (Advocacy Campaigns)
**Files:**
- `api/src/modules/influence/campaigns/` — Campaign CRUD + public routes
- `api/src/modules/influence/representatives/` — Represent API client + cache
- `api/src/modules/influence/responses/` — Response wall + moderation + upvoting
- `api/src/services/email-queue.service.ts` — BullMQ queue + worker
- `admin/src/pages/CampaignsPage.tsx` — Campaign management
- `admin/src/pages/public/CampaignPage.tsx` — Public campaign page
**Features:** Postal code → representative lookup, email campaigns, response wall with moderation, BullMQ async email queue
**Routes:**
- Admin: `/app/influence/campaigns`, `/app/influence/responses`, `/app/influence/email-queue`
- Public: `/campaigns`, `/campaigns/:id`, `/responses/:campaignId`
### Map Module (Locations & Canvassing)
**Files:**
- `api/src/modules/map/locations/` — Location CRUD + geocoding + NAR import
- `api/src/modules/map/geocoding/geocoding.service.ts` — Multi-provider geocoding (6 providers)
- `api/src/modules/map/cuts/` — Polygon CRUD + spatial queries
- `api/src/modules/map/shifts/` — Shift CRUD + signups
- `api/src/modules/map/canvass/` — Canvassing sessions + visits + routes
- `api/src/modules/map/tracking/` — GPS tracking sessions (volunteer + admin routes)
- `api/src/utils/spatial.ts` — Point-in-polygon, haversine, bounds, centroids
- `admin/src/pages/LocationsPage.tsx` — Location CRUD + CSV + geocoding
- `admin/src/pages/CutsPage.tsx` — Cut table + map drawing editor
- `admin/src/pages/CanvassDashboardPage.tsx` — Admin canvass overview
- `admin/src/pages/volunteer/VolunteerMapPage.tsx` — Full-screen GPS canvass map
**Features:** Multi-provider geocoding, NAR 2025 import (Canadian electoral data), polygon cuts, volunteer shifts, canvassing system with GPS tracking, walking route algorithm, printable walk sheets
**Routes:**
- Admin: `/app/map/locations`, `/app/map/cuts`, `/app/map/shifts`, `/app/canvass/dashboard`
- Public: `/map`, `/shifts`
- Volunteer: `/volunteer/canvass/:cutId`, `/volunteer/assignments`, `/volunteer/activity`
### Landing Pages & Email Templates
**Files:**
- `api/src/modules/pages/` — Landing page CRUD + block library (3 route files)
- `api/src/modules/email-templates/` — Email template CRUD + rendering
- `admin/src/components/GrapesJSEditor.tsx` — GrapesJS wrapper (forwardRef, Ctrl+S)
- `admin/src/pages/PageEditorPage.tsx` — Full-screen page editor
- `admin/src/pages/EmailTemplateEditorPage.tsx` — Email template editor
**Features:** GrapesJS WYSIWYG editor, page/template CRUD, MkDocs export (Jinja2 Material overrides), public renderer, desktop-only editor warning
**Routes:**
- Admin: `/app/pages`, `/app/pages/:id/edit`, `/app/email-templates`
- Public: `/p/:slug`
### Media Manager (Dual API)
**Files:**
- `api/src/modules/media/` — Fastify media API (videos, reactions, jobs, analytics)
- `api/src/modules/media/services/` — FFprobe, video analytics service
- `api/src/modules/media/routes/` — Video CRUD, actions, schedule, analytics, tracking, upload
- `api/src/services/video-schedule-queue.service.ts` — BullMQ queue for scheduled publishing
- `admin/src/lib/media-api.ts` — Dedicated axios instance for Media API
- `admin/src/pages/media/LibraryPage.tsx` — Video library with quick actions + calendar
- `admin/src/pages/media/AnalyticsDashboardPage.tsx` — Global analytics dashboard
- `admin/src/pages/media/SharedMediaPage.tsx` — Public gallery admin
- `admin/src/pages/public/MediaGalleryPage.tsx` — Public video gallery
- `admin/src/components/media/` — VideoCard, VideoActions, modals, charts
**Features:**
- **Video CRUD:** Upload with FFprobe metadata extraction (duration, dimensions, orientation, quality), bulk operations
- **Quick Actions** (Feb 2026): Edit, preview, analytics, schedule, duplicate, preview links (24h JWT), reset analytics
- **Scheduled Publishing** (Feb 2026): BullMQ job queue, timezone support (11 zones), calendar view, publish/unpublish automation
- **Analytics** (Feb 2026): Views, watch time, completion rate, traffic sources, registered viewers, GDPR-compliant (IP hashing, 90-day retention)
- **Tracking:** Public endpoints for view/event recording, 10s heartbeat, navigator.sendBeacon for reliability
- **UI Features:** Keyboard shortcuts (E/P/A/S), hover overlays, skeleton loading, error handling, mobile responsive
**Routes:**
- Admin: `/app/media/library`, `/app/media/analytics`, `/app/media/shared`, `/app/media/jobs`
- Public: `/gallery` (public video gallery), `/gallery/watch/:id` (video viewer), `/media/:id` (backwards compatible viewer route)
- Tracking (public): `/track/view`, `/track/event`, `/track/heartbeat`
**Note:** The public gallery is served at `/gallery` via the admin app using `MediaPublicLayout`. This provides a unified purple interface for both authenticated and unauthenticated users. The gallery supports optional authentication (session-based upvoting/commenting for anonymous users).
**Documentation:**
- [Media Admin Features Guide](./docs/MEDIA_ADMIN_FEATURES.md) — Complete feature documentation
- [Video Analytics Guide](./docs/VIDEO_ANALYTICS_GUIDE.md) — Analytics setup and interpretation
- [Media API README](./api/src/modules/media/README.md) — Architecture overview
### Services & Integrations
**Listmonk Newsletter Sync:**
- `api/src/services/listmonk.client.ts` — Listmonk REST API client (native fetch)
- `api/src/services/listmonk-sync.service.ts` — Sync participants/locations → lists
- `admin/src/pages/ListmonkPage.tsx` — Newsletter sync management
- Opt-in sync: `LISTMONK_SYNC_ENABLED=true`
**Pangolin Tunnel Management:**
- `api/src/services/pangolin.client.ts` — Pangolin Integration API client
- `api/src/modules/pangolin/pangolin.routes.ts` — Tunnel management routes (includes `/setup-automated`)
- `admin/src/pages/PangolinPage.tsx` — Setup wizard + status dashboard + automated setup button
- `scripts/pangolin-setup.sh` — CLI wrapper for automated setup
- `configs/pangolin/resources.yml` — Central resource definitions (12 services)
- Newt container integration (Cloudflare alternative)
- **Automated setup:** One-command deployment (creates site, updates .env, restarts Newt)
- **Continuous sync:** Hourly resource sync via nginx cron job
**MkDocs + Code Server:**
- `api/src/modules/docs/docs.routes.ts` — Health checks + export routes
- `admin/src/pages/DocsPage.tsx` — MkDocs export management
- `admin/src/pages/CodeEditorPage.tsx` — Code Server management
- Embedded iframes in admin (CSP `frame-ancestors` for embedding)
**Mini QR Service:**
- `api/src/modules/qr/qr.routes.ts` — QR code PNG generation (public, no auth)
- `admin/src/pages/MiniQRPage.tsx` — Mini QR iframe
- Used by walk sheets + cut exports
### Observability & Monitoring
**Files:**
- `api/src/modules/observability/observability.routes.ts` — Prometheus/Grafana/Alertmanager integration
- `api/src/utils/metrics.ts` — 12 custom `cm_*` Prometheus metrics
- `admin/src/pages/ObservabilityPage.tsx` — Monitoring dashboard (3 tabs)
- `admin/src/pages/DataQualityDashboardPage.tsx` — Geocoding quality metrics
- `configs/prometheus/` — Scrape targets, alert rules
- `configs/grafana/` — 3 pre-configured dashboards
**Features:** 12 custom `cm_*` metrics (API uptime, queue size, sessions, etc.), HTTP request metrics, external service health gauges, 3 Grafana dashboards, alert rules, auto-start banner
**Routes:**
- Admin: `/app/observability`, `/app/map/data-quality`
- Direct: `localhost:9090` (Prometheus), `localhost:3001` (Grafana)
---
## Port Reference
| Port | Service | Notes |
|------|---------|-------|
| **Core Services** | | |
| 3000 | Admin GUI | Vite dev / React production |
| 4000 | Express API | Main V2 API (Prisma) |
| 4100 | Fastify Media API | Video library (Drizzle) |
| 5433 | V2 PostgreSQL | Localhost (container: 5432) |
| 6379 | Redis | Cache, rate limit, BullMQ |
| **Supporting Services** | | |
| 3001 | Grafana | Metrics visualization |
| 3010 | Homepage | Service dashboard |
| 3030 | Gitea | Git hosting |
| 4001 | MkDocs Site | Served docs |
| 4003 | MkDocs Dev | Live preview |
| 5432 | Listmonk PostgreSQL | Listmonk DB |
| 5678 | n8n | Workflow automation |
| 8025 | MailHog | Email capture (dev) |
| 8089 | Mini QR | QR generator |
| 8091 | NocoDB | Data browser |
| 8885 | Mini QR Proxy | Iframe-friendly |
| 8888 | Code Server | Web IDE |
| 9001 | Listmonk | Newsletter platform |
| **Monitoring** (profile: `monitoring`) | | |
| 8080 | cAdvisor | Container metrics |
| 8889 | Gotify | Notifications |
| 9090 | Prometheus | Metrics collection |
| 9093 | Alertmanager | Alert routing |
| 9100 | Node Exporter | Host metrics |
| 9121 | Redis Exporter | Redis metrics |
---
## Nginx Routing
| Subdomain | Target | Purpose |
|-----------|--------|---------|
| `app.cmlite.org` | Admin (3000) | **All application routes** (admin + public pages, campaigns, map, shifts, media) |
| `api.cmlite.org` | Express (4000) | Main API |
| `media.cmlite.org` | Fastify (4100) | Media API |
| `db.cmlite.org` | NocoDB (8091) | Data browser |
| `docs.cmlite.org` | MkDocs (4003) | Docs site |
| `code.cmlite.org` | Code Server (8888) | Web IDE |
| `n8n.cmlite.org` | n8n (5678) | Workflow automation |
| `git.cmlite.org` | Gitea (3030) | Git hosting |
| `home.cmlite.org` | Homepage (3010) | Dashboard |
| `grafana.cmlite.org` | Grafana (3001) | Metrics viz |
| `listmonk.cmlite.org` | Listmonk (9001) | Newsletters |
| `qr.cmlite.org` | Mini QR (8089) | QR generator |
| `cmlite.org` | MkDocs Static (4004) | **Documentation/marketing site only** |
**Clean separation:** Root domain (`${DOMAIN}`) serves MkDocs documentation site. All application functionality (admin GUI, public campaigns, map, shifts, media gallery) is accessible via `app.${DOMAIN}` subdomain. This provides clear separation between public documentation and the application.
---
## Common Patterns
**Note:** See `MEMORY.md` for comprehensive development patterns, gotchas, and lessons learned. Below are V2-specific patterns only.
### API Router Structure
- Service layer (`*.service.ts`) — business logic, database queries
- Routes (`*.routes.ts`) — Express router, middleware, validation
- Schemas (`*.schemas.ts`) — Zod validation schemas
- Split admin/public routes when needed (e.g., `campaigns.routes.ts` + `campaigns-public.routes.ts`)
### Authentication Middleware
- `authenticate` — requires any logged-in user
- `requireRole(...roles)` — requires specific role(s)
- `requireNonTemp` — blocks TEMP users
- Login redirects: ADMIN_ROLES → `/app`, USER/TEMP → `/volunteer`
### Frontend Architecture
- Admin pages: `admin/src/pages/` (AppLayout)
- Public pages: `admin/src/pages/public/` (PublicLayout, dark theme)
- Volunteer pages: `admin/src/pages/volunteer/` (VolunteerLayout)
- Zustand stores: `auth.store.ts`, `canvass.store.ts`
- API clients: `{ api }` from `lib/api.ts`, `mediaApi` from `lib/media-api.ts`
### Database ORMs
- **Prisma** (main API): Use `UncheckedCreateInput`/`UncheckedUpdateInput` for foreign keys, `Prisma.InputJsonValue` for JSON arrays
- **Drizzle** (media API): Separate schema file, push with `npx drizzle-kit push`, no migrations generated
### Prisma Migration Workflow
- **Always use `prisma migrate dev`** for schema changes (not `prisma db push`) — `db push` applies changes directly but doesn't create migration files, causing drift
- **Migration history:** 14 migrations in `api/prisma/migrations/` fully cover the schema (baseline catch-up applied Feb 2026)
- **Fixing drift:** If `db push` was used and migrations are out of sync:
1. Drop any stray indexes/objects in DB not in schema: `DROP INDEX IF EXISTS <name>;`
2. Create a temp shadow DB: `docker compose exec -T v2-postgres createdb -U changemaker prisma_shadow_diff`
3. Generate catch-up SQL: `docker compose exec -T api npx prisma migrate diff --from-migrations ./prisma/migrations --to-schema-datamodel ./prisma/schema.prisma --shadow-database-url "postgresql://..." --script`
4. Save to `api/prisma/migrations/<timestamp>_<name>/migration.sql`
5. Mark as applied: `docker compose exec -T api npx prisma migrate resolve --applied <migration_name>`
6. Verify: `docker compose exec -T api npx prisma migrate status` → "Database schema is up to date!"
7. Clean up: `docker compose exec -T v2-postgres dropdb -U changemaker prisma_shadow_diff`
- **Gotcha:** `--from-migrations` replays all migration files on a shadow DB. If a migration references tables created by `db push` (no migration file), it will fail. Fix: temporarily move the dependent migration aside, generate the catch-up (which includes the missing tables), then remove the old migration
- **Production deploys:** Use `prisma migrate deploy` (not `migrate dev`) — it applies pending migrations without creating a shadow DB
### V2-Specific Gotchas
- **Prisma migrations:** Never use `db push` on the v2 branch — always use `migrate dev` to keep migration history in sync. The baseline catch-up migration (`20260224100000_baseline_catchup`) covers all schema changes from Feb 18–24 that were previously applied via `db push`
- Fastify media API on port 4100, separate from Express on 4000 (same DB, different ORM)
- Volunteer page naming: `VolunteerShiftsPage.tsx` (not "MyAssignmentsPage")
- Tracking module: `api/src/modules/map/tracking/` (volunteer + admin routes)
- Pages module: 3 route files (pages-admin, pages-public, blocks)
- Vite proxy: `VITE_API_URL`, `VITE_MKDOCS_URL` env vars (Docker sets to container hostnames)
- Nginx media API block must come BEFORE general API block
- MkDocs port 4003 (was 4000, conflicted with API)
- Media upload: requires separate RW volume mount for inbox directory (`:rw` on `/media/local/inbox`), library remains read-only
- FFmpeg/FFprobe: installed in media-api container (Alpine `apk add --no-cache ffmpeg`), used for metadata extraction
- **Registry mode:** `IMAGE_TAG=local` (default) never pulls from registry; set to a commit SHA or `latest` to use pre-built images; `--use-registry` in upgrade.sh auto-sets `IMAGE_TAG` to the new commit SHA
- **Registry auth:** `docker login gitea.bnkops.com` required once per machine; `GITEA_REGISTRY_USER`/`GITEA_REGISTRY_PASS` in `.env` used by the API's registry status endpoint only (not Docker itself)
- **Cloudflare + registry:** gitea.bnkops.com must be DNS-only (no proxy) to push image layers >100MB; Cloudflare free plan blocks large blobs with 413 errors. See `docs/REGISTRY_GUIDE.md`
- **Production image size:** API Dockerfile production stage uses `npm ci --omit=dev` + `npx prisma generate` (not copying from build stage) to avoid including TypeScript devDeps in the final image
- **Release vs source installs:** Detected by `VERSION` file + absence of `.git/`. Release installs use `docker-compose.prod.yml` (no build blocks, no source mounts, `IMAGE_TAG:-latest`). Source installs use `docker-compose.yml` (build blocks + source mounts for dev). `config.sh`, `upgrade.sh`, and `upgrade-check.sh` all auto-detect the mode
- **`api/dist/` is gitignored:** Compiled output must NOT be committed. It's generated by `npm run build` (dev) or baked into Docker images (prod). If root-owned (from container builds), fix with `docker run --rm -v ./api:/api alpine chown -R $(id -u):$(id -g) /api/dist/`
- **`api/.dockerignore` excludes dist/:** Prevents stale local compiled JS from being copied into Docker build context. Without this, cached Docker layers may serve old module sets even after adding new TypeScript modules
---
## Security & Configuration
### Security Audit
Comprehensive security audit completed 2025-02-11, addressing 13 findings. See `SECURITY_AUDIT_2025-02-11.md` for full report.
**Key improvements:**
- Password policy: 12+ chars, uppercase, lowercase, digit (schema-enforced)
- Rate limits on auth endpoints (10/min per IP)
- Refresh token rotation (atomic transaction)
- User enumeration prevention (401 not 404)
- Redis authentication required
- XSS/injection prevention (HTML escaping)
- Path traversal protection
- Encryption key for DB secrets (`ENCRYPTION_KEY` required in production)
- Nginx security headers (HSTS, Permissions-Policy, CSP)
### Required Environment Variables
See `.env.example` for all 100+ variables. Critical ones:
- `V2_POSTGRES_PASSWORD`, `REDIS_PASSWORD`
- `JWT_ACCESS_SECRET`, `JWT_REFRESH_SECRET`
- `ENCRYPTION_KEY` (must differ from JWT secrets)
- `LISTMONK_SYNC_ENABLED` (opt-in newsletter sync)
- `EMAIL_TEST_MODE` (MailHog vs SMTP)
- `ENABLE_MEDIA_FEATURES` (media manager)
### Production Deployment
- **Tunneling:** Pangolin with Newt container (Cloudflare alternative)
- **SSL/TLS:** Handled by tunnel provider (Pangolin/Cloudflare)
- **Docker Networking:** All containers share `changemaker-lite` bridge network, reference by container name
- **Monitoring:** Enable with `docker compose --profile monitoring up -d`
- **Backups:** Run `./scripts/backup.sh` (PostgreSQL + Listmonk + uploads, optional S3 upload)
#### Production CORS Configuration
When deploying to a production domain via Pangolin tunnel, you MUST update the `.env` file to include the production domain in `CORS_ORIGINS`:
```bash
# Example for betteredmonton.org
CORS_ORIGINS=http://app.betteredmonton.org,https://app.betteredmonton.org,http://localhost:3000,http://localhost
# Also set production mode
NODE_ENV=production
```
Without this, API requests from the production domain will fail CORS validation. After updating `.env`, restart the API container:
```bash
docker compose restart api
```
---
## Troubleshooting
### Production 403/302 Errors - Pangolin Resources
**Symptom:** All API endpoints return 302 redirects to Pangolin authentication page, or 403 Forbidden errors.
**Root Cause:** Pangolin tunnel resources are configured with authentication enabled (default behavior).
**Fix:** Log in to your Pangolin dashboard and edit each resource:
1. Navigate to **Resources** → **Public**
2. For each resource (app, api, media, docs, etc.), click **Edit**
3. Change **Authentication** setting to **"Not Protected"** (or "Public Access"/"No Authentication")
4. Save changes
**Critical resources to fix first:**
- `api.betteredmonton.org` - Main API (all endpoints fail without this)
- `app.betteredmonton.org` - Admin GUI + public pages
- `media.betteredmonton.org` - Media API
**Verification:**
```bash
# Should return JSON, NOT a 302 redirect
curl https://api.betteredmonton.org/api/health
```
**See Also:** `PRODUCTION_403_FIX.md` for detailed step-by-step instructions.
### CORS Errors in Production
**Symptom:** Browser console shows CORS errors when accessing production domain.
**Fix:** Add production domain to `CORS_ORIGINS` in `.env` file (see Production CORS Configuration above).
### API Works Locally But Not Via Tunnel
Check in order:
1. **Newt container running:** `docker compose ps newt`
2. **Newt connected:** `docker compose logs newt --tail 50` (should show successful connection)
3. **Environment variables set:** `PANGOLIN_SITE_ID`, `PANGOLIN_NEWT_ID`, `PANGOLIN_NEWT_SECRET` in `.env`
4. **Pangolin resources configured:** All resources set to "Not Protected"
5. **Nginx running:** `docker compose ps nginx`
### Database Connection Failures
**Symptom:** API logs show database connection errors.
**Fix:**
1. Check PostgreSQL container: `docker compose ps v2-postgres`
2. Verify `DATABASE_URL` in `.env` matches container name and port
3. Check PostgreSQL logs: `docker compose logs v2-postgres --tail 50`
4. Test connection: `docker compose exec api npx prisma db execute --stdin <<< "SELECT 1"`
### Redis Connection Failures
**Symptom:** API logs show Redis connection errors, rate limiting doesn't work.
**Fix:**
1. Check Redis container: `docker compose ps redis-changemaker`
2. Verify `REDIS_PASSWORD` matches in `.env` and `REDIS_URL` format
3. Check Redis logs: `docker compose logs redis-changemaker --tail 50`
4. Test connection: `docker compose exec redis-changemaker redis-cli -a $REDIS_PASSWORD ping`
---
## V1 Reference (Legacy)
V1 code archived in `influence/`, `map/`, and `docker-compose.v1.yml`. Two independent Express apps using NocoDB REST API. See individual README files for V1 documentation:
- `influence/README.MD` — Features, config, campaign management
- `map/README.md` — Features, config, setup instructions
- Both use session-based auth, bcryptjs passwords, Bull job queues
---
## Key Configuration Files
### Infrastructure
- `docker-compose.yml` — Development orchestration (build blocks + source mounts, 20+ services)
- `docker-compose.prod.yml` — Production orchestration (image-only, no source mounts, `IMAGE_TAG:-latest`)
- `.env` / `.env.example` — Environment variables (100+ vars)
- `config.sh` — Interactive setup wizard (14 steps, release-mode aware)
### Database
- `api/prisma/schema.prisma` — Main schema (30+ Prisma models)
- `api/prisma/migrations/` — 14 migration files (fully cover schema as of Feb 2026)
- `api/drizzle.config.ts` — Drizzle config for media tables
- `api/prisma/seed.ts` — Database seeding
### Nginx
- `nginx/nginx.conf` — Global config + security headers
- `nginx/conf.d/default.conf` — Subdomain routing (12+ subdomains)
- `nginx/conf.d/api.conf` — API reverse proxy (Express + Fastify)
- `nginx/conf.d/services.conf` — Service proxies
### Monitoring
- `configs/prometheus/prometheus.yml` — Scrape targets + global config
- `configs/prometheus/alerts.yml` — Alert rules (12 rules)
- `configs/grafana/` — 3 pre-configured dashboards
- `configs/alertmanager/alertmanager.yml` — Alert routing
### Documentation
- `CLAUDE.md` — Project-wide instructions (this file)
- `V2_PLAN.md` — Full 14-phase roadmap
- `SECURITY_AUDIT_2025-02-11.md` — Security audit report
- `MEMORY.md` — Development patterns and gotchas
Reference in New Issue View Git Blame Copy Permalink