# Development & Release Workflow

How code changes move from development to production deployments across all installation methods.

---

## Overview

There are **three ways** Changemaker Lite gets deployed:

| Method | Who uses it | Images from | Compose file |
|--------|------------|-------------|--------------|
| **Source install** | Developers, contributors | Built locally from source | `docker-compose.yml` |
| **Release install** | Production servers, evaluators | Gitea registry (pre-built) | `docker-compose.prod.yml` (ships as `docker-compose.yml` in tarball) |
| **CCP provisioned** | Fleet operators (Control Panel) | Gitea registry (pre-built) | Rendered from `templates/docker-compose.yml.hbs` |

All three methods share the same Gitea container registry at `gitea.bnkops.com/admin`.

---

## The Pipeline

```
 ┌──────────────────────────────────────────────────────────────────┐
 │  DEVELOPMENT (your machine)                                     │
 │                                                                  │
 │  Edit code → docker compose up -d → test locally                │
 │  Uses: docker-compose.yml (build: blocks + ./api:/app mounts)   │
 └──────────────────┬───────────────────────────────────────────────┘
                    │ git push
                    ▼
 ┌──────────────────────────────────────────────────────────────────┐
 │  BUILD & PUBLISH                                                │
 │                                                                  │
 │  Step 1: ./scripts/build-and-push.sh                            │
 │          Builds 4 production images, pushes to Gitea registry   │
 │          (api, admin, media-api, nginx) tagged :SHA + :latest   │
 │                                                                  │
 │  Step 2: ./scripts/mirror-images.sh          (run once/rarely)  │
 │          Mirrors 36 third-party images to Gitea registry        │
 │          (postgres, redis, nocodb, jitsi, grafana, etc.)        │
 │                                                                  │
 │  Step 3: ./scripts/build-release.sh --tag vX.Y.Z --upload      │
 │          Packages runtime files into ~9MB tarball, uploads to   │
 │          Gitea Releases                                         │
 └──────────────────┬─────────────────100.90.78.47──────────────────────────────┘
                    │
        ┌───────────┴───────────┐
        ▼                       ▼
 ┌─────────────────┐   ┌──────────────────┐
 │ RELEASE INSTALL  │   │ CCP PROVISIONED  │
 │                  │   │                  │
 │ curl installer   │   │ Control Panel    │
 │ or manual tarball│   │ creates instance │
 │ → config.sh      │   │ via web UI       │
 │ → docker compose │   │ → renders config │
 │   up -d          │   │ → docker compose │
 │                  │   │   up -d          │
 └─────────────────┘   └──────────────────┘
        │                       │
        └───────────┬───────────┘
                    ▼
         All images pulled from
         gitea.bnkops.com/admin
         (zero external dependencies)
```

---

## Step-by-Step

### 1. Local Development

Standard Docker Compose workflow with hot-reload:

```bash
# Start core services
docker compose up -d v2-postgres redis api admin

# API logs (watch for errors)
docker compose logs -f api

# Run with media API
docker compose up -d media-api

# Run with monitoring stack
docker compose --profile monitoring up -d
```

**Key:** `docker-compose.yml` uses `build:` blocks to compile TypeScript from source and mounts `./api:/app` for live code changes. This is the only compose file that builds from source.

### 2. Build & Push Production Images

After code changes are tested locally:

```bash
# Build production images and push to Gitea registry
./scripts/build-and-push.sh
```

This builds **4 services** with multi-stage Dockerfiles (production target, no dev dependencies), tags each image with `:SHA` and `:latest`, and pushes to `gitea.bnkops.com/admin/changemaker-{service}`:

| Service | Dockerfile | What it produces |
|---------|-----------|-----------------|
| `api` | `api/Dockerfile` | Express + Prisma (compiled JS, no TS) |
| `admin` | `admin/Dockerfile` | Nginx serving React build output |
| `media-api` | `api/Dockerfile.media` | Fastify + FFmpeg (compiled JS) |
| `nginx` | `nginx/Dockerfile` | Nginx with `envsubst` domain templating |

```bash
# Build specific services only
./scripts/build-and-push.sh --services api,admin

# Build without pushing (verify first)
./scripts/build-and-push.sh --no-push

# Include code-server (~9GB, only when Dockerfile changes)
./scripts/build-and-push.sh --include-code-server
```

### 3. Mirror Third-Party Images (Run Once / On Version Bumps)

Copies all third-party Docker images used by the platform to the Gitea registry, so deployments never depend on Docker Hub, GHCR, LSCR, or GCR:

```bash
# Mirror all 36 images (core + platform + comms + monitoring)
./scripts/mirror-images.sh

# Mirror only essential infrastructure (postgres, redis, alpine)
./scripts/mirror-images.sh --core-only

# Preview without executing
./scripts/mirror-images.sh --dry-run
```

**When to re-run:** Only when upgrading a third-party image version. The script has explicit version pins — update the version in `mirror-images.sh`, then re-run.

Images are organized into 4 groups:

| Group | Count | Examples |
|-------|-------|---------|
| Core Infrastructure | 5 | postgres:16-alpine, redis:7-alpine, alpine:3 |
| Platform Services | 16 | nocodb, listmonk, gitea, n8n, vaultwarden, nginx, code-server |
| Communication | 8 | rocket.chat, mongo, nats, gancio, jitsi (4 containers) |
| Monitoring | 7 | prometheus, grafana, alertmanager, cadvisor, exporters, gotify |

### 4. Build Release Tarball

Packages only runtime files (~9 MB) — no source code, no node_modules:

```bash
# Build tarball
./scripts/build-release.sh --tag v2.2.0

# Build and upload to Gitea Releases
./scripts/build-release.sh --tag v2.2.0 --upload

# Preview contents without creating tarball
./scripts/build-release.sh --dry-run

# --upload refuses to overwrite an existing tag. To deliberately replace
# a release (destructive — users on that tag see no upgrade signal):
./scripts/build-release.sh --tag v2.2.0 --upload --replace
```

**Version hygiene:** bump the tag when changing release contents. Overwriting
an existing release silently breaks upgrade checks for users already on that
version — they see "no update available" even though the tarball they'd
download differs.

The tarball contains:
- `docker-compose.yml` (copy of `docker-compose.prod.yml` — image-only, no build blocks)
- `.env.example`, `config.sh` (configuration wizard)
- `scripts/` (init scripts, backup, upgrade, systemd units)
- `configs/` (prometheus, grafana, alertmanager, homepage, pangolin)
- `nginx/conf.d/` (templates for reference)
- `mkdocs/` (starter documentation)
- Empty data directories

### 5. Deploying

#### New Release Install (End Users)

```bash
# One-liner
curl -fsSL https://gitea.bnkops.com/admin/changemaker.lite/raw/branch/main/scripts/install.sh | bash

# Or manual
curl -LO https://gitea.bnkops.com/admin/changemaker.lite/releases/latest/download/changemaker-lite-latest.tar.gz
tar xzf changemaker-lite-latest.tar.gz
cd changemaker-lite
bash config.sh
docker compose up -d
```

All images (custom + third-party) pull from `gitea.bnkops.com/admin`. No external registry access needed.

#### New CCP Instance (Fleet Operators)

The Control Panel provisions instances via its web UI:

1. Operator fills in the Create Instance wizard (domain, features, email, tunnel)
2. CCP copies source files, renders templates (Handlebars), generates secrets
3. With `USE_REGISTRY_IMAGES=true` (default): pulls pre-built images from Gitea (~2 min)
4. With `USE_REGISTRY_IMAGES=false`: builds from source (~10+ min)
5. Starts infrastructure → runs migrations → starts all services

CCP registry settings (in `changemaker-control-panel/.env`):
```bash
GITEA_REGISTRY=gitea.bnkops.com/admin    # Registry URL for all images
USE_REGISTRY_IMAGES=true                   # true = pull pre-built, false = build from source
IMAGE_TAG=latest                           # Tag for custom images (api, admin, media-api)
```

### 6. Upgrading Existing Installations

#### Source Installs

```bash
./scripts/upgrade.sh                    # Standard: git pull + rebuild from source
./scripts/upgrade.sh --use-registry     # Fast: pull pre-built images instead of rebuilding
./scripts/upgrade.sh --dry-run          # Preview changes
```

#### Release Installs

```bash
./scripts/upgrade.sh                    # Auto-detects release mode, downloads latest tarball
```

Release installs are detected by the presence of a `VERSION` file and absence of `.git/`. The upgrade script automatically downloads the latest tarball from Gitea instead of running `git pull`.

---

## Image Naming Conventions

All images live under `gitea.bnkops.com/admin/`:

| Type | Naming Pattern | Example |
|------|---------------|---------|
| Custom services | `changemaker-{service}:{sha\|latest}` | `changemaker-api:latest` |
| Simple names | Same as upstream | `postgres:16-alpine`, `redis:7-alpine` |
| Namespaced → short | Org removed | `nocodb/nocodb` → `nocodb:0.301.3` |
| Conflict resolution | Explicit short name | `gotify/server` → `gotify`, `vaultwarden/server` → `vaultwarden` |
| Jitsi suite | `jitsi-{component}` | `jitsi-web:stable-9823`, `jitsi-prosody:stable-9823` |
| LinuxServer nginx | `ls-nginx` (avoids nginx conflict) | `ls-nginx:1.28.2` |

---

## Two Compose Files

| File | Purpose | Build? | Source mounts? | Image source |
|------|---------|--------|---------------|-------------|
| `docker-compose.yml` | Development | Yes (`build:` blocks) | Yes (`./api:/app`) | Built locally |
| `docker-compose.prod.yml` | Production | No | No | `${GITEA_REGISTRY:-gitea.bnkops.com/admin}/...` |

Release tarballs ship `docker-compose.prod.yml` renamed as `docker-compose.yml`.

The CCP template (`templates/docker-compose.yml.hbs`) generates a compose file that works like `docker-compose.prod.yml` when `USE_REGISTRY_IMAGES=true`, or like `docker-compose.yml` when `false`.

---

## Quick Reference

```bash
# ── Development ──
docker compose up -d v2-postgres redis api admin     # Start dev stack
docker compose logs -f api                            # Watch API logs
docker compose exec api npx prisma migrate dev        # Create migration

# ── Build & Publish ──
./scripts/build-and-push.sh                           # Build + push 4 images
./scripts/mirror-images.sh                            # Mirror 36 third-party images
git tag --sort=-v:refname | head -3                   # Check latest version tags
./scripts/build-release.sh --tag vX.Y.Z --upload      # Package + upload release

# ── Deploy ──
curl -fsSL .../install.sh | bash                      # New install (release)
./scripts/upgrade.sh                                  # Upgrade existing install
./scripts/upgrade.sh --use-registry                   # Fast upgrade (registry images)

# ── Verify ──
curl -s http://localhost:4000/api/health              # API health check
docker compose ps                                     # Container status
```

---

## Gitea API Tokens

There are **two separate Gitea tokens** with different purposes. Using the wrong one is a common mistake:

| Variable | Target | Used by | Create at |
|----------|--------|---------|-----------|
| `GITEA_REGISTRY_API_TOKEN` | Remote registry (`gitea.bnkops.com`) | `build-release.sh --upload`, release API calls | `https://gitea.bnkops.com/user/settings/applications` |
| `GITEA_API_TOKEN` | Local Gitea instance | Docs comments, user provisioning, SSO | `http://localhost:3030/user/settings/applications` |

**Key:** Release uploads and the Gitea Releases API require `GITEA_REGISTRY_API_TOKEN`. If you get `"user does not exist"` from the API, you're using the wrong token.

---

## Checklist: Cutting a New Release

1. [ ] All code changes committed and pushed to `main` branch
2. [ ] `docker compose up -d` works locally (smoke test)
3. [ ] **Determine version tag:**
   ```bash
   # Check the latest existing tag to pick the next version
   git tag --sort=-v:refname | head -5
   # Check commits since the last tag
   git log $(git tag --sort=-v:refname | head -1)..HEAD --oneline
   ```
4. [ ] `./scripts/build-and-push.sh` — builds and pushes 4 production images
5. [ ] `./scripts/mirror-images.sh` — only if third-party versions changed
6. [ ] `./scripts/build-release.sh --tag vX.Y.Z --upload` — packages and uploads tarball
7. [ ] **Add release notes** (via Gitea web UI or API):
   ```bash
   # Update release body via API (use GITEA_REGISTRY_API_TOKEN, not GITEA_API_TOKEN)
   GITEA_TOKEN=$(grep -oP 'GITEA_REGISTRY_API_TOKEN=\K.*' .env)
   # Find release ID
   curl -s "https://gitea.bnkops.com/api/v1/repos/admin/changemaker.lite/releases?limit=1" \
     -H "Authorization: token $GITEA_TOKEN" | python3 -c "import sys,json; r=json.load(sys.stdin)[0]; print(f'ID: {r[\"id\"]}, Tag: {r[\"tag_name\"]}')"
   # Update with release notes (write JSON body to /tmp/release-notes.json first)
   curl -s -X PATCH "https://gitea.bnkops.com/api/v1/repos/admin/changemaker.lite/releases/RELEASE_ID" \
     -H "Authorization: token $GITEA_TOKEN" \
     -H "Content-Type: application/json" \
     -d @/tmp/release-notes.json
   ```
8. [ ] Test clean install: `tar xzf ... && cd changemaker-lite && bash config.sh && docker compose up -d`
9. [ ] Test upgrade: `./scripts/upgrade.sh` on an existing installation
10. [ ] Verify: `curl http://localhost:4000/api/health` returns `{"status":"ok"}`