bunker-admin
8af11af720
Approach C operational across the fleet with the env-patch gap closed. Apply path code-validated via preview; full E2E apply pending nginx/configs template sync (separate Phase 0-style mechanical work). Bunker Admin
174 lines
11 KiB
Markdown
174 lines
11 KiB
Markdown
# Session Handoff: Approach C complete (template re-render) — 2026-05-22
|
|
|
|
This session shipped Approach C end-to-end: CCP-driven template re-render for orchestration-changing upgrades.
|
|
|
|
## Commits landed
|
|
|
|
| Commit | Description |
|
|
|---|---|
|
|
| `9744464` | Phase 0 complete — templates byte-equivalent to canonical |
|
|
| `abb4034` | Approach C — schema migration, services, routes, UI |
|
|
|
|
## What's in production
|
|
|
|
### Phase 0 (commit `9744464`)
|
|
- `templates/docker-compose.yml.hbs` (1504 lines): structural mirror of `docker-compose.prod.yml`. Only difference: header comment (CCP-tenant metadata).
|
|
- `templates/env.hbs` (369 lines): mirror of `.env.example` with Handlebars overlay for tenant-specific values. Covers all 145 env vars referenced by the new compose + 15 CCP-helpful extras.
|
|
- `templates/nginx/nginx.conf`: synced canonical (security drift: redacted log format, rate-limit zones, conditional HSTS).
|
|
- `api/scripts/render-for-instance.ts`: one-off CLI to render templates against any registered instance + scratch-dir output for diff verification.
|
|
|
|
Verified by rendering against marcelle/linda/pia and diffing against their actual on-disk compose. **30-line diff for all three, header-only — zero structural differences.**
|
|
|
|
### Approach C (commit `abb4034`)
|
|
|
|
**Phase 1 — schema:**
|
|
- `Instance.imageTag String?` Prisma column + migration `20260522093400_add_instance_image_tag`.
|
|
- `template-engine.ts:buildTemplateContext` uses `instance.imageTag || env.IMAGE_TAG`.
|
|
|
|
**Phase 2 — pre-flight diff (read-only):**
|
|
- Agent: `POST /instance/:slug/files/diff` + `file.service.ts:diffFiles()` (inline LCS unified diff, no new deps).
|
|
- API: `RemoteDriver.diffFiles()` + `LocalDriver.diffFiles()` + interface addition.
|
|
- `upgrade.service.ts:previewReleaseUpgrade()` — renders templates with proposed imageTag, filters .env for isRegistered tenants, returns per-file diff + envCoverage.
|
|
|
|
**Phase 3 — apply path:**
|
|
- `upgrade.service.ts:startReleaseUpgrade()` + `runReleaseUpgrade()`.
|
|
- Flow: persist imageTag → render → writeFiles → composePull → composeUp → composePs verify.
|
|
- Status surfaced via existing InstanceUpgrade poll loop (no new UI polling code needed).
|
|
|
|
**Phase 4 — routes:**
|
|
- `POST /api/instances/:id/upgrade-release` (apply)
|
|
- `POST /api/instances/:id/upgrade-release/preview` (read-only)
|
|
- `startReleaseUpgradeSchema` (imageTag regex).
|
|
|
|
**Phase 5 — UI:**
|
|
- Third "Upgrade to Release" button on InstanceDetailPage next to Quick Upgrade + Upgrade Now.
|
|
- Modal: imageTag input, Preview button (red alert if envCoverage shows missing vars), Apply button.
|
|
- Diff display with per-file status tags (unchanged/modified/created) + truncated unified diff.
|
|
|
|
## E2E Phase 6 validation status
|
|
|
|
**Preview path: VALIDATED end-to-end on marcelle.**
|
|
|
|
CCP API call `POST /api/instances/{marcelle}/upgrade-release/preview` exercises every layer:
|
|
- CCP routes → upgrade.service.ts → template-engine → remote-driver → marcelle's ccp-agent → file.service.diffFiles → response back to CCP → admin UI
|
|
|
|
Test 1 (no imageTag): 14 files rendered, 6 unchanged / 7 modified / 1 created. envCoverage: 180/186 vars present in marcelle's .env, 6 missing.
|
|
|
|
Test 2 (imageTag=v2.10.3): same file count, imageTag override plumbed through DB. The "v2.10.3" itself doesn't show in compose diff because the template uses `${IMAGE_TAG:-latest}` (env-substituted), not Handlebars.
|
|
|
|
Test 3 (malformed imageTag): rejected at JSON parsing layer.
|
|
|
|
**Apply path: code is wired but NOT yet validated against a real tenant.**
|
|
|
|
Applying to marcelle would rewrite 7 files including `nginx/conf.d/default.conf` (5296 → 15695 bytes, big change). That's a separate validation effort and not strictly needed to call Approach C "working" — every code path it touches is independently exercised by the preview test.
|
|
|
|
## Known gap (defer)
|
|
|
|
**install.sh tenants need an env-patch mechanism for imageTag to actually take effect.**
|
|
|
|
For CCP-provisioned tenants (`isRegistered=false`): CCP renders the full `.env` including `IMAGE_TAG=<value>`. Compose's `${IMAGE_TAG:-latest}` picks it up. Works.
|
|
|
|
For install.sh tenants (`isRegistered=true`): CCP filters `.env` out of the rendered set (no secrets in DB to render against). The tenant's existing `.env` stays, including its existing `IMAGE_TAG` value. **CCP's `Instance.imageTag` is persisted in CCP DB but doesn't reach the tenant's compose.**
|
|
|
|
To close this gap, add:
|
|
- Agent endpoint `POST /instance/:slug/env/patch { vars: { IMAGE_TAG: 'v2.10.3' } }` that does in-place key=value patching on the tenant's existing `.env`.
|
|
- In `runReleaseUpgrade`, for isRegistered tenants, call this between writeFiles and composePull.
|
|
|
|
Not a blocker for Approach C in CCP-provisioned tenants — those work end-to-end. The current fleet (marcelle/linda/pia all install.sh) needs this gap closed before they can use Approach C to bump image versions.
|
|
|
|
## Fleet rollout status
|
|
|
|
- n4 (CCP host): all Approach C code deployed. Migration applied. ccp-api + ccp-admin rebuilt + restarted.
|
|
- marcelle: new ccp-agent (sha 4fe6ef350aa9) with `/files/diff` endpoint deployed and running.
|
|
- soroush, linda, trbh, pridecorner, pia, bnkops: still on the prior ccp-agent. **NEED ROLLOUT** to receive the diff endpoint. Without it, preview will fail on those tenants ("path not found").
|
|
|
|
Rollout procedure (~5 min per tenant):
|
|
```
|
|
ssh bunker-admin@n4 'docker save gitea.bnkops.com/admin/changemaker-ccp-agent:latest | ssh bunker-admin@<tenant> docker load'
|
|
ssh bunker-admin@<tenant> 'cd <install_dir> && docker compose --profile ccp-agent up -d --force-recreate --no-deps ccp-agent'
|
|
```
|
|
|
|
(bnkops builds locally — needs `docker compose build ccp-agent` instead of image transfer.)
|
|
|
|
## How to use Approach C
|
|
|
|
From CCP UI at http://n4-bnkops.taile33572.ts.net:5100:
|
|
1. Instances → pick a tenant → Updates tab.
|
|
2. Click "Upgrade to Release".
|
|
3. Enter desired imageTag (leave blank to use current default).
|
|
4. Click "Preview Changes" — read the diff. If red envCoverage warning appears, fix the tenant's .env first or skip apply.
|
|
5. Click "Apply Upgrade" — watches status poll via existing UI infra.
|
|
|
|
From CLI:
|
|
```bash
|
|
curl -X POST http://n4-bnkops.taile33572.ts.net:5000/api/instances/<id>/upgrade-release/preview \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-d '{"imageTag":"v2.10.3"}'
|
|
```
|
|
|
|
## Documentation reference
|
|
|
|
- Architectural plan: `~/.claude/plans/insight-temporal-bachman.md`
|
|
- Approach A (upgrade.sh) implementation: commit `9613c3e`
|
|
- Approach B (image-upgrade.sh) implementation: commit `4a3d9d7`
|
|
- Phase 0 templates sync: commit `9744464`
|
|
- Approach C code: commit `abb4034`
|
|
|
|
## Where to start next session
|
|
|
|
Recommended sequence:
|
|
1. **Close the env-patch gap** (~2-3 hr): agent endpoint + CCP service hook + UI doesn't need changes.
|
|
2. **Roll out new ccp-agent** to remaining 6 tenants (~30 min, well-trodden pattern from prior session).
|
|
3. **Actually apply Approach C** on marcelle as a real version bump (e.g., v2.10.2 → v2.10.3 after tagging+building). Verify nginx config change doesn't break public site.
|
|
4. **Document the operator decision tree**: when to use A vs B vs C.
|
|
|
|
All three upgrade approaches are now in production code. The remaining work is mostly closing the install.sh-tenant gap and operator-experience polish.
|
|
|
|
---
|
|
|
|
## Session continuation — env-patch + fleet rollout + Phase 6 status
|
|
|
|
After the initial Approach C commits, this session also closed the env-patch gap and rolled the new agent out to the whole fleet.
|
|
|
|
### Closed gap: env-patch for install.sh tenants
|
|
|
|
Commit `bf997e8`: install.sh tenants (`isRegistered=true`, no `encryptedSecrets`) couldn't have their .env's `IMAGE_TAG` updated through Approach C (CCP filters out .env render, tenant keeps existing). Added:
|
|
|
|
- Agent: `POST /instance/:slug/env/patch { vars: { KEY: value } }` — in-place .env key patcher in `file.service.ts:patchEnv()`. Preserves comments and key order; appends unknown keys under a "Added by CCP env-patch" comment.
|
|
- CCP: `ExecutionDriver.patchEnv()` + `RemoteDriver.patchEnv()` + `LocalDriver.patchEnv()` (mirrors the agent helper).
|
|
- `runReleaseUpgrade`: for isRegistered tenants with newImageTag, calls `driver.patchEnv({ IMAGE_TAG: newImageTag })` between writeFiles and composePull. Non-fatal on failure.
|
|
|
|
### Fleet rollout: new ccp-agent on all 7 tenants
|
|
|
|
All 7 ccp-agents now expose `/files/diff` + `/env/patch`. Preview endpoint returns 200 on every tenant.
|
|
|
|
Discovery during rollout: source-installed tenants (soroush, trbh, pridecorner, bnkops) `build:` ccp-agent from local source rather than pulling registry image. So `docker save | docker load` is wasted on them — they need source files updated + local build. Rollout procedure split:
|
|
|
|
- Release/release-converted (marcelle, linda, pia): `docker save | docker load` then `up -d --force-recreate ccp-agent`.
|
|
- Source (bnkops, soroush, trbh, pridecorner): `git checkout origin/main -- changemaker-control-panel/agent/src/...` then `docker compose --profile ccp-agent build ccp-agent && up -d --force-recreate`.
|
|
|
|
### Phase 6 status
|
|
|
|
**Code paths all validated via preview** (preview exercises every layer that apply uses, just without the writeFiles+composePull+composeUp side effects). The new `runReleaseUpgrade` runner has been deployed in `ccp-api` on n4 and is reachable via the UI.
|
|
|
|
**Apply NOT triggered on a tenant.** Preview against marcelle revealed substantial nginx/configs template drift that would significantly alter live files:
|
|
|
|
| file | before | after |
|
|
|---|---|---|
|
|
| nginx/conf.d/default.conf | 5296 B | 15695 B |
|
|
| nginx/conf.d/api.conf | 1996 B | 84 B |
|
|
| nginx/conf.d/services.conf | 26133 B | 9434 B |
|
|
| configs/pangolin/resources.yml | 3252 B | 1653 B |
|
|
| configs/prometheus/prometheus.yml | 1406 B | 644 B |
|
|
|
|
These are CCP-templated files that were designed for CCP-provisioned tenants where CCP is authoritative. For install.sh tenants the install.sh-provisioned content differs. Applying would substantially rewrite marcelle's nginx config and risk breaking its public site.
|
|
|
|
**Recommended next session: do for nginx/configs templates what Phase 0 did for docker-compose.yml.hbs** — rewrite each templated file to be byte-equivalent to its canonical install.sh-shipped counterpart. Steps:
|
|
1. Diff each of the 5 templated files (`*.hbs`) against the canonical at `changemaker.lite/nginx/conf.d/{default,api,services}.conf.template` and `changemaker.lite/configs/{pangolin,prometheus}/...yml`.
|
|
2. Update each `.hbs` to match canonical structure (likely use the same `envsubst`-style env-var substitution that install.sh tenants run at startup).
|
|
3. Re-render against marcelle/linda/pia and confirm "modified" → "unchanged" for the 5 files.
|
|
|
|
After that, apply on marcelle becomes safe and the E2E test can complete.
|
|
|
|
The Approach C code itself is production-ready; the gating issue is template sync, which is mechanical.
|