bunker-admin 7895ce683e Tonne of debugging - getting ready for the production builds

2026-02-16 10:44:18 -07:00

34 KiB

Raw Blame History

Nginx Reverse Proxy Configuration

Overview

Nginx serves as the central reverse proxy for Changemaker Lite V2, routing traffic to 15+ backend services via subdomain-based routing. It handles SSL termination, security headers, static file serving, and WebSocket upgrades.

Key Responsibilities:

Subdomain Routing: api.cmlite.org, app.cmlite.org, db.cmlite.org, etc.
SSL/TLS Termination: Handles HTTPS certificates (Let's Encrypt, Cloudflare, or Pangolin)
Security Headers: CSP, HSTS, X-Frame-Options, Permissions-Policy
Proxy Pass: Forwards requests to backend Docker containers
Static File Serving: Serves admin GUI production builds + MkDocs site
WebSocket Support: Upgrades connections for n8n, MailHog, MkDocs live reload
Iframe Embedding: CSP policies allow admin to embed services (NocoDB, Gitea, etc.)

Architecture:

Internet → Nginx (:80, :443) → [Docker Internal Network]
                                  ├─ api:4000 (Express)
                                  ├─ media-api:4100 (Fastify)
                                  ├─ admin:3000 (Vite / static)
                                  ├─ nocodb:8080
                                  ├─ listmonk:9000
                                  ├─ gitea:3000
                                  ├─ n8n:5678
                                  ├─ mkdocs:8000
                                  ├─ code-server:8080
                                  ├─ mailhog:8025
                                  ├─ mini-qr:8080
                                  ├─ homepage:3000
                                  ├─ grafana:3000
                                  └─ public-media:80

Architecture

graph LR
    subgraph "External Access"
        USER[User Browser]
        TUNNEL[Pangolin Tunnel]
    end

    subgraph "Nginx Proxy :80, :443"
        NGINX{Nginx<br/>Subdomain Router}
    end

    subgraph "Backend Services (Docker Network)"
        API[api:4000<br/>Express]
        MEDIA[media-api:4100<br/>Fastify]
        ADMIN[admin:3000<br/>Vite]
        NOCODB[nocodb:8080]
        LISTMONK[listmonk:9000]
        GITEA[gitea:3000]
        N8N[n8n:5678]
        MKDOCS[mkdocs:8000]
        CODE[code-server:8080]
    end

    USER -->|HTTP/HTTPS| NGINX
    TUNNEL -->|HTTP| NGINX

    NGINX -->|api.cmlite.org| API
    NGINX -->|api.cmlite.org/media| MEDIA
    NGINX -->|app.cmlite.org| ADMIN
    NGINX -->|db.cmlite.org| NOCODB
    NGINX -->|listmonk.cmlite.org| LISTMONK
    NGINX -->|git.cmlite.org| GITEA
    NGINX -->|n8n.cmlite.org| N8N
    NGINX -->|docs.cmlite.org| MKDOCS
    NGINX -->|code.cmlite.org| CODE

Configuration Files

Nginx configuration split across multiple files:

File	Purpose	Type
`nginx/nginx.conf`	Global settings, gzip, security headers	Main config
`nginx/conf.d/default.conf`	Localhost fallback, path-based routing	Server block
`nginx/conf.d/api.conf`	API subdomain routing (Express + Fastify)	Server block
`nginx/conf.d/services.conf`	Supporting service subdomains	Server blocks (12+)

Configuration hierarchy:

nginx.conf
├─ Global: worker_processes, events, http
├─ Security headers (applied to all)
├─ Gzip compression
├─ Docker DNS resolver (127.0.0.11)
└─ Include conf.d/*.conf
    ├─ default.conf (localhost)
    ├─ api.conf (api.cmlite.org)
    └─ services.conf (all other subdomains)

Global Configuration (nginx.conf)

File: nginx/nginx.conf

Worker Configuration

worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

Explanation:

worker_processes auto: Detects CPU cores (1 worker per core)
worker_connections 1024: Max 1024 concurrent connections per worker
Total capacity: auto × 1024 (e.g., 4 cores = 4096 connections)

HTTP Block

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /var/log/nginx/access.log main;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    client_max_body_size 50m;  # Default max upload size

    # Include server blocks
    include /etc/nginx/conf.d/*.conf;
}

Key Settings:

sendfile on: Optimized file serving (kernel-level copy)
tcp_nopush on: Sends HTTP headers in single packet
client_max_body_size 50m: Default upload limit (overridden per location)

Gzip Compression

# Gzip compression
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css application/json application/javascript
           text/xml application/xml application/xml+rss text/javascript
           image/svg+xml;

Performance Impact:

CPU usage: Level 6 provides 80% compression with moderate CPU cost
Bandwidth savings: ~60-80% reduction for text/JSON responses
Excluded: Images, video (already compressed)

Security Headers

# Security headers (applied globally)
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header Permissions-Policy "geolocation=(self), microphone=(), camera=()" always;

Header Explanation:

X-Content-Type-Options: Prevents MIME sniffing attacks
X-XSS-Protection: Enables browser XSS filter (legacy browsers)
Referrer-Policy: Controls referer header sent to external sites
HSTS: Forces HTTPS for 1 year (31536000 seconds)
Permissions-Policy: Restricts geolocation/media access

Note: X-Frame-Options set per server block (not global).

Docker DNS Resolver

# Docker internal DNS — enables runtime resolution
resolver 127.0.0.11 valid=30s;

Purpose: Docker's embedded DNS server at 127.0.0.11 resolves container names.

Why needed: Allows Nginx to start even when optional services are down. Without this, Nginx fails to start if any upstream is missing.

Usage pattern:

location / {
    set $upstream_api http://changemaker-v2-api:4000;
    proxy_pass $upstream_api;  # Resolves at request time, not config parse
}

Alternative (fails if container missing):

proxy_pass http://changemaker-v2-api:4000;  # Resolved at config parse — fails if down

Subdomain Routing

Default Server (localhost)

File: nginx/conf.d/default.conf

server {
    listen 80 default_server;
    server_name localhost _;
    add_header X-Frame-Options "SAMEORIGIN" always;

    # Admin GUI (default)
    location / {
        set $upstream_admin http://changemaker-v2-admin:3000;
        proxy_pass $upstream_admin;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    # Media API (must come BEFORE /api/ for longest prefix match)
    location /api/media/ {
        set $upstream_media http://changemaker-media-api:4100;
        proxy_pass $upstream_media;
        # ... (proxy headers)

        # Large upload support
        client_max_body_size 10G;
        proxy_read_timeout 3600s;
        proxy_connect_timeout 75s;
        proxy_request_buffering off;
    }

    # API (Express)
    location /api/ {
        set $upstream_api http://changemaker-v2-api:4000;
        proxy_pass $upstream_api;
        # ... (proxy headers)
    }

    # Public Media Gallery
    location /gallery/ {
        proxy_pass http://changemaker-public-media:80/;
        # ... (proxy headers)
    }
}

Routing Logic:

Request to http://localhost/api/media/videos → media-api:4100
Request to http://localhost/api/campaigns → api:4000
Request to http://localhost/ → admin:3000
Request to http://localhost/gallery/ → public-media:80

Important: /api/media/ location must come before /api/ in config file (longest prefix match).

API Subdomain (api.cmlite.org)

File: nginx/conf.d/api.conf

server {
    listen 80;
    server_name api.cmlite.org;
    add_header X-Frame-Options "SAMEORIGIN" always;

    # Media API endpoints (must come BEFORE / for longest prefix match)
    location /media/ {
        set $upstream_media http://changemaker-media-api:4100/api/;
        proxy_pass $upstream_media;
        # ... (proxy headers)

        # Large upload support
        client_max_body_size 10G;
        proxy_read_timeout 3600s;
        proxy_connect_timeout 75s;
        proxy_request_buffering off;

        # WebSocket support
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    # Main API (Express)
    location / {
        set $upstream_api http://changemaker-v2-api:4000;
        proxy_pass $upstream_api;
        # ... (proxy headers)
        proxy_read_timeout 300s;
        proxy_connect_timeout 75s;
    }
}

URL Mapping:

http://api.cmlite.org/media/videos → http://changemaker-media-api:4100/api/videos
http://api.cmlite.org/auth/login → http://changemaker-v2-api:4000/auth/login

Critical: Media API location includes /api/ in proxy_pass to rewrite path.

Service Subdomains

File: nginx/conf.d/services.conf

Gitea (git.cmlite.org)

server {
    listen 80;
    server_name git.cmlite.org;
    add_header Content-Security-Policy "frame-ancestors 'self' app.cmlite.org" always;

    # Increase max body size for large git pushes (2GB)
    client_max_body_size 2048M;

    location / {
        set $upstream_gitea http://gitea-changemaker:3000;
        proxy_pass $upstream_gitea;
        proxy_hide_header X-Frame-Options;  # Allow iframe embedding
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Key Features:

CSP frame-ancestors: Allows embedding in app.cmlite.org (admin GUI)
proxy_hide_header X-Frame-Options: Strips Gitea's default DENY policy
2GB upload limit: For large repository pushes

n8n (n8n.cmlite.org)

server {
    listen 80;
    server_name n8n.cmlite.org;
    add_header Content-Security-Policy "frame-ancestors 'self' app.cmlite.org" always;

    location / {
        set $upstream_n8n http://n8n-changemaker:5678;
        proxy_pass $upstream_n8n;
        proxy_hide_header X-Frame-Options;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";  # WebSocket support
    }
}

WebSocket Headers:

Upgrade: $http_upgrade: Passes WebSocket upgrade header
Connection: "upgrade": Indicates protocol upgrade

Required for: n8n workflow editor, MailHog live updates, MkDocs live reload

NocoDB (db.cmlite.org)

server {
    listen 80;
    server_name db.cmlite.org;
    add_header Content-Security-Policy "frame-ancestors 'self' app.cmlite.org" always;

    location / {
        set $upstream_nocodb http://changemaker-v2-nocodb:8080;
        proxy_pass $upstream_nocodb;
        proxy_hide_header X-Frame-Options;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Iframe Embedding:

frame-ancestors 'self' app.cmlite.org: Allows admin GUI to embed NocoDB
proxy_hide_header X-Frame-Options: Removes NocoDB's default SAMEORIGIN policy

MkDocs (docs.cmlite.org)

server {
    listen 80;
    server_name docs.cmlite.org;
    add_header Content-Security-Policy "frame-ancestors 'self' app.cmlite.org" always;

    location / {
        set $upstream_mkdocs http://mkdocs-changemaker:8000;
        proxy_pass $upstream_mkdocs;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";  # Live reload WebSocket
    }
}

Live Reload: MkDocs Material theme uses WebSocket for live reload during development.

Code Server (code.cmlite.org)

server {
    listen 80;
    server_name code.cmlite.org;
    add_header Content-Security-Policy "frame-ancestors 'self' app.cmlite.org" always;

    location / {
        set $upstream_code http://code-server-changemaker:8080;
        proxy_pass $upstream_code;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";  # VS Code WebSocket
    }
}

WebSocket Usage: Code Server uses WebSockets for terminal, file watching, language server.

MailHog (mail.cmlite.org)

server {
    listen 80;
    server_name mail.cmlite.org;
    add_header Content-Security-Policy "frame-ancestors 'self' app.cmlite.org" always;

    location / {
        set $upstream_mailhog http://mailhog-changemaker:8025;
        proxy_pass $upstream_mailhog;
        proxy_hide_header X-Frame-Options;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # WebSocket support for MailHog live updates
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Live Updates: MailHog uses WebSocket to push new emails to browser without polling.

Listmonk (listmonk.cmlite.org)

server {
    listen 80;
    server_name listmonk.cmlite.org;
    add_header X-Frame-Options "SAMEORIGIN" always;

    location / {
        set $upstream_listmonk http://listmonk-app:9000;
        proxy_pass $upstream_listmonk;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

No Iframe: Listmonk not embedded in admin (accessed directly), so SAMEORIGIN policy kept.

Grafana (grafana.cmlite.org)

server {
    listen 80;
    server_name grafana.cmlite.org;
    add_header X-Frame-Options "SAMEORIGIN" always;

    location / {
        set $upstream_grafana http://grafana-changemaker:3000;
        proxy_pass $upstream_grafana;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";  # Grafana live updates
    }
}

WebSocket: Grafana uses WebSocket for live dashboard updates.

Mini QR (qr.cmlite.org)

server {
    listen 80;
    server_name qr.cmlite.org;
    add_header Content-Security-Policy "frame-ancestors 'self' app.cmlite.org" always;

    location / {
        set $upstream_miniqr http://mini-qr:8080;
        proxy_pass $upstream_miniqr;
        proxy_hide_header X-Frame-Options;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Iframe Embedding: Admin GUI embeds Mini QR for walk sheet previews.

Root Domain (cmlite.org)

server {
    listen 80;
    server_name cmlite.org;

    location / {
        set $upstream_site http://mkdocs-site-server-changemaker:80;
        proxy_pass $upstream_site;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Purpose: Serves MkDocs static site (production build) on root domain.

Homepage (home.cmlite.org)

server {
    listen 80;
    server_name home.cmlite.org;
    add_header X-Frame-Options "SAMEORIGIN" always;

    location / {
        set $upstream_homepage http://homepage-changemaker:3000;
        proxy_pass $upstream_homepage;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Dashboard: Service status dashboard with Docker integration.

Embed Proxy Ports

Purpose: Allow admin GUI to iframe services via localhost ports (bypassing subdomain requirements).

Ports: 8881-8885 (NocoDB, n8n, Gitea, MailHog, Mini QR)

Configuration (in services.conf):

# NocoDB embed proxy (port 8881)
server {
    listen 8881;
    location / {
        set $upstream_nocodb http://changemaker-v2-nocodb:8080;
        proxy_pass $upstream_nocodb;
        proxy_hide_header X-Frame-Options;
        proxy_hide_header Content-Security-Policy;  # Strip all frame restrictions
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# n8n embed proxy (port 8882)
server {
    listen 8882;
    location / {
        set $upstream_n8n http://n8n-changemaker:5678;
        proxy_pass $upstream_n8n;
        proxy_hide_header X-Frame-Options;
        proxy_hide_header Content-Security-Policy;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

# Gitea embed proxy (port 8883)
server {
    listen 8883;
    client_max_body_size 2048M;  # Large git pushes
    location / {
        set $upstream_gitea http://gitea-changemaker:3000;
        proxy_pass $upstream_gitea;
        proxy_hide_header X-Frame-Options;
        proxy_hide_header Content-Security-Policy;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# MailHog embed proxy (port 8884)
server {
    listen 8884;
    location / {
        set $upstream_mailhog http://mailhog-changemaker:8025;
        proxy_pass $upstream_mailhog;
        proxy_hide_header X-Frame-Options;
        proxy_hide_header Content-Security-Policy;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

# Mini QR embed proxy (port 8885)
server {
    listen 8885;
    location / {
        set $upstream_miniqr http://mini-qr:8080;
        proxy_pass $upstream_miniqr;
        proxy_hide_header X-Frame-Options;
        proxy_hide_header Content-Security-Policy;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Usage in Admin GUI:

<iframe src="http://localhost:8881" />  {/* NocoDB */}
<iframe src="http://localhost:8882" />  {/* n8n */}
<iframe src="http://localhost:8883" />  {/* Gitea */}
<iframe src="http://localhost:8884" />  {/* MailHog */}
<iframe src="http://localhost:8885" />  {/* Mini QR */}

Exposed in docker-compose.yml:

nginx:
  ports:
    - "80:80"
    - "443:443"
    - "8881:8881"  # NocoDB
    - "8882:8882"  # n8n
    - "8883:8883"  # Gitea
    - "8884:8884"  # MailHog
    - "8885:8885"  # Mini QR

Proxy Configuration

Standard Proxy Headers

All proxy locations should include:

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

Header Explanation:

Host: Preserves original hostname (e.g., api.cmlite.org)
X-Real-IP: Client's IP address
X-Forwarded-For: Chain of proxy IPs (adds to existing list)
X-Forwarded-Proto: HTTP or HTTPS (used by backend for redirect logic)

WebSocket Upgrade

Required for: n8n, MailHog, MkDocs, Code Server, Grafana

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

Explanation:

Upgrade: websocket: Browser requests protocol upgrade
Connection: upgrade: Indicates connection will persist

Without these headers: WebSocket connections fail with 400 Bad Request.

Timeouts

Default timeouts:

proxy_read_timeout 300s;     # 5 minutes
proxy_connect_timeout 75s;   # 75 seconds

Media API timeouts (video uploads):

proxy_read_timeout 3600s;    # 1 hour
proxy_connect_timeout 75s;

Why longer: FFprobe video analysis + large file uploads take time.

Upload Size Limits

Global default (nginx.conf):

client_max_body_size 50m;

Per-location overrides:

Media API: client_max_body_size 10G; (video uploads)
Gitea: client_max_body_size 2048M; (large git pushes)

Request Buffering

Media API (disable buffering for streaming uploads):

proxy_request_buffering off;

Effect: Nginx streams request body directly to backend (no temp file).

Benefits:

Lower disk I/O on Nginx server
Faster upload start time
Reduced memory usage

Trade-off: Backend must handle slow clients (Fastify multipart does this).

SSL/TLS Configuration

Certificate Paths

Recommended structure:

/etc/letsencrypt/live/cmlite.org/
├─ fullchain.pem  (certificate + intermediate)
├─ privkey.pem    (private key)
└─ chain.pem      (intermediate CA)

Nginx SSL block:

server {
    listen 443 ssl http2;
    server_name api.cmlite.org;

    ssl_certificate /etc/letsencrypt/live/cmlite.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/cmlite.org/privkey.pem;

    # Strong TLS configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;

    # ... location blocks
}

HTTP to HTTPS Redirect

server {
    listen 80;
    server_name api.cmlite.org;

    # Redirect all HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name api.cmlite.org;
    # ... SSL config + locations
}

HSTS Header

Already applied globally (in nginx.conf):

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

Effect: Browser caches HTTPS requirement for 1 year.

Important: Only enable after verifying HTTPS works (can't easily undo).

Wildcard Certificates

For *.cmlite.org (Let's Encrypt DNS challenge):

certbot certonly --dns-cloudflare \
  --dns-cloudflare-credentials ~/.secrets/cloudflare.ini \
  -d cmlite.org -d "*.cmlite.org"

Single cert covers all subdomains:

api.cmlite.org
app.cmlite.org
db.cmlite.org
etc.

See SSL/TLS for complete certificate management.

Static File Serving

Admin GUI Production Build

Dockerfile multi-stage build (admin/Dockerfile):

# Build stage
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Production stage
FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/nginx.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Nginx serves static files (no Node.js in production):

server {
    listen 80;
    server_name app.cmlite.org;

    root /usr/share/nginx/html;
    index index.html;

    # React Router support (all routes → index.html)
    location / {
        try_files $uri $uri/ /index.html;
    }

    # API proxy
    location /api/ {
        proxy_pass http://changemaker-v2-api:4000;
        # ... proxy headers
    }

    # Cache static assets
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
}

MkDocs Static Site

Build process (via admin GUI or CLI):

docker compose exec mkdocs mkdocs build

Output: mkdocs/site/ directory with static HTML

Served by mkdocs-site-server (Nginx Alpine container):

mkdocs-site-server:
  image: lscr.io/linuxserver/nginx:latest
  volumes:
    - ./mkdocs/site:/config/www
  ports:
    - "4004:80"

Nginx config (in configs/mkdocs-site/default.conf):

server {
    listen 80;
    root /config/www;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

Performance Optimization

Gzip Compression

Already enabled globally (see nginx.conf above).

Compression ratio:

JSON responses: ~75% reduction
HTML/CSS/JS: ~60-70% reduction
Images/video: No compression (already compressed)

Trade-off: Slight CPU increase (~5-10%) for bandwidth savings.

Caching Static Assets

location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
    expires 1y;
    add_header Cache-Control "public, immutable";
}

Effect: Browsers cache static assets for 1 year.

Caveat: Use content hashing in filenames (Vite does this automatically).

Proxy Caching

Optional (not enabled by default):

# In http block
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:10m max_size=1g inactive=60m;

# In location block
location /api/campaigns {
    proxy_cache api_cache;
    proxy_cache_valid 200 10m;
    proxy_cache_key "$scheme$request_method$host$request_uri";
    proxy_pass http://changemaker-v2-api:4000;
}

Use cases:

Public campaign listing (10-minute cache)
Public map data (5-minute cache)
Representative lookup (1-hour cache)

Avoid caching:

Authenticated endpoints
POST/PUT/DELETE requests
Real-time data (canvass sessions, email queue)

Connection Pooling

Keep-alive to backends:

upstream api {
    server changemaker-v2-api:4000;
    keepalive 32;  # Maintain 32 idle connections
}

location /api/ {
    proxy_pass http://api;
    proxy_http_version 1.1;
    proxy_set_header Connection "";  # Clear close header
}

Benefits:

Reduced latency (no TCP handshake)
Lower CPU (fewer connection setups)
Better throughput under load

Troubleshooting

502 Bad Gateway

Symptoms: 502 Bad Gateway error

Causes:

Backend container not running
Backend healthcheck failing
Backend listening on wrong port
Network connectivity issue

Diagnosis:

# Check backend status
docker compose ps api

# Check backend logs
docker compose logs --tail=50 api

# Test backend directly
docker compose exec nginx curl http://changemaker-v2-api:4000/api/health

# Check Nginx error log
docker compose exec nginx cat /var/log/nginx/error.log

Solution:

# Restart backend
docker compose restart api

# Check healthcheck
docker inspect changemaker-v2-api | jq '.[0].State.Health'

# Verify port in docker-compose.yml
grep -A5 "api:" docker-compose.yml

504 Gateway Timeout

Symptoms: Request times out after 60 seconds

Cause: Backend processing too slow, proxy timeout too short

Solution:

# Increase timeout for slow endpoints
location /api/locations/geocode {
    proxy_read_timeout 300s;  # 5 minutes
    proxy_pass http://changemaker-v2-api:4000;
}

SSL Certificate Errors

Symptoms: SSL_ERROR_RX_RECORD_TOO_LONG or ERR_SSL_PROTOCOL_ERROR

Cause: Accessing HTTPS port via HTTP or vice versa

Diagnosis:

# Test HTTPS
curl -I https://api.cmlite.org

# Check certificate
openssl s_client -connect api.cmlite.org:443 -servername api.cmlite.org

# Verify Nginx config
docker compose exec nginx nginx -t

Solution:

# Reload Nginx after cert renewal
docker compose exec nginx nginx -s reload

# Check cert paths in config
grep ssl_certificate /path/to/nginx/conf.d/*.conf

CORS Errors

Symptoms: Browser console shows CORS policy: No 'Access-Control-Allow-Origin' header

Cause: Backend not setting CORS headers

Diagnosis:

# Test from browser console
fetch('http://api.cmlite.org/api/campaigns')

# Check response headers
curl -H "Origin: http://example.com" -I http://api.cmlite.org/api/campaigns

Solution: CORS headers set by backend (not Nginx). Check api/src/server.ts:

app.use(cors({
  origin: process.env.CORS_ORIGINS?.split(',') || ['http://localhost:3000'],
  credentials: true,
}));

Nginx passthrough (don't modify CORS headers):

# DO NOT add these in Nginx (backend handles CORS)
# add_header Access-Control-Allow-Origin "*";  # ❌ WRONG

WebSocket Connection Failures

Symptoms: WebSocket upgrade fails with 400 Bad Request

Cause: Missing Upgrade/Connection headers

Diagnosis:

# Check Nginx config
grep -A5 "Upgrade" nginx/conf.d/services.conf

# Test WebSocket
wscat -c ws://localhost:5678

Solution:

# Add to location block
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

Large Upload Failures

Symptoms: Upload fails with 413 Request Entity Too Large

Cause: client_max_body_size too small

Solution:

# Increase limit for specific location
location /api/media/videos {
    client_max_body_size 10G;
    proxy_pass http://changemaker-media-api:4100;
}

Iframe Not Displaying

Symptoms: Service loads in new tab but not in iframe

Cause: X-Frame-Options: DENY or CSP frame-ancestors blocking

Diagnosis:

# Check response headers
curl -I http://db.cmlite.org

# Look for X-Frame-Options or Content-Security-Policy

Solution:

# Hide backend's X-Frame-Options
proxy_hide_header X-Frame-Options;

# Add CSP allowing admin
add_header Content-Security-Policy "frame-ancestors 'self' app.cmlite.org" always;

Nginx Won't Start

Symptoms: docker compose up fails with Nginx error

Diagnosis:

# Test config syntax
docker compose run --rm nginx nginx -t

# Check for duplicate server_name
grep server_name nginx/conf.d/*.conf | sort

# Check for port conflicts
docker compose config | grep -A2 "ports:"

Common mistakes:

Missing semicolon
Duplicate server_name (same subdomain in multiple files)
Invalid regex in location
Unclosed { bracket

Production Best Practices

Rate Limiting

Limit requests per IP (prevents abuse):

# In http block
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;

# In location block
location /api/ {
    limit_req zone=api_limit burst=20 nodelay;
    proxy_pass http://changemaker-v2-api:4000;
}

Explanation:

rate=10r/s: 10 requests per second average
burst=20: Allow bursts up to 20 requests
nodelay: Process burst immediately (don't queue)

Security Headers Review

Production checklist:

HSTS enabled (max-age=31536000)
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
CSP frame-ancestors for embeddable services
X-Frame-Options: SAMEORIGIN for non-embedded services
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy restricts sensors

Optional enhancements:

# Stricter CSP
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline';" always;

# Expect-CT (certificate transparency)
add_header Expect-CT "max-age=86400, enforce" always;

Access Logging

Production log format (JSON for parsing):

log_format json_combined escape=json
  '{'
    '"time_local":"$time_local",'
    '"remote_addr":"$remote_addr",'
    '"request":"$request",'
    '"status": $status,'
    '"body_bytes_sent":$body_bytes_sent,'
    '"request_time":$request_time,'
    '"http_referrer":"$http_referer",'
    '"http_user_agent":"$http_user_agent"'
  '}';

access_log /var/log/nginx/access.log json_combined;

Benefits: Easy parsing with tools like jq, Logstash, Loki.

Error Page Customization

Custom error pages:

error_page 404 /404.html;
error_page 500 502 503 504 /50x.html;

location = /404.html {
    root /usr/share/nginx/html;
    internal;
}

location = /50x.html {
    root /usr/share/nginx/html;
    internal;
}

Create files:

cat > nginx/html/404.html <<EOF
<!DOCTYPE html>
<html>
<head><title>404 Not Found</title></head>
<body>
<h1>404 - Page Not Found</h1>
<p>Return to <a href="/">homepage</a></p>
</body>
</html>
EOF

Docker Compose — Container orchestration
Environment Variables — Configuration reference
SSL/TLS — Certificate management
Tunneling — Pangolin tunnel setup
Scaling — Load balancing strategies

34 KiB Raw Blame History Unescape Escape

Nginx Reverse Proxy Configuration

Overview

Architecture

Configuration Files

Global Configuration (nginx.conf)

Worker Configuration

HTTP Block

Gzip Compression

Security Headers

Docker DNS Resolver

Subdomain Routing

Default Server (localhost)

API Subdomain (api.cmlite.org)

Service Subdomains

Gitea (git.cmlite.org)

n8n (n8n.cmlite.org)

NocoDB (db.cmlite.org)

MkDocs (docs.cmlite.org)

Code Server (code.cmlite.org)

MailHog (mail.cmlite.org)

Listmonk (listmonk.cmlite.org)

Grafana (grafana.cmlite.org)

Mini QR (qr.cmlite.org)

Root Domain (cmlite.org)

Homepage (home.cmlite.org)

Embed Proxy Ports

Proxy Configuration

Standard Proxy Headers

WebSocket Upgrade

Timeouts

Upload Size Limits

Request Buffering

SSL/TLS Configuration

Certificate Paths

HTTP to HTTPS Redirect

HSTS Header

Wildcard Certificates

Static File Serving

Admin GUI Production Build

MkDocs Static Site

Performance Optimization

Gzip Compression

Caching Static Assets

Proxy Caching

Connection Pooling

Troubleshooting

502 Bad Gateway

504 Gateway Timeout

SSL Certificate Errors

CORS Errors

WebSocket Connection Failures

Large Upload Failures

Iframe Not Displaying

Nginx Won't Start

Production Best Practices

Rate Limiting

Security Headers Review

Access Logging

Error Page Customization

Related Documentation

34 KiB

Raw Blame History