admin/changemaker.lite

Fork 0

bunker-admin 7895ce683e Tonne of debugging - getting ready for the production builds

2026-02-16 10:44:18 -07:00

35 KiB

Raw Blame History

Docker and Container Issues

This guide covers Docker-specific problems in Changemaker Lite V2.

Overview

Docker Troubleshooting Approach

Check status - Are containers running?
Read logs - What do container logs show?
Inspect configuration - Is docker-compose.yml correct?
Test connectivity - Can containers communicate?
Resource check - Enough CPU/memory/disk?

Essential Docker Commands

# View running containers
docker compose ps

# View all containers (including stopped)
docker compose ps -a

# View logs
docker compose logs [service-name]

# Follow logs in real-time
docker compose logs -f [service-name]

# Execute command in container
docker compose exec [service-name] [command]

# Restart service
docker compose restart [service-name]

# Stop all services
docker compose down

# Start services
docker compose up -d

# Rebuild and start
docker compose up -d --build [service-name]

Container Won't Start

Port Already in Use

Severity: 🔴 Critical

Symptoms

Error response from daemon: driver failed programming external connectivity
on endpoint changemaker-lite-admin-1: Bind for 0.0.0.0:3000 failed:
port is already allocated

Or:

ERROR: for api  Cannot start service api: Ports are not available:
exposing port TCP 0.0.0.0:4000 -> 0.0.0.0:0: listen tcp 0.0.0.0:4000:
bind: address already in use

Common Causes

Another container using port - Different Docker project
Host process using port - npm dev server running
Previous container not stopped - Old container still running
Port conflict in docker-compose.yml - Two services same port

Solutions

Solution 1: Find what's using the port

# Linux/Mac
sudo lsof -i :4000

# Or with netstat
netstat -tuln | grep :4000

# Windows
netstat -ano | findstr :4000

Output shows:

COMMAND   PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
node    12345  user   23u  IPv4 123456      0t0  TCP *:4000 (LISTEN)

Solution 2: Stop conflicting process

# Kill process by PID
kill 12345

# Or kill all node processes (careful!)
killall node

# Or stop other Docker containers
docker ps  # List all running containers
docker stop container-name-or-id

Solution 3: Change port in docker-compose.yml

# In docker-compose.yml
api:
  ports:
    - "4002:4000"  # Changed from 4000:4000

Then:

# Restart with new port
docker compose up -d api

# Update .env to use new port
VITE_API_URL=http://localhost:4002

Solution 4: Stop all and restart

# Stop all Changemaker Lite containers
docker compose down

# Verify nothing running
docker compose ps

# Start fresh
docker compose up -d

Prevention

Use unique ports - Avoid common ports (3000, 4000, 8000, 8080)
Stop properly - Always use docker compose down
Check before start - Run docker compose ps first
Document ports - Keep port reference updated

Volume Mount Errors

Severity: 🔴 Critical

Symptoms

Error response from daemon: invalid mount config for type "bind":
bind source path does not exist: /home/user/changemaker.lite/uploads

Or:

Error: EACCES: permission denied, open '/media/local/inbox/video.mp4'

Common Causes

Path doesn't exist - Directory not created
Permission denied - Container can't access directory
Wrong path - Typo in docker-compose.yml
SELinux blocking - Linux security policy

Solutions

Solution 1: Create missing directories

# Create all required directories
mkdir -p uploads
mkdir -p media/local/inbox
mkdir -p media/local/library
mkdir -p data
mkdir -p configs/prometheus
mkdir -p configs/grafana

# Verify they exist
ls -la

Solution 2: Fix permissions

# Make directories writable
chmod -R 777 uploads
chmod -R 777 media/local/inbox

# Or set ownership to container user
# Check container user ID
docker compose exec api id
# uid=1000(node) gid=1000(node)

# Set ownership
sudo chown -R 1000:1000 uploads
sudo chown -R 1000:1000 media

Solution 3: Check volume configuration

In docker-compose.yml:

api:
  volumes:
    # Correct format:
    - ./uploads:/app/uploads:rw        # Read-write
    - ./media:/media:ro                 # Read-only

    # Wrong formats:
    # - uploads:/app/uploads            # Named volume, not bind mount
    # - /uploads:/app/uploads           # Absolute path on host

Solution 4: Disable SELinux (last resort)

# Check if SELinux is the issue
getenforce
# If "Enforcing":

# Option 1: Add :z flag to volume
# In docker-compose.yml:
    - ./uploads:/app/uploads:z

# Option 2: Temporarily disable (not recommended)
sudo setenforce 0

Solution 5: Verify mount inside container

# Check if mount exists
docker compose exec api ls -la /app/uploads

# Check permissions
docker compose exec api ls -ld /app/uploads

# Try creating file
docker compose exec api touch /app/uploads/test.txt

Prevention

Create directories first - Before docker compose up
Set permissions early - In setup script
Use relative paths - Start with ./ in docker-compose.yml
Document requirements - List all required directories

Missing Environment Variables

Severity: 🔴 Critical

Symptoms

Container logs show:

Error: DATABASE_URL is required

Or:

ZodError: [
  {
    "code": "invalid_type",
    "expected": "string",
    "received": "undefined",
    "path": ["SMTP_HOST"],
    "message": "Required"
  }
]

Or container exits immediately:

changemaker-lite-api-1 exited with code 1

Common Causes

.env not found - Missing .env file
Variable not set - Missing required variable
Wrong .env location - .env not in project root
Syntax error - Malformed .env file

Solutions

Solution 1: Check .env exists

# Verify .env file
ls -la .env

# If missing, copy from example
cp .env.example .env

Solution 2: Find missing variables

# View container logs to see which variable
docker compose logs api | grep -i "required\|undefined"

# Example output:
# Error: SMTP_HOST is required

Solution 3: Add missing variables

# Edit .env
nano .env

# Add missing variable
SMTP_HOST=smtp.gmail.com

# Save and restart
docker compose restart api

Solution 4: Validate .env format

# Check for common issues:
# - No spaces around =
# - Quotes for values with spaces
# - No trailing commas
# - No comments on same line as value

# Good:
DATABASE_URL="postgresql://user:pass@host:5432/db"
CORS_ORIGINS=http://localhost:3000,http://localhost:4000

# Bad:
DATABASE_URL = "postgresql://..."  # Space around =
CORS_ORIGINS=http://localhost:3000, http://localhost:4000  # Space after comma
SMTP_HOST=smtp.gmail.com # Gmail  # Comment on same line

Solution 5: Check which variables are loaded

# View environment inside container
docker compose exec api env | grep -E "DATABASE_URL|SMTP_HOST|JWT_"

# Should show actual values (not undefined)

Prevention

Use .env.example - Keep template updated
Validation on startup - Zod validates env in config/env.ts
Documentation - Document all required variables
Setup script - Validate .env before starting

Health Check Failures

Severity: 🟠 High

Symptoms

docker compose ps

Shows:

NAME                    STATUS
api                     Up 30 seconds (unhealthy)
v2-postgres            Up 1 minute (healthy)

Or logs show:

Health check failed

Common Causes

Service not ready - Still starting up
Health check endpoint failing - /health returns error
Timeout too short - Service needs more time
Dependencies not ready - Database not connected

Solutions

Solution 1: Check health check configuration

In docker-compose.yml:

api:
  healthcheck:
    test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:4000/api/health"]
    interval: 30s
    timeout: 10s
    retries: 3
    start_period: 40s

Solution 2: Test health endpoint manually

# From inside container
docker compose exec api wget -O- http://localhost:4000/api/health

# Should return:
# {"status":"healthy","timestamp":"2026-02-13T..."}

# From host
curl http://localhost:4000/api/health

Solution 3: View health check logs

# Detailed health check output
docker inspect changemaker-lite-api-1 --format='{{json .State.Health}}' | jq

# Shows:
# {
#   "Status": "unhealthy",
#   "FailingStreak": 3,
#   "Log": [
#     {
#       "Start": "2026-02-13T...",
#       "End": "2026-02-13T...",
#       "ExitCode": 1,
#       "Output": "Error: Connection refused"
#     }
#   ]
# }

Solution 4: Increase timeout/interval

api:
  healthcheck:
    interval: 60s      # Check less frequently
    timeout: 30s       # Allow more time
    start_period: 90s  # Wait longer before first check

Solution 5: Check service logs

# Real issue is usually in service logs
docker compose logs api | tail -50

# Common issues:
# - Database connection failed
# - Missing environment variable
# - Port already in use

Prevention

Reasonable timeouts - Allow enough time for startup
Accurate health checks - Check actual readiness
Monitor health - Alert on unhealthy containers
Dependencies - Use depends_on with condition: service_healthy

Container Crashes

Out of Memory

Severity: 🔴 Critical

Symptoms

Container logs show:

<--- Last few GCs --->
[1:0x5588e4f8e000]    65432 ms: Mark-sweep 2048.0 (2048.4) -> 2047.9 (2048.4) MB, 1845.2 / 0.0 ms  (average mu = 0.123, current mu = 0.001) allocation failure scavenge might not succeed

<--- JS stacktrace --->
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Or:

Killed

Or docker compose ps shows:

api   Exit 137

Common Causes

Memory leak - Application leaking memory
Large dataset - Processing too much data
Too many connections - Database connection pool too large
Container limit - Memory limit too low

Solutions

Solution 1: Check memory usage

# View container memory usage
docker stats

# Shows:
# CONTAINER           CPU %    MEM USAGE / LIMIT     MEM %
# api                 15.5%    1.2GiB / 2GiB        60%

Solution 2: Increase Node.js heap size

In docker-compose.yml:

api:
  environment:
    - NODE_OPTIONS=--max-old-space-size=4096  # 4GB heap

Or in api/package.json:

{
  "scripts": {
    "start": "node --max-old-space-size=4096 dist/server.js"
  }
}

Solution 3: Increase container memory limit

api:
  deploy:
    resources:
      limits:
        memory: 4G  # Increase from 2G
      reservations:
        memory: 2G

Solution 4: Find memory leak

# Enable heap snapshots
docker compose exec api node --inspect dist/server.js

# Or use clinic.js
npm install -g clinic
clinic doctor -- node dist/server.js

Solution 5: Reduce memory usage

// Reduce database connection pool
// In prisma/schema.prisma
datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
  // Add connection limit
}

// In DATABASE_URL:
DATABASE_URL="postgresql://...?connection_limit=5"

// Process data in batches
const users = await prisma.user.findMany({
  take: 100,  // Limit batch size
  skip: offset
});

Prevention

Monitor memory - Alert on high usage
Generous limits - Set limits higher than expected usage
Memory profiling - Regular memory audits
Optimize queries - Reduce data fetched

Application Errors

Severity: 🔴 Critical

Symptoms

Container exits immediately:

api-1 exited with code 1

Logs show:

Error: Cannot find module 'express'

Or:

SyntaxError: Unexpected token 'export'

Common Causes

Missing dependencies - npm install not run
Build not run - TypeScript not compiled
Syntax error - Code has errors
Wrong Node version - Incompatible Node.js version

Solutions

Solution 1: Rebuild container

# Rebuild with no cache
docker compose build --no-cache api

# Start
docker compose up -d api

# View logs
docker compose logs -f api

Solution 2: Check dependencies

# Verify package.json and package-lock.json exist
docker compose exec api ls -la package*.json

# Verify node_modules exists
docker compose exec api ls -la node_modules | head

# If missing, install
docker compose exec api npm install

Solution 3: Verify build

# Check if TypeScript compiled
docker compose exec api ls -la dist/

# If missing, build
docker compose exec api npm run build

# Or rebuild container
docker compose up -d --build api

Solution 4: Check Node version

# Check version in container
docker compose exec api node --version

# Should match Dockerfile
cat api/Dockerfile | grep "FROM node:"

# Example:
# FROM node:20-alpine

Solution 5: Test locally

# Test build locally
cd api
npm install
npm run build
npm start

# If works locally but not in Docker, check:
# - Dockerfile COPY commands
# - .dockerignore file
# - Volume mounts

Prevention

Multi-stage builds - Separate build and runtime
Lock files - Commit package-lock.json
CI/CD - Automated build testing
Version pinning - Pin Node.js version

Database Connection Failures

Severity: 🔴 Critical

Symptoms

API logs show:

Error: Can't reach database server at `v2-postgres:5432`
Error: connect ECONNREFUSED 172.18.0.2:5432

Container restarts repeatedly.

Common Causes

Database not ready - API started before database
Wrong host - Incorrect database hostname
Network issue - Containers on different networks
Database crashed - PostgreSQL container down

Solutions

Solution 1: Check database status

# Is database running?
docker compose ps v2-postgres

# Should show "Up" status
# If not:
docker compose up -d v2-postgres

# Check logs
docker compose logs v2-postgres | tail -50

Solution 2: Verify DATABASE_URL

# Check .env
cat .env | grep DATABASE_URL

# From API container, should use container name:
DATABASE_URL="postgresql://changemaker:password@v2-postgres:5432/changemaker_v2"

# From host, use localhost:
DATABASE_URL="postgresql://changemaker:password@localhost:5433/changemaker_v2"

Solution 3: Test database connection

# From API container
docker compose exec api sh -c 'psql $DATABASE_URL -c "SELECT NOW();"'

# Should return current timestamp
# If fails, database connection is broken

Solution 4: Check Docker network

# List networks
docker network ls

# Inspect changemaker-lite network
docker network inspect changemaker-lite

# All containers should be on same network

Solution 5: Use depends_on with health check

In docker-compose.yml:

api:
  depends_on:
    v2-postgres:
      condition: service_healthy
  # ...

v2-postgres:
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U changemaker"]
    interval: 10s
    timeout: 5s
    retries: 5

Prevention

Health checks - Wait for database to be ready
Retry logic - Retry connection on startup
Connection pooling - Handle connection failures gracefully
Monitoring - Alert on connection failures

Networking Issues

Containers Can't Communicate

Severity: 🔴 Critical

Symptoms

Error: getaddrinfo ENOTFOUND v2-postgres

Or:

Error: connect EHOSTUNREACH 172.18.0.2:5432

Containers can't ping each other.

Common Causes

Different networks - Containers on separate Docker networks
Wrong hostname - Using IP instead of container name
Firewall - Host firewall blocking
DNS issue - Docker DNS not working

Solutions

Solution 1: Verify same network

# Check container networks
docker inspect changemaker-lite-api-1 | grep NetworkMode
docker inspect changemaker-lite-v2-postgres-1 | grep NetworkMode

# Should both show "changemaker-lite"

Solution 2: Use container names

# Correct - use service names
api:
  environment:
    - DATABASE_URL=postgresql://user:pass@v2-postgres:5432/db

# Wrong - using IPs
api:
  environment:
    - DATABASE_URL=postgresql://user:pass@172.18.0.2:5432/db

Solution 3: Test connectivity

# Ping from one container to another
docker compose exec api ping v2-postgres

# DNS lookup
docker compose exec api nslookup v2-postgres

# Telnet to port
docker compose exec api telnet v2-postgres 5432

Solution 4: Recreate network

# Stop all containers
docker compose down

# Remove network
docker network rm changemaker-lite

# Start fresh (network auto-created)
docker compose up -d

Solution 5: Check firewall

# Temporarily disable firewall (Linux)
sudo ufw disable

# Test if containers can communicate
# If yes, firewall is blocking

# Re-enable and add rules
sudo ufw enable
sudo ufw allow from 172.18.0.0/16 to any

Prevention

Use service names - Never hardcode IPs
Single network - All services on same network
Docker DNS - Rely on Docker's built-in DNS
Health checks - Verify connectivity on startup

Port Not Accessible from Host

Severity: 🟠 High

Symptoms

From host:

curl http://localhost:4000/api/health
# curl: (7) Failed to connect to localhost port 4000: Connection refused

But from inside container:

docker compose exec api curl http://localhost:4000/api/health
# {"status":"healthy"}

Common Causes

Port not published - Missing ports: in docker-compose.yml
Bound to 127.0.0.1 - Only listening on localhost inside container
Firewall blocking - Host firewall blocking port
Wrong port - Trying different port than published

Solutions

Solution 1: Check port publishing

In docker-compose.yml:

api:
  ports:
    - "4000:4000"  # host:container

Verify:

docker compose ps api

# Should show:
# PORTS: 0.0.0.0:4000->4000/tcp

Solution 2: Bind to 0.0.0.0

In api/src/server.ts:

// Wrong - only localhost
app.listen(4000, '127.0.0.1');

// Right - all interfaces
app.listen(4000, '0.0.0.0');

// Or just
app.listen(4000);  // Defaults to 0.0.0.0

Solution 3: Check firewall

# Check if port allowed (Linux)
sudo ufw status

# Allow port
sudo ufw allow 4000/tcp

# Or disable temporarily for testing
sudo ufw disable

Solution 4: Verify correct port

# Check what ports are actually listening
docker compose exec api netstat -tuln

# Should show:
# tcp6  0  0  :::4000  :::*  LISTEN

Solution 5: Restart with port forwarding

# Stop container
docker compose stop api

# Remove container
docker compose rm -f api

# Start fresh
docker compose up -d api

# Verify port
curl http://localhost:4000/api/health

Prevention

Always publish ports - In docker-compose.yml
Bind to 0.0.0.0 - Not 127.0.0.1
Test from host - Verify accessibility
Document ports - Keep port reference updated

DNS Resolution Failures

Severity: 🟠 High

Symptoms

Error: getaddrinfo ENOTFOUND smtp.gmail.com

Or:

Error: getaddrinfo EAI_AGAIN api.represent.org

Container can't resolve external hostnames.

Common Causes

Docker DNS issue - Docker DNS not working
No internet - Container has no internet access
Firewall blocking DNS - Port 53 blocked
Wrong DNS servers - Using invalid DNS servers

Solutions

Solution 1: Test DNS resolution

# From inside container
docker compose exec api nslookup google.com

# Should return IP address
# If not, DNS is broken

Solution 2: Check Docker DNS

# View container DNS config
docker compose exec api cat /etc/resolv.conf

# Should show:
# nameserver 127.0.0.11  # Docker's embedded DNS

Solution 3: Use custom DNS servers

In docker-compose.yml:

api:
  dns:
    - 8.8.8.8      # Google DNS
    - 8.8.4.4

Or in /etc/docker/daemon.json:

{
  "dns": ["8.8.8.8", "8.8.4.4"]
}

Then restart Docker:

sudo systemctl restart docker

Solution 4: Check internet connectivity

# Ping external host
docker compose exec api ping -c 3 8.8.8.8

# If fails, no internet access
# Check host internet connection
ping -c 3 8.8.8.8

Solution 5: Restart Docker daemon

# Sometimes Docker DNS gets stuck
sudo systemctl restart docker

# Then restart containers
docker compose down
docker compose up -d

Prevention

Reliable DNS - Use public DNS servers as backup
Monitor connectivity - Alert on DNS failures
Health checks - Include external connectivity checks
Retry logic - Handle transient DNS failures

Volume Issues

Permission Denied

Severity: 🔴 Critical

Symptoms

Error: EACCES: permission denied, open '/app/uploads/image.jpg'

Or:

Error: EACCES: permission denied, mkdir '/media/local/inbox'

File operations fail inside container.

Common Causes

Wrong ownership - Host directory owned by different user
Wrong permissions - Directory not writable
SELinux - Linux security policy blocking
Read-only mount - Volume mounted as read-only

Solutions

Solution 1: Check ownership

# On host
ls -la uploads/

# Shows:
# drwxr-xr-x  2 root   root   4096 Feb 13 10:00 uploads

# Check container user
docker compose exec api id
# uid=1000(node) gid=1000(node)

# Fix ownership
sudo chown -R 1000:1000 uploads/

Solution 2: Fix permissions

# Make writable
chmod -R 755 uploads/

# Or more permissive (dev only)
chmod -R 777 uploads/

Solution 3: Check mount mode

In docker-compose.yml:

api:
  volumes:
    - ./uploads:/app/uploads:rw  # Read-write
    # Not:
    # - ./uploads:/app/uploads:ro  # Read-only

Solution 4: SELinux labels

# Add :z flag to volume
# In docker-compose.yml:
    - ./uploads:/app/uploads:z

# Or relabel directory
sudo chcon -Rt svirt_sandbox_file_t uploads/

Solution 5: Run as root (not recommended)

# In docker-compose.yml (last resort)
api:
  user: "0:0"  # Run as root

Prevention

Set permissions early - In setup script
Match UIDs - Container user matches host user
SELinux-aware - Use :z flag on volumes
Document requirements - List permission requirements

Volume Not Mounted

Severity: 🟠 High

Symptoms

Container can't see files that exist on host.

# On host
ls uploads/
# image.jpg  video.mp4

# In container
docker compose exec api ls /app/uploads/
# (empty)

Common Causes

Wrong path - Volume path incorrect
Typo - Syntax error in docker-compose.yml
Not mounted - Volume mount missing
Cached old config - Using old container

Solutions

Solution 1: Verify volume configuration

In docker-compose.yml:

api:
  volumes:
    - ./uploads:/app/uploads  # host:container

Solution 2: Check mounts in running container

# Inspect container mounts
docker inspect changemaker-lite-api-1 | grep -A 10 Mounts

# Should show:
# "Mounts": [
#   {
#     "Type": "bind",
#     "Source": "/home/user/changemaker.lite/uploads",
#     "Destination": "/app/uploads",
#     "Mode": "",
#     "RW": true,
#     "Propagation": "rprivate"
#   }
# ]

Solution 3: Recreate container

# Stop and remove container
docker compose down api

# Start fresh
docker compose up -d api

# Verify mount
docker compose exec api ls /app/uploads/

Solution 4: Use absolute path

# Sometimes relative paths don't work
api:
  volumes:
    - /home/user/changemaker.lite/uploads:/app/uploads

Solution 5: Check Docker Compose version

# Check version
docker compose version

# Should be v2+
# If v1, syntax might differ

Prevention

Test mounts - Verify after container start
Use relative paths - Start with ./
Documentation - Document all volume mounts
Health checks - Verify critical files exist

Data Persistence Problems

Severity: 🔴 Critical

Symptoms

Data disappears after docker compose down:

Database data lost
Uploaded files missing
Configuration reset

Common Causes

Using containers, not volumes - Data stored in container filesystem
Anonymous volumes - Volume not named or bound
Deleting volumes - docker compose down -v removes volumes
Wrong volume type - tmpfs instead of volume

Solutions

Solution 1: Use named volumes

In docker-compose.yml:

v2-postgres:
  volumes:
    - postgres-data:/var/lib/postgresql/data  # Named volume

volumes:
  postgres-data:  # Declare named volume

Solution 2: Use bind mounts

v2-postgres:
  volumes:
    - ./data/postgres:/var/lib/postgresql/data  # Bind to host directory

Solution 3: Don't use -v flag

# Wrong - deletes volumes
docker compose down -v

# Right - keeps volumes
docker compose down

Solution 4: Check volume exists

# List volumes
docker volume ls

# Should show:
# changemaker-lite_postgres-data

# Inspect volume
docker volume inspect changemaker-lite_postgres-data

Solution 5: Backup before down

# Backup database before stopping
docker compose exec v2-postgres pg_dump -U changemaker changemaker_v2 > backup.sql

# Then safe to:
docker compose down -v

# Restore after up:
docker compose up -d v2-postgres
docker compose exec -T v2-postgres psql -U changemaker changemaker_v2 < backup.sql

Prevention

Named volumes - For all persistent data
Regular backups - Automated backup script
Never use -v - Unless intentionally resetting
Documentation - Document what data persists where

Performance Issues

Slow Container Startup

Severity: 🟡 Medium

Symptoms

Container takes minutes to start:

docker compose up -d api
# Creating api ... (2 minutes)
# Creating api ... done

Common Causes

Large image - Downloading/extracting large image
Many dependencies - npm install taking long
Health check delay - Waiting for health checks
Slow disk - I/O bottleneck

Solutions

Solution 1: Use pre-built image

# Instead of building locally
api:
  build: ./api

# Use pre-built image from registry
api:
  image: ghcr.io/yourorg/changemaker-api:latest

Solution 2: Layer caching

# In Dockerfile, copy package files first
COPY package*.json ./
RUN npm ci

# Then copy code (changes more frequently)
COPY . .
RUN npm run build

Solution 3: Multi-stage builds

# Build stage
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Runtime stage (smaller)
FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY package*.json ./
CMD ["node", "dist/server.js"]

Solution 4: Increase Docker resources

In Docker Desktop settings:

CPU: 4+ cores
Memory: 8GB+
Disk: Fast SSD

Solution 5: Parallel builds

# Build all services in parallel
docker compose build --parallel

Prevention

Optimize Dockerfile - Layer caching, multi-stage
Small base images - Alpine instead of full images
Registry caching - Pull from registry instead of building
Resource allocation - Adequate CPU/memory for Docker

High CPU Usage

Severity: 🟠 High

Symptoms

docker stats
# CONTAINER   CPU %
# api         95%

Container consuming excessive CPU.

Common Causes

Infinite loop - Bug causing tight loop
Heavy computation - Processing large dataset
Too many workers - Worker threads maxed out
Memory thrashing - Swapping due to low memory

Solutions

Solution 1: Identify process

# Top inside container
docker compose exec api top

# Shows process using CPU

Solution 2: Check for loops

# View logs for repeated messages
docker compose logs api | tail -100

# Restart if stuck
docker compose restart api

Solution 3: Limit worker threads

// In BullMQ worker
new Worker('queueName', processor, {
  concurrency: 2,  // Reduce from 10
  limiter: {
    max: 10,
    duration: 1000  // Max 10 jobs per second
  }
});

Solution 4: Set CPU limits

api:
  deploy:
    resources:
      limits:
        cpus: '2.0'  # Max 2 CPUs

Solution 5: Profile application

# Use Node.js profiler
docker compose exec api node --prof dist/server.js

# Or clinic.js
npm install -g clinic
clinic doctor -- node dist/server.js

Prevention

Monitor CPU - Alert on high usage
Rate limiting - Limit request rate
Queue management - Control worker concurrency
Performance testing - Load test regularly

High Memory Usage

Severity: 🟠 High

Symptoms

docker stats
# CONTAINER   MEM USAGE / LIMIT
# api         3.8GiB / 4GiB

Memory usage keeps increasing.

Common Causes

Memory leak - Not releasing memory
Large cache - Caching too much data
Database connections - Too many open connections
Large response bodies - Sending huge payloads

Solutions

Solution 1: Identify memory usage

# Memory breakdown inside container
docker compose exec api sh -c 'cat /proc/meminfo'

# Node.js heap stats
docker compose exec api node -e "console.log(process.memoryUsage())"

Solution 2: Restart to free memory

# Temporary fix
docker compose restart api

# Memory should drop
docker stats api

Solution 3: Reduce cache size

// In Redis cache
redis.set(key, value, 'EX', 3600);  // Expire after 1 hour

// Limit cache size
const cache = new LRU({
  max: 1000,  // Max 1000 entries
  maxAge: 3600000  // 1 hour
});

Solution 4: Set memory limit

api:
  deploy:
    resources:
      limits:
        memory: 2G  # Hard limit
      reservations:
        memory: 1G  # Reserved amount

Solution 5: Find memory leak

# Take heap snapshot
docker compose exec api node --expose-gc --inspect dist/server.js

# Use Chrome DevTools to analyze
# chrome://inspect

Prevention

Monitor memory - Alert on high usage
Memory limits - Prevent runaway processes
Regular restarts - Restart daily if leaking
Memory profiling - Profile in staging

Useful Commands

Viewing Logs

# Last 100 lines
docker compose logs api --tail=100

# Follow logs (real-time)
docker compose logs -f api

# All services
docker compose logs

# Since timestamp
docker compose logs --since="2026-02-13T10:00:00"

# Filter by keyword
docker compose logs api | grep -i error

# Save to file
docker compose logs api > api-logs.txt

Executing Commands

# Run command in running container
docker compose exec api npm run migrate

# Interactive shell
docker compose exec api sh

# Run as different user
docker compose exec -u root api sh

# Run in new container (one-off)
docker compose run --rm api npm test

Inspecting Containers

# View container details
docker inspect changemaker-lite-api-1

# View specific field
docker inspect changemaker-lite-api-1 --format='{{.State.Status}}'

# View environment variables
docker inspect changemaker-lite-api-1 --format='{{range .Config.Env}}{{println .}}{{end}}'

# View mounts
docker inspect changemaker-lite-api-1 --format='{{json .Mounts}}' | jq

Container Management

# Start all services
docker compose up -d

# Start specific service
docker compose up -d api

# Stop all services
docker compose stop

# Stop specific service
docker compose stop api

# Restart service
docker compose restart api

# Remove stopped containers
docker compose rm

# Stop and remove
docker compose down

Rebuilding

# Rebuild single service
docker compose build api

# Rebuild without cache
docker compose build --no-cache api

# Build all services
docker compose build

# Build and start
docker compose up -d --build

# Force recreate containers
docker compose up -d --force-recreate

Log Analysis

Reading Container Logs

Logs follow this pattern:

[timestamp] [level] [message]
2026-02-13T10:30:00.000Z INFO Server started on port 4000

Common Log Patterns

Successful startup:

INFO Connecting to database...
INFO Database connected
INFO Registered route: GET /api/health
INFO Registered route: POST /api/auth/login
INFO Server started on port 4000

Database connection error:

INFO Connecting to database...
ERROR Can't reach database server at `v2-postgres:5432`
ERROR Retrying in 5 seconds...

Missing environment variable:

ERROR Environment validation failed:
ERROR   SMTP_HOST is required
ERROR   JWT_ACCESS_SECRET is required

Health check failure:

WARN Health check failed: Database not connected

Filtering Logs

# Only errors
docker compose logs api | grep ERROR

# Only warnings and errors
docker compose logs api | grep -E "ERROR|WARN"

# Exclude health checks
docker compose logs api | grep -v "GET /api/health"

# Find specific request
docker compose logs api | grep "POST /api/users"

# Find by request ID
docker compose logs api | grep "req-abc123"

Cleanup Commands

Remove Stopped Containers

# Remove all stopped containers
docker compose down

# Remove specific service containers
docker compose rm api

# Force remove running containers
docker compose rm -f api

Remove Images

# Remove all images for project
docker compose down --rmi all

# Remove only project-built images (not postgres, redis, etc.)
docker compose down --rmi local

# Remove specific image
docker rmi changemaker-lite-api

# Remove dangling images
docker image prune

Remove Volumes

# ⚠️ WARNING: Deletes all data!
docker compose down -v

# Remove specific volume
docker volume rm changemaker-lite_postgres-data

# Remove unused volumes
docker volume prune

Remove Networks

# Remove project network (containers must be stopped first)
docker network rm changemaker-lite

# Remove unused networks
docker network prune

Full Cleanup

# ⚠️ DANGER: Removes everything!
docker compose down -v --rmi all
docker system prune -a --volumes

# This deletes:
# - All containers
# - All volumes (data lost!)
# - All images
# - All networks
# - All build cache

Safe Cleanup

# Safe cleanup (keeps volumes)
docker compose down
docker image prune -a
docker network prune

# This keeps:
# - Volumes (data safe)
# - .env file
# - Application code

Docker Documentation

Docker Issues - This guide
Installation Guide - Initial setup
Architecture Overview - System design

Other Troubleshooting

Common Errors - General errors
Database Issues - PostgreSQL problems
Monitoring Issues - Observability problems

Docker Resources

Last Updated: February 2026 Version: V2.0 Status: Complete

35 KiB Raw Blame History

Docker and Container Issues

Overview

Docker Troubleshooting Approach

Essential Docker Commands

Container Won't Start

Port Already in Use

Symptoms

Common Causes

Solutions

Prevention

Volume Mount Errors

Symptoms

Common Causes

Solutions

Prevention

Missing Environment Variables

Symptoms

Common Causes

Solutions

Prevention

Health Check Failures

Symptoms

Common Causes

Solutions

Prevention

Container Crashes

Out of Memory

Symptoms

Common Causes

Solutions

Prevention

Application Errors

Symptoms

Common Causes

Solutions

Prevention

Database Connection Failures

Symptoms

Common Causes

Solutions

Prevention

Networking Issues

Containers Can't Communicate

Symptoms

Common Causes

Solutions

Prevention

Port Not Accessible from Host

Symptoms

Common Causes

Solutions

Prevention

DNS Resolution Failures

Symptoms

Common Causes

Solutions

Prevention

Volume Issues

Permission Denied

Symptoms

Common Causes

Solutions

Prevention

Volume Not Mounted

Symptoms

Common Causes

Solutions

Prevention

Data Persistence Problems

Symptoms

Common Causes

Solutions

Prevention

Performance Issues

Slow Container Startup

Symptoms

Common Causes

Solutions

Prevention

35 KiB

Raw Blame History