admin/changemaker.lite

Fork 0

bunker-admin 7895ce683e Tonne of debugging - getting ready for the production builds

2026-02-16 10:44:18 -07:00

42 KiB

Raw Blame History

Database and PostgreSQL Issues

This guide covers PostgreSQL and database-related problems in Changemaker Lite V2.

Overview

Database Architecture

Changemaker Lite V2 uses:

PostgreSQL 16 - Primary database
Prisma ORM - Main API (Express)
Drizzle ORM - Media API (Fastify)
Same database - Shared by both APIs
Separate schemas - Tables owned by different ORMs

Database Connection Info

# From API container
DATABASE_URL="postgresql://changemaker:password@v2-postgres:5432/changemaker_v2"

# From host
DATABASE_URL="postgresql://changemaker:password@localhost:5433/changemaker_v2"

# Connection details:
# User: changemaker
# Password: set in V2_POSTGRES_PASSWORD env var
# Host: v2-postgres (container) or localhost (host)
# Port: 5432 (inside Docker), 5433 (host)
# Database: changemaker_v2

Essential Commands

# Connect to database
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2

# Run single query
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "SELECT NOW();"

# Run SQL file
docker compose exec -T v2-postgres psql -U changemaker -d changemaker_v2 < script.sql

# Database logs
docker compose logs v2-postgres

# Prisma Studio (GUI)
docker compose exec api npx prisma studio

Connection Errors

Connection Refused

Severity: 🔴 Critical

Symptoms

API logs:

Error: connect ECONNREFUSED 127.0.0.1:5433
Error: Can't reach database server at `v2-postgres:5432`

Or direct connection:

psql: error: connection to server at "localhost" (127.0.0.1), port 5433 failed:
Connection refused

Common Causes

Database not running - Container stopped
Wrong connection string - Incorrect host/port
Port not exposed - Missing port mapping
Network issue - Container can't reach database

Solutions

Solution 1: Check database status

# Is database running?
docker compose ps v2-postgres

# Should show:
# NAME                          STATUS
# changemaker-lite-v2-postgres-1   Up 5 minutes

# If not running:
docker compose up -d v2-postgres

Solution 2: Wait for database to be ready

# Check logs for "ready to accept connections"
docker compose logs v2-postgres | grep "ready"

# Should show:
# database system is ready to accept connections

# If not ready, wait 10-20 seconds and check again

Solution 3: Verify connection string

# Check .env
cat .env | grep DATABASE_URL

# From API container should use container name:
DATABASE_URL="postgresql://changemaker:password@v2-postgres:5432/changemaker_v2"

# From host should use localhost:
DATABASE_URL="postgresql://changemaker:password@localhost:5433/changemaker_v2"

# Common mistakes:
# ❌ Using localhost from container
# ❌ Using v2-postgres from host
# ❌ Wrong port (5432 vs 5433)
# ❌ Wrong password

Solution 4: Test connection manually

# From API container
docker compose exec api sh -c 'psql $DATABASE_URL -c "SELECT NOW();"'

# From host
psql "postgresql://changemaker:password@localhost:5433/changemaker_v2" -c "SELECT NOW();"

# If fails, connection string is wrong

Solution 5: Check port mapping

In docker-compose.yml:

v2-postgres:
  ports:
    - "5433:5432"  # host:container

Verify:

docker compose ps v2-postgres

# Should show:
# PORTS: 0.0.0.0:5433->5432/tcp

Prevention

Health checks - Wait for database health before starting API
Connection retry - Retry connection on startup
Correct env vars - Validate DATABASE_URL format
Monitoring - Alert on connection failures

Too Many Clients

Severity: 🟠 High

Symptoms

FATAL: sorry, too many clients already

Or:

Error: remaining connection slots are reserved for non-replication superuser connections

Common Causes

Connection leak - Connections not closed
Pool too large - Connection pool size too high
Multiple Prisma instances - Each creates own pool
Long-running transactions - Holding connections

Solutions

Solution 1: Check active connections

-- View all connections
SELECT count(*) FROM pg_stat_activity;

-- View connections by state
SELECT state, count(*)
FROM pg_stat_activity
WHERE datname = 'changemaker_v2'
GROUP BY state;

-- View connection details
SELECT pid, usename, application_name, state, query_start, query
FROM pg_stat_activity
WHERE datname = 'changemaker_v2'
ORDER BY query_start;

Solution 2: Kill idle connections

-- Find idle connections
SELECT pid, usename, state, state_change
FROM pg_stat_activity
WHERE datname = 'changemaker_v2'
  AND state = 'idle'
  AND state_change < NOW() - INTERVAL '5 minutes';

-- Kill specific connection
SELECT pg_terminate_backend(12345);  -- Replace with actual PID

-- Kill all idle connections (careful!)
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname = 'changemaker_v2'
  AND state = 'idle'
  AND state_change < NOW() - INTERVAL '5 minutes';

Solution 3: Adjust connection pool

In DATABASE_URL:

# Limit connection pool size
DATABASE_URL="postgresql://changemaker:password@v2-postgres:5432/changemaker_v2?connection_limit=10"

Or in Prisma code:

// api/src/config/database.ts
import { PrismaClient } from '@prisma/client';

export const prisma = new PrismaClient({
  datasources: {
    db: {
      url: process.env.DATABASE_URL
    }
  }
  // Connection pool defaults:
  // connection_limit: 10
  // pool_timeout: 10 (seconds)
});

Solution 4: Increase max connections

In docker-compose.yml:

v2-postgres:
  command: postgres -c max_connections=200
  # Default is 100

Restart:

docker compose up -d v2-postgres

Verify:

SHOW max_connections;

Solution 5: Restart API to release connections

# Restart API releases all connections
docker compose restart api
docker compose restart media-api

# Check connection count dropped
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 \
  -c "SELECT count(*) FROM pg_stat_activity WHERE datname = 'changemaker_v2';"

Prevention

Proper cleanup - Always close Prisma clients in tests
Appropriate pool size - Balance performance vs connections
Monitor connections - Alert when approaching max
Idle timeout - Automatically close idle connections

!!! warning "Connection Math" Total connections = (number of API instances) × (connection pool size) + (other clients)

Example:
- 2 API instances × 10 pool size = 20 connections
- 1 media API × 5 pool size = 5 connections
- Prisma Studio = 1 connection
- Total = 26 connections

Set max_connections to 2-3× expected usage.

Authentication Failed

Severity: 🔴 Critical

Symptoms

FATAL: password authentication failed for user "changemaker"

Or:

FATAL: role "changemaker" does not exist

Common Causes

Wrong password - PASSWORD in DATABASE_URL doesn't match
Wrong username - User doesn't exist
Password changed - Database password changed but not .env
Case sensitivity - PostgreSQL usernames are case-sensitive

Solutions

Solution 1: Verify credentials

# Check .env
cat .env | grep V2_POSTGRES_PASSWORD

# Check DATABASE_URL
cat .env | grep DATABASE_URL

# Password in DATABASE_URL must match V2_POSTGRES_PASSWORD

Solution 2: Test connection directly

# Test with password
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2

# If prompted for password, enter V2_POSTGRES_PASSWORD
# If fails, credentials are wrong

Solution 3: Check user exists

# Connect as postgres superuser
docker compose exec v2-postgres psql -U postgres -c "\du"

# Should show changemaker user:
# Role name | Attributes
# changemaker |

# If missing, create user:
docker compose exec v2-postgres psql -U postgres -c \
  "CREATE USER changemaker WITH PASSWORD 'your-password';"

docker compose exec v2-postgres psql -U postgres -c \
  "GRANT ALL PRIVILEGES ON DATABASE changemaker_v2 TO changemaker;"

Solution 4: Reset password

# As postgres superuser
docker compose exec v2-postgres psql -U postgres -c \
  "ALTER USER changemaker WITH PASSWORD 'new-password';"

# Update .env
V2_POSTGRES_PASSWORD=new-password
DATABASE_URL="postgresql://changemaker:new-password@v2-postgres:5432/changemaker_v2"

# Restart API
docker compose restart api

Solution 5: Recreate database

If completely broken:

# Backup first!
docker compose exec v2-postgres pg_dump -U postgres changemaker_v2 > backup.sql

# Stop database
docker compose down v2-postgres

# Remove volume (⚠️ DELETES DATA!)
docker volume rm changemaker-lite_postgres-data

# Start fresh
docker compose up -d v2-postgres

# Wait for ready
docker compose logs -f v2-postgres | grep "ready"

# Run migrations
docker compose exec api npx prisma migrate deploy
docker compose exec api npx prisma db seed

Prevention

Secure passwords - Strong passwords in .env
Consistent credentials - Same password in all places
Version control .env.example - Template with placeholders
Documentation - Document credential structure

Database Does Not Exist

Severity: 🟠 High

Symptoms

FATAL: database "changemaker_v2" does not exist

Common Causes

First run - Database not created yet
Wrong database name - Typo in DATABASE_URL
Database deleted - Volume was removed
Wrong postgres instance - Connected to different database

Solutions

Solution 1: Check database exists

# List databases
docker compose exec v2-postgres psql -U postgres -l

# Should show:
# Name          | Owner
# changemaker_v2 | changemaker

# If missing, database wasn't created

Solution 2: Create database

# Create database
docker compose exec v2-postgres psql -U postgres -c \
  "CREATE DATABASE changemaker_v2 OWNER changemaker;"

# Verify
docker compose exec v2-postgres psql -U postgres -l | grep changemaker_v2

Solution 3: Run migrations

# Prisma migrations create tables
docker compose exec api npx prisma migrate deploy

# Drizzle push creates media tables
docker compose exec api npx drizzle-kit push

# Seed initial data
docker compose exec api npx prisma db seed

Solution 4: Check DATABASE_URL

# Verify database name in URL
cat .env | grep DATABASE_URL

# Should end with /changemaker_v2
# Not:
# /changemaker (missing _v2)
# /postgres (wrong database)

Solution 5: Full reset

# ⚠️ Deletes all data!
docker compose down -v
docker compose up -d v2-postgres

# Wait for ready
sleep 10

# Create and migrate
docker compose exec api npx prisma migrate deploy
docker compose exec api npx drizzle-kit push
docker compose exec api npx prisma db seed

Prevention

Initialization scripts - Auto-create database on first run
Health checks - Verify database exists before app starts
Migrations - Run migrations in deployment script
Documentation - Clear setup instructions

Migration Errors

Migration Conflict

Severity: 🟠 High

Symptoms

Error: Migration failed to apply cleanly to the shadow database.
Error: P3006 Migration `20260101000000_init` failed to apply cleanly to a temporary database.

Or:

Error: The migration `20260201000000_add_field` cannot be applied to the database:
- Added the required column `fieldName` to the `User` table without a default value.

Common Causes

Schema drift - Database schema doesn't match Prisma schema
Non-nullable column - Adding required field to table with data
Conflicting migration - Different migration with same name
Shadow database issue - Can't create shadow database

Solutions

Solution 1: Check migration status

# View migration history
docker compose exec api npx prisma migrate status

# Shows:
# - Applied migrations
# - Pending migrations
# - Failed migrations

Solution 2: Add default value for new field

If adding non-nullable column to table with existing data:

// In prisma/schema.prisma
model User {
  id    String @id @default(uuid())
  email String @unique
  name  String @default("")  // Add default for existing rows
}

Or use two-step migration:

-- Migration 1: Add nullable field
ALTER TABLE "User" ADD COLUMN "name" TEXT;

-- Migration 2: Make non-nullable (after backfilling)
UPDATE "User" SET "name" = 'Unknown' WHERE "name" IS NULL;
ALTER TABLE "User" ALTER COLUMN "name" SET NOT NULL;

Solution 3: Reset database (dev only)

# ⚠️ DELETES ALL DATA!
docker compose exec api npx prisma migrate reset

# This:
# 1. Drops database
# 2. Creates database
# 3. Applies all migrations
# 4. Runs seed

Solution 4: Manually fix schema drift

# Compare database schema to Prisma schema
docker compose exec api npx prisma db pull

# This creates a new schema.prisma from database
# Compare with your current schema.prisma
# Manually fix differences

Solution 5: Mark migration as applied (if already applied manually)

# If you manually ran migration SQL, mark as applied:
docker compose exec api npx prisma migrate resolve --applied "20260201000000_migration_name"

Prevention

Development workflow - Use prisma migrate dev in dev
Production workflow - Use prisma migrate deploy in prod
Never edit migrations - Don't modify files in migrations/
Test migrations - Test on copy of prod data first

Schema Drift

Severity: 🟡 Medium

Symptoms

Warning: Your database schema is not in sync with your Prisma schema.

Or:

Error: P2021 The table `main.NewTable` does not exist in the current database.

Common Causes

Manual schema changes - Changed database without migration
Missing migrations - Migrations not run on this database
Different environment - Prod vs dev schema mismatch
Failed migration - Migration partially applied

Solutions

Solution 1: Detect drift

# Check for drift
docker compose exec api npx prisma migrate diff \
  --from-schema-datamodel prisma/schema.prisma \
  --to-schema-datasource prisma/schema.prisma \
  --script

# If output is empty, no drift
# If shows SQL, that's the drift

Solution 2: Create migration from drift

# Generate migration to fix drift
docker compose exec api npx prisma migrate dev --name fix_drift

# Reviews changes and creates migration

Solution 3: Pull schema from database

# Update Prisma schema from database
docker compose exec api npx prisma db pull

# This overwrites schema.prisma with actual database schema
# Review changes before committing

Solution 4: Deploy missing migrations

# Apply all pending migrations
docker compose exec api npx prisma migrate deploy

# Check status
docker compose exec api npx prisma migrate status

Solution 5: Reset and re-migrate (dev only)

# ⚠️ DELETES ALL DATA!
docker compose exec api npx prisma migrate reset

# Applies all migrations fresh

Prevention

Never manual schema changes - Always use migrations
Consistent workflow - Same process in all environments
CI/CD validation - Check for drift in CI pipeline
Documentation - Document migration process

Failed Migration Rollback

Severity: 🔴 Critical

Symptoms

Error: Migration failed. Cannot rollback without losing data.

Or:

Error: Database is in an inconsistent state after a failed migration

Common Causes

Data migration failed - Migration includes data changes that failed
Constraint violation - Migration violates database constraints
No rollback - Prisma doesn't support automatic rollback
Partial application - Migration partially applied before error

Solutions

Solution 1: Mark migration as rolled back

# Mark as failed (doesn't undo changes)
docker compose exec api npx prisma migrate resolve --rolled-back "20260201000000_migration_name"

Solution 2: Manually revert changes

-- Find what migration did
cat api/prisma/migrations/20260201000000_migration_name/migration.sql

-- Write reverse SQL
-- If migration did:
ALTER TABLE "User" ADD COLUMN "newField" TEXT;

-- Reverse is:
ALTER TABLE "User" DROP COLUMN "newField";

-- Apply reverse
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 \
  -c 'ALTER TABLE "User" DROP COLUMN "newField";'

Solution 3: Restore from backup

# If you have backup before migration
docker compose exec -T v2-postgres psql -U changemaker -d changemaker_v2 < backup-before-migration.sql

# Then mark migration as rolled back
docker compose exec api npx prisma migrate resolve --rolled-back "20260201000000_migration_name"

Solution 4: Fix forward

Instead of rolling back, fix the issue and continue:

# Fix the issue (e.g., add missing default value)
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 \
  -c 'ALTER TABLE "User" ALTER COLUMN "newField" SET DEFAULT '\''value'\'';'

# Retry migration
docker compose exec api npx prisma migrate deploy

Solution 5: Baseline from current state

If database is in unknown state:

# Create new migration from current state
docker compose exec api npx prisma migrate dev --name baseline --create-only

# Review generated migration
# If it looks correct, apply:
docker compose exec api npx prisma migrate deploy

Prevention

Test migrations - Test on copy of prod data first
Backup before migrate - Always backup before production migration
Reversible migrations - Design migrations to be reversible
Small migrations - Small, focused migrations easier to fix

!!! danger "Prisma Doesn't Auto-Rollback" Prisma Migrate does NOT automatically rollback failed migrations. You must manually fix issues.

Query Performance

Slow Queries

Severity: 🟡 Medium to 🟠 High

Symptoms

API requests taking seconds to respond:

GET /api/users - 5000ms

Database logs show slow queries:

LOG: duration: 4521.234 ms statement: SELECT * FROM "User" WHERE ...

Common Causes

Missing indexes - Querying without index
Full table scan - WHERE clause doesn't use index
N+1 queries - Multiple queries instead of JOIN
Large result set - Fetching too many rows
Complex query - Too many JOINs or subqueries

Solutions

Solution 1: Enable slow query logging

In docker-compose.yml:

v2-postgres:
  command: postgres -c log_min_duration_statement=1000
  # Logs queries taking > 1 second

Restart:

docker compose up -d v2-postgres

# View slow query log
docker compose logs v2-postgres | grep "duration:"

Solution 2: Analyze query

-- Use EXPLAIN to see query plan
EXPLAIN ANALYZE
SELECT * FROM "User"
WHERE email LIKE '%@example.com%';

-- Output shows:
-- Seq Scan on "User"  (cost=0.00..20.00 rows=1000 width=100) (actual time=0.123..5.234 rows=50 loops=1)
--   Filter: (email ~~ '%@example.com%'::text)
--   Rows Removed by Filter: 950
-- Planning Time: 0.456 ms
-- Execution Time: 5.678 ms

-- "Seq Scan" = full table scan (slow)
-- "Index Scan" = using index (fast)

Solution 3: Add indexes

// In prisma/schema.prisma
model User {
  id    String @id @default(uuid())
  email String @unique  // Creates index automatically
  name  String

  @@index([name])  // Add index for name searches
}

Create migration:

docker compose exec api npx prisma migrate dev --name add_user_name_index

Verify index used:

EXPLAIN SELECT * FROM "User" WHERE name = 'John';
-- Should show: Index Scan using User_name_idx

Solution 4: Fix N+1 queries

// Bad - N+1 queries
const campaigns = await prisma.campaign.findMany();
for (const campaign of campaigns) {
  const emails = await prisma.campaignEmail.findMany({
    where: { campaignId: campaign.id }
  });
}
// 1 query for campaigns + N queries for emails = N+1

// Good - single query with include
const campaigns = await prisma.campaign.findMany({
  include: {
    emails: true
  }
});
// 1 query total

Solution 5: Limit result size

// Bad - fetch all users
const users = await prisma.user.findMany();

// Good - paginate
const users = await prisma.user.findMany({
  take: 50,  // Limit to 50 rows
  skip: page * 50,  // Offset for pagination
});

Prevention

Index frequently queried fields - email, createdAt, etc.
Use includes - Avoid N+1 queries
Paginate results - Never fetch all rows
Monitor query performance - Alert on slow queries

Missing Indexes

Severity: 🟡 Medium

Symptoms

Slow queries on filtered/sorted columns:

SELECT * FROM "Location" WHERE "postalCode" = 'M5H 2N2';
-- Slow without index on postalCode

Common Causes

No index on filter column - WHERE clause column not indexed
No index on sort column - ORDER BY column not indexed
No index on foreign key - JOIN column not indexed
Composite index needed - Multiple columns in WHERE

Solutions

Solution 1: Identify missing indexes

-- Find tables without indexes
SELECT schemaname, tablename, indexname
FROM pg_indexes
WHERE schemaname = 'public'
ORDER BY tablename;

-- Find columns used in WHERE but not indexed
-- (requires pg_stat_statements extension)

Solution 2: Add single-column index

model Location {
  id         String @id @default(uuid())
  address    String
  postalCode String

  @@index([postalCode])  // Add index
}

Solution 3: Add composite index

For queries filtering on multiple columns:

model Location {
  id         String @id @default(uuid())
  province   String
  city       String
  postalCode String

  @@index([province, city])  // Composite index
  // Speeds up: WHERE province = 'ON' AND city = 'Toronto'
  // Also speeds up: WHERE province = 'ON'
  // Does NOT speed up: WHERE city = 'Toronto' (must start with first column)
}

Solution 4: Add index on foreign key

model CampaignEmail {
  id         String @id @default(uuid())
  campaignId String

  campaign Campaign @relation(fields: [campaignId], references: [id])

  @@index([campaignId])  // Index foreign key for JOINs
}

Solution 5: Create migration

# Generate migration for index
docker compose exec api npx prisma migrate dev --name add_indexes

# Apply to production
docker compose exec api npx prisma migrate deploy

Prevention

Index foreign keys - Always index foreign keys
Index filter columns - Index columns used in WHERE
Index sort columns - Index columns used in ORDER BY
Monitor query patterns - Add indexes based on actual usage

!!! tip "Index Guidelines" - Unique constraints auto-create indexes - Foreign keys should be indexed - Columns in WHERE/ORDER BY/GROUP BY are candidates - Don't over-index (slows down writes)

N+1 Queries

Severity: 🟠 High

Symptoms

API slow when fetching related data:

GET /api/campaigns - 2000ms

Database logs show many similar queries:

SELECT * FROM "CampaignEmail" WHERE "campaignId" = 'uuid1'
SELECT * FROM "CampaignEmail" WHERE "campaignId" = 'uuid2'
SELECT * FROM "CampaignEmail" WHERE "campaignId" = 'uuid3'
...

Common Causes

No eager loading - Fetching relations in loop
Separate queries - Not using include/select
Nested loops - Multiple levels of relations

Solutions

Solution 1: Detect N+1 queries

Enable query logging:

// In api/src/config/database.ts
export const prisma = new PrismaClient({
  log: ['query'],  // Log all queries
});

Look for repeated patterns:

Query: SELECT * FROM "Campaign"
Query: SELECT * FROM "CampaignEmail" WHERE "campaignId" = '...'
Query: SELECT * FROM "CampaignEmail" WHERE "campaignId" = '...'
Query: SELECT * FROM "CampaignEmail" WHERE "campaignId" = '...'

Solution 2: Use include

// Bad - N+1
const campaigns = await prisma.campaign.findMany();
for (const campaign of campaigns) {
  campaign.emails = await prisma.campaignEmail.findMany({
    where: { campaignId: campaign.id }
  });
}
// 1 + N queries

// Good - single query
const campaigns = await prisma.campaign.findMany({
  include: {
    emails: true
  }
});
// 2 queries (1 for campaigns, 1 for all emails with JOIN)

Solution 3: Nested includes

// Multi-level relations
const campaigns = await prisma.campaign.findMany({
  include: {
    emails: {
      include: {
        user: true  // Include user who sent email
      }
    },
    createdBy: true
  }
});

Solution 4: Select only needed fields

// Fetch only needed data
const campaigns = await prisma.campaign.findMany({
  select: {
    id: true,
    name: true,
    emails: {
      select: {
        id: true,
        sentAt: true
      }
    }
  }
});

Solution 5: Use findUnique with include for single record

// Bad
const campaign = await prisma.campaign.findUnique({
  where: { id }
});
const emails = await prisma.campaignEmail.findMany({
  where: { campaignId: id }
});

// Good
const campaign = await prisma.campaign.findUnique({
  where: { id },
  include: { emails: true }
});

Prevention

Always use include - Load relations in single query
Enable query logging - Monitor for N+1 patterns
Code review - Check for loops with queries
Testing - Load test with realistic data

Connection Pool Exhaustion

Severity: 🟠 High

Symptoms

Error: Timed out fetching a new connection from the connection pool.

Or:

Error: Can't create connection pool - all connections are in use

API becomes unresponsive.

Common Causes

Pool too small - Not enough connections for load
Connections not released - Long-running transactions
Too many workers - BullMQ workers using all connections
Connection leak - Connections never closed

Solutions

Solution 1: Check pool size

# View DATABASE_URL
cat .env | grep DATABASE_URL

# Default connection_limit is 10
# Check if you've set it:
postgresql://user:pass@host:5432/db?connection_limit=10

Solution 2: Increase pool size

# In .env
DATABASE_URL="postgresql://changemaker:password@v2-postgres:5432/changemaker_v2?connection_limit=20"

# Restart API
docker compose restart api

Solution 3: Check active connections

-- View connection pool usage
SELECT count(*), state
FROM pg_stat_activity
WHERE datname = 'changemaker_v2'
GROUP BY state;

-- Should show:
-- count | state
--    5  | active
--    2  | idle
--    3  | idle in transaction

Solution 4: Find long-running transactions

-- Find transactions running > 1 minute
SELECT pid, usename, state, NOW() - xact_start AS duration, query
FROM pg_stat_activity
WHERE datname = 'changemaker_v2'
  AND state = 'idle in transaction'
  AND NOW() - xact_start > INTERVAL '1 minute';

-- Kill if stuck
SELECT pg_terminate_backend(pid);

Solution 5: Configure pool timeout

# Increase timeout from 10s to 30s
DATABASE_URL="postgresql://...?connection_limit=20&pool_timeout=30"

Prevention

Appropriate pool size - Size based on load
Release connections - Always close transactions
Monitor pool usage - Alert when near limit
Connection timeout - Kill stuck connections

!!! tip "Pool Sizing" Recommended pool size = (CPU cores × 2) + effective_spindle_count

For most applications: 10-20 connections per API instance

Data Issues

Duplicate Records

Severity: 🟡 Medium

Symptoms

Error: Unique constraint failed on the fields: (`email`)

Or finding multiple records:

SELECT email, count(*)
FROM "User"
GROUP BY email
HAVING count(*) > 1;
-- Returns duplicates

Common Causes

Race condition - Two creates at exact same time
Import error - CSV import created duplicates
Migration bug - Migration didn't handle duplicates
No unique constraint - Database allows duplicates

Solutions

Solution 1: Find duplicates

-- Find duplicate emails
SELECT email, array_agg(id) AS ids, count(*)
FROM "User"
GROUP BY email
HAVING count(*) > 1;

-- Example output:
-- email              | ids                                    | count
-- john@example.com   | {uuid1, uuid2}                        | 2

Solution 2: Delete duplicates (keep oldest)

-- Delete newer duplicates, keep oldest
DELETE FROM "User" u1
WHERE EXISTS (
  SELECT 1 FROM "User" u2
  WHERE u2.email = u1.email
    AND u2."createdAt" < u1."createdAt"
);

-- Or keep newest:
DELETE FROM "User" u1
WHERE EXISTS (
  SELECT 1 FROM "User" u2
  WHERE u2.email = u1.email
    AND u2."createdAt" > u1."createdAt"
);

Solution 3: Merge duplicates

-- If duplicates have different data, merge:
-- 1. Update foreign keys to point to kept record
UPDATE "Campaign" SET "createdByUserId" = 'uuid-to-keep'
WHERE "createdByUserId" = 'uuid-to-delete';

-- 2. Delete duplicate
DELETE FROM "User" WHERE id = 'uuid-to-delete';

Solution 4: Add unique constraint

model User {
  id    String @id @default(uuid())
  email String @unique  // Ensures uniqueness
}

Create migration:

# This will fail if duplicates exist
# Delete duplicates first (Solution 2)
docker compose exec api npx prisma migrate dev --name add_unique_email

Solution 5: Prevent in application code

// Use upsert instead of create
const user = await prisma.user.upsert({
  where: { email },
  update: {},  // Don't change if exists
  create: { email, name, password }
});

Prevention

Unique constraints - Database enforces uniqueness
Use upsert - Update or create atomically
Validation - Check existence before creating
Transaction isolation - Prevent race conditions

Constraint Violations

Severity: 🟡 Medium

Symptoms

Error: Foreign key constraint failed on the field: `campaignId`

Or:

Error: Null value in column "name" violates not-null constraint

Or:

Error: Check constraint "positive_age" is violated

Common Causes

Foreign key missing - Referenced record doesn't exist
Null in required field - NULL when NOT NULL constraint
Check constraint - Value violates CHECK constraint
Data type mismatch - Wrong type for column

Solutions

Solution 1: Verify foreign key exists

-- Check if campaign exists
SELECT id FROM "Campaign" WHERE id = 'campaign-uuid';

-- If not found, create parent first

Solution 2: Provide required fields

// Bad - missing required field
await prisma.user.create({
  data: {
    email: 'user@example.com'
    // Missing: name (required)
  }
});

// Good - all required fields
await prisma.user.create({
  data: {
    email: 'user@example.com',
    name: 'User Name',
    password: 'hashed-password'
  }
});

Solution 3: Handle check constraints

-- If schema has:
ALTER TABLE "User" ADD CONSTRAINT age_check CHECK (age >= 0);

-- Ensure value meets constraint:
INSERT INTO "User" (email, age) VALUES ('user@example.com', 25);
-- Not: VALUES ('user@example.com', -5);

Solution 4: Fix data type

// Bad - passing string for number
await prisma.location.create({
  data: {
    latitude: "43.65" as any  // Wrong type
  }
});

// Good - use number
await prisma.location.create({
  data: {
    latitude: 43.65  // Correct type
  }
});

Solution 5: Use transactions for dependent creates

// Create parent and child atomically
await prisma.$transaction(async (tx) => {
  const campaign = await tx.campaign.create({
    data: { name: 'My Campaign' }
  });

  const email = await tx.campaignEmail.create({
    data: {
      campaignId: campaign.id,
      subject: 'Email Subject'
    }
  });
});

Prevention

TypeScript types - Catch type errors at compile time
Zod validation - Validate before database operations
Foreign key checks - Verify parent exists
Transactions - Atomic multi-step operations

Data Corruption

Severity: 🔴 Critical

Symptoms

Invalid JSON in JSON columns
Truncated text
Wrong character encoding
Inconsistent relationships

SELECT * FROM "Campaign" WHERE "settings"::text LIKE '%\\u0000%';
-- Null bytes in JSON

Common Causes

Bad import - CSV/JSON import with bad data
Encoding issues - Wrong character encoding
Failed migration - Migration partially applied
Application bug - Code writing bad data

Solutions

Solution 1: Detect corruption

-- Find invalid JSON
SELECT id, settings
FROM "Campaign"
WHERE settings IS NOT NULL
  AND settings::text !~ '^[\[\{].*[\]\}]$';

-- Find null bytes
SELECT id, name
FROM "Location"
WHERE name LIKE '%' || chr(0) || '%';

-- Find wrong encoding
SELECT id, address
FROM "Location"
WHERE address ~ '[^\x00-\x7F]' AND address !~ '[À-ÿ]';

Solution 2: Fix invalid JSON

-- Replace invalid JSON with valid default
UPDATE "Campaign"
SET settings = '{}'::jsonb
WHERE settings IS NOT NULL
  AND settings::text !~ '^[\[\{].*[\]\}]$';

Solution 3: Fix encoding

-- Convert encoding
UPDATE "Location"
SET address = convert_from(convert_to(address, 'LATIN1'), 'UTF8')
WHERE address ~ '[^\x00-\x7F]';

Solution 4: Restore from backup

# If corruption is widespread, restore from backup
docker compose exec -T v2-postgres psql -U changemaker -d changemaker_v2 < backup-before-corruption.sql

Solution 5: Prevent future corruption

// Validate data before saving
import { z } from 'zod';

const settingsSchema = z.object({
  key: z.string(),
  value: z.any()
});

// Before save
const validated = settingsSchema.parse(settings);
await prisma.campaign.update({
  where: { id },
  data: { settings: validated as any }
});

Prevention

Input validation - Validate all inputs with Zod
UTF-8 encoding - Use UTF-8 everywhere
Regular backups - Daily backups
Data integrity checks - Regular validation scripts

Prisma Studio Issues

Won't Connect

Severity: 🟢 Low

Symptoms

docker compose exec api npx prisma studio

Opens browser but shows:

Error connecting to database

Solutions

Solution 1: Check DATABASE_URL

# Verify DATABASE_URL in container
docker compose exec api sh -c 'echo $DATABASE_URL'

# Should be valid connection string

Solution 2: Test connection

# Test database connection
docker compose exec api npx prisma db pull

# If fails, connection string is wrong

Solution 3: Use correct port

Prisma Studio runs on port 5555 by default. If port conflicts:

# Use different port
docker compose exec api npx prisma studio --port 5556

Solution 4: Check database is running

docker compose ps v2-postgres
# Must be "Up"

Slow Loading

Severity: 🟢 Low

Symptoms

Prisma Studio takes minutes to load tables with many rows.

Solutions

Solution 1: Limit rows

Prisma Studio loads all rows. For large tables, use SQL instead:

# Instead of Prisma Studio for large tables
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2

Solution 2: Add pagination

-- In psql, paginate manually
SELECT * FROM "Location" LIMIT 50 OFFSET 0;
SELECT * FROM "Location" LIMIT 50 OFFSET 50;

Drizzle Kit Issues

Push Failures

Severity: 🟠 High

Symptoms

docker compose exec api npx drizzle-kit push

Fails with:

Error: Failed to push schema changes

Solutions

Solution 1: Check Drizzle config

// In api/drizzle.config.ts
import { defineConfig } from 'drizzle-kit';

export default defineConfig({
  schema: './src/modules/media/db/schema.ts',
  out: './drizzle',
  dialect: 'postgresql',
  dbCredentials: {
    url: process.env.DATABASE_URL!
  }
});

Solution 2: Verify schema file

# Check schema file exists
docker compose exec api ls -la src/modules/media/db/schema.ts

# Check for syntax errors
docker compose exec api npx tsc --noEmit src/modules/media/db/schema.ts

Solution 3: Check for conflicts with Prisma tables

Drizzle and Prisma share same database. Ensure table names don't conflict:

// Drizzle tables
export const videos = pgTable('media_videos', { ... });
export const reactions = pgTable('media_reactions', { ... });

// Prisma uses: User, Campaign, etc. (no conflict)

Solution 4: Manually apply schema

# Generate SQL
docker compose exec api npx drizzle-kit generate:pg

# Review SQL in drizzle/ directory
# Apply manually if needed
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 < drizzle/0000_schema.sql

Backup/Restore Issues

pg_dump Errors

Severity: 🟠 High

Symptoms

docker compose exec v2-postgres pg_dump -U changemaker changemaker_v2 > backup.sql

Fails with:

pg_dump: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory

Solutions

Solution 1: Use correct connection

# From inside container
docker compose exec v2-postgres pg_dump -U changemaker changemaker_v2 > backup.sql

# Or specify host explicitly
docker compose exec v2-postgres pg_dump -U changemaker -h v2-postgres changemaker_v2 > backup.sql

Solution 2: Backup to file inside container

# Dump to file inside container
docker compose exec v2-postgres pg_dump -U changemaker changemaker_v2 -f /tmp/backup.sql

# Copy to host
docker cp changemaker-lite-v2-postgres-1:/tmp/backup.sql ./backup.sql

Solution 3: Use backup script

# Use provided backup script
./scripts/backup.sh

Restore Failures

Severity: 🔴 Critical

Symptoms

docker compose exec -T v2-postgres psql -U changemaker -d changemaker_v2 < backup.sql

Fails with errors:

ERROR: relation "User" already exists
ERROR: duplicate key value violates unique constraint

Solutions

Solution 1: Drop database first

# ⚠️ DELETES ALL DATA!
docker compose exec v2-postgres psql -U postgres -c "DROP DATABASE changemaker_v2;"
docker compose exec v2-postgres psql -U postgres -c "CREATE DATABASE changemaker_v2 OWNER changemaker;"

# Then restore
docker compose exec -T v2-postgres psql -U changemaker -d changemaker_v2 < backup.sql

Solution 2: Use --clean flag

# Create backup with clean option
docker compose exec v2-postgres pg_dump -U changemaker --clean changemaker_v2 > backup.sql

# Restore (drops existing objects first)
docker compose exec -T v2-postgres psql -U changemaker -d changemaker_v2 < backup.sql

Solution 3: Ignore errors for existing objects

# Restore and ignore "already exists" errors
docker compose exec -T v2-postgres psql -U changemaker -d changemaker_v2 < backup.sql 2>&1 | grep -v "already exists"

Useful Commands

Query Database

# Connect to database
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2

# Run single query
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "SELECT NOW();"

# Run SQL file
docker compose exec -T v2-postgres psql -U changemaker -d changemaker_v2 < script.sql

# Export query results to CSV
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 \
  -c "COPY (SELECT * FROM \"User\") TO STDOUT WITH CSV HEADER" > users.csv

Database Inspection

# List databases
docker compose exec v2-postgres psql -U postgres -l

# List tables
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "\dt"

# Describe table
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "\d \"User\""

# List indexes
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "\di"

# View table sizes
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "
SELECT
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
"

Performance Analysis

# Current activity
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "
SELECT pid, usename, application_name, state, query_start, query
FROM pg_stat_activity
WHERE datname = 'changemaker_v2'
ORDER BY query_start;
"

# Table statistics
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "
SELECT schemaname, tablename, n_live_tup, n_dead_tup, last_autovacuum
FROM pg_stat_user_tables
ORDER BY n_live_tup DESC;
"

# Index usage
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "
SELECT schemaname, tablename, indexname, idx_scan, idx_tup_read, idx_tup_fetch
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC;
"

# Unused indexes
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "
SELECT schemaname, tablename, indexname, idx_scan
FROM pg_stat_user_indexes
WHERE idx_scan = 0 AND indexname NOT LIKE '%pkey'
ORDER BY pg_relation_size(indexname::regclass) DESC;
"

Database Documentation

Database Issues - This guide
Installation Guide - Initial database setup
Architecture Overview - Database architecture

Other Troubleshooting

Common Errors - General errors
Docker Issues - Container problems
Performance Optimization - Database tuning

PostgreSQL Resources

Last Updated: February 2026 Version: V2.0 Status: Complete

42 KiB Raw Blame History Unescape Escape

Database and PostgreSQL Issues

Overview

Database Architecture

Database Connection Info

Essential Commands

Connection Errors

Connection Refused

Symptoms

Common Causes

Solutions

Prevention

Too Many Clients

Symptoms

Common Causes

Solutions

Prevention

Authentication Failed

Symptoms

Common Causes

Solutions

Prevention

Database Does Not Exist

Symptoms

Common Causes

Solutions

Prevention

Migration Errors

Migration Conflict

Symptoms

Common Causes

Solutions

Prevention

Schema Drift

Symptoms

Common Causes

Solutions

Prevention

Failed Migration Rollback

Symptoms

Common Causes

Solutions

Prevention

Query Performance

Slow Queries

Symptoms

Common Causes

Solutions

Prevention

Missing Indexes

Symptoms

Common Causes

Solutions

Prevention

N+1 Queries

Symptoms

Common Causes

Solutions

Prevention

Connection Pool Exhaustion

Symptoms

Common Causes

Solutions

Prevention

Data Issues

Duplicate Records

Symptoms

Common Causes

Solutions

Prevention

Constraint Violations

Symptoms

Common Causes

Solutions

Prevention

Data Corruption

Symptoms

Common Causes

Solutions

Prevention

42 KiB

Raw Blame History