changemaker.lite/mkdocs/docs/v2/development/migrations.md

# Database Migrations Guide

Complete guide to managing database schema changes in Changemaker Lite V2 using Prisma Migrate and Drizzle Kit.

## Overview

Changemaker Lite V2 uses **two ORMs** for different parts of the application:

- **Prisma** (Main API) - Full-featured ORM with migration tracking
- **Drizzle** (Media API) - Lightweight ORM with schema push (no migrations)

This guide covers both workflows.

## Prisma Migrations (Main API)

### Migration Workflow Overview

```
1. Edit schema.prisma
        ↓
2. Create migration (npx prisma migrate dev)
        ↓
3. Review generated SQL
        ↓
4. Test migration locally
        ↓
5. Commit migration files
        ↓
6. Deploy to production (npx prisma migrate deploy)
```

### Understanding Prisma Migrate

**Prisma Migrate:**
- Tracks schema changes as SQL migration files
- Stores migration history in `_prisma_migrations` table
- Ensures schema consistency across environments
- Supports rollback via version control

**Migration Files:**
- Located in `api/prisma/migrations/`
- Named with timestamp: `20260213123456_description/`
- Contains `migration.sql` (SQL commands)

**Migration States:**
- **Pending:** Not yet applied
- **Applied:** Successfully executed
- **Failed:** Execution error (requires manual fix)

### Creating Migrations

#### Step 1: Edit Prisma Schema

Edit `api/prisma/schema.prisma`:

```prisma
// Before
model User {
  id        Int      @id @default(autoincrement())
  email     String   @unique
  password  String
  role      Role     @default(USER)
  createdAt DateTime @default(now())
}

// After (add name field)
model User {
  id        Int      @id @default(autoincrement())
  email     String   @unique
  password  String
  name      String?  // New field (nullable)
  role      Role     @default(USER)
  createdAt DateTime @default(now())
}
```

#### Step 2: Validate Schema

```bash
cd api
npx prisma validate
```

**Expected output:**
```
Environment variables loaded from .env
Prisma schema loaded from prisma/schema.prisma
The schema is valid ✔
```

**If errors:**
```
Error validating model "User": Field "foo" references unknown model "Bar"
```

Fix errors before proceeding.

#### Step 3: Create Migration

```bash
cd api
npx prisma migrate dev --name add_user_name
```

**What happens:**
1. Prisma detects schema changes
2. Generates SQL migration file
3. Prompts for migration name (or uses `--name` argument)
4. Applies migration to development database
5. Regenerates Prisma Client

**Expected output:**
```
Environment variables loaded from .env
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "changemaker_v2_db"

Applying migration `20260213123456_add_user_name`
Running seed command `tsx prisma/seed.ts` ...

✔ Generated Prisma Client to ./node_modules/@prisma/client
```

**Migration file created:**
```
api/prisma/migrations/
└── 20260213123456_add_user_name/
    └── migration.sql
```

#### Step 4: Review Generated SQL

```bash
cd api
cat prisma/migrations/20260213123456_add_user_name/migration.sql
```

**Example SQL:**
```sql
-- AlterTable
ALTER TABLE "users" ADD COLUMN "name" TEXT;
```

**Verify SQL is correct:**
- Check table names match expectations
- Ensure data types are correct
- Look for unexpected `DROP` commands

#### Step 5: Test Migration

Migration already applied to development DB. Verify:

```bash
# Check schema with Prisma Studio
cd api
npx prisma studio
```

Or query directly:

```bash
# PostgreSQL shell
docker compose exec v2-postgres psql -U changemaker_v2 -d changemaker_v2_db

# Describe users table
changemaker_v2_db=# \d users;
```

**Expected output:**
```
Column    |  Type   | Nullable | Default
----------+---------+----------+---------
id        | integer | not null | nextval(...)
email     | text    | not null |
password  | text    | not null |
name      | text    |          |  <-- New field
role      | text    | not null | 'USER'
created_at| timestamp| not null | now()
```

#### Step 6: Commit Migration

```bash
git add prisma/migrations/20260213123456_add_user_name/
git add prisma/schema.prisma
git commit -m "feat(db): add name field to User model"
```

**Always commit:**
- Migration directory (`prisma/migrations/*/`)
- Updated `schema.prisma`

### Applying Migrations (Production)

#### In Production Environment

```bash
cd api
npx prisma migrate deploy
```

**What it does:**
- Checks `_prisma_migrations` table for applied migrations
- Applies only **pending** migrations
- Does NOT create new migrations
- Safe for production

**Expected output:**
```
Environment variables loaded from .env
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "changemaker_v2_prod_db"

2 migrations found in prisma/migrations

Applying migration `20260213123456_add_user_name`
Applying migration `20260214000000_add_user_avatar`

All migrations have been successfully applied.
```

#### In Docker

```bash
# Apply migrations in Docker container
docker compose exec api npx prisma migrate deploy

# Or during container startup (Dockerfile)
CMD npx prisma migrate deploy && npm start
```

#### CI/CD Deployment

```yaml
# GitHub Actions example
- name: Run migrations
  run: |
    cd api
    npx prisma migrate deploy
```

### Migration Best Practices

#### 1. Incremental Changes

Make small, focused migrations:

**Good:**
```bash
# Separate migrations
npx prisma migrate dev --name add_user_name
npx prisma migrate dev --name add_user_avatar
npx prisma migrate dev --name add_user_bio
```

**Bad:**
```bash
# One huge migration
npx prisma migrate dev --name update_user_model
# (adds 10 fields, 3 relations, 5 indexes)
```

#### 2. Descriptive Names

Use clear migration names:

**Good:**
```bash
npx prisma migrate dev --name add_user_name
npx prisma migrate dev --name make_email_unique
npx prisma migrate dev --name create_posts_table
npx prisma migrate dev --name add_user_posts_relation
```

**Bad:**
```bash
npx prisma migrate dev --name update
npx prisma migrate dev --name fix
npx prisma migrate dev --name changes
```

#### 3. Review SQL Before Committing

Always review generated SQL:

```bash
cat prisma/migrations/*/migration.sql
```

**Watch for:**
- Unexpected `DROP TABLE` or `DROP COLUMN`
- Missing `NOT NULL` constraints
- Incorrect data types
- Missing indexes on foreign keys

#### 4. Backup Before Migration (Production)

```bash
# Backup database before deploy
docker compose exec v2-postgres pg_dump -U changemaker_v2 changemaker_v2_db > backup-$(date +%Y%m%d).sql

# Apply migration
npx prisma migrate deploy

# If migration fails, restore:
cat backup-20260213.sql | docker compose exec -T v2-postgres psql -U changemaker_v2 changemaker_v2_db
```

#### 5. Test on Staging First

Never deploy migrations directly to production:

```
1. Create migration in development
2. Test locally
3. Commit to version control
4. Deploy to staging environment
5. Test on staging
6. Deploy to production
```

### Common Migration Scenarios

#### Add New Field

```prisma
// schema.prisma
model User {
  id    Int    @id @default(autoincrement())
  email String @unique
  name  String? // New nullable field
}
```

```bash
npx prisma migrate dev --name add_user_name
```

**Generated SQL:**
```sql
ALTER TABLE "users" ADD COLUMN "name" TEXT;
```

#### Add Required Field (with Default)

```prisma
model User {
  id        Int      @id @default(autoincrement())
  email     String   @unique
  createdAt DateTime @default(now()) // New required field with default
}
```

```bash
npx prisma migrate dev --name add_created_at
```

**Generated SQL:**
```sql
ALTER TABLE "users" ADD COLUMN "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP;
```

#### Add New Table

```prisma
model Post {
  id        Int      @id @default(autoincrement())
  title     String
  content   String?
  published Boolean  @default(false)
  authorId  Int
  author    User     @relation(fields: [authorId], references: [id])
  createdAt DateTime @default(now())
}

model User {
  id    Int    @id @default(autoincrement())
  email String @unique
  posts Post[]
}
```

```bash
npx prisma migrate dev --name create_posts_table
```

**Generated SQL:**
```sql
CREATE TABLE "posts" (
    "id" SERIAL NOT NULL,
    "title" TEXT NOT NULL,
    "content" TEXT,
    "published" BOOLEAN NOT NULL DEFAULT false,
    "author_id" INTEGER NOT NULL,
    "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,

    CONSTRAINT "posts_pkey" PRIMARY KEY ("id")
);

CREATE INDEX "posts_author_id_idx" ON "posts"("author_id");

ALTER TABLE "posts" ADD CONSTRAINT "posts_author_id_fkey"
    FOREIGN KEY ("author_id") REFERENCES "users"("id")
    ON DELETE RESTRICT ON UPDATE CASCADE;
```

#### Add Relation

```prisma
model Campaign {
  id           Int    @id @default(autoincrement())
  title        String
  createdByUserId Int // New foreign key
  createdBy    User   @relation(fields: [createdByUserId], references: [id])
}

model User {
  id        Int        @id @default(autoincrement())
  email     String     @unique
  campaigns Campaign[]
}
```

```bash
npx prisma migrate dev --name add_campaign_user_relation
```

**Generated SQL:**
```sql
ALTER TABLE "campaigns" ADD COLUMN "created_by_user_id" INTEGER NOT NULL;

CREATE INDEX "campaigns_created_by_user_id_idx" ON "campaigns"("created_by_user_id");

ALTER TABLE "campaigns" ADD CONSTRAINT "campaigns_created_by_user_id_fkey"
    FOREIGN KEY ("created_by_user_id") REFERENCES "users"("id")
    ON DELETE RESTRICT ON UPDATE CASCADE;
```

#### Change Field Type

```prisma
// Before
model User {
  age Int
}

// After
model User {
  age String // Changed from Int to String
}
```

```bash
npx prisma migrate dev --name change_user_age_to_string
```

**Generated SQL:**
```sql
ALTER TABLE "users" ALTER COLUMN "age" SET DATA TYPE TEXT;
```

**Warning:** This may fail if data cannot be cast. Consider data migration first.

#### Add Unique Constraint

```prisma
model User {
  email String @unique // Add unique constraint
}
```

```bash
npx prisma migrate dev --name make_email_unique
```

**Generated SQL:**
```sql
CREATE UNIQUE INDEX "users_email_key" ON "users"("email");
```

#### Add Index

```prisma
model User {
  email String

  @@index([email]) // Add index
}
```

```bash
npx prisma migrate dev --name add_email_index
```

**Generated SQL:**
```sql
CREATE INDEX "users_email_idx" ON "users"("email");
```

### Migration History and Status

#### Check Migration Status

```bash
cd api
npx prisma migrate status
```

**Expected output:**
```
Environment variables loaded from .env
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "changemaker_v2_db"

Database schema is up to date!

Following migrations have been applied:

20260101000000_init
20260105000000_add_campaigns
20260110000000_add_locations
20260213123456_add_user_name
```

#### View Migration History

```bash
# List migration files
ls -la api/prisma/migrations/

# View specific migration
cat api/prisma/migrations/20260213123456_add_user_name/migration.sql
```

#### Check Database Migration Table

```bash
docker compose exec v2-postgres psql -U changemaker_v2 -d changemaker_v2_db -c "SELECT * FROM _prisma_migrations;"
```

**Output:**
```
id | checksum | finished_at | migration_name | logs
---+----------+-------------+----------------+-----
1  | abc123   | 2026-01-01  | 20260101000000_init | NULL
2  | def456   | 2026-01-05  | 20260105000000_add_campaigns | NULL
```

### Rollback Strategies

Prisma Migrate does **NOT** have automatic rollback. Use these strategies:

#### 1. Version Control Rollback

```bash
# Revert schema changes
git revert <commit-hash>

# Create new migration to undo changes
npx prisma migrate dev --name revert_user_name

# This creates a new migration that undoes the previous one
```

#### 2. Manual Rollback Migration

Create a new migration to reverse changes:

```prisma
// If you added a field, remove it
model User {
  id    Int    @id @default(autoincrement())
  email String @unique
  // name  String? // Remove this
}
```

```bash
npx prisma migrate dev --name remove_user_name
```

**Generated SQL:**
```sql
ALTER TABLE "users" DROP COLUMN "name";
```

#### 3. Database Restore (Last Resort)

```bash
# Restore from backup
cat backup-20260213.sql | docker compose exec -T v2-postgres psql -U changemaker_v2 changemaker_v2_db

# Mark migrations as rolled back
docker compose exec v2-postgres psql -U changemaker_v2 -d changemaker_v2_db -c "
  DELETE FROM _prisma_migrations
  WHERE migration_name = '20260213123456_add_user_name';
"
```

#### 4. Reset Development Database

**WARNING: Deletes all data!**

```bash
cd api
npx prisma migrate reset
```

This:
1. Drops all tables
2. Re-applies all migrations from scratch
3. Runs seed script

### Handling Migration Conflicts

#### Schema Drift

**Problem:** Database schema doesn't match Prisma schema.

**Symptoms:**
```
Error: Database schema is not in sync with the migration history
```

**Solution:**

```bash
# Check what's different
npx prisma migrate diff \
  --from-schema-datamodel prisma/schema.prisma \
  --to-schema-datasource prisma/schema.prisma

# Create migration to fix drift
npx prisma migrate dev --name fix_schema_drift
```

#### Failed Migration

**Problem:** Migration fails during apply.

**Symptoms:**
```
Error: Migration failed with error:
  ALTER TABLE "users" ADD COLUMN "age" INTEGER NOT NULL;
  ERROR: column "age" contains null values
```

**Solution:**

```bash
# 1. Mark migration as rolled back
npx prisma migrate resolve --rolled-back 20260213123456_add_user_age

# 2. Fix migration SQL manually
vi prisma/migrations/20260213123456_add_user_age/migration.sql

# Change to:
ALTER TABLE "users" ADD COLUMN "age" INTEGER; -- Make nullable first
UPDATE "users" SET "age" = 0 WHERE "age" IS NULL; -- Set default
ALTER TABLE "users" ALTER COLUMN "age" SET NOT NULL; -- Then make required

# 3. Apply migration again
npx prisma migrate deploy
```

#### Conflicting Migrations (Team Environment)

**Problem:** Two developers create migrations simultaneously.

**Solution:**

```bash
# 1. Pull latest changes
git pull origin v2

# 2. Prisma detects conflict
npx prisma migrate dev

# 3. Resolve by creating merge migration
# Prisma will prompt you to create a migration that includes both changes
```

### Data Migrations

Prisma Migrate handles **schema** changes, not **data** changes. For data transformations:

#### Option 1: Custom SQL in Migration

Edit generated migration file:

```sql
-- Add column (Prisma-generated)
ALTER TABLE "users" ADD COLUMN "full_name" TEXT;

-- Populate from existing data (manual addition)
UPDATE "users" SET "full_name" = "first_name" || ' ' || "last_name";

-- Remove old columns (Prisma-generated)
ALTER TABLE "users" DROP COLUMN "first_name";
ALTER TABLE "users" DROP COLUMN "last_name";
```

#### Option 2: Separate Data Migration Script

```typescript
// api/prisma/data-migrations/20260213-populate-full-name.ts
import { PrismaClient } from '@prisma/client';

const prisma = new PrismaClient();

async function main() {
  const users = await prisma.user.findMany();

  for (const user of users) {
    await prisma.user.update({
      where: { id: user.id },
      data: {
        fullName: `${user.firstName} ${user.lastName}`
      }
    });
  }

  console.log(`Updated ${users.length} users`);
}

main()
  .catch(console.error)
  .finally(() => prisma.$disconnect());
```

Run after migration:

```bash
npx tsx prisma/data-migrations/20260213-populate-full-name.ts
```

## Drizzle Push (Media API)

### Drizzle Overview

**Drizzle Kit Push:**
- Syncs schema directly to database
- No migration files generated
- Fast iteration for prototyping
- Used only for Media API tables

**Schema Location:**
- `api/src/modules/media/db/schema.ts`

**When to Use:**
- Rapid prototyping
- Development only
- Media API tables (videos, jobs, reactions)

**When NOT to Use:**
- Production deployments
- Main API tables (use Prisma)
- When migration history is needed

### Drizzle Push Workflow

#### Step 1: Edit Schema

Edit `api/src/modules/media/db/schema.ts`:

```typescript
// Before
export const videos = pgTable('videos', {
  id: serial('id').primaryKey(),
  filename: text('filename').notNull(),
  title: text('title'),
  duration: integer('duration'),
  createdAt: timestamp('created_at').defaultNow().notNull(),
});

// After (add description field)
export const videos = pgTable('videos', {
  id: serial('id').primaryKey(),
  filename: text('filename').notNull(),
  title: text('title'),
  description: text('description'), // New field
  duration: integer('duration'),
  createdAt: timestamp('created_at').defaultNow().notNull(),
});
```

#### Step 2: Push Schema

```bash
cd api
npm run drizzle:push
```

Or directly:

```bash
cd api
npx drizzle-kit push
```

**What happens:**
1. Drizzle compares schema to database
2. Generates SQL for changes
3. Applies changes immediately
4. No migration files created

**Expected output:**
```
Reading config from drizzle.config.ts
Using 'pg' driver for database querying

Pulling schema from database...
[✓] Schema pulled successfully

Comparing schemas...
[!] Changes detected:
  - ALTER TABLE "videos" ADD COLUMN "description" TEXT;

Do you want to execute these changes? [y/N]: y

Applying changes...
[✓] Schema pushed successfully
```

#### Step 3: Verify Changes

```bash
# Check with Drizzle Studio
cd api
npx drizzle-kit studio
```

Or query directly:

```bash
docker compose exec v2-postgres psql -U changemaker_v2 -d changemaker_v2_db -c "\d videos"
```

### Drizzle Best Practices

#### 1. Development Only

Use Drizzle Push only in development:

**Good:**
```bash
# Development
npm run drizzle:push
```

**Bad:**
```bash
# Production (use Prisma migrate for production schema changes)
npm run drizzle:push
```

#### 2. Backup Before Push

Always backup before pushing schema:

```bash
# Backup database
docker compose exec v2-postgres pg_dump -U changemaker_v2 changemaker_v2_db > backup.sql

# Push schema
npm run drizzle:push

# If something breaks, restore:
cat backup.sql | docker compose exec -T v2-postgres psql -U changemaker_v2 changemaker_v2_db
```

#### 3. Test Changes Locally

Never push untested schema changes:

```bash
# 1. Edit schema
vi src/modules/media/db/schema.ts

# 2. Push to dev database
npm run drizzle:push

# 3. Test with Drizzle Studio
npm run drizzle:studio

# 4. Test API endpoints
curl http://localhost:4100/api/media/videos
```

### Drizzle vs Prisma

| Feature | Prisma Migrate | Drizzle Push |
|---------|----------------|--------------|
| Migration files | ✅ Yes | ❌ No |
| Migration history | ✅ Tracked | ❌ Not tracked |
| Rollback | ✅ Via version control | ❌ Manual only |
| Production use | ✅ Recommended | ⚠️ Not recommended |
| Prototyping | ⚠️ Slower | ✅ Faster |
| Use case | Main API tables | Media API tables |

## Seeding After Migration

### Running Seed Script

After migrations, seed database:

```bash
cd api
npx prisma db seed
```

**What it does:**
- Runs `prisma/seed.ts`
- Creates admin user
- Creates default settings
- Creates sample blocks

**Expected output:**
```
Running seed command `tsx prisma/seed.ts` ...
Seeding database...
Created default settings
Created admin user: admin@example.com
Created 10 sample blocks
Seed completed successfully
```

### Custom Seed Data

Edit `api/prisma/seed.ts`:

```typescript
import { PrismaClient } from '@prisma/client';

const prisma = new PrismaClient();

async function main() {
  // Create admin user
  await prisma.user.upsert({
    where: { email: 'admin@example.com' },
    update: {},
    create: {
      email: 'admin@example.com',
      password: await hashPassword('Admin123!'),
      role: 'SUPER_ADMIN',
      name: 'Admin User'
    }
  });

  // Create sample campaign
  await prisma.campaign.create({
    data: {
      title: 'Sample Campaign',
      description: 'This is a sample campaign',
      active: true,
      createdByUserId: 1
    }
  });

  console.log('Seed completed');
}

main()
  .catch(console.error)
  .finally(() => prisma.$disconnect());
```

## CI/CD Integration

### GitHub Actions Example

```yaml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '20'

      - name: Install dependencies
        working-directory: ./api
        run: npm ci

      - name: Run migrations
        working-directory: ./api
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
        run: npx prisma migrate deploy

      - name: Seed database
        working-directory: ./api
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
        run: npx prisma db seed
```

### Docker Deployment

```dockerfile
# api/Dockerfile
FROM node:20-alpine

WORKDIR /app

COPY package*.json ./
RUN npm ci --production

COPY . .

# Generate Prisma Client
RUN npx prisma generate

# Run migrations on startup
CMD npx prisma migrate deploy && npm start
```

## Troubleshooting

### Migration Fails with "Column Already Exists"

**Problem:**
```
Error: column "name" of relation "users" already exists
```

**Solution:**

```bash
# Mark migration as applied
npx prisma migrate resolve --applied 20260213123456_add_user_name

# Or drop column manually and re-run
docker compose exec v2-postgres psql -U changemaker_v2 -d changemaker_v2_db -c "ALTER TABLE users DROP COLUMN name;"
npx prisma migrate deploy
```

### Migration Fails with "Relation Does Not Exist"

**Problem:**
```
Error: relation "posts" does not exist
```

**Solution:**

```bash
# Check migration history
npx prisma migrate status

# Apply missing migrations
npx prisma migrate deploy

# Or reset (development only)
npx prisma migrate reset
```

### Schema Out of Sync

**Problem:**
```
Error: Database schema is not in sync
```

**Solution:**

```bash
# Generate migration to fix drift
npx prisma migrate dev --name fix_drift

# Or in production, create explicit migration
npx prisma migrate diff \
  --from-schema-datamodel prisma/schema.prisma \
  --to-schema-datasource prisma/schema.prisma \
  --script > fix-drift.sql

# Review fix-drift.sql and apply manually
```

### Drizzle Push Fails

**Problem:**
```
Error: Could not push schema
```

**Solution:**

```bash
# Check Drizzle config
cat api/drizzle.config.ts

# Verify DATABASE_URL
echo $DATABASE_URL

# Test database connection
docker compose exec v2-postgres psql -U changemaker_v2 -d changemaker_v2_db -c "SELECT 1"

# Clear Drizzle cache and retry
rm -rf api/.drizzle
npm run drizzle:push
```

## Related Documentation

- **Setup:** [Local Development Setup](local-setup.md)
- **Commands:** [NPM Commands Reference](npm-commands.md)
- **Docker:** [Docker Workflow](docker-workflow.md)
- **Database:** [Database Schema](../architecture/database-schema.md)
- **Deployment:** [Production Deployment](../deployment/production.md)

## Summary

You now know:
- ✅ How Prisma Migrate tracks schema changes
- ✅ How to create and apply migrations
- ✅ Common migration scenarios (add field, table, relation)
- ✅ Migration best practices
- ✅ How to handle migration conflicts
- ✅ How to perform data migrations
- ✅ How Drizzle Push works for Media API
- ✅ When to use Prisma vs Drizzle
- ✅ How to seed database after migrations
- ✅ How to integrate migrations in CI/CD

**Quick Reference:**
```bash
# Prisma: Create migration
npx prisma migrate dev --name description

# Prisma: Apply migrations (production)
npx prisma migrate deploy

# Prisma: Check status
npx prisma migrate status

# Drizzle: Push schema (dev only)
npx drizzle-kit push

# Seed database
npx prisma db seed

# Reset (dev only, DELETES DATA)
npx prisma migrate reset
```