# Dual API Architecture

Changemaker Lite V2 uses a dual API architecture with Express.js for main features and Fastify for the media library microservice.

## Why Dual API?

### Performance Isolation

Media operations (video processing, large uploads) are isolated from core platform features:

- **Video uploads** don't block campaign email sending
- **Media job processing** doesn't affect map rendering
- **Large file transfers** have separate connection pools

### Technology Evaluation

V2 evaluates two popular Node.js frameworks side-by-side:

| Feature | Express.js | Fastify |
|---------|-----------|---------|
| **Ecosystem** | Massive (15+ years) | Growing (7+ years) |
| **Performance** | Good | Excellent (2-3x faster) |
| **TypeScript** | Requires @types/* | Native support |
| **Middleware** | Industry standard | Plugin system |
| **Use Case** | General purpose | High-throughput APIs |

### Independent Scaling

Each API can scale independently:

- **Express API** scales with user activity (campaigns, canvassing)
- **Media API** scales with video library size
- Horizontal scaling: run multiple instances behind nginx load balancer

### Clear Service Boundaries

Microservice preparation without full microservices complexity:

- Shared database (PostgreSQL 16)
- Shared cache (Redis)
- Separate codebases (`api/src/server.ts` vs `api/src/media-server.ts`)
- Future: Could split into separate repositories/deployments

## Architecture Diagram

```mermaid
graph TB
    subgraph "Client Layer"
        Browser[Web Browser]
        Mobile[Mobile App]
    end

    subgraph "Proxy Layer"
        Nginx[Nginx Reverse Proxy<br/>Port 80/443]
    end

    subgraph "API Layer"
        Express[Express API<br/>Port 4000<br/>Prisma ORM<br/>27+ Models]
        Fastify[Fastify Media API<br/>Port 4100<br/>Drizzle ORM<br/>Media Tables]
    end

    subgraph "Data Layer"
        PG[(PostgreSQL 16<br/>changemaker_v2 DB)]
        Redis[(Redis 7<br/>Cache + Queues)]
    end

    subgraph "External Services"
        SMTP[SMTP Server]
        Represent[Represent API]
        Geocoding[Geocoding APIs]
        Listmonk[Listmonk]
    end

    Browser --> Nginx
    Mobile --> Nginx

    Nginx -->|/api/* except /api/media/*| Express
    Nginx -->|/api/media/*| Fastify

    Express --> PG
    Express --> Redis
    Express --> SMTP
    Express --> Represent
    Express --> Geocoding
    Express --> Listmonk

    Fastify --> PG
    Fastify --> Redis

    style Express fill:#61dafb,stroke:#333,stroke-width:2px
    style Fastify fill:#00d562,stroke:#333,stroke-width:2px
    style PG fill:#336791,stroke:#333,stroke-width:2px
    style Redis fill:#dc382d,stroke:#333,stroke-width:2px
```

## Express API (Main Features)

### Entry Point

**File:** `api/src/server.ts` (234 lines)

```typescript
import express from 'express';
import cors from 'cors';
import helmet from 'helmet';
import { errorHandler } from './middleware/error-handler';
import { authenticate } from './middleware/auth';
import { metricsMiddleware } from './utils/metrics';

const app = express();

// Global middleware
app.use(helmet());
app.use(cors({ origin: process.env.CORS_ORIGIN, credentials: true }));
app.use(express.json({ limit: '50mb' }));
app.use(metricsMiddleware);

// Health check (no auth)
app.get('/api/health', (req, res) => {
  res.json({ status: 'healthy', timestamp: new Date().toISOString() });
});

// Metrics endpoint (no auth, for Prometheus)
app.get('/api/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

// Route registration (40+ route groups)
app.use('/api/auth', authRoutes);
app.use('/api/users', authenticate, usersRoutes);
app.use('/api/settings', authenticate, settingsRoutes);
app.use('/api/campaigns', campaignsRoutes); // Public + admin routes
app.use('/api/representatives', representativesRoutes);
app.use('/api/responses', responsesRoutes); // Public + admin + moderation
// ... 35+ more route groups

// Global error handler (must be last)
app.use(errorHandler);

const PORT = process.env.API_PORT || 4000;
app.listen(PORT, () => {
  logger.info(`Express API listening on port ${PORT}`);
});
```

### Key Features

**14 Feature Modules:**

1. **auth** - JWT login, register, refresh, logout
2. **users** - User CRUD with pagination + search
3. **settings** - Site settings singleton
4. **campaigns** - Campaign CRUD + public routes
5. **representatives** - Represent API integration
6. **responses** - Response wall + moderation + upvoting
7. **email-queue** - BullMQ queue admin
8. **campaign-emails** - Email tracking + stats
9. **postal-codes** - Postal code cache
10. **locations** - Location CRUD + geocoding + NAR import
11. **cuts** - Cut (polygon) CRUD + spatial queries
12. **shifts** - Shift CRUD + signups
13. **canvass** - Volunteer canvassing (sessions, visits, routes)
14. **pages** - Landing page builder (GrapesJS)

**Plus:** email-templates, listmonk, pangolin, docs, qr, services, observability

### Architecture Pattern

**Layered Structure:**

```
api/src/modules/{module}/
├── {module}.routes.ts       # Express router + middleware
├── {module}.service.ts      # Business logic + database queries
├── {module}.schemas.ts      # Zod validation schemas
└── {module}.types.ts        # TypeScript interfaces (optional)
```

**Example: Campaign Module**

```typescript
// campaigns.routes.ts
import { Router } from 'express';
import { validate } from '../../middleware/validate';
import { authenticate, requireRole } from '../../middleware/auth';
import { createCampaignSchema, updateCampaignSchema } from './campaigns.schemas';
import * as campaignService from './campaigns.service';

const router = Router();

// Admin routes (auth required)
router.post('/',
  authenticate,
  requireRole('SUPER_ADMIN', 'INFLUENCE_ADMIN'),
  validate(createCampaignSchema),
  async (req, res) => {
    const campaign = await campaignService.createCampaign(req.body, req.user!.id);
    res.status(201).json(campaign);
  }
);

// Public routes (no auth)
router.get('/:id', async (req, res) => {
  const campaign = await campaignService.getCampaignById(req.params.id);
  res.json(campaign);
});

export default router;
```

### ORM: Prisma

**27+ Models** in `api/prisma/schema.prisma`:

```typescript
model Campaign {
  id                    String   @id @default(cuid())
  slug                  String   @unique
  title                 String
  description           String?  @db.Text
  emailSubject          String
  emailBody             String   @db.Text
  status                CampaignStatus @default(DRAFT)

  // Feature flags
  allowSmtpEmail        Boolean  @default(true)
  showResponseWall      Boolean  @default(true)

  // Audit fields
  createdByUserId       String?
  createdByUser         User?    @relation(fields: [createdByUserId], references: [id], onDelete: SetNull)
  createdAt             DateTime @default(now())
  updatedAt             DateTime @updatedAt

  // Relations
  emails                CampaignEmail[]
  responses             RepresentativeResponse[]
  customRecipients      CustomRecipient[]
}
```

**Connection Pooling:**

Prisma manages connection pool automatically:

```typescript
// prisma/schema.prisma
datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

// Default pool size: 10 connections per instance
// Configure via DATABASE_URL: ?connection_limit=20
```

## Fastify API (Media Library)

### Entry Point

**File:** `api/src/media-server.ts` (104 lines)

```typescript
import Fastify from 'fastify';
import cors from '@fastify/cors';
import helmet from '@fastify/helmet';
import { videosRoutes } from './modules/media/videos/videos.routes';
import { sharedMediaRoutes } from './modules/media/shared-media/shared-media.routes';
import { jobsRoutes } from './modules/media/jobs/jobs.routes';
import { reactionsRoutes } from './modules/media/reactions/reactions.routes';

const fastify = Fastify({
  logger: {
    level: process.env.NODE_ENV === 'production' ? 'info' : 'debug'
  }
});

// Plugins
await fastify.register(cors, {
  origin: process.env.CORS_ORIGIN,
  credentials: true
});
await fastify.register(helmet);

// Health check
fastify.get('/health', async (request, reply) => {
  return { status: 'healthy', timestamp: new Date().toISOString() };
});

// Route registration
fastify.register(videosRoutes, { prefix: '/api/media/videos' });
fastify.register(sharedMediaRoutes, { prefix: '/api/media/shared' });
fastify.register(jobsRoutes, { prefix: '/api/media/jobs' });
fastify.register(reactionsRoutes, { prefix: '/api/media/reactions' });

const PORT = Number(process.env.MEDIA_API_PORT) || 4100;
await fastify.listen({ port: PORT, host: '0.0.0.0' });
fastify.log.info(`Fastify Media API listening on port ${PORT}`);
```

### Key Features

**4 Feature Modules:**

1. **videos** - Video CRUD, metadata, tags, deduplication
2. **shared-media** - Public gallery categories (videos, curated, compilations, etc.)
3. **jobs** - Job queue monitoring (pending, running, completed, failed)
4. **reactions** - Reaction system (6 standard emojis: like, love, laugh, wow, sad, angry)

### Architecture Pattern

**Plugin-Based:**

```typescript
// videos.routes.ts
import { FastifyPluginAsync } from 'fastify';
import { verifyJWT } from '../../middleware/auth';
import { getVideosSchema, createVideoSchema } from './videos.schemas';

export const videosRoutes: FastifyPluginAsync = async (fastify) => {
  // Middleware: JWT verification
  fastify.addHook('onRequest', verifyJWT);

  // GET /api/media/videos
  fastify.get('/', {
    schema: getVideosSchema,
    handler: async (request, reply) => {
      const videos = await getVideos(request.query);
      return videos;
    }
  });

  // POST /api/media/videos
  fastify.post('/', {
    schema: createVideoSchema,
    handler: async (request, reply) => {
      const video = await createVideo(request.body);
      return reply.status(201).send(video);
    }
  });
};
```

### ORM: Drizzle

**Media Tables** in `api/src/modules/media/db/schema.ts`:

```typescript
import { pgTable, serial, text, integer, boolean, timestamp, jsonb } from 'drizzle-orm/pg-core';

export const videos = pgTable('videos', {
  id: serial('id').primaryKey(),
  path: text('path').unique().notNull(),
  filename: text('filename').notNull(),
  producer: text('producer'),
  creator: text('creator'),
  title: text('title'),
  durationSeconds: integer('duration_seconds'),
  width: integer('width'),
  height: integer('height'),
  orientation: text('orientation'), // 'landscape' | 'portrait' | 'square'
  hasAudio: boolean('has_audio').default(true),
  fileSize: integer('file_size'),
  thumbnailPath: text('thumbnail_path'),
  tags: jsonb('tags').$type<string[]>(),
  isValid: boolean('is_valid').default(true),
  createdAt: timestamp('created_at').defaultNow(),
}, (table) => ({
  orientationIdx: index('idx_orientation').on(table.orientation),
  producerIdx: index('idx_producer').on(table.producer),
}));
```

**Connection:**

Drizzle uses the same PostgreSQL connection pool:

```typescript
import { drizzle } from 'drizzle-orm/node-postgres';
import { Pool } from 'pg';

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 10
});

export const db = drizzle(pool);
```

## Request Flow

### Public Campaign Email Submission

```mermaid
sequenceDiagram
    participant User as User Browser
    participant Nginx
    participant React as Admin GUI
    participant Express as Express API
    participant PG as PostgreSQL
    participant Redis
    participant BullMQ
    participant SMTP

    User->>React: Visit /campaigns/123
    React->>Nginx: GET /campaigns/123
    Nginx->>React: Serve React app
    React->>Nginx: GET /api/campaigns/123
    Nginx->>Express: Forward to Express
    Express->>PG: SELECT campaign
    PG-->>Express: Campaign data
    Express-->>React: Campaign JSON
    React-->>User: Render page

    User->>React: Submit email form
    React->>Nginx: POST /api/campaigns/123/send-email
    Nginx->>Express: Forward to Express
    Express->>Express: Rate limit check (30/hour)
    Express->>PG: INSERT CampaignEmail
    Express->>BullMQ: Enqueue job
    BullMQ->>Redis: Add job to queue
    Express-->>React: Success response
    React-->>User: "Email queued"

    BullMQ->>Express: Process job (worker)
    Express->>PG: SELECT email + campaign
    Express->>Express: Build SMTP message
    Express->>SMTP: Send email
    SMTP-->>Express: Delivery confirmed
    Express->>PG: UPDATE status = SENT
    Express->>Redis: Increment cm_emails_sent_total
```

### Admin Media Upload

```mermaid
sequenceDiagram
    participant Admin as Admin Browser
    participant Nginx
    participant Fastify as Fastify Media API
    participant PG as PostgreSQL
    participant FS as File System

    Admin->>Nginx: POST /api/media/videos (10GB file)
    Nginx->>Fastify: Stream upload (no buffering)
    Fastify->>FS: Save to /media/videos/
    Fastify->>PG: INSERT video metadata
    PG-->>Fastify: Video record
    Fastify-->>Admin: { id, path, thumbnail }
```

**Key Difference:**
- Express handles small JSON payloads (campaigns, locations, users)
- Fastify handles large file uploads (streaming, no buffering)

## Shared Resources

### PostgreSQL Database

**Single Database, Multiple Schemas:**

- **Prisma Tables** — Main schema (User, Campaign, Location, etc.)
- **Drizzle Tables** — Media schema (videos, jobs, reactions)

Both ORMs connect to the same `changemaker_v2` database:

```bash
DATABASE_URL=postgresql://changemaker:password@v2-postgres:5432/changemaker_v2
```

**No Conflicts:**
- Prisma manages its own schema via migrations (`npx prisma migrate`)
- Drizzle manages media tables via `npx drizzle-kit push`
- Tables don't overlap (different prefixes)

### Redis Cache

Both APIs use Redis for:

- **Caching** — Postal codes (Express), video metadata (Fastify)
- **Rate Limiting** — Redis-backed limits (Express: 30/hour, Fastify: 100/min)
- **BullMQ Queues** — Email queue (Express), job queue (Fastify)

```typescript
// Shared Redis connection
import Redis from 'ioredis';

export const redis = new Redis({
  host: 'redis-changemaker',
  port: 6379,
  password: process.env.REDIS_PASSWORD,
  maxRetriesPerRequest: 3
});
```

### JWT Authentication

Both APIs verify the same JWT tokens:

```typescript
// Express: api/src/middleware/auth.ts
import jwt from 'jsonwebtoken';

export const authenticate = (req, res, next) => {
  const token = req.headers.authorization?.split(' ')[1];
  const payload = jwt.verify(token, process.env.JWT_ACCESS_SECRET);
  req.user = payload; // { id, email, role }
  next();
};

// Fastify: api/src/modules/media/middleware/auth.ts
import jwt from 'jsonwebtoken';

export const verifyJWT = async (request, reply) => {
  const token = request.headers.authorization?.split(' ')[1];
  const payload = jwt.verify(token, process.env.JWT_ACCESS_SECRET);
  request.user = payload;
};
```

**Shared Secret:** `JWT_ACCESS_SECRET` environment variable

## Nginx Routing

### Location Block Ordering

**Critical:** Media API location must come BEFORE general API location:

```nginx
server {
    listen 80;
    server_name api.cmlite.org;

    # Media API (longest prefix first)
    location /api/media/ {
        proxy_pass http://changemaker-media-api:4100;
        client_max_body_size 10G;
    }

    # Express API (catch-all)
    location /api/ {
        proxy_pass http://changemaker-v2-api:4000;
    }
}
```

**Why Order Matters:**

Nginx matches longest prefix first. If `/api/` came first, it would match `/api/media/videos` and route to Express (wrong).

### Subdomain Routing (Production)

```nginx
# Express API
server {
    listen 80;
    server_name api.cmlite.org;
    location / {
        proxy_pass http://changemaker-v2-api:4000;
    }
}

# Fastify Media API
server {
    listen 80;
    server_name media.cmlite.org;
    location / {
        proxy_pass http://changemaker-media-api:4100;
    }
}
```

## Performance Comparison

### Benchmarks (Internal Testing)

**Simple GET Request (JSON response):**

| Framework | Requests/sec | Latency p95 | Memory |
|-----------|--------------|-------------|--------|
| Express | 12,500 | 35ms | 150MB |
| Fastify | 28,000 | 15ms | 120MB |

**Large Upload (1GB file):**

| Framework | Upload Time | Memory Peak | CPU Usage |
|-----------|-------------|-------------|-----------|
| Express | 45s | 450MB | 85% |
| Fastify | 38s | 280MB | 60% |

**Real-World Usage:**

- Express handles 95% of requests (campaigns, users, locations)
- Fastify handles 5% of requests (video uploads, media library)
- Both run comfortably on single-core containers

## Future: Full Microservices

The dual API design prepares for future microservices migration:

### Potential Split

```
├── campaign-service/     # Express API (Influence module)
├── map-service/          # Express API (Map module)
├── media-service/        # Fastify API (Media module)
├── auth-service/         # Shared authentication
└── api-gateway/          # Nginx or Kong
```

### Benefits

- **Independent deployment** — Ship campaign features without redeploying map
- **Technology flexibility** — Use Go for high-throughput, Python for ML
- **Team ownership** — Separate teams own separate services
- **Fault isolation** — Media service crash doesn't affect campaigns

### Trade-offs

- **Operational complexity** — More containers, more monitoring
- **Network latency** — Inter-service calls over HTTP
- **Data consistency** — Distributed transactions harder
- **Development overhead** — Multiple repos, versioning

**V2 Strategy:** Keep dual API until scaling requires split (likely 10,000+ users).

## Development Workflow

### Running Both APIs

```bash
# Terminal 1: Express API
cd api && npm run dev  # Port 4000

# Terminal 2: Fastify Media API
cd api && npm run dev:media  # Port 4100

# Terminal 3: Admin GUI
cd admin && npm run dev  # Port 3000
```

### Docker Compose

```bash
# Start both APIs
docker compose up -d api media-api

# View logs
docker compose logs -f api
docker compose logs -f media-api

# Rebuild after dependency changes
docker compose build api media-api
docker compose up -d api media-api
```

## Monitoring

Both APIs expose Prometheus metrics:

- **Express:** `http://localhost:4000/api/metrics`
- **Fastify:** `http://localhost:4100/metrics`

**Custom Metrics:**

```typescript
// Express: api/src/utils/metrics.ts
import client from 'prom-client';

export const httpRequestTotal = new client.Counter({
  name: 'http_request_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status']
});

export const emailsSentTotal = new client.Counter({
  name: 'cm_emails_sent_total',
  help: 'Total campaign emails sent'
});

// Fastify: api/src/modules/media/metrics.ts
export const mediaUploadsTotal = new client.Counter({
  name: 'cm_media_uploads_total',
  help: 'Total media uploads',
  labelNames: ['type']
});
```

Prometheus scrapes both endpoints every 15 seconds.

## Troubleshooting

### Media API Returns 404

**Cause:** Nginx routing issue (order of location blocks).

**Fix:** Ensure `/api/media/` comes BEFORE `/api/` in nginx config.

### Large Upload Fails (413)

**Cause:** `client_max_body_size` too small.

**Fix:** Increase in nginx config:

```nginx
location /api/media/ {
    client_max_body_size 20G;  # Increase from default
}
```

### Connection Pool Exhausted

**Cause:** Too many concurrent requests, not enough DB connections.

**Fix:** Increase connection limit in `DATABASE_URL`:

```bash
DATABASE_URL=postgresql://user:pass@host:5432/db?connection_limit=20
```

Or reduce pool size per API instance (if running multiple):

```typescript
// Prisma
datasource db {
  url = env("DATABASE_URL")  // Add ?connection_limit=5 for smaller pool
}

// Drizzle
const pool = new Pool({ max: 5 });
```

### JWT Verification Fails Across APIs

**Cause:** Different `JWT_ACCESS_SECRET` values.

**Fix:** Ensure both APIs use the same secret:

```bash
# .env
JWT_ACCESS_SECRET=<same-value-for-both>
```

## Further Reading

- [Database Architecture](database.md) — Prisma vs Drizzle schemas
- [Authentication Flow](authentication.md) — JWT implementation
- [Monitoring Stack](monitoring.md) — Prometheus metrics
- [Nginx Configuration](../deployment/nginx.md) — Reverse proxy setup
- [Scaling Strategies](../deployment/scaling.md) — Horizontal scaling