changemaker.lite/mkdocs/docs/v2/migration/index.md

# Migration Guide: V1 to V2 Overview

This comprehensive guide covers the complete migration process from Changemaker Lite V1 to V2, including architectural changes, data migration, and rollback procedures.

## Overview

Changemaker Lite V2 is a **complete rebuild** of the platform, not an incremental upgrade. The migration represents a fundamental shift in architecture, technology stack, and approach to campaign management.

### V1 vs V2 at a Glance

| Aspect | V1 | V2 |
|--------|----|----|
| **Architecture** | Two separate Express apps | Single unified Express + Fastify API |
| **Data Layer** | NocoDB REST API | Prisma ORM + PostgreSQL 16 |
| **Frontend** | Embedded EJS templates | React SPA (Vite + Ant Design) |
| **Authentication** | Session cookies + bcrypt | JWT tokens (access + refresh) |
| **API Style** | REST via NocoDB | REST with Zod validation |
| **State Management** | Server-side sessions | Zustand client state + JWT |
| **Job Queue** | Bull (Redis) | BullMQ (Redis) |
| **Database** | NocoDB tables | Prisma migrations |
| **Email** | Nodemailer + Bull | BullMQ + Listmonk integration |
| **Ports** | 3333 (influence), 3000 (map) | 4000 (API), 3000 (admin) |

## Why Migrate to V2?

### Technical Benefits

1. **Unified Codebase**: Single API codebase instead of two separate applications
2. **Type Safety**: Full TypeScript coverage with Prisma type generation
3. **Modern Stack**: Latest React, Vite build tooling, Ant Design components
4. **Better Performance**: Direct database access via Prisma vs REST API abstraction
5. **Improved Security**: JWT refresh token rotation, RBAC, comprehensive audit trail
6. **Scalability**: Separation of concerns (dual API architecture for media)
7. **Developer Experience**: Hot reload, better tooling, comprehensive documentation

### Feature Enhancements

1. **New Features**:
   - Landing page builder with GrapesJS
   - Email template system with versioning
   - Media library with video uploads and reactions
   - Volunteer canvassing system with GPS tracking
   - Data quality dashboard for geocoding
   - Comprehensive monitoring (Prometheus + Grafana)
   - NAR 2025 electoral data import
   - Pangolin tunnel integration

2. **Enhanced Existing Features**:
   - Response wall with upvoting and moderation
   - Multi-provider geocoding (6 providers)
   - Advanced shift management with cut assignments
   - Printable walk sheets with QR codes
   - Listmonk newsletter sync

3. **Improved Admin Experience**:
   - Modern React UI with consistent design
   - Real-time updates with optimistic UI
   - Advanced filtering and search
   - Bulk operations
   - Responsive mobile support

## Migration Timeline

### Planned Phases (from V2_PLAN.md)

- **Phase 1-14**: ✅ **COMPLETE** (Foundation through Monitoring)
- **Phase 15**: 🚧 Testing + Polish (current)

### Actual Development Timeline

- **2025-01**: V2 rebuild initiated (clean-room approach)
- **2025-02**: Security audit completed, NAR import, media upload
- **2026-02**: Phase 14 complete, ready for production migration

### Migration Duration Estimate

| Migration Step | Duration | Downtime Required |
|----------------|----------|-------------------|
| V1 data export | 1-2 hours | No |
| Data transformation | 2-4 hours | No |
| V2 database setup | 30 minutes | No |
| V2 data import | 1-3 hours | No |
| Testing & validation | 2-4 hours | No |
| DNS/service switchover | 15 minutes | **Yes** |
| Post-migration verification | 1 hour | No |
| **Total** | **8-15 hours** | **15 minutes** |

!!! tip "Minimize Downtime"
    Perform all data export, transformation, and testing on a separate V2 staging environment. Only switch production traffic after full validation.

## Risk Assessment

### High Risk Areas

1. **Data Loss**
   - **Risk**: Campaign data, locations, or user accounts lost during migration
   - **Mitigation**: Full V1 backup before migration, validation checksums, rollback plan
   - **Impact**: High (business-critical data)

2. **Authentication Disruption**
   - **Risk**: Users unable to login after migration (password hash incompatibility)
   - **Mitigation**: Test password migration with sample users, password reset flow ready
   - **Impact**: High (blocks all access)

3. **Email Delivery Failure**
   - **Risk**: Campaign emails stop sending after migration
   - **Mitigation**: Test SMTP configuration, BullMQ queue verification, MailHog testing
   - **Impact**: High (core feature)

### Medium Risk Areas

4. **Representative Data**
   - **Risk**: Cached representative data doesn't migrate correctly
   - **Mitigation**: Cache can be rebuilt from Represent API, non-critical
   - **Impact**: Medium (cacheable data)

5. **Location Geocoding**
   - **Risk**: Geocoded coordinates lost or corrupted
   - **Mitigation**: V2 multi-provider geocoding can re-geocode, bulk geocode endpoint
   - **Impact**: Medium (can be re-geocoded)

6. **Shift Signups**
   - **Risk**: Volunteer shift assignments lost
   - **Mitigation**: Export signups separately, manual verification, confirmation emails
   - **Impact**: Medium (time-sensitive data)

### Low Risk Areas

7. **Response Wall Data**
   - **Risk**: Public responses or upvotes lost
   - **Mitigation**: CSV export, manual re-entry if needed
   - **Impact**: Low (public-facing only)

8. **Custom Settings**
   - **Risk**: V1 settings don't map to V2 schema
   - **Mitigation**: Manual reconfiguration in V2 SettingsPage
   - **Impact**: Low (quick to reconfigure)

## Rollback Plan

### If Migration Fails

1. **Immediate Actions** (within 15 minutes):
   ```bash
   # Stop V2 services
   docker compose down

   # Restore V1 services
   docker compose -f docker-compose.v1.yml up -d

   # Restore DNS (point back to V1)
   # Update tunnel/proxy configuration
   ```

2. **Data Restoration** (if V2 data was modified):
   ```bash
   # Restore V1 database from backup
   docker compose -f docker-compose.v1.yml exec -T v1-postgres \
     psql -U nocodb nocodb < backups/v1-nocodb-backup.sql

   # Verify data integrity
   docker compose -f docker-compose.v1.yml logs -f
   ```

3. **Verification**:
   - Test V1 login
   - Verify campaign data visible
   - Check location map loads
   - Send test campaign email
   - Verify response wall displays

### Rollback Window

- **First 24 hours**: Simple rollback (V1 backup unchanged)
- **After 24 hours**: Complex rollback (may need to merge V2 changes back to V1)
- **After 1 week**: Rollback not recommended (significant V2 data divergence)

!!! warning "Rollback Deadline"
    Plan your migration with a clear rollback deadline. After this window, V2 becomes the source of truth.

## Support Resources

### Documentation

- **Migration Docs**:
  - [Breaking Changes](breaking-changes.md) - Detailed V1→V2 differences
  - [Data Migration](data-migration.md) - Step-by-step data transfer
  - [API Changes](api-changes.md) - Endpoint mapping table
  - [Feature Parity](feature-parity.md) - Feature comparison matrix

- **V2 Docs**:
  - [Getting Started](../getting-started/index.md) - V2 installation
  - [Architecture](../architecture/index.md) - System design
  - [Deployment](../deployment/production.md) - Production setup

### Community & Support

- **GitHub Issues**: Report bugs or migration problems
- **Discussions**: Ask questions, share migration experiences
- **Email**: support@cmlite.org for direct assistance

### Professional Services

For organizations requiring:
- Custom data migration scripts
- Zero-downtime migration
- Training for administrators
- Priority support during migration

Contact: enterprise@cmlite.org

## Prerequisites

Before beginning migration, ensure you have:

### V1 Environment

- [ ] **V1 backup completed** (database + uploads)
- [ ] **V1 environment variables documented** (`.env` file)
- [ ] **V1 access credentials** (NocoDB admin, database passwords)
- [ ] **V1 running and healthy** (all services operational)
- [ ] **V1 data export tested** (able to export NocoDB tables)

### V2 Environment

- [ ] **V2 repository cloned** (`git checkout v2`)
- [ ] **Docker and Docker Compose installed** (20.10+, 2.0+)
- [ ] **PostgreSQL 16 compatible** (for V2 database)
- [ ] **4GB+ RAM available** (8GB recommended)
- [ ] **20GB+ disk space** (for database + uploads)

### Migration Planning

- [ ] **Downtime window scheduled** (notify users)
- [ ] **Rollback plan reviewed** (tested on staging)
- [ ] **Team assigned** (minimum 2 people recommended)
- [ ] **Backup storage ready** (S3 bucket or local storage)
- [ ] **Testing checklist prepared** (critical workflows to verify)

## Migration Steps Overview

This is a high-level overview. Detailed steps are in [Data Migration](data-migration.md).

### Phase 1: Preparation (No Downtime)

1. **Export V1 Data**
   ```bash
   # Export all NocoDB tables to JSON
   ./scripts/export-v1-data.sh

   # Backup file uploads
   tar -czf v1-uploads.tar.gz ./uploads/
   ```

2. **Set Up V2 Environment**
   ```bash
   git checkout v2
   cp .env.example .env
   # Edit .env with V2 configuration
   ```

3. **Start V2 Services** (parallel to V1)
   ```bash
   docker compose up -d v2-postgres redis
   docker compose exec api npx prisma migrate deploy
   ```

### Phase 2: Data Transformation (No Downtime)

4. **Transform V1 Data for V2**
   ```bash
   # Run transformation scripts
   node scripts/transform-users.js
   node scripts/transform-campaigns.js
   node scripts/transform-locations.js
   ```

5. **Import into V2 Database**
   ```bash
   # Import transformed data
   docker compose exec api node scripts/import-data.js
   ```

6. **Validate Data Integrity**
   ```bash
   # Compare record counts
   docker compose exec api node scripts/validate-migration.js
   ```

### Phase 3: Testing (No Downtime)

7. **Test V2 Functionality**
   - Login with test users (verify password migration)
   - View campaigns, locations, shifts
   - Submit test response
   - Send test email
   - Check admin permissions

8. **Performance Testing**
   - Load campaigns page (check query performance)
   - Geocode sample addresses
   - Test map rendering with all locations
   - Verify Redis caching

### Phase 4: Switchover (15 Minutes Downtime)

9. **Enable Maintenance Mode** (V1)
   ```bash
   # Stop V1 services
   docker compose -f docker-compose.v1.yml down
   ```

10. **Start V2 Services**
    ```bash
    # Start all V2 services
    docker compose up -d
    ```

11. **Update DNS/Proxy**
    - Point `cmlite.org` to V2 nginx
    - Update Pangolin tunnel endpoint
    - Verify SSL certificates

### Phase 5: Verification (Post-Migration)

12. **Smoke Tests**
    - Admin login works
    - Campaign list loads
    - Location map renders
    - Email sending functional
    - Response wall displays

13. **Monitor for Issues**
    ```bash
    # Watch logs for errors
    docker compose logs -f api admin

    # Check metrics
    open http://localhost:3001  # Grafana
    ```

14. **Announce Migration Complete**
    - Email all users with V2 login URL
    - Update documentation links
    - Monitor support channels

## Post-Migration Checklist

After successful migration, complete these tasks:

### Immediate (Day 1)

- [ ] Verify all user accounts can login
- [ ] Test campaign email sending (real SMTP, not MailHog)
- [ ] Confirm location geocoding works
- [ ] Check shift signup flow (public)
- [ ] Verify response wall displays correctly
- [ ] Test admin CRUD operations (create campaign, location, shift)
- [ ] Monitor error logs for exceptions
- [ ] Verify Prometheus metrics collecting

### First Week

- [ ] Review Grafana dashboards for anomalies
- [ ] Check BullMQ job queue (no stuck jobs)
- [ ] Verify geocoding cache hit rate
- [ ] Test all user roles (SUPER_ADMIN, MAP_ADMIN, etc.)
- [ ] Confirm Listmonk sync working (if enabled)
- [ ] Validate backup script runs successfully
- [ ] Review user feedback and support tickets

### First Month

- [ ] Optimize slow queries (check Prometheus API duration metrics)
- [ ] Review disk usage (PostgreSQL, uploads, logs)
- [ ] Audit user permissions (remove temp accounts)
- [ ] Update documentation based on issues encountered
- [ ] Train administrators on new V2 features
- [ ] Plan rollout of new features (landing pages, canvassing)
- [ ] Schedule security audit

## Common Migration Scenarios

### Scenario 1: Small Organization (< 1000 locations)

- **Migration Duration**: 4-6 hours
- **Downtime**: 10 minutes
- **Recommended Approach**:
  - Export V1 data Friday evening
  - Transform and import over weekend
  - Test Saturday/Sunday
  - Switchover Monday morning
  - Rollback window: 48 hours

### Scenario 2: Medium Organization (1000-10000 locations)

- **Migration Duration**: 8-12 hours
- **Downtime**: 15 minutes
- **Recommended Approach**:
  - Set up V2 staging environment 1 week prior
  - Perform test migration on staging
  - Document issues and solutions
  - Schedule production migration for low-traffic period
  - Rollback window: 24 hours

### Scenario 3: Large Organization (10000+ locations)

- **Migration Duration**: 12-20 hours
- **Downtime**: 20-30 minutes
- **Recommended Approach**:
  - Hire professional services (enterprise@cmlite.org)
  - Perform multiple test migrations on staging
  - Use incremental data sync (minimize final catchup)
  - Blue-green deployment (parallel V1/V2 for 1 week)
  - Rollback window: 1 week with data sync

### Scenario 4: Active Campaign During Migration

**Problem**: Can't afford downtime during critical campaign period.

**Solution**:
1. Set up V2 as read-only mirror (import V1 data, disable writes)
2. Continue using V1 for all active operations
3. Schedule final catchup migration after campaign concludes
4. Or: Use blue-green deployment with manual data sync

!!! danger "Active Campaign Warning"
    Do NOT migrate during active campaign periods. Schedule migration between campaigns or during organizational downtime.

## Migration Validation Checklist

Use this checklist to verify successful migration:

### Data Integrity

- [ ] **User count matches**: V1 users = V2 users (excluding duplicates)
- [ ] **Campaign count matches**: V1 campaigns = V2 campaigns
- [ ] **Location count matches**: V1 locations = V2 locations
- [ ] **Shift count matches**: V1 shifts = V2 shifts
- [ ] **Response count matches**: V1 responses = V2 responses
- [ ] **Representative cache count**: V1 reps = V2 reps (approximate, can refresh)

### Functional Testing

- [ ] **Login works**: Test with 5 different user accounts
- [ ] **Password authentication**: All migrated passwords validate correctly
- [ ] **Campaign email sends**: Queue job, verify SMTP delivery
- [ ] **Representative lookup**: Postal code returns correct reps
- [ ] **Location geocoding**: Bulk geocode 10 addresses successfully
- [ ] **Map rendering**: All locations display on map
- [ ] **Shift signup**: Public user can sign up for shift
- [ ] **Response submission**: Can submit and view responses
- [ ] **Admin CRUD**: Create, edit, delete test records

### Performance Testing

- [ ] **Campaign list loads < 2 seconds**: 100+ campaigns
- [ ] **Location map loads < 3 seconds**: 1000+ locations
- [ ] **Search response time < 500ms**: User, campaign, location search
- [ ] **Geocoding batch < 30 seconds**: 100 addresses
- [ ] **Email queue processing**: 10 emails/minute minimum
- [ ] **No N+1 queries**: Check Prisma logs for query count

### Security Testing

- [ ] **JWT authentication works**: Access + refresh token flow
- [ ] **RBAC enforced**: SUPER_ADMIN vs USER vs TEMP roles
- [ ] **Rate limiting active**: Auth endpoints limited to 10/min
- [ ] **Password policy enforced**: 12+ chars, complexity requirements
- [ ] **Redis authenticated**: Connection requires password
- [ ] **Encryption key set**: ENCRYPTION_KEY env var different from JWT secrets

## Troubleshooting Migration Issues

Common problems and solutions:

### Issue: User Login Fails After Migration

**Symptoms**: Users receive "Invalid credentials" error despite correct password.

**Causes**:
- Bcrypt hash corruption during export/import
- Password field length truncation
- Character encoding issues

**Solutions**:
```bash
# Check password hash format in V2
docker compose exec api npx prisma studio
# User table → password field should start with $2b$

# Reset affected user password
docker compose exec api node scripts/reset-password.js user@example.com
```

### Issue: Missing Data After Import

**Symptoms**: User count, campaign count, or location count lower than V1.

**Causes**:
- Incomplete V1 export (pagination issues)
- Transformation script errors (check logs)
- Unique constraint violations (duplicates skipped)

**Solutions**:
```bash
# Compare record counts
docker compose exec api node scripts/compare-counts.js

# Re-run import for specific table
docker compose exec api node scripts/import-data.js --table=users

# Check import logs for errors
docker compose logs api | grep ERROR
```

### Issue: Geocoding Data Lost

**Symptoms**: Locations missing latitude/longitude coordinates.

**Causes**:
- V1 geocoding provider different from V2
- Coordinates not exported from V1
- Transformation script didn't map geocoding fields

**Solutions**:
```bash
# Bulk re-geocode all locations
curl -X POST http://localhost:4000/api/map/locations/bulk-geocode \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Check geocoding provider configuration
docker compose exec api node scripts/test-geocoding.js
```

### Issue: Campaign Emails Not Sending

**Symptoms**: BullMQ queue shows "failed" jobs.

**Causes**:
- SMTP configuration incorrect
- EMAIL_TEST_MODE still enabled (sends to MailHog)
- Nodemailer authentication failure

**Solutions**:
```bash
# Check SMTP configuration
docker compose exec api node scripts/test-smtp.js

# View failed job details
# Visit http://localhost:4000/api/influence/email-queue/stats

# Retry failed jobs
curl -X POST http://localhost:4000/api/influence/email-queue/retry-failed \
  -H "Authorization: Bearer $ADMIN_TOKEN"
```

### Issue: High Memory Usage After Migration

**Symptoms**: V2 services consuming > 4GB RAM, slow response times.

**Causes**:
- Prisma connection pool too large
- Redis cache not evicting old entries
- Large JSON fields in database (campaign data, page blocks)

**Solutions**:
```bash
# Reduce Prisma connection pool
# Edit .env: DATABASE_URL="...?connection_limit=5"

# Clear Redis cache
docker compose exec redis redis-cli FLUSHDB

# Optimize database
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "VACUUM ANALYZE;"
```

## Related Documentation

### Migration Guides

- [Breaking Changes](breaking-changes.md) - Comprehensive V1→V2 differences
- [Data Migration](data-migration.md) - Detailed migration procedures
- [API Changes](api-changes.md) - Endpoint reference mapping
- [Feature Parity](feature-parity.md) - Feature comparison matrix

### V2 Setup Guides

- [Quick Start](../getting-started/quick-start.md) - 5-minute V2 setup
- [Production Deployment](../deployment/production.md) - Production configuration
- [Environment Variables](../deployment/environment.md) - .env reference

### V2 Architecture

- [Architecture Overview](../architecture/index.md) - System design
- [Dual API Design](../architecture/dual-api.md) - Express + Fastify
- [Authentication](../architecture/authentication.md) - JWT flow
- [Database Schema](../database/schema.md) - Prisma models

### Post-Migration

- [Admin Guide](../user-guides/admin-guide.md) - Platform administration
- [Monitoring](../features/observability/monitoring.md) - Prometheus + Grafana
- [Backups](../deployment/backup-restore.md) - Backup procedures
- [Troubleshooting](../troubleshooting/common-errors.md) - Common issues

## Next Steps

Ready to begin migration?

1. **[Review Breaking Changes](breaking-changes.md)** - Understand all V1→V2 differences
2. **[Plan Data Migration](data-migration.md)** - Create migration timeline
3. **[Set Up V2 Staging](../getting-started/quick-start.md)** - Test environment
4. **[Perform Test Migration](data-migration.md#testing-migration)** - Validate process
5. **[Execute Production Migration](data-migration.md#production-migration)** - Go live

!!! success "Migration Support"
    Need help with your migration? Email support@cmlite.org or open a GitHub discussion.