admin/changemaker.lite
1
0
Fork 0
Code Issues Packages Projects Releases 31 Wiki Activity
changemaker.lite/mkdocs/docs/v2/migration/index.md

20 KiB
Raw Blame History

Migration Guide: V1 to V2 Overview

This comprehensive guide covers the complete migration process from Changemaker Lite V1 to V2, including architectural changes, data migration, and rollback procedures.

Overview

Changemaker Lite V2 is a complete rebuild of the platform, not an incremental upgrade. The migration represents a fundamental shift in architecture, technology stack, and approach to campaign management.

V1 vs V2 at a Glance

Aspect V1 V2
Architecture Two separate Express apps Single unified Express + Fastify API
Data Layer NocoDB REST API Prisma ORM + PostgreSQL 16
Frontend Embedded EJS templates React SPA (Vite + Ant Design)
Authentication Session cookies + bcrypt JWT tokens (access + refresh)
API Style REST via NocoDB REST with Zod validation
State Management Server-side sessions Zustand client state + JWT
Job Queue Bull (Redis) BullMQ (Redis)
Database NocoDB tables Prisma migrations
Email Nodemailer + Bull BullMQ + Listmonk integration
Ports 3333 (influence), 3000 (map) 4000 (API), 3000 (admin)

Why Migrate to V2?

Technical Benefits

  1. Unified Codebase: Single API codebase instead of two separate applications
  2. Type Safety: Full TypeScript coverage with Prisma type generation
  3. Modern Stack: Latest React, Vite build tooling, Ant Design components
  4. Better Performance: Direct database access via Prisma vs REST API abstraction
  5. Improved Security: JWT refresh token rotation, RBAC, comprehensive audit trail
  6. Scalability: Separation of concerns (dual API architecture for media)
  7. Developer Experience: Hot reload, better tooling, comprehensive documentation

Feature Enhancements

  1. New Features:

    • Landing page builder with GrapesJS
    • Email template system with versioning
    • Media library with video uploads and reactions
    • Volunteer canvassing system with GPS tracking
    • Data quality dashboard for geocoding
    • Comprehensive monitoring (Prometheus + Grafana)
    • NAR 2025 electoral data import
    • Pangolin tunnel integration

  2. Enhanced Existing Features:

    • Response wall with upvoting and moderation
    • Multi-provider geocoding (6 providers)
    • Advanced shift management with cut assignments
    • Printable walk sheets with QR codes
    • Listmonk newsletter sync

  3. Improved Admin Experience:

    • Modern React UI with consistent design
    • Real-time updates with optimistic UI
    • Advanced filtering and search
    • Bulk operations
    • Responsive mobile support

Migration Timeline

Planned Phases (from V2_PLAN.md)

  • Phase 1-14: ✅ COMPLETE (Foundation through Monitoring)
  • Phase 15: 🚧 Testing + Polish (current)

Actual Development Timeline

  • 2025-01: V2 rebuild initiated (clean-room approach)
  • 2025-02: Security audit completed, NAR import, media upload
  • 2026-02: Phase 14 complete, ready for production migration

Migration Duration Estimate

Migration Step Duration Downtime Required
V1 data export 1-2 hours No
Data transformation 2-4 hours No
V2 database setup 30 minutes No
V2 data import 1-3 hours No
Testing & validation 2-4 hours No
DNS/service switchover 15 minutes Yes
Post-migration verification 1 hour No
Total 8-15 hours 15 minutes

!!! tip "Minimize Downtime" Perform all data export, transformation, and testing on a separate V2 staging environment. Only switch production traffic after full validation.

Risk Assessment

High Risk Areas

  1. Data Loss

    • Risk: Campaign data, locations, or user accounts lost during migration
    • Mitigation: Full V1 backup before migration, validation checksums, rollback plan
    • Impact: High (business-critical data)

  2. Authentication Disruption

    • Risk: Users unable to login after migration (password hash incompatibility)
    • Mitigation: Test password migration with sample users, password reset flow ready
    • Impact: High (blocks all access)

  3. Email Delivery Failure

    • Risk: Campaign emails stop sending after migration
    • Mitigation: Test SMTP configuration, BullMQ queue verification, MailHog testing
    • Impact: High (core feature)

Medium Risk Areas

  1. Representative Data

    • Risk: Cached representative data doesn't migrate correctly
    • Mitigation: Cache can be rebuilt from Represent API, non-critical
    • Impact: Medium (cacheable data)

  2. Location Geocoding

    • Risk: Geocoded coordinates lost or corrupted
    • Mitigation: V2 multi-provider geocoding can re-geocode, bulk geocode endpoint
    • Impact: Medium (can be re-geocoded)

  3. Shift Signups

    • Risk: Volunteer shift assignments lost
    • Mitigation: Export signups separately, manual verification, confirmation emails
    • Impact: Medium (time-sensitive data)

Low Risk Areas

  1. Response Wall Data

    • Risk: Public responses or upvotes lost
    • Mitigation: CSV export, manual re-entry if needed
    • Impact: Low (public-facing only)

  2. Custom Settings

    • Risk: V1 settings don't map to V2 schema
    • Mitigation: Manual reconfiguration in V2 SettingsPage
    • Impact: Low (quick to reconfigure)

Rollback Plan

If Migration Fails

  1. Immediate Actions (within 15 minutes):

    # Stop V2 services
docker compose down

# Restore V1 services
docker compose -f docker-compose.v1.yml up -d

# Restore DNS (point back to V1)
# Update tunnel/proxy configuration

  2. Data Restoration (if V2 data was modified):

    # Restore V1 database from backup
docker compose -f docker-compose.v1.yml exec -T v1-postgres \
  psql -U nocodb nocodb < backups/v1-nocodb-backup.sql

# Verify data integrity
docker compose -f docker-compose.v1.yml logs -f

  3. Verification:

    • Test V1 login
    • Verify campaign data visible
    • Check location map loads
    • Send test campaign email
    • Verify response wall displays

Rollback Window

  • First 24 hours: Simple rollback (V1 backup unchanged)
  • After 24 hours: Complex rollback (may need to merge V2 changes back to V1)
  • After 1 week: Rollback not recommended (significant V2 data divergence)

!!! warning "Rollback Deadline" Plan your migration with a clear rollback deadline. After this window, V2 becomes the source of truth.

Support Resources

Documentation

Community & Support

  • GitHub Issues: Report bugs or migration problems
  • Discussions: Ask questions, share migration experiences
  • Email: support@cmlite.org for direct assistance

Professional Services

For organizations requiring:

  • Custom data migration scripts
  • Zero-downtime migration
  • Training for administrators
  • Priority support during migration

Contact: enterprise@cmlite.org

Prerequisites

Before beginning migration, ensure you have:

V1 Environment

  • V1 backup completed (database + uploads)
  • V1 environment variables documented (.env file)
  • V1 access credentials (NocoDB admin, database passwords)
  • V1 running and healthy (all services operational)
  • V1 data export tested (able to export NocoDB tables)

V2 Environment

  • V2 repository cloned (git checkout v2)
  • Docker and Docker Compose installed (20.10+, 2.0+)
  • PostgreSQL 16 compatible (for V2 database)
  • 4GB+ RAM available (8GB recommended)
  • 20GB+ disk space (for database + uploads)

Migration Planning

  • Downtime window scheduled (notify users)
  • Rollback plan reviewed (tested on staging)
  • Team assigned (minimum 2 people recommended)
  • Backup storage ready (S3 bucket or local storage)
  • Testing checklist prepared (critical workflows to verify)

Migration Steps Overview

This is a high-level overview. Detailed steps are in Data Migration.

Phase 1: Preparation (No Downtime)

  1. Export V1 Data

    # Export all NocoDB tables to JSON
./scripts/export-v1-data.sh

# Backup file uploads
tar -czf v1-uploads.tar.gz ./uploads/

  2. Set Up V2 Environment

    git checkout v2
cp .env.example .env
# Edit .env with V2 configuration

  3. Start V2 Services (parallel to V1)

    docker compose up -d v2-postgres redis
docker compose exec api npx prisma migrate deploy

Phase 2: Data Transformation (No Downtime)

  1. Transform V1 Data for V2

    # Run transformation scripts
node scripts/transform-users.js
node scripts/transform-campaigns.js
node scripts/transform-locations.js

  2. Import into V2 Database

    # Import transformed data
docker compose exec api node scripts/import-data.js

  3. Validate Data Integrity

    # Compare record counts
docker compose exec api node scripts/validate-migration.js

Phase 3: Testing (No Downtime)

  1. Test V2 Functionality

    • Login with test users (verify password migration)
    • View campaigns, locations, shifts
    • Submit test response
    • Send test email
    • Check admin permissions

  2. Performance Testing

    • Load campaigns page (check query performance)
    • Geocode sample addresses
    • Test map rendering with all locations
    • Verify Redis caching

Phase 4: Switchover (15 Minutes Downtime)

  1. Enable Maintenance Mode (V1)

    # Stop V1 services
docker compose -f docker-compose.v1.yml down

  2. Start V2 Services

    # Start all V2 services
docker compose up -d

  3. Update DNS/Proxy

    • Point cmlite.org to V2 nginx
    • Update Pangolin tunnel endpoint
    • Verify SSL certificates

Phase 5: Verification (Post-Migration)

  1. Smoke Tests

    • Admin login works
    • Campaign list loads
    • Location map renders
    • Email sending functional
    • Response wall displays

  2. Monitor for Issues

    # Watch logs for errors
docker compose logs -f api admin

# Check metrics
open http://localhost:3001  # Grafana

  3. Announce Migration Complete

    • Email all users with V2 login URL
    • Update documentation links
    • Monitor support channels

Post-Migration Checklist

After successful migration, complete these tasks:

Immediate (Day 1)

  • Verify all user accounts can login
  • Test campaign email sending (real SMTP, not MailHog)
  • Confirm location geocoding works
  • Check shift signup flow (public)
  • Verify response wall displays correctly
  • Test admin CRUD operations (create campaign, location, shift)
  • Monitor error logs for exceptions
  • Verify Prometheus metrics collecting

First Week

  • Review Grafana dashboards for anomalies
  • Check BullMQ job queue (no stuck jobs)
  • Verify geocoding cache hit rate
  • Test all user roles (SUPER_ADMIN, MAP_ADMIN, etc.)
  • Confirm Listmonk sync working (if enabled)
  • Validate backup script runs successfully
  • Review user feedback and support tickets

First Month

  • Optimize slow queries (check Prometheus API duration metrics)
  • Review disk usage (PostgreSQL, uploads, logs)
  • Audit user permissions (remove temp accounts)
  • Update documentation based on issues encountered
  • Train administrators on new V2 features
  • Plan rollout of new features (landing pages, canvassing)
  • Schedule security audit

Common Migration Scenarios

Scenario 1: Small Organization (< 1000 locations)

  • Migration Duration: 4-6 hours
  • Downtime: 10 minutes
  • Recommended Approach:
    • Export V1 data Friday evening
    • Transform and import over weekend
    • Test Saturday/Sunday
    • Switchover Monday morning
    • Rollback window: 48 hours

Scenario 2: Medium Organization (1000-10000 locations)

  • Migration Duration: 8-12 hours
  • Downtime: 15 minutes
  • Recommended Approach:
    • Set up V2 staging environment 1 week prior
    • Perform test migration on staging
    • Document issues and solutions
    • Schedule production migration for low-traffic period
    • Rollback window: 24 hours

Scenario 3: Large Organization (10000+ locations)

  • Migration Duration: 12-20 hours
  • Downtime: 20-30 minutes
  • Recommended Approach:
    • Hire professional services (enterprise@cmlite.org)
    • Perform multiple test migrations on staging
    • Use incremental data sync (minimize final catchup)
    • Blue-green deployment (parallel V1/V2 for 1 week)
    • Rollback window: 1 week with data sync

Scenario 4: Active Campaign During Migration

Problem: Can't afford downtime during critical campaign period.

Solution:

  1. Set up V2 as read-only mirror (import V1 data, disable writes)
  2. Continue using V1 for all active operations
  3. Schedule final catchup migration after campaign concludes
  4. Or: Use blue-green deployment with manual data sync

!!! danger "Active Campaign Warning" Do NOT migrate during active campaign periods. Schedule migration between campaigns or during organizational downtime.

Migration Validation Checklist

Use this checklist to verify successful migration:

Data Integrity

  • User count matches: V1 users = V2 users (excluding duplicates)
  • Campaign count matches: V1 campaigns = V2 campaigns
  • Location count matches: V1 locations = V2 locations
  • Shift count matches: V1 shifts = V2 shifts
  • Response count matches: V1 responses = V2 responses
  • Representative cache count: V1 reps = V2 reps (approximate, can refresh)

Functional Testing

  • Login works: Test with 5 different user accounts
  • Password authentication: All migrated passwords validate correctly
  • Campaign email sends: Queue job, verify SMTP delivery
  • Representative lookup: Postal code returns correct reps
  • Location geocoding: Bulk geocode 10 addresses successfully
  • Map rendering: All locations display on map
  • Shift signup: Public user can sign up for shift
  • Response submission: Can submit and view responses
  • Admin CRUD: Create, edit, delete test records

Performance Testing

  • Campaign list loads < 2 seconds: 100+ campaigns
  • Location map loads < 3 seconds: 1000+ locations
  • Search response time < 500ms: User, campaign, location search
  • Geocoding batch < 30 seconds: 100 addresses
  • Email queue processing: 10 emails/minute minimum
  • No N+1 queries: Check Prisma logs for query count

Security Testing

  • JWT authentication works: Access + refresh token flow
  • RBAC enforced: SUPER_ADMIN vs USER vs TEMP roles
  • Rate limiting active: Auth endpoints limited to 10/min
  • Password policy enforced: 12+ chars, complexity requirements
  • Redis authenticated: Connection requires password
  • Encryption key set: ENCRYPTION_KEY env var different from JWT secrets

Troubleshooting Migration Issues

Common problems and solutions:

Issue: User Login Fails After Migration

Symptoms: Users receive "Invalid credentials" error despite correct password.

Causes:

  • Bcrypt hash corruption during export/import
  • Password field length truncation
  • Character encoding issues

Solutions:

# Check password hash format in V2
docker compose exec api npx prisma studio
# User table → password field should start with $2b$

# Reset affected user password
docker compose exec api node scripts/reset-password.js user@example.com

Issue: Missing Data After Import

Symptoms: User count, campaign count, or location count lower than V1.

Causes:

  • Incomplete V1 export (pagination issues)
  • Transformation script errors (check logs)
  • Unique constraint violations (duplicates skipped)

Solutions:

# Compare record counts
docker compose exec api node scripts/compare-counts.js

# Re-run import for specific table
docker compose exec api node scripts/import-data.js --table=users

# Check import logs for errors
docker compose logs api | grep ERROR

Issue: Geocoding Data Lost

Symptoms: Locations missing latitude/longitude coordinates.

Causes:

  • V1 geocoding provider different from V2
  • Coordinates not exported from V1
  • Transformation script didn't map geocoding fields

Solutions:

# Bulk re-geocode all locations
curl -X POST http://localhost:4000/api/map/locations/bulk-geocode \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Check geocoding provider configuration
docker compose exec api node scripts/test-geocoding.js

Issue: Campaign Emails Not Sending

Symptoms: BullMQ queue shows "failed" jobs.

Causes:

  • SMTP configuration incorrect
  • EMAIL_TEST_MODE still enabled (sends to MailHog)
  • Nodemailer authentication failure

Solutions:

# Check SMTP configuration
docker compose exec api node scripts/test-smtp.js

# View failed job details
# Visit http://localhost:4000/api/influence/email-queue/stats

# Retry failed jobs
curl -X POST http://localhost:4000/api/influence/email-queue/retry-failed \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Issue: High Memory Usage After Migration

Symptoms: V2 services consuming > 4GB RAM, slow response times.

Causes:

  • Prisma connection pool too large
  • Redis cache not evicting old entries
  • Large JSON fields in database (campaign data, page blocks)

Solutions:

# Reduce Prisma connection pool
# Edit .env: DATABASE_URL="...?connection_limit=5"

# Clear Redis cache
docker compose exec redis redis-cli FLUSHDB

# Optimize database
docker compose exec v2-postgres psql -U changemaker -d changemaker_v2 -c "VACUUM ANALYZE;"

Related Documentation

Migration Guides

V2 Setup Guides

V2 Architecture

Post-Migration

Next Steps

Ready to begin migration?

  1. Review Breaking Changes - Understand all V1→V2 differences
  2. Plan Data Migration - Create migration timeline
  3. Set Up V2 Staging - Test environment
  4. Perform Test Migration - Validate process
  5. Execute Production Migration - Go live

!!! success "Migration Support" Need help with your migration? Email support@cmlite.org or open a GitHub discussion.