setFilters({ ...filters, search: value })} allowClear /> setPagination({ ...pagination, page }), }} /> ); } ``` --- ## Troubleshooting ### Problem: Media API Not Accessible **Symptoms:** - Admin GUI shows "Cannot connect to media API" - Browser console shows CORS errors or network failures - Public gallery doesn't load **Solutions:** 1. **Check Fastify server running:** ```bash docker compose ps media-api # Should show "Up" status docker compose logs media-api # Look for "Fastify server listening on port 4100" ``` 2. **Verify port 4100 not in use:** ```bash lsof -i :4100 # Should show only media-api container # If another process using port, stop it or change MEDIA_API_PORT in .env ``` 3. **Check nginx proxy configuration:** ```nginx # nginx/conf.d/api.conf # Media API block must come BEFORE general API block server { listen 80; server_name media.cmlite.org; location / { proxy_pass http://localhost:4100; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } } ``` 4. **Test direct API access:** ```bash # From host machine curl http://localhost:4100/api/media/videos # From inside container docker compose exec media-api curl http://localhost:4100/api/media/videos ``` 5. **Check Docker networking:** ```bash docker network inspect changemaker-lite # Verify media-api container connected ``` --- ### Problem: Scan Finds No Videos **Symptoms:** - Scan completes with "Created 0 new records" - Directory known to contain video files - Scan reports 0 files scanned **Solutions:** 1. **Verify MEDIA_LIBRARY_PATH correct:** ```bash # Check environment variable docker compose exec media-api printenv MEDIA_LIBRARY_PATH # Should output: /media/local/library # List directory contents docker compose exec media-api ls -la /media/local/library/videos # Should show video files ``` 2. **Check directory exists:** ```bash # Create missing directory docker compose exec media-api mkdir -p /media/local/library/videos # Copy test videos docker cp test.mp4 $(docker compose ps -q media-api):/media/local/library/videos/ ``` 3. **Verify Docker volume mounted:** ```yaml # docker-compose.yml services: media-api: volumes: - /media/local/library:/media/local/library:ro # Check path correct ``` ```bash # Inspect volume mounts docker compose config | grep -A 5 media-api ``` 4. **Check file extensions supported:** Only these extensions scanned: - `.mp4` - `.mov` - `.avi` - `.mkv` - `.webm` - `.m4v` - `.flv` Rename files if using other extensions: ```bash # Rename .MP4 to .mp4 (case-sensitive) docker compose exec media-api sh -c 'cd /media/local/library/videos && rename "s/.MP4$/.mp4/" *.MP4' ``` 5. **Check file permissions:** ```bash # Verify readable by container user docker compose exec media-api ls -la /media/local/library/videos # Fix permissions if needed (on host) sudo chmod -R 755 /media/local/library ``` --- ### Problem: FFprobe Validation Fails **Symptoms:** - Validation returns error "FFprobe command failed" - Videos marked `isValid = false` - Timeout errors after 30 seconds **Solutions:** 1. **Check FFmpeg installed in container:** ```bash # Verify FFprobe available docker compose exec media-api which ffprobe # Should output: /usr/bin/ffprobe docker compose exec media-api ffprobe -version # Should show FFmpeg version info ``` 2. **Install FFmpeg if missing:** ```dockerfile # api/Dockerfile.media FROM node:20-alpine # Install FFmpeg (both dev and production stages) RUN apk add --no-cache ffmpeg # ... rest of Dockerfile ``` ```bash # Rebuild container docker compose build media-api docker compose up -d media-api ``` 3. **Test FFprobe directly on video:** ```bash # Run FFprobe manually docker compose exec media-api ffprobe -v quiet -print_format json -show_streams -show_format /media/local/library/videos/test.mp4 # If this fails, video file corrupt or unsupported ``` 4. **Check timeout not exceeded:** Default timeout: 30 seconds ```bash # For very large files (>5GB), increase timeout # api/src/modules/media/services/ffprobe.service.ts const FFPROBE_TIMEOUT = 60000; // 60 seconds ``` 5. **Verify video file not corrupt:** ```bash # Test playback docker compose exec media-api ffplay /media/local/library/videos/test.mp4 # Or copy to host and test in VLC docker cp $(docker compose ps -q media-api):/media/local/library/videos/test.mp4 ./test.mp4 vlc test.mp4 ``` 6. **Check for special characters in filename:** ```bash # Rename files with spaces or special chars docker compose exec media-api sh -c 'cd /media/local/library/videos && rename "s/ /_/g" *.mp4' ``` --- ### Problem: Drizzle Schema Changes Not Applied **Symptoms:** - Code references new column but database doesn't have it - Error: "column does not exist" - Schema changes made but not reflected **Solutions:** 1. **Push schema changes:** ```bash # Drizzle uses push (not migrations) cd api npx drizzle-kit push # Confirm changes ``` 2. **Verify connection:** ```bash # Check DATABASE_URL correct docker compose exec media-api printenv DATABASE_URL # Test connection docker compose exec media-api npx drizzle-kit studio # Opens DB browser on http://localhost:4983 ``` 3. **Compare with Prisma migrations:** Media tables exist in same database as Prisma tables. If conflict: ```bash # Check both schemas npx prisma db pull # Prisma introspection npx drizzle-kit introspect # Drizzle introspection # Resolve conflicts manually ``` --- ### Problem: Large Library Performance **Symptoms:** - Library page loads slowly (5+ seconds) - Pagination sluggish - Scan operations timeout **Solutions:** 1. **Add database indexes:** ```sql -- Index for common filters CREATE INDEX idx_videos_directory_type ON videos(directory_type); CREATE INDEX idx_videos_orientation ON videos(orientation); CREATE INDEX idx_videos_quality ON videos(quality); CREATE INDEX idx_videos_is_valid ON videos(is_valid); CREATE INDEX idx_videos_created_at ON videos(created_at DESC); -- Composite index for filtered queries CREATE INDEX idx_videos_filters ON videos(directory_type, is_valid, created_at DESC); -- Full-text search index CREATE INDEX idx_videos_search ON videos USING gin(to_tsvector('english', coalesce(title, '') || ' ' || coalesce(producer, '') || ' ' || coalesce(creator, ''))); ``` 2. **Reduce page size:** ```typescript // admin/src/pages/media/LibraryPage.tsx const [pagination, setPagination] = useState({ page: 1, limit: 10, total: 0 }); // Reduced from 20 to 10 ``` 3. **Enable query caching:** ```typescript // api/src/modules/media/routes/videos.routes.ts import { redisClient } from '@/config/redis'; app.get('/api/media/videos', async (req, reply) => { const cacheKey = `videos:list:${JSON.stringify(req.query)}`; // Check cache const cached = await redisClient.get(cacheKey); if (cached) { return reply.send(JSON.parse(cached)); } // Fetch from database const results = await db.select()...; // Cache for 5 minutes await redisClient.setex(cacheKey, 300, JSON.stringify(results)); reply.send(results); }); ``` 4. **Use virtual scrolling:** ```typescript // Replace Ant Design Table with react-window for large datasets import { FixedSizeList } from 'react-window'; ``` --- ## Performance Considerations ### Directory Scans **Scaling Factors:** - 100 files: ~2 seconds - 1,000 files: ~15 seconds - 10,000 files: ~2.5 minutes **Optimization Strategies:** 1. **Incremental Scans** — Use `skipExisting: true` to only process new files 2. **Parallel Processing** — Scan multiple directories simultaneously 3. **Background Jobs** — Queue scans as async jobs instead of synchronous requests 4. **Caching** — Cache directory listings in Redis ### FFprobe Extraction **Timing:** - Small video (<100MB): ~50-100ms - Medium video (500MB): ~150-250ms - Large video (2GB+): ~500ms-1s **Batch Processing:** For 100 videos: ~10-20 seconds total **Optimization:** ```typescript // Parallel extraction (limit concurrency) import pLimit from 'p-limit'; const limit = pLimit(5); // Max 5 concurrent FFprobe calls const results = await Promise.all( videoFiles.map((file) => limit(() => ffprobeService.extract(file)) ) ); ``` ### Database Queries **Query Performance:** - List 20 videos (no filters): ~5-10ms - List 20 videos (with filters): ~10-20ms - Full-text search: ~20-50ms - Count total videos: ~5ms (with index) **Optimization:** 1. **Always use pagination** — Never fetch all records 2. **Index heavily filtered columns** — directoryType, orientation, quality, isValid 3. **Use SELECT only needed columns** — Avoid `SELECT *` for large tables 4. **Cache counts** — Total video count changes infrequently, cache in Redis ### Thumbnail Generation **Deferred Loading:** Don't generate thumbnails during scan. Instead: 1. Create video record without thumbnail 2. Queue thumbnail generation job 3. Worker processes job asynchronously 4. Update record with `thumbnailPath` **Lazy Loading:** Frontend requests thumbnails only when visible (IntersectionObserver). --- ## Dual API Architecture ### Why Separate Fastify API? The media system was introduced as a **Phase 14 enhancement** after V2 core functionality stabilized. A separate Fastify microservice was chosen to: 1. **Avoid Disrupting Stable Express API** — V2 Express API battle-tested with 30+ models, introducing media directly risked regressions 2. **Test Drizzle ORM Migration** — Fastify+Drizzle serves as proof-of-concept for potential future Prisma→Drizzle migration 3. **Isolate Video Processing** — CPU/GPU-intensive FFprobe, encoding jobs isolated from main API request handling 4. **Independent Scaling** — Media API can be horizontally scaled separately based on video processing load 5. **Technology Experimentation** — Fastify's performance benefits evaluated for potential broader adoption ### Database Sharing Strategy **Same PostgreSQL, Different ORMs:** ``` ┌─────────────────┐ │ PostgreSQL 16 │ │ v2_changemaker │ └─────────────────┘ ↑ ┌────┴────┐ │ │ ┌───┴───┐ ┌──┴────┐ │Prisma │ │Drizzle│ │ ORM │ │ ORM │ └───┬───┘ └──┬────┘ │ │ ┌───┴────┐ ┌─┴─────┐ │Express │ │Fastify│ │ API │ │ Media │ │ :4000 │ │ API │ │ │ │ :4100 │ └────────┘ └───────┘ ``` **Benefits:** - **Single Source of Truth** — All data in one database - **Cross-API Queries** — Main API can query media tables via Prisma raw queries - **Unified Backups** — One PostgreSQL dump includes both APIs - **Shared Connections** — Connection pooling optimizations benefit both **Challenges:** - **Schema Coordination** — Must manually sync schema changes between Prisma migrations and Drizzle pushes - **Type Conflicts** — Same table, different type definitions (Prisma vs Drizzle types) - **Migration Complexity** — Prisma generates migrations, Drizzle uses push (no migration files) ### Migration Strategy Roadmap **Short Term (Current):** - Keep dual API architecture - Synchronize schemas manually - Document shared tables in both ORMs **Medium Term (6-12 months):** - Evaluate Fastify+Drizzle performance vs Express+Prisma - If Fastify superior, migrate select Express routes to Fastify - If no significant benefit, consolidate media into Express+Prisma **Long Term (12+ months):** - Unified API (either all Express or all Fastify) - Single ORM (either all Prisma or all Drizzle) - Deprecate less performant stack **Migration Effort Estimate:** - **Media to Express+Prisma:** 3-5 days (convert Drizzle queries to Prisma, merge Fastify routes into Express) - **All to Fastify+Drizzle:** 2-3 weeks (convert 30+ Prisma models to Drizzle, rewrite Express routes for Fastify) --- ## Related Documentation ### Backend Documentation - **API Server:** `backend/api/media-server.md` — Fastify server setup, middleware, error handling - **Videos Module:** `backend/modules/media/videos.md` — Video routes, service layer, business logic - **FFprobe Service:** `backend/modules/media/ffprobe.md` — Metadata extraction implementation - **Jobs System:** `backend/modules/media/jobs.md` — Job queue architecture, worker processes ### Frontend Documentation - **Library Page:** `frontend/pages/media/library.md` — Video library management UI - **Shared Media Page:** `frontend/pages/media/shared.md` — Public gallery admin UI - **Media Components:** `frontend/components/media.md` — Reusable video components ### Database Documentation - **Media Models:** `database/models/media.md` — Drizzle schema definitions for videos, compilations, jobs - **Drizzle Setup:** `database/drizzle.md` — Drizzle ORM configuration, connection management ### Feature Documentation - **Video Upload:** `features/media/upload.md` — Upload system workflow, FFprobe integration - **Media Jobs:** `features/media/jobs.md` — Job queue system, processing pipeline - **Public Gallery:** `features/media/public-gallery.md` — Public video sharing system ### Integration Documentation - **Dual API Architecture:** `architecture/dual-api.md` — Express+Prisma vs Fastify+Drizzle comparison - **Nginx Routing:** `deployment/nginx.md` — Reverse proxy configuration for media.cmlite.org - **Docker Setup:** `deployment/docker.md` — Media API container, volume mounts, healthchecks --- ## Next Steps After mastering video library management: 1. **Upload System** — Read `features/media/upload.md` to understand video upload workflow 2. **Jobs Queue** — Review `features/media/jobs.md` for video processing automation 3. **Public Gallery** — Explore `features/media/public-gallery.md` for sharing videos publicly 4. **Custom Integrations** — Use Media API endpoints to build custom video features For hands-on practice, try: ```bash # 1. Upload test videos curl -X POST http://localhost:4100/api/media/upload/single \ -H "Authorization: Bearer YOUR_TOKEN" \ -F "video=@test.mp4" \ -F "producer=Test Studio" \ -F "title=Test Video" # 2. Scan directory curl -X POST http://localhost:4100/api/media/videos/scan \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"directoryType": "videos"}' # 3. List videos curl http://localhost:4100/api/media/videos?page=1&limit=10 # 4. Validate video curl -X POST http://localhost:4100/api/media/videos/VIDEO_ID/validate \ -H "Authorization: Bearer YOUR_TOKEN" ``` --- **Last Updated:** 2026-02-13 **Version:** V2.0 **Maintainer:** Changemaker Lite Team