); } ``` --- ## Troubleshooting ### Problem: Jobs Stuck in Pending **Symptoms:** - Jobs created but never start - Status remains "pending" for hours - No "running" jobs visible **Solutions:** 1. **Check worker process running:** ```bash docker compose ps media-api # Should show "Up" status docker compose logs media-api | grep "Job worker" # Should show "Job worker started" ``` 2. **Manually trigger worker:** ```bash # Restart media-api container docker compose restart media-api # Worker starts automatically on container boot ``` 3. **Check worker logs for errors:** ```bash docker compose logs -f media-api | grep ERROR # Look for database connection errors, permission issues ``` 4. **Verify database connection:** ```bash # Test database accessible from container docker compose exec media-api psql $DATABASE_URL -c "SELECT COUNT(*) FROM jobs WHERE status='pending';" ``` --- ### Problem: Job Fails Immediately **Symptoms:** - Job status changes from pending → running → failed within seconds - No meaningful progress - Error in log: "Command not found" or "Permission denied" **Solutions:** 1. **Check job log in database:** ```sql SELECT log FROM jobs WHERE id = 'JOB_ID'; ``` 2. **Verify FFmpeg installed:** ```bash docker compose exec media-api which ffmpeg # Should output: /usr/bin/ffmpeg docker compose exec media-api ffmpeg -version ``` 3. **Check file paths valid:** ```bash # Verify input file exists docker compose exec media-api ls -la /media/local/library/inbox/original.mp4 # Check output directory writable docker compose exec media-api touch /media/local/playback/test.txt ``` 4. **Test FFmpeg command manually:** ```bash # Copy command from job log, run manually docker compose exec media-api ffmpeg -i /media/local/inbox/test.mp4 -c:v libx264 /media/local/playback/test-output.mp4 ``` --- ### Problem: Re-encode Job Hangs at Same Progress **Symptoms:** - Job progress reaches 25%, 50%, or 75% then stops updating - Status remains "running" for hours - No CPU/GPU activity visible **Solutions:** 1. **Check FFmpeg process still running:** ```bash docker compose exec media-api ps aux | grep ffmpeg # Should show ffmpeg process # If not running, worker crashed docker compose logs media-api --tail 100 ``` 2. **Kill hung FFmpeg process:** ```bash docker compose exec media-api pkill -9 ffmpeg # Job will fail and can be retried ``` 3. **Check disk space:** ```bash df -h /media/local/playback # If 100% full, encoding fails # Free space docker compose exec media-api rm /media/local/playback/*.partial ``` 4. **Increase FFmpeg timeout (if very large file):** ```typescript // api/src/modules/media/services/job-worker.service.ts const FFMPEG_TIMEOUT = 3600000; // 1 hour (from 30 minutes) ``` --- ### Problem: GPU Out of Memory Errors **Symptoms:** - Multiple GPU jobs running simultaneously - Error in log: "CUDA out of memory" or "Cannot allocate memory" - System becomes unresponsive **Solutions:** 1. **Check total VRAM available:** ```bash nvidia-smi # Shows GPU memory usage # Should show < 16GB used (adjust based on your GPU) ``` 2. **Reduce concurrent GPU job limit:** ```typescript // api/src/modules/media/services/job-worker.service.ts const limits = { cpu: 5, gpu_encode: 1, // Reduced from 2 gpu_ai: 1, }; ``` 3. **Increase VRAM requirements for jobs:** ```typescript // Jobs require more VRAM than specified // Update job creation to use higher vramRequired values { type: 'reencode_streaming', vramRequired: 6000, // Increased from 4000 } ``` 4. **Kill running GPU jobs:** ```bash # Stop all media jobs docker compose exec media-api pkill -9 ffmpeg # Update stuck jobs to failed status docker compose exec v2-postgres psql -U changemaker -d v2_changemaker \ -c "UPDATE jobs SET status='failed' WHERE status='running';" ``` --- ## Performance Considerations ### Job Queue Throughput **Scaling Factors:** - CPU jobs: 5 concurrent = ~10-20 jobs/minute (scans, validations) - GPU encode: 2 concurrent = ~4-8 videos/hour (depends on length) - GPU AI: 1 concurrent = ~2-6 videos/hour (depends on complexity) **Bottlenecks:** 1. **GPU Memory** — Limits concurrent GPU jobs 2. **Disk I/O** — Reading/writing large video files 3. **CPU** — FFmpeg encoding uses all available cores **Optimization:** - **Distribute workers across multiple machines** — Each machine runs separate worker process - **Use job priority** — Urgent jobs (priority 1-3) run first - **Batch similar jobs** — Group scan jobs, re-encode jobs, etc. for efficiency --- ### Database Performance **Job Queue Index:** ```sql CREATE INDEX idx_jobs_status_priority ON jobs(status, priority, created_at); ``` **Query Performance:** - Find next pending job: ~1-5ms (with index) - Update job status: ~2-10ms - Fetch job logs: ~5-20ms **Optimization:** - **Partition jobs table by date** — Move old completed/failed jobs to archive table - **Limit log size** — Truncate logs > 10KB to prevent bloat --- ## Monitoring & Observability ### Prometheus Metrics ```typescript // api/src/utils/metrics.ts import { Counter, Gauge } from 'prom-client'; export const mediaJobsTotal = new Counter({ name: 'media_jobs_total', help: 'Total media jobs created', labelNames: ['type', 'status'], }); export const mediaJobsPending = new Gauge({ name: 'media_jobs_pending', help: 'Number of pending media jobs', }); export const mediaJobsRunning = new Gauge({ name: 'media_jobs_running', help: 'Number of running media jobs', labelNames: ['resourceCategory'], }); export const mediaVramUsed = new Gauge({ name: 'media_vram_used_mb', help: 'Total VRAM used by running jobs (MB)', }); // Update metrics in worker mediaJobsPending.set(pendingCount); mediaJobsRunning.set({ resourceCategory: 'gpu_encode' }, gpuEncodeCount); mediaVramUsed.set(totalVramUsed); ``` ### Grafana Dashboard Panel **Job Queue Status:** ```promql # Pending jobs count media_jobs_pending # Running jobs by category sum(media_jobs_running) by (resourceCategory) # VRAM usage percentage (media_vram_used_mb / 16000) * 100 ``` **Alert Rules:** ```yaml # configs/prometheus/alerts.yml groups: - name: media_jobs rules: - alert: MediaJobQueueBacklog expr: media_jobs_pending > 50 for: 30m labels: severity: warning annotations: summary: "Media job queue backlog" description: "{{ $value }} jobs pending for 30+ minutes" - alert: MediaJobsStuckRunning expr: sum(media_jobs_running) == 0 AND media_jobs_pending > 0 for: 10m labels: severity: critical annotations: summary: "Media jobs stuck" description: "Jobs pending but worker not processing" ``` --- ## Related Documentation ### Backend Documentation - **Job Worker:** `backend/modules/media/job-worker.md` — Worker process implementation - **Job Processors:** `backend/modules/media/processors/` — Individual job type processors (reencode, scan, etc.) - **Jobs Routes:** `backend/modules/media/jobs.md` — API endpoints for job management ### Frontend Documentation - **Jobs Page:** `frontend/pages/media/jobs.md` — Job queue monitoring UI - **Job Detail Modal:** `frontend/components/media/job-detail.md` — Log viewer component ### Feature Documentation - **Video Library:** `features/media/video-library.md` — Triggering jobs from library actions - **Upload System:** `features/media/upload.md` — Post-upload job creation --- ## Next Steps After mastering the job queue: 1. **Create Custom Jobs** — Implement new job types for domain-specific processing 2. **Optimize Scheduling** — Tune resource limits and priority settings for your workload 3. **Monitor Performance** — Set up Grafana dashboards and alerts for job queue health 4. **Distributed Workers** — Scale horizontally by running workers on multiple machines **Hands-On Practice:** ```bash # 1. Create re-encode job curl -X POST http://localhost:4100/api/media/jobs \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "type": "reencode_streaming", "params": { "videoId": "VIDEO_ID", "targetBitrate": 2000 }, "priority": 5 }' # 2. Monitor job progress watch -n 2 'curl -s http://localhost:4100/api/media/jobs/JOB_ID | jq ".progress"' # 3. View job logs curl http://localhost:4100/api/media/jobs/JOB_ID | jq -r ".log" # 4. Check queue stats curl http://localhost:4100/api/media/jobs/stats | jq ``` --- **Last Updated:** 2026-02-13 **Version:** V2.0 **Maintainer:** Changemaker Lite Team