bunker-admin 7895ce683e Tonne of debugging - getting ready for the production builds

2026-02-16 10:44:18 -07:00

33 KiB

Raw Blame History

Video Upload System

Overview

The Video Upload System provides a modern drag-and-drop interface for uploading video files with automatic metadata extraction, progress tracking, and batch processing capabilities. Built on Fastify's multipart plugin with FFprobe integration, it supports large files up to 10GB while maintaining server stability through streaming.

Key Features:

Drag-and-Drop Interface — Intuitive file selection with visual drop zone
Automatic Metadata Extraction — FFprobe extracts duration, dimensions, orientation, quality, and audio detection
Single & Batch Upload — Upload one video or queue multiple files
Large File Support — Handles files up to 10GB via streaming (no memory buffering)
Progress Tracking — Real-time upload progress with percentage and speed
Format Validation — Supports MP4, MOV, AVI, MKV, WebM, M4V, FLV
UUID Filenames — Prevents conflicts and path traversal attacks
Inbox Staging — Videos uploaded to /inbox directory before processing
Manual Metadata — Admin can override auto-detected fields (producer, creator, title, tags)

Technology Stack:

Frontend: Ant Design Upload component with custom drag-drop styling
Backend: Fastify @fastify/multipart plugin for streaming uploads
Metadata: FFprobe for video analysis (duration, dimensions, codec, bitrate)
Storage: Direct filesystem writes to /media/local/inbox directory

Architecture

sequenceDiagram
    participant U as User
    participant UI as UploadVideoModal
    participant API as Fastify Media API
    participant FS as Filesystem
    participant FFP as FFprobe Service
    participant DB as PostgreSQL

    U->>UI: Drag video file(s)
    UI->>UI: Validate file type/size
    UI->>U: Show file in queue

    U->>UI: Click "Upload"
    UI->>API: POST /api/media/upload/single<br/>(multipart/form-data)

    API->>API: Generate UUID filename
    API->>FS: Stream to /inbox/{uuid}.mp4
    FS-->>API: Write complete

    API->>FFP: Extract metadata
    FFP->>FS: Analyze video file
    FFP-->>API: Return metadata JSON

    API->>DB: INSERT video record
    DB-->>API: Return video ID

    API-->>UI: Upload success + metadata
    UI-->>U: Show success message
    UI->>UI: Refresh library table

    Note over API,FS: File remains in /inbox<br/>until moved by admin

Upload Flow:

Client Validation — Browser checks file extension and size before upload
Streaming Upload — File streamed to disk in chunks (no memory buffer)
Metadata Extraction — FFprobe analyzes video (30s timeout)
Database Record — Video record created with auto-detected metadata
Response — Frontend receives video ID and metadata
Library Update — Table refreshes to show new video

Key Design Decisions:

Streaming vs Buffering — Streaming prevents memory exhaustion on large files (10GB would require 10GB RAM if buffered)
Inbox Staging — New uploads go to /inbox directory instead of final location, allowing admin review before publishing
UUID Filenames — Prevents filename conflicts and path traversal attacks (../../etc/passwd.mp4)
Synchronous FFprobe — Metadata extracted immediately (not deferred to job queue) for instant feedback

Upload Workflow

User Workflow (Admin)

Open Upload Modal
- Navigate to Media → Library page
- Click "Upload Video" button in top toolbar
- Modal opens with drag-drop zone
Select Files
- Drag files from desktop into blue dashed zone
- OR click "Click to browse" link to open file picker
- Multiple files can be selected for batch upload
Review Queue
- Selected files appear in list with:
  - Filename and size
  - File type icon
  - Remove button (X)
- Invalid files (wrong extension, too large) highlighted in red
Enter Metadata (Optional)
- Producer — Studio or production company name
- Creator — Director or primary creator
- Title — Display title (defaults to filename if blank)
- Tags — Comma-separated tags (e.g., "action, sports, highlight")
Upload
- Click "Upload" button
- Files upload sequentially (not parallel)
- Progress bar shows:
  - Current file name
  - Upload percentage (0-100%)
  - Upload speed (MB/s)
  - Estimated time remaining
Metadata Extraction
- After upload completes, FFprobe runs automatically
- Spinner shows "Extracting metadata..."
- Auto-fills: duration, dimensions, orientation, quality, audio
Success
- Green checkmark appears
- Success message: "Uploaded: {filename}"
- Modal can be closed or kept open for more uploads
- Library table refreshes showing new video

Error Handling

Invalid File Type:

Error: File type not supported
Allowed: MP4, MOV, AVI, MKV, WebM, M4V, FLV

File Too Large:

Error: File exceeds 10GB limit
Selected file: 12.5 GB

Upload Failed:

Error: Upload failed
Network error or server unavailable

FFprobe Extraction Failed:

Warning: Metadata extraction failed
Video uploaded but metadata incomplete
You can manually enter duration and dimensions

API Endpoints

Upload Single Video

POST /api/media/upload/single
Content-Type: multipart/form-data
Authorization: Bearer <admin_token>

Request (Multipart Form Data):

--boundary
Content-Disposition: form-data; name="video"; filename="my-video.mp4"
Content-Type: video/mp4

<binary video data>
--boundary
Content-Disposition: form-data; name="producer"

Studio A
--boundary
Content-Disposition: form-data; name="creator"

Director B
--boundary
Content-Disposition: form-data; name="title"

My Awesome Video
--boundary
Content-Disposition: form-data; name="tags"

action,sports,highlight
--boundary--

Response (Success):

{
  "id": "660e8400-e29b-41d4-a716-446655440000",
  "path": "inbox/660e8400-e29b-41d4-a716-446655440000.mp4",
  "filename": "660e8400-e29b-41d4-a716-446655440000.mp4",
  "originalFilename": "my-video.mp4",
  "directoryType": "inbox",
  "producer": "Studio A",
  "creator": "Director B",
  "title": "My Awesome Video",
  "tags": ["action", "sports", "highlight"],
  "durationSeconds": 125,
  "width": 1920,
  "height": 1080,
  "quality": "FHD",
  "orientation": "landscape",
  "hasAudio": true,
  "fileSize": 45678912,
  "isValid": true,
  "createdAt": "2026-02-13T14:30:00Z"
}

Response (Error):

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Invalid file type. Allowed: mp4, mov, avi, mkv, webm, m4v, flv"
}

Upload Batch (Multiple Videos)

POST /api/media/upload/batch
Content-Type: multipart/form-data
Authorization: Bearer <admin_token>

Request:

--boundary
Content-Disposition: form-data; name="videos"; filename="video1.mp4"
Content-Type: video/mp4

<binary data>
--boundary
Content-Disposition: form-data; name="videos"; filename="video2.mp4"
Content-Type: video/mp4

<binary data>
--boundary
Content-Disposition: form-data; name="producer"

Studio A
--boundary--

Response:

{
  "uploaded": 2,
  "failed": 0,
  "results": [
    {
      "id": "660e8400-e29b-41d4-a716-446655440000",
      "filename": "video1.mp4",
      "status": "success"
    },
    {
      "id": "770e8400-e29b-41d4-a716-446655440001",
      "filename": "video2.mp4",
      "status": "success"
    }
  ]
}

Configuration

Environment Variables

# Upload Limits
MEDIA_MAX_FILE_SIZE=10737418240  # 10GB in bytes
MEDIA_MAX_FILES_BATCH=10         # Max files per batch upload

# Upload Paths
MEDIA_INBOX_PATH=/media/local/inbox
MEDIA_LIBRARY_PATH=/media/local/library

# FFprobe
FFPROBE_TIMEOUT=30000  # 30 seconds
FFPROBE_PATH=/usr/bin/ffprobe  # Auto-detected if not set

# Allowed Extensions (comma-separated)
MEDIA_ALLOWED_EXTENSIONS=mp4,mov,avi,mkv,webm,m4v,flv

Fastify Multipart Configuration

// api/src/media-server.ts
import multipart from '@fastify/multipart';

app.register(multipart, {
  limits: {
    fieldNameSize: 100,      // Max field name size (bytes)
    fieldSize: 1000000,      // Max field value size (bytes) - for text fields
    fields: 10,              // Max number of non-file fields
    fileSize: 10 * 1024 * 1024 * 1024, // 10GB max file size
    files: 10,               // Max number of files per request
    headerPairs: 2000,       // Max header key-value pairs
  },
  attachFieldsToBody: false, // Don't parse all fields into body (use req.file())
});

Docker Volume Mounts

Critical: Inbox directory must be mounted as read-write (:rw):

# docker-compose.yml
services:
  media-api:
    volumes:
      - /media/local/library:/media/local/library:ro  # Read-only library
      - /media/local/inbox:/media/local/inbox:rw      # READ-WRITE inbox

Without :rw suffix, uploads fail with permission errors.

Code Examples

// admin/src/components/media/UploadVideoModal.tsx
import { Modal, Upload, Form, Input, Button, Progress, message } from 'antd';
import { InboxOutlined } from '@ant-design/icons';
import { useState } from 'react';
import { mediaApi } from '@/lib/media-api';

interface UploadVideoModalProps {
  visible: boolean;
  onClose: () => void;
  onSuccess: () => void;
}

export default function UploadVideoModal({ visible, onClose, onSuccess }: UploadVideoModalProps) {
  const [form] = Form.useForm();
  const [fileList, setFileList] = useState<any[]>([]);
  const [uploading, setUploading] = useState(false);
  const [uploadProgress, setUploadProgress] = useState(0);

  const handleUpload = async () => {
    if (fileList.length === 0) {
      message.error('Please select at least one video file');
      return;
    }

    setUploading(true);

    try {
      const values = await form.validateFields();

      for (const fileItem of fileList) {
        const formData = new FormData();
        formData.append('video', fileItem.originFileObj);
        formData.append('producer', values.producer || '');
        formData.append('creator', values.creator || '');
        formData.append('title', values.title || fileItem.name);
        formData.append('tags', values.tags || '');

        const { data } = await mediaApi.post('/api/media/upload/single', formData, {
          headers: {
            'Content-Type': 'multipart/form-data',
          },
          onUploadProgress: (progressEvent) => {
            const percent = Math.round((progressEvent.loaded * 100) / progressEvent.total!);
            setUploadProgress(percent);
          },
        });

        message.success(`Uploaded: ${fileItem.name}`);
      }

      onSuccess();
      handleClose();
    } catch (error: any) {
      message.error(error.response?.data?.message || 'Upload failed');
    } finally {
      setUploading(false);
      setUploadProgress(0);
    }
  };

  const handleClose = () => {
    form.resetFields();
    setFileList([]);
    setUploadProgress(0);
    onClose();
  };

  return (
    <Modal
      title="Upload Video"
      open={visible}
      onCancel={handleClose}
      footer={[
        <Button key="cancel" onClick={handleClose} disabled={uploading}>
          Cancel
        </Button>,
        <Button key="upload" type="primary" onClick={handleUpload} loading={uploading}>
          Upload
        </Button>,
      ]}
      width={600}
      destroyOnClose
    >
      <Upload.Dragger
        multiple
        fileList={fileList}
        onChange={({ fileList }) => setFileList(fileList)}
        beforeUpload={(file) => {
          const isVideo = [
            'video/mp4',
            'video/quicktime',
            'video/x-msvideo',
            'video/x-matroska',
            'video/webm',
            'video/x-m4v',
            'video/x-flv',
          ].includes(file.type);

          if (!isVideo) {
            message.error(`${file.name} is not a supported video format`);
            return Upload.LIST_IGNORE;
          }

          const isLt10GB = file.size / 1024 / 1024 / 1024 < 10;
          if (!isLt10GB) {
            message.error(`${file.name} exceeds 10GB limit`);
            return Upload.LIST_IGNORE;
          }

          return false; // Prevent auto-upload
        }}
        disabled={uploading}
      >
        <p className="ant-upload-drag-icon">
          <InboxOutlined />
        </p>
        <p className="ant-upload-text">Click or drag video files to this area</p>
        <p className="ant-upload-hint">
          Supports MP4, MOV, AVI, MKV, WebM, M4V, FLV. Max 10GB per file.
        </p>
      </Upload.Dragger>

      {uploading && (
        <div style={{ marginTop: 16 }}>
          <Progress percent={uploadProgress} status="active" />
        </div>
      )}

      <Form form={form} layout="vertical" style={{ marginTop: 24 }}>
        <Form.Item label="Producer" name="producer">
          <Input placeholder="Studio or production company" />
        </Form.Item>

        <Form.Item label="Creator" name="creator">
          <Input placeholder="Director or creator name" />
        </Form.Item>

        <Form.Item label="Title" name="title">
          <Input placeholder="Display title (defaults to filename)" />
        </Form.Item>

        <Form.Item label="Tags" name="tags">
          <Input placeholder="Comma-separated tags (e.g., action, sports)" />
        </Form.Item>
      </Form>
    </Modal>
  );
}

Backend: Single Upload Route

// api/src/modules/media/routes/upload.routes.ts
import { FastifyInstance } from 'fastify';
import path from 'path';
import fs from 'fs/promises';
import { randomUUID } from 'crypto';
import { db } from '@/modules/media/db';
import { videos } from '@/modules/media/db/schema';
import { ffprobeService } from '@/modules/media/services/ffprobe.service';
import { requireRole } from '@/middleware/auth';

export default async function (app: FastifyInstance) {
  app.post(
    '/api/media/upload/single',
    {
      preHandler: [requireRole('SUPER_ADMIN')],
    },
    async (req, reply) => {
      try {
        // Get uploaded file
        const data = await req.file();

        if (!data) {
          return reply.code(400).send({ error: 'No file uploaded' });
        }

        // Validate file extension
        const ext = path.extname(data.filename).toLowerCase();
        const allowedExtensions = ['.mp4', '.mov', '.avi', '.mkv', '.webm', '.m4v', '.flv'];

        if (!allowedExtensions.includes(ext)) {
          return reply.code(400).send({
            error: 'Invalid file type',
            message: `Allowed extensions: ${allowedExtensions.join(', ')}`,
          });
        }

        // Generate UUID filename
        const uuid = randomUUID();
        const filename = `${uuid}${ext}`;
        const relativePath = `inbox/${filename}`;
        const absolutePath = path.join(process.env.MEDIA_INBOX_PATH!, filename);

        // Stream to disk
        const writeStream = fs.createWriteStream(absolutePath);
        await data.file.pipe(writeStream);

        app.log.info(`File uploaded to ${absolutePath}`);

        // Extract metadata
        let metadata;
        try {
          metadata = await ffprobeService.extract(absolutePath);
          app.log.info('FFprobe metadata extracted', metadata);
        } catch (error: any) {
          app.log.warn('FFprobe extraction failed', error);
          // Continue without metadata (can be validated later)
          metadata = {
            duration: null,
            width: null,
            height: null,
            orientation: null,
            quality: null,
            hasAudio: false,
          };
        }

        // Get file size
        const stats = await fs.stat(absolutePath);

        // Parse metadata from request body
        const body = data.fields as any;
        const producer = body.producer?.value || null;
        const creator = body.creator?.value || null;
        const title = body.title?.value || data.filename;
        const tagsString = body.tags?.value || '';
        const tags = tagsString
          ? tagsString.split(',').map((t: string) => t.trim())
          : [];

        // Create database record
        const [video] = await db
          .insert(videos)
          .values({
            path: relativePath,
            filename,
            originalFilename: data.filename,
            directoryType: 'inbox',
            producer,
            creator,
            title,
            tags,
            durationSeconds: metadata.duration,
            width: metadata.width,
            height: metadata.height,
            orientation: metadata.orientation,
            quality: metadata.quality,
            hasAudio: metadata.hasAudio,
            fileSize: stats.size,
            isValid: true,
          })
          .returning();

        reply.send(video);
      } catch (error: any) {
        app.log.error('Upload failed', error);
        reply.code(500).send({
          error: 'Upload failed',
          message: error.message,
        });
      }
    }
  );
}

FFprobe Metadata Extraction

// api/src/modules/media/services/ffprobe.service.ts
import { exec } from 'child_process';
import { promisify } from 'util';

const execAsync = promisify(exec);

interface VideoMetadata {
  duration: number | null;
  width: number | null;
  height: number | null;
  orientation: string | null;
  quality: string | null;
  hasAudio: boolean;
  fileSize: number | null;
  fileHash: string | null;
}

export class FFprobeService {
  private timeout = parseInt(process.env.FFPROBE_TIMEOUT || '30000', 10);
  private ffprobePath = process.env.FFPROBE_PATH || 'ffprobe';

  async extract(filePath: string): Promise<VideoMetadata> {
    try {
      const command = `${this.ffprobePath} -v quiet -print_format json -show_streams -show_format "${filePath}"`;

      const { stdout } = await execAsync(command, {
        timeout: this.timeout,
        maxBuffer: 1024 * 1024 * 10, // 10MB buffer
      });

      const data = JSON.parse(stdout);

      // Find video stream
      const videoStream = data.streams.find((s: any) => s.codec_type === 'video');
      if (!videoStream) {
        throw new Error('No video stream found');
      }

      // Find audio stream
      const audioStream = data.streams.find((s: any) => s.codec_type === 'audio');

      // Extract metadata
      const width = parseInt(videoStream.width, 10);
      const height = parseInt(videoStream.height, 10);
      const duration = parseFloat(data.format.duration);
      const fileSize = parseInt(data.format.size, 10);

      // Detect orientation
      const orientation = this.detectOrientation(width, height);

      // Detect quality
      const quality = this.detectQuality(height);

      return {
        duration: isNaN(duration) ? null : Math.round(duration),
        width: isNaN(width) ? null : width,
        height: isNaN(height) ? null : height,
        orientation,
        quality,
        hasAudio: !!audioStream,
        fileSize: isNaN(fileSize) ? null : fileSize,
        fileHash: null, // Can be computed separately if needed
      };
    } catch (error: any) {
      throw new Error(`FFprobe extraction failed: ${error.message}`);
    }
  }

  private detectOrientation(width: number, height: number): string {
    if (isNaN(width) || isNaN(height)) return 'unknown';

    const ratio = width / height;
    if (ratio > 1.1) return 'landscape';
    if (ratio < 0.9) return 'portrait';
    return 'square';
  }

  private detectQuality(height: number): string {
    if (isNaN(height)) return 'unknown';

    if (height < 720) return 'SD';
    if (height < 1080) return 'HD';
    if (height < 2160) return 'FHD';
    return 'UHD';
  }
}

export const ffprobeService = new FFprobeService();

Troubleshooting

Problem: Upload Fails with "File Too Large"

Symptoms:

Upload progress reaches 100% then fails
Error message: "File exceeds maximum size"
Browser console shows 413 Payload Too Large

Solutions:

Check file size:

# On macOS/Linux
ls -lh video.mp4
# Should show size < 10GB

# If larger, compress video first:
ffmpeg -i large-video.mp4 -vcodec h264 -acodec aac compressed.mp4

Verify Fastify limit:

// api/src/media-server.ts
app.register(multipart, {
  limits: {
    fileSize: 10 * 1024 * 1024 * 1024, // 10GB
  },
});

Check nginx client_max_body_size:

# nginx/nginx.conf or nginx/conf.d/api.conf
client_max_body_size 10G;

Increase timeout for large files:

# nginx/conf.d/api.conf
server {
    location / {
        proxy_pass http://localhost:4100;
        proxy_read_timeout 600s;  # 10 minutes
        proxy_send_timeout 600s;
    }
}

Problem: FFprobe Metadata Extraction Fails

Symptoms:

Upload succeeds but metadata fields null
Warning: "Metadata extraction failed"
Duration, dimensions missing in library

Solutions:

Check FFmpeg installed:

docker compose exec media-api which ffprobe
# Should output: /usr/bin/ffprobe

docker compose exec media-api ffprobe -version
# Should show FFmpeg version

Install FFmpeg if missing:

# api/Dockerfile.media
FROM node:20-alpine

# Install FFmpeg
RUN apk add --no-cache ffmpeg

# ... rest of Dockerfile

# Rebuild container
docker compose build media-api
docker compose up -d media-api

Test FFprobe manually:

# Run FFprobe on uploaded file
docker compose exec media-api ffprobe \
  -v quiet \
  -print_format json \
  -show_streams \
  -show_format \
  /media/local/inbox/test.mp4

# Should output JSON with streams and format info

Check video file not corrupt:

# Try playing video
docker compose exec media-api ffplay /media/local/inbox/test.mp4

# Or copy to host and test
docker cp $(docker compose ps -q media-api):/media/local/inbox/test.mp4 ./
vlc test.mp4

Increase timeout for large files:

# .env
FFPROBE_TIMEOUT=60000  # 60 seconds (from 30)

Problem: Upload Hangs at 100%

Symptoms:

Progress bar reaches 100% but never completes
No success or error message
Browser tab freezes

Solutions:

Check nginx proxy timeout:

# nginx/conf.d/api.conf
server {
    location / {
        proxy_pass http://localhost:4100;
        proxy_read_timeout 600s;  # 10 minutes for large uploads
    }
}

Verify disk space available:

df -h /media/local/inbox
# Should show available space > file size

# Clear space if needed
docker compose exec media-api rm /media/local/inbox/*.mp4

Check backend logs:

docker compose logs -f media-api | grep upload
# Look for errors or timeouts

Test with smaller file:

# Create 100MB test video
ffmpeg -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 -pix_fmt yuv420p test-100mb.mp4

# Upload test file
# If succeeds, issue likely large file timeout

Problem: Inbox Directory Not Writable

Symptoms:

Upload fails with "Permission denied"
Error: "EACCES: permission denied, open '/media/local/inbox/...'"
Upload never starts

Solutions:

Check Docker volume mount:

# docker-compose.yml
services:
  media-api:
    volumes:
      - /media/local/inbox:/media/local/inbox:rw  # MUST have :rw suffix

Verify mount in running container:

docker compose exec media-api mount | grep inbox
# Should show /media/local/inbox mounted as rw (read-write)

Check directory permissions:

# On host machine
ls -la /media/local/inbox
# Should show drwxrwxrwx or drwxr-xr-x

# Fix permissions if needed
sudo chmod 777 /media/local/inbox

# Or set ownership to container user (usually node:node)
sudo chown -R 1000:1000 /media/local/inbox

Create directory if missing:

# On host
sudo mkdir -p /media/local/inbox
sudo chmod 777 /media/local/inbox

# Restart container
docker compose restart media-api

Test write access:

# Try writing test file from container
docker compose exec media-api sh -c 'echo "test" > /media/local/inbox/test.txt'

# If fails, permissions issue
# If succeeds, issue elsewhere

Problem: Invalid File Type Error

Symptoms:

Upload rejected immediately
Error: "File type not supported"
File is valid MP4/MOV/etc

Solutions:

Check MIME type:

// Browser console
const file = document.querySelector('input[type=file]').files[0];
console.log(file.type);
// Should be video/mp4, video/quicktime, etc.

Verify file extension:

# Rename file to ensure correct extension
mv video.MP4 video.mp4  # Case-sensitive on Linux

Add MIME type to allowed list:

// admin/src/components/media/UploadVideoModal.tsx
const isVideo = [
  'video/mp4',
  'video/quicktime',
  'video/x-msvideo',
  'video/x-matroska',
  'video/webm',
  'video/x-m4v',
  'video/x-flv',
  'video/mpeg',  // Add MPEG
  'video/ogg',   // Add OGG
].includes(file.type);

Bypass frontend validation (testing only):

// Temporarily comment out beforeUpload validation
beforeUpload={() => false}

Check backend extension validation:

// api/src/modules/media/routes/upload.routes.ts
const allowedExtensions = ['.mp4', '.mov', '.avi', '.mkv', '.webm', '.m4v', '.flv'];
// Add more if needed

Problem: Batch Upload Only Uploads First File

Symptoms:

Multiple files selected
Only first file uploads
Others disappear from queue

Solutions:

Check sequential upload logic:

// admin/src/components/media/UploadVideoModal.tsx
// Should use for loop, not forEach with async
for (const fileItem of fileList) {
  await mediaApi.post(...);  // Await each upload
}

Verify batch endpoint:

# Use /api/media/upload/batch for multiple files
# Not multiple calls to /api/media/upload/single

Check Fastify file limit:

// api/src/media-server.ts
app.register(multipart, {
  limits: {
    files: 10,  // Max 10 files per request
  },
});

Frontend: prevent early unmount:

// Don't close modal while uploading
<Modal
  closable={!uploading}
  maskClosable={!uploading}
  ...
/>

Performance Considerations

Upload Speed

Factors:

Network bandwidth — 100 Mbps = ~12 MB/s theoretical max
Disk write speed — SSD: 500+ MB/s, HDD: 100-150 MB/s
Nginx buffering — Can slow large uploads if enabled
Docker overlay network — ~10% overhead vs host networking

Typical Speeds:

File Size	Upload Time (100 Mbps)	Upload Time (1 Gbps)
100 MB	~10 seconds	~1 second
1 GB	~1.5 minutes	~10 seconds
5 GB	~7 minutes	~50 seconds
10 GB	~14 minutes	~1.5 minutes

Optimization:

Disable nginx buffering:

# nginx/conf.d/api.conf
location /api/media/upload {
    proxy_pass http://localhost:4100;
    proxy_request_buffering off;  # Stream directly to backend
    client_max_body_size 10G;
}

Use faster disk:

Mount /media/local/inbox on SSD instead of HDD.

Increase network MTU:

# Increase Docker network MTU
docker network create --opt com.docker.network.driver.mtu=9000 changemaker-lite

FFprobe Extraction Time

Benchmarks:

Video Size	Resolution	Extraction Time
50 MB	720p	~50-100ms
200 MB	1080p	~100-200ms
1 GB	1080p	~200-400ms
5 GB	4K	~500ms-1s

Optimization:

FFprobe only reads video metadata (not entire file), so extraction time scales sub-linearly with file size.

For very large files (10GB+), consider deferring extraction to job queue:

// Upload endpoint returns immediately
const video = await db.insert(videos).values({ ... }).returning();

// Queue FFprobe job
await jobQueue.add('extract-metadata', { videoId: video.id });

reply.send({ id: video.id, status: 'pending-metadata' });

Streaming vs Buffering

Memory Usage Comparison:

Upload Method	Memory Usage (10GB file)
Streaming (current)	~10 MB
Buffering (alternative)	~10 GB

Why Streaming:

Constant memory — Uses fixed ~10 MB buffer regardless of file size
Server stability — 10 concurrent uploads = ~100 MB RAM vs 100 GB if buffered
No 32-bit limit — Buffering fails on Node.js for files > 2GB on 32-bit systems

Tradeoff:

Streaming writes directly to disk, so failed uploads leave partial files in /inbox. Cleanup script required:

# Cron job to clean incomplete uploads (files with 0 size)
find /media/local/inbox -type f -size 0 -mtime +1 -delete

Security Considerations

Admin-Only Access

All upload endpoints require SUPER_ADMIN role:

// api/src/modules/media/routes/upload.routes.ts
app.post('/api/media/upload/single', {
  preHandler: [requireRole('SUPER_ADMIN')],
}, async (req, reply) => {
  // ...
});

Regular users, volunteers, and public cannot upload videos.

File Extension Validation

Backend enforces strict whitelist:

const allowedExtensions = ['.mp4', '.mov', '.avi', '.mkv', '.webm', '.m4v', '.flv'];

if (!allowedExtensions.includes(ext)) {
  return reply.code(400).send({ error: 'Invalid file type' });
}

No executable extensions allowed:

❌ .exe
❌ .sh
❌ .bat
❌ .php
❌ .js (only video extensions)

Path Traversal Prevention

UUID filenames prevent directory traversal:

// User-supplied filename: ../../etc/passwd.mp4
// Actual filename: 660e8400-e29b-41d4-a716-446655440000.mp4

const uuid = randomUUID();
const filename = `${uuid}${ext}`;  // No user input in filename

Original filename preserved in database:

originalFilename: data.filename,  // Stored for reference, not used for filepath

Virus Scanning (Future)

Recommended Integration:

// api/src/modules/media/services/virus-scan.service.ts
import { exec } from 'child_process';

class VirusScanService {
  async scan(filePath: string): Promise<{ clean: boolean; threat?: string }> {
    // Use ClamAV
    const { stdout } = await execAsync(`clamscan --no-summary ${filePath}`);

    if (stdout.includes('FOUND')) {
      return { clean: false, threat: stdout };
    }

    return { clean: true };
  }
}

// In upload route:
const scanResult = await virusScanService.scan(absolutePath);
if (!scanResult.clean) {
  await fs.unlink(absolutePath);  // Delete infected file
  return reply.code(400).send({ error: 'File contains malware' });
}

Rate Limiting

Upload endpoint has stricter rate limits:

// api/src/modules/media/routes/upload.routes.ts
import rateLimit from '@fastify/rate-limit';

app.register(rateLimit, {
  max: 10,        // 10 uploads
  timeWindow: '1 hour',
});

Prevents abuse (uploading hundreds of large files).

Backend Documentation

Upload Routes: backend/modules/media/upload.md — Upload endpoint implementation
FFprobe Service: backend/modules/media/ffprobe.md — Metadata extraction service
Fastify Multipart: backend/api/media-server.md — Multipart plugin configuration

Frontend Documentation

Upload Modal: frontend/components/media/upload-modal.md — Upload UI component
Library Page: frontend/pages/media/library.md — Integration with library table

Feature Documentation

Video Library: features/media/video-library.md — Video management system overview
Media Jobs: features/media/jobs.md — Background processing for uploads

Deployment Documentation

Docker Volumes: deployment/docker.md — Volume mount configuration for inbox
Nginx: deployment/nginx.md — Reverse proxy upload timeout settings

Next Steps

After mastering video upload:

Move Videos — Learn how to move uploaded videos from /inbox to target directories
Thumbnail Generation — Create thumbnails for video previews
Encoding Jobs — Queue re-encoding jobs for web-optimized playback
Public Sharing — Share videos in public gallery (see public-gallery.md)

Hands-On Practice:

# 1. Create test video (FFmpeg)
ffmpeg -f lavfi -i testsrc=duration=30:size=1920x1080:rate=30 -pix_fmt yuv420p test-video.mp4

# 2. Upload via curl
curl -X POST http://localhost:4100/api/media/upload/single \
  -H "Authorization: Bearer YOUR_ADMIN_TOKEN" \
  -F "video=@test-video.mp4" \
  -F "producer=Test Studio" \
  -F "title=Test Video"

# 3. Verify in database
docker compose exec v2-postgres psql -U changemaker -d v2_changemaker \
  -c "SELECT id, filename, duration_seconds, quality FROM videos ORDER BY created_at DESC LIMIT 1;"

# 4. Check file on disk
docker compose exec media-api ls -lh /media/local/inbox/

Last Updated: 2026-02-13 Version: V2.0 Maintainer: Changemaker Lite Team

33 KiB Raw Blame History

Video Upload System

Overview

Architecture

Upload Workflow

User Workflow (Admin)

Error Handling

API Endpoints

Upload Single Video

Upload Batch (Multiple Videos)

Configuration

Environment Variables

Fastify Multipart Configuration

Docker Volume Mounts

Code Examples

Frontend: Upload Modal Component

Backend: Single Upload Route

FFprobe Metadata Extraction

Troubleshooting

Problem: Upload Fails with "File Too Large"

Problem: FFprobe Metadata Extraction Fails

Problem: Upload Hangs at 100%

Problem: Inbox Directory Not Writable

Problem: Invalid File Type Error

Problem: Batch Upload Only Uploads First File

Performance Considerations

Upload Speed

FFprobe Extraction Time

Streaming vs Buffering

Security Considerations

Admin-Only Access

File Extension Validation

Path Traversal Prevention

Virus Scanning (Future)

Rate Limiting

Related Documentation

Backend Documentation

Frontend Documentation

Feature Documentation

Deployment Documentation

Next Steps

33 KiB

Raw Blame History