admin/changemaker.lite
1
0
Fork 0
Code Issues Packages Projects Releases 31 Wiki Activity
changemaker.lite/mkdocs/docs/v2/features/media/upload.md

1341 lines
33 KiB
Markdown
Raw Blame History

# Video Upload System
## Overview
The Video Upload System provides a modern drag-and-drop interface for uploading video files with automatic metadata extraction, progress tracking, and batch processing capabilities. Built on Fastify's multipart plugin with FFprobe integration, it supports large files up to 10GB while maintaining server stability through streaming.
**Key Features:**
- **Drag-and-Drop Interface** — Intuitive file selection with visual drop zone
- **Automatic Metadata Extraction** — FFprobe extracts duration, dimensions, orientation, quality, and audio detection
- **Single & Batch Upload** — Upload one video or queue multiple files
- **Large File Support** — Handles files up to 10GB via streaming (no memory buffering)
- **Progress Tracking** — Real-time upload progress with percentage and speed
- **Format Validation** — Supports MP4, MOV, AVI, MKV, WebM, M4V, FLV
- **UUID Filenames** — Prevents conflicts and path traversal attacks
- **Inbox Staging** — Videos uploaded to `/inbox` directory before processing
- **Manual Metadata** — Admin can override auto-detected fields (producer, creator, title, tags)
**Technology Stack:**
- **Frontend:** Ant Design Upload component with custom drag-drop styling
- **Backend:** Fastify @fastify/multipart plugin for streaming uploads
- **Metadata:** FFprobe for video analysis (duration, dimensions, codec, bitrate)
- **Storage:** Direct filesystem writes to `/media/local/inbox` directory
---
## Architecture
```mermaid
sequenceDiagram
participant U as User
participant UI as UploadVideoModal
participant API as Fastify Media API
participant FS as Filesystem
participant FFP as FFprobe Service
participant DB as PostgreSQL
U->>UI: Drag video file(s)
UI->>UI: Validate file type/size
UI->>U: Show file in queue
U->>UI: Click "Upload"
UI->>API: POST /api/media/upload/single<br/>(multipart/form-data)
API->>API: Generate UUID filename
API->>FS: Stream to /inbox/{uuid}.mp4
FS-->>API: Write complete
API->>FFP: Extract metadata
FFP->>FS: Analyze video file
FFP-->>API: Return metadata JSON
API->>DB: INSERT video record
DB-->>API: Return video ID
API-->>UI: Upload success + metadata
UI-->>U: Show success message
UI->>UI: Refresh library table
Note over API,FS: File remains in /inbox<br/>until moved by admin
```
**Upload Flow:**
1. **Client Validation** — Browser checks file extension and size before upload
2. **Streaming Upload** — File streamed to disk in chunks (no memory buffer)
3. **Metadata Extraction** — FFprobe analyzes video (30s timeout)
4. **Database Record** — Video record created with auto-detected metadata
5. **Response** — Frontend receives video ID and metadata
6. **Library Update** — Table refreshes to show new video
**Key Design Decisions:**
- **Streaming vs Buffering** — Streaming prevents memory exhaustion on large files (10GB would require 10GB RAM if buffered)
- **Inbox Staging** — New uploads go to `/inbox` directory instead of final location, allowing admin review before publishing
- **UUID Filenames** — Prevents filename conflicts and path traversal attacks (`../../etc/passwd.mp4`)
- **Synchronous FFprobe** — Metadata extracted immediately (not deferred to job queue) for instant feedback
---
## Upload Workflow
### User Workflow (Admin)
1. **Open Upload Modal**
- Navigate to **Media → Library** page
- Click **"Upload Video"** button in top toolbar
- Modal opens with drag-drop zone
2. **Select Files**
- **Drag files** from desktop into blue dashed zone
- **OR click** "Click to browse" link to open file picker
- Multiple files can be selected for batch upload
3. **Review Queue**
- Selected files appear in list with:
- Filename and size
- File type icon
- Remove button (X)
- Invalid files (wrong extension, too large) highlighted in red
4. **Enter Metadata (Optional)**
- **Producer** — Studio or production company name
- **Creator** — Director or primary creator
- **Title** — Display title (defaults to filename if blank)
- **Tags** — Comma-separated tags (e.g., "action, sports, highlight")
5. **Upload**
- Click **"Upload"** button
- Files upload sequentially (not parallel)
- Progress bar shows:
- Current file name
- Upload percentage (0-100%)
- Upload speed (MB/s)
- Estimated time remaining
6. **Metadata Extraction**
- After upload completes, FFprobe runs automatically
- Spinner shows "Extracting metadata..."
- Auto-fills: duration, dimensions, orientation, quality, audio
7. **Success**
- Green checkmark appears
- Success message: "Uploaded: {filename}"
- Modal can be closed or kept open for more uploads
- Library table refreshes showing new video
### Error Handling
**Invalid File Type:**
```
Error: File type not supported
Allowed: MP4, MOV, AVI, MKV, WebM, M4V, FLV
```
**File Too Large:**
```
Error: File exceeds 10GB limit
Selected file: 12.5 GB
```
**Upload Failed:**
```
Error: Upload failed
Network error or server unavailable
```
**FFprobe Extraction Failed:**
```
Warning: Metadata extraction failed
Video uploaded but metadata incomplete
You can manually enter duration and dimensions
```
---
## API Endpoints
### Upload Single Video
```http
POST /api/media/upload/single
Content-Type: multipart/form-data
Authorization: Bearer <admin_token>
```
**Request (Multipart Form Data):**
```
--boundary
Content-Disposition: form-data; name="video"; filename="my-video.mp4"
Content-Type: video/mp4
<binary video data>
--boundary
Content-Disposition: form-data; name="producer"
Studio A
--boundary
Content-Disposition: form-data; name="creator"
Director B
--boundary
Content-Disposition: form-data; name="title"
My Awesome Video
--boundary
Content-Disposition: form-data; name="tags"
action,sports,highlight
--boundary--
```
**Response (Success):**
```json
{
"id": "660e8400-e29b-41d4-a716-446655440000",
"path": "inbox/660e8400-e29b-41d4-a716-446655440000.mp4",
"filename": "660e8400-e29b-41d4-a716-446655440000.mp4",
"originalFilename": "my-video.mp4",
"directoryType": "inbox",
"producer": "Studio A",
"creator": "Director B",
"title": "My Awesome Video",
"tags": ["action", "sports", "highlight"],
"durationSeconds": 125,
"width": 1920,
"height": 1080,
"quality": "FHD",
"orientation": "landscape",
"hasAudio": true,
"fileSize": 45678912,
"isValid": true,
"createdAt": "2026-02-13T14:30:00Z"
}
```
**Response (Error):**
```json
{
"statusCode": 400,
"error": "Bad Request",
"message": "Invalid file type. Allowed: mp4, mov, avi, mkv, webm, m4v, flv"
}
```
---
### Upload Batch (Multiple Videos)
```http
POST /api/media/upload/batch
Content-Type: multipart/form-data
Authorization: Bearer <admin_token>
```
**Request:**
```
--boundary
Content-Disposition: form-data; name="videos"; filename="video1.mp4"
Content-Type: video/mp4
<binary data>
--boundary
Content-Disposition: form-data; name="videos"; filename="video2.mp4"
Content-Type: video/mp4
<binary data>
--boundary
Content-Disposition: form-data; name="producer"
Studio A
--boundary--
```
**Response:**
```json
{
"uploaded": 2,
"failed": 0,
"results": [
{
"id": "660e8400-e29b-41d4-a716-446655440000",
"filename": "video1.mp4",
"status": "success"
},
{
"id": "770e8400-e29b-41d4-a716-446655440001",
"filename": "video2.mp4",
"status": "success"
}
]
}
```
---
## Configuration
### Environment Variables
```bash
# Upload Limits
MEDIA_MAX_FILE_SIZE=10737418240 # 10GB in bytes
MEDIA_MAX_FILES_BATCH=10 # Max files per batch upload
# Upload Paths
MEDIA_INBOX_PATH=/media/local/inbox
MEDIA_LIBRARY_PATH=/media/local/library
# FFprobe
FFPROBE_TIMEOUT=30000 # 30 seconds
FFPROBE_PATH=/usr/bin/ffprobe # Auto-detected if not set
# Allowed Extensions (comma-separated)
MEDIA_ALLOWED_EXTENSIONS=mp4,mov,avi,mkv,webm,m4v,flv
```
### Fastify Multipart Configuration
```typescript
// api/src/media-server.ts
import multipart from '@fastify/multipart';
app.register(multipart, {
limits: {
fieldNameSize: 100, // Max field name size (bytes)
fieldSize: 1000000, // Max field value size (bytes) - for text fields
fields: 10, // Max number of non-file fields
fileSize: 10 * 1024 * 1024 * 1024, // 10GB max file size
files: 10, // Max number of files per request
headerPairs: 2000, // Max header key-value pairs
},
attachFieldsToBody: false, // Don't parse all fields into body (use req.file())
});
```
### Docker Volume Mounts
**Critical:** Inbox directory must be mounted as **read-write** (`:rw`):
```yaml
# docker-compose.yml
services:
media-api:
volumes:
- /media/local/library:/media/local/library:ro # Read-only library
- /media/local/inbox:/media/local/inbox:rw # READ-WRITE inbox
```
**Without `:rw` suffix, uploads fail with permission errors.**
---
## Code Examples
### Frontend: Upload Modal Component
```typescript
// admin/src/components/media/UploadVideoModal.tsx
import { Modal, Upload, Form, Input, Button, Progress, message } from 'antd';
import { InboxOutlined } from '@ant-design/icons';
import { useState } from 'react';
import { mediaApi } from '@/lib/media-api';
interface UploadVideoModalProps {
visible: boolean;
onClose: () => void;
onSuccess: () => void;
}
export default function UploadVideoModal({ visible, onClose, onSuccess }: UploadVideoModalProps) {
const [form] = Form.useForm();
const [fileList, setFileList] = useState<any[]>([]);
const [uploading, setUploading] = useState(false);
const [uploadProgress, setUploadProgress] = useState(0);
const handleUpload = async () => {
if (fileList.length === 0) {
message.error('Please select at least one video file');
return;
}
setUploading(true);
try {
const values = await form.validateFields();
for (const fileItem of fileList) {
const formData = new FormData();
formData.append('video', fileItem.originFileObj);
formData.append('producer', values.producer || '');
formData.append('creator', values.creator || '');
formData.append('title', values.title || fileItem.name);
formData.append('tags', values.tags || '');
const { data } = await mediaApi.post('/api/media/upload/single', formData, {
headers: {
'Content-Type': 'multipart/form-data',
},
onUploadProgress: (progressEvent) => {
const percent = Math.round((progressEvent.loaded * 100) / progressEvent.total!);
setUploadProgress(percent);
},
});
message.success(`Uploaded: ${fileItem.name}`);
}
onSuccess();
handleClose();
} catch (error: any) {
message.error(error.response?.data?.message || 'Upload failed');
} finally {
setUploading(false);
setUploadProgress(0);
}
};
const handleClose = () => {
form.resetFields();
setFileList([]);
setUploadProgress(0);
onClose();
};
return (
<Modal
title="Upload Video"
open={visible}
onCancel={handleClose}
footer={[
<Button key="cancel" onClick={handleClose} disabled={uploading}>
Cancel
</Button>,
<Button key="upload" type="primary" onClick={handleUpload} loading={uploading}>
Upload
</Button>,
]}
width={600}
destroyOnClose
>
<Upload.Dragger
multiple
fileList={fileList}
onChange={({ fileList }) => setFileList(fileList)}
beforeUpload={(file) => {
const isVideo = [
'video/mp4',
'video/quicktime',
'video/x-msvideo',
'video/x-matroska',
'video/webm',
'video/x-m4v',
'video/x-flv',
].includes(file.type);
if (!isVideo) {
message.error(`${file.name} is not a supported video format`);
return Upload.LIST_IGNORE;
}
const isLt10GB = file.size / 1024 / 1024 / 1024 < 10;
if (!isLt10GB) {
message.error(`${file.name} exceeds 10GB limit`);
return Upload.LIST_IGNORE;
}
return false; // Prevent auto-upload
}}
disabled={uploading}
>
<p className="ant-upload-drag-icon">
<InboxOutlined />
</p>
<p className="ant-upload-text">Click or drag video files to this area</p>
<p className="ant-upload-hint">
Supports MP4, MOV, AVI, MKV, WebM, M4V, FLV. Max 10GB per file.
</p>
</Upload.Dragger>
{uploading && (
<div style={{ marginTop: 16 }}>
<Progress percent={uploadProgress} status="active" />
</div>
)}
<Form form={form} layout="vertical" style={{ marginTop: 24 }}>
<Form.Item label="Producer" name="producer">
<Input placeholder="Studio or production company" />
</Form.Item>
<Form.Item label="Creator" name="creator">
<Input placeholder="Director or creator name" />
</Form.Item>
<Form.Item label="Title" name="title">
<Input placeholder="Display title (defaults to filename)" />
</Form.Item>
<Form.Item label="Tags" name="tags">
<Input placeholder="Comma-separated tags (e.g., action, sports)" />
</Form.Item>
</Form>
</Modal>
);
}
```
---
### Backend: Single Upload Route
```typescript
// api/src/modules/media/routes/upload.routes.ts
import { FastifyInstance } from 'fastify';
import path from 'path';
import fs from 'fs/promises';
import { randomUUID } from 'crypto';
import { db } from '@/modules/media/db';
import { videos } from '@/modules/media/db/schema';
import { ffprobeService } from '@/modules/media/services/ffprobe.service';
import { requireRole } from '@/middleware/auth';
export default async function (app: FastifyInstance) {
app.post(
'/api/media/upload/single',
{
preHandler: [requireRole('SUPER_ADMIN')],
},
async (req, reply) => {
try {
// Get uploaded file
const data = await req.file();
if (!data) {
return reply.code(400).send({ error: 'No file uploaded' });
}
// Validate file extension
const ext = path.extname(data.filename).toLowerCase();
const allowedExtensions = ['.mp4', '.mov', '.avi', '.mkv', '.webm', '.m4v', '.flv'];
if (!allowedExtensions.includes(ext)) {
return reply.code(400).send({
error: 'Invalid file type',
message: `Allowed extensions: ${allowedExtensions.join(', ')}`,
});
}
// Generate UUID filename
const uuid = randomUUID();
const filename = `${uuid}${ext}`;
const relativePath = `inbox/${filename}`;
const absolutePath = path.join(process.env.MEDIA_INBOX_PATH!, filename);
// Stream to disk
const writeStream = fs.createWriteStream(absolutePath);
await data.file.pipe(writeStream);
app.log.info(`File uploaded to ${absolutePath}`);
// Extract metadata
let metadata;
try {
metadata = await ffprobeService.extract(absolutePath);
app.log.info('FFprobe metadata extracted', metadata);
} catch (error: any) {
app.log.warn('FFprobe extraction failed', error);
// Continue without metadata (can be validated later)
metadata = {
duration: null,
width: null,
height: null,
orientation: null,
quality: null,
hasAudio: false,
};
}
// Get file size
const stats = await fs.stat(absolutePath);
// Parse metadata from request body
const body = data.fields as any;
const producer = body.producer?.value || null;
const creator = body.creator?.value || null;
const title = body.title?.value || data.filename;
const tagsString = body.tags?.value || '';
const tags = tagsString
? tagsString.split(',').map((t: string) => t.trim())
: [];
// Create database record
const [video] = await db
.insert(videos)
.values({
path: relativePath,
filename,
originalFilename: data.filename,
directoryType: 'inbox',
producer,
creator,
title,
tags,
durationSeconds: metadata.duration,
width: metadata.width,
height: metadata.height,
orientation: metadata.orientation,
quality: metadata.quality,
hasAudio: metadata.hasAudio,
fileSize: stats.size,
isValid: true,
})
.returning();
reply.send(video);
} catch (error: any) {
app.log.error('Upload failed', error);
reply.code(500).send({
error: 'Upload failed',
message: error.message,
});
}
}
);
}
```
---
### FFprobe Metadata Extraction
```typescript
// api/src/modules/media/services/ffprobe.service.ts
import { exec } from 'child_process';
import { promisify } from 'util';
const execAsync = promisify(exec);
interface VideoMetadata {
duration: number | null;
width: number | null;
height: number | null;
orientation: string | null;
quality: string | null;
hasAudio: boolean;
fileSize: number | null;
fileHash: string | null;
}
export class FFprobeService {
private timeout = parseInt(process.env.FFPROBE_TIMEOUT || '30000', 10);
private ffprobePath = process.env.FFPROBE_PATH || 'ffprobe';
async extract(filePath: string): Promise<VideoMetadata> {
try {
const command = `${this.ffprobePath} -v quiet -print_format json -show_streams -show_format "${filePath}"`;
const { stdout } = await execAsync(command, {
timeout: this.timeout,
maxBuffer: 1024 * 1024 * 10, // 10MB buffer
});
const data = JSON.parse(stdout);
// Find video stream
const videoStream = data.streams.find((s: any) => s.codec_type === 'video');
if (!videoStream) {
throw new Error('No video stream found');
}
// Find audio stream
const audioStream = data.streams.find((s: any) => s.codec_type === 'audio');
// Extract metadata
const width = parseInt(videoStream.width, 10);
const height = parseInt(videoStream.height, 10);
const duration = parseFloat(data.format.duration);
const fileSize = parseInt(data.format.size, 10);
// Detect orientation
const orientation = this.detectOrientation(width, height);
// Detect quality
const quality = this.detectQuality(height);
return {
duration: isNaN(duration) ? null : Math.round(duration),
width: isNaN(width) ? null : width,
height: isNaN(height) ? null : height,
orientation,
quality,
hasAudio: !!audioStream,
fileSize: isNaN(fileSize) ? null : fileSize,
fileHash: null, // Can be computed separately if needed
};
} catch (error: any) {
throw new Error(`FFprobe extraction failed: ${error.message}`);
}
}
private detectOrientation(width: number, height: number): string {
if (isNaN(width) || isNaN(height)) return 'unknown';
const ratio = width / height;
if (ratio > 1.1) return 'landscape';
if (ratio < 0.9) return 'portrait';
return 'square';
}
private detectQuality(height: number): string {
if (isNaN(height)) return 'unknown';
if (height < 720) return 'SD';
if (height < 1080) return 'HD';
if (height < 2160) return 'FHD';
return 'UHD';
}
}
export const ffprobeService = new FFprobeService();
```
---
## Troubleshooting
### Problem: Upload Fails with "File Too Large"
**Symptoms:**
- Upload progress reaches 100% then fails
- Error message: "File exceeds maximum size"
- Browser console shows 413 Payload Too Large
**Solutions:**
1. **Check file size:**
```bash
# On macOS/Linux
ls -lh video.mp4
# Should show size < 10GB
# If larger, compress video first:
ffmpeg -i large-video.mp4 -vcodec h264 -acodec aac compressed.mp4
```
2. **Verify Fastify limit:**
```typescript
// api/src/media-server.ts
app.register(multipart, {
limits: {
fileSize: 10 * 1024 * 1024 * 1024, // 10GB
},
});
```
3. **Check nginx client_max_body_size:**
```nginx
# nginx/nginx.conf or nginx/conf.d/api.conf
client_max_body_size 10G;
```
4. **Increase timeout for large files:**
```nginx
# nginx/conf.d/api.conf
server {
location / {
proxy_pass http://localhost:4100;
proxy_read_timeout 600s; # 10 minutes
proxy_send_timeout 600s;
}
}
```
---
### Problem: FFprobe Metadata Extraction Fails
**Symptoms:**
- Upload succeeds but metadata fields null
- Warning: "Metadata extraction failed"
- Duration, dimensions missing in library
**Solutions:**
1. **Check FFmpeg installed:**
```bash
docker compose exec media-api which ffprobe
# Should output: /usr/bin/ffprobe
docker compose exec media-api ffprobe -version
# Should show FFmpeg version
```
2. **Install FFmpeg if missing:**
```dockerfile
# api/Dockerfile.media
FROM node:20-alpine
# Install FFmpeg
RUN apk add --no-cache ffmpeg
# ... rest of Dockerfile
```
```bash
# Rebuild container
docker compose build media-api
docker compose up -d media-api
```
3. **Test FFprobe manually:**
```bash
# Run FFprobe on uploaded file
docker compose exec media-api ffprobe \
-v quiet \
-print_format json \
-show_streams \
-show_format \
/media/local/inbox/test.mp4
# Should output JSON with streams and format info
```
4. **Check video file not corrupt:**
```bash
# Try playing video
docker compose exec media-api ffplay /media/local/inbox/test.mp4
# Or copy to host and test
docker cp $(docker compose ps -q media-api):/media/local/inbox/test.mp4 ./
vlc test.mp4
```
5. **Increase timeout for large files:**
```bash
# .env
FFPROBE_TIMEOUT=60000 # 60 seconds (from 30)
```
---
### Problem: Upload Hangs at 100%
**Symptoms:**
- Progress bar reaches 100% but never completes
- No success or error message
- Browser tab freezes
**Solutions:**
1. **Check nginx proxy timeout:**
```nginx
# nginx/conf.d/api.conf
server {
location / {
proxy_pass http://localhost:4100;
proxy_read_timeout 600s; # 10 minutes for large uploads
}
}
```
2. **Verify disk space available:**
```bash
df -h /media/local/inbox
# Should show available space > file size
# Clear space if needed
docker compose exec media-api rm /media/local/inbox/*.mp4
```
3. **Check backend logs:**
```bash
docker compose logs -f media-api | grep upload
# Look for errors or timeouts
```
4. **Test with smaller file:**
```bash
# Create 100MB test video
ffmpeg -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 -pix_fmt yuv420p test-100mb.mp4
# Upload test file
# If succeeds, issue likely large file timeout
```
---
### Problem: Inbox Directory Not Writable
**Symptoms:**
- Upload fails with "Permission denied"
- Error: "EACCES: permission denied, open '/media/local/inbox/...'"
- Upload never starts
**Solutions:**
1. **Check Docker volume mount:**
```yaml
# docker-compose.yml
services:
media-api:
volumes:
- /media/local/inbox:/media/local/inbox:rw # MUST have :rw suffix
```
2. **Verify mount in running container:**
```bash
docker compose exec media-api mount | grep inbox
# Should show /media/local/inbox mounted as rw (read-write)
```
3. **Check directory permissions:**
```bash
# On host machine
ls -la /media/local/inbox
# Should show drwxrwxrwx or drwxr-xr-x
# Fix permissions if needed
sudo chmod 777 /media/local/inbox
# Or set ownership to container user (usually node:node)
sudo chown -R 1000:1000 /media/local/inbox
```
4. **Create directory if missing:**
```bash
# On host
sudo mkdir -p /media/local/inbox
sudo chmod 777 /media/local/inbox
# Restart container
docker compose restart media-api
```
5. **Test write access:**
```bash
# Try writing test file from container
docker compose exec media-api sh -c 'echo "test" > /media/local/inbox/test.txt'
# If fails, permissions issue
# If succeeds, issue elsewhere
```
---
### Problem: Invalid File Type Error
**Symptoms:**
- Upload rejected immediately
- Error: "File type not supported"
- File is valid MP4/MOV/etc
**Solutions:**
1. **Check MIME type:**
```javascript
// Browser console
const file = document.querySelector('input[type=file]').files[0];
console.log(file.type);
// Should be video/mp4, video/quicktime, etc.
```
2. **Verify file extension:**
```bash
# Rename file to ensure correct extension
mv video.MP4 video.mp4 # Case-sensitive on Linux
```
3. **Add MIME type to allowed list:**
```typescript
// admin/src/components/media/UploadVideoModal.tsx
const isVideo = [
'video/mp4',
'video/quicktime',
'video/x-msvideo',
'video/x-matroska',
'video/webm',
'video/x-m4v',
'video/x-flv',
'video/mpeg', // Add MPEG
'video/ogg', // Add OGG
].includes(file.type);
```
4. **Bypass frontend validation (testing only):**
```typescript
// Temporarily comment out beforeUpload validation
beforeUpload={() => false}
```
5. **Check backend extension validation:**
```typescript
// api/src/modules/media/routes/upload.routes.ts
const allowedExtensions = ['.mp4', '.mov', '.avi', '.mkv', '.webm', '.m4v', '.flv'];
// Add more if needed
```
---
### Problem: Batch Upload Only Uploads First File
**Symptoms:**
- Multiple files selected
- Only first file uploads
- Others disappear from queue
**Solutions:**
1. **Check sequential upload logic:**
```typescript
// admin/src/components/media/UploadVideoModal.tsx
// Should use for loop, not forEach with async
for (const fileItem of fileList) {
await mediaApi.post(...); // Await each upload
}
```
2. **Verify batch endpoint:**
```bash
# Use /api/media/upload/batch for multiple files
# Not multiple calls to /api/media/upload/single
```
3. **Check Fastify file limit:**
```typescript
// api/src/media-server.ts
app.register(multipart, {
limits: {
files: 10, // Max 10 files per request
},
});
```
4. **Frontend: prevent early unmount:**
```typescript
// Don't close modal while uploading
<Modal
closable={!uploading}
maskClosable={!uploading}
...
/>
```
---
## Performance Considerations
### Upload Speed
**Factors:**
- **Network bandwidth** — 100 Mbps = ~12 MB/s theoretical max
- **Disk write speed** — SSD: 500+ MB/s, HDD: 100-150 MB/s
- **Nginx buffering** — Can slow large uploads if enabled
- **Docker overlay network** — ~10% overhead vs host networking
**Typical Speeds:**
| File Size | Upload Time (100 Mbps) | Upload Time (1 Gbps) |
|-----------|----------------------|---------------------|
| 100 MB | ~10 seconds | ~1 second |
| 1 GB | ~1.5 minutes | ~10 seconds |
| 5 GB | ~7 minutes | ~50 seconds |
| 10 GB | ~14 minutes | ~1.5 minutes |
**Optimization:**
1. **Disable nginx buffering:**
```nginx
# nginx/conf.d/api.conf
location /api/media/upload {
proxy_pass http://localhost:4100;
proxy_request_buffering off; # Stream directly to backend
client_max_body_size 10G;
}
```
2. **Use faster disk:**
Mount `/media/local/inbox` on SSD instead of HDD.
3. **Increase network MTU:**
```bash
# Increase Docker network MTU
docker network create --opt com.docker.network.driver.mtu=9000 changemaker-lite
```
---
### FFprobe Extraction Time
**Benchmarks:**
| Video Size | Resolution | Extraction Time |
|-----------|-----------|----------------|
| 50 MB | 720p | ~50-100ms |
| 200 MB | 1080p | ~100-200ms |
| 1 GB | 1080p | ~200-400ms |
| 5 GB | 4K | ~500ms-1s |
**Optimization:**
FFprobe only reads video metadata (not entire file), so extraction time scales sub-linearly with file size.
For very large files (10GB+), consider deferring extraction to job queue:
```typescript
// Upload endpoint returns immediately
const video = await db.insert(videos).values({ ... }).returning();
// Queue FFprobe job
await jobQueue.add('extract-metadata', { videoId: video.id });
reply.send({ id: video.id, status: 'pending-metadata' });
```
---
### Streaming vs Buffering
**Memory Usage Comparison:**
| Upload Method | Memory Usage (10GB file) |
|--------------|-------------------------|
| **Streaming** (current) | ~10 MB |
| **Buffering** (alternative) | ~10 GB |
**Why Streaming:**
- **Constant memory** — Uses fixed ~10 MB buffer regardless of file size
- **Server stability** — 10 concurrent uploads = ~100 MB RAM vs 100 GB if buffered
- **No 32-bit limit** — Buffering fails on Node.js for files > 2GB on 32-bit systems
**Tradeoff:**
Streaming writes directly to disk, so failed uploads leave partial files in `/inbox`. Cleanup script required:
```bash
# Cron job to clean incomplete uploads (files with 0 size)
find /media/local/inbox -type f -size 0 -mtime +1 -delete
```
---
## Security Considerations
### Admin-Only Access
**All upload endpoints require `SUPER_ADMIN` role:**
```typescript
// api/src/modules/media/routes/upload.routes.ts
app.post('/api/media/upload/single', {
preHandler: [requireRole('SUPER_ADMIN')],
}, async (req, reply) => {
// ...
});
```
Regular users, volunteers, and public cannot upload videos.
---
### File Extension Validation
**Backend enforces strict whitelist:**
```typescript
const allowedExtensions = ['.mp4', '.mov', '.avi', '.mkv', '.webm', '.m4v', '.flv'];
if (!allowedExtensions.includes(ext)) {
return reply.code(400).send({ error: 'Invalid file type' });
}
```
**No executable extensions allowed:**
- ❌ `.exe`
- ❌ `.sh`
- ❌ `.bat`
- ❌ `.php`
- ❌ `.js` (only video extensions)
---
### Path Traversal Prevention
**UUID filenames prevent directory traversal:**
```typescript
// User-supplied filename: ../../etc/passwd.mp4
// Actual filename: 660e8400-e29b-41d4-a716-446655440000.mp4
const uuid = randomUUID();
const filename = `${uuid}${ext}`; // No user input in filename
```
**Original filename preserved in database:**
```typescript
originalFilename: data.filename, // Stored for reference, not used for filepath
```
---
### Virus Scanning (Future)
**Recommended Integration:**
```typescript
// api/src/modules/media/services/virus-scan.service.ts
import { exec } from 'child_process';
class VirusScanService {
async scan(filePath: string): Promise<{ clean: boolean; threat?: string }> {
// Use ClamAV
const { stdout } = await execAsync(`clamscan --no-summary ${filePath}`);
if (stdout.includes('FOUND')) {
return { clean: false, threat: stdout };
}
return { clean: true };
}
}
// In upload route:
const scanResult = await virusScanService.scan(absolutePath);
if (!scanResult.clean) {
await fs.unlink(absolutePath); // Delete infected file
return reply.code(400).send({ error: 'File contains malware' });
}
```
---
### Rate Limiting
**Upload endpoint has stricter rate limits:**
```typescript
// api/src/modules/media/routes/upload.routes.ts
import rateLimit from '@fastify/rate-limit';
app.register(rateLimit, {
max: 10, // 10 uploads
timeWindow: '1 hour',
});
```
Prevents abuse (uploading hundreds of large files).
---
## Related Documentation
### Backend Documentation
- **Upload Routes:** `backend/modules/media/upload.md` — Upload endpoint implementation
- **FFprobe Service:** `backend/modules/media/ffprobe.md` — Metadata extraction service
- **Fastify Multipart:** `backend/api/media-server.md` — Multipart plugin configuration
### Frontend Documentation
- **Upload Modal:** `frontend/components/media/upload-modal.md` — Upload UI component
- **Library Page:** `frontend/pages/media/library.md` — Integration with library table
### Feature Documentation
- **Video Library:** `features/media/video-library.md` — Video management system overview
- **Media Jobs:** `features/media/jobs.md` — Background processing for uploads
### Deployment Documentation
- **Docker Volumes:** `deployment/docker.md` — Volume mount configuration for inbox
- **Nginx:** `deployment/nginx.md` — Reverse proxy upload timeout settings
---
## Next Steps
After mastering video upload:
1. **Move Videos** — Learn how to move uploaded videos from `/inbox` to target directories
2. **Thumbnail Generation** — Create thumbnails for video previews
3. **Encoding Jobs** — Queue re-encoding jobs for web-optimized playback
4. **Public Sharing** — Share videos in public gallery (see `public-gallery.md`)
**Hands-On Practice:**
```bash
# 1. Create test video (FFmpeg)
ffmpeg -f lavfi -i testsrc=duration=30:size=1920x1080:rate=30 -pix_fmt yuv420p test-video.mp4
# 2. Upload via curl
curl -X POST http://localhost:4100/api/media/upload/single \
-H "Authorization: Bearer YOUR_ADMIN_TOKEN" \
-F "video=@test-video.mp4" \
-F "producer=Test Studio" \
-F "title=Test Video"
# 3. Verify in database
docker compose exec v2-postgres psql -U changemaker -d v2_changemaker \
-c "SELECT id, filename, duration_seconds, quality FROM videos ORDER BY created_at DESC LIMIT 1;"
# 4. Check file on disk
docker compose exec media-api ls -lh /media/local/inbox/
```
---
**Last Updated:** 2026-02-13
**Version:** V2.0
**Maintainer:** Changemaker Lite Team
Reference in New Issue View Git Blame Copy Permalink