1719 lines
43 KiB
Markdown
1719 lines
43 KiB
Markdown
# Video Library Management
|
|
|
|
## Overview
|
|
|
|
The Video Library system provides comprehensive video asset management through a dedicated Fastify microservice running on port 4100, separate from the main Express API. This dual API architecture allows the media system to operate independently while sharing the same PostgreSQL database.
|
|
|
|
**Key Features:**
|
|
|
|
- **Dual API Architecture** — Fastify media API (port 4100) separate from Express API (port 4000)
|
|
- **Drizzle ORM** — Media tables use Drizzle ORM instead of Prisma for schema flexibility
|
|
- **9 Directory Types** — Organized library structure (studios, gifs, private, inbox, curated, playback, compilations, videos, highlights)
|
|
- **FFprobe Integration** — Automatic metadata extraction (duration, dimensions, orientation, quality, audio detection)
|
|
- **Video CRUD** — Full create, read, update, delete operations (admin-only)
|
|
- **Directory Scanning** — Bulk import videos from filesystem with automatic record creation
|
|
- **Validation System** — Re-validate videos to refresh metadata and check file integrity
|
|
- **File Hashing** — Duplicate detection via SHA-256 file hashing
|
|
- **Soft Delete** — Videos marked invalid instead of hard deletion (preserves history)
|
|
- **Thumbnail Support** — Custom thumbnail paths for video previews
|
|
|
|
**Access Control:**
|
|
|
|
- All video library operations require `SUPER_ADMIN` role
|
|
- Public video viewing handled separately via Shared Media system (see `public-gallery.md`)
|
|
|
|
**Technology Stack:**
|
|
|
|
- **Fastify 4.x** — High-performance Node.js web framework
|
|
- **Drizzle ORM** — TypeScript-first ORM with zero-runtime overhead
|
|
- **FFprobe** — FFmpeg's media file analyzer for metadata extraction
|
|
- **PostgreSQL 16** — Shared database with main API
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
The Media API operates as an independent microservice while maintaining data consistency through shared database access:
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph "Client Layer"
|
|
Admin[Admin GUI :3000]
|
|
Public[Public Users]
|
|
end
|
|
|
|
subgraph "API Layer"
|
|
Express[Express API :4000<br/>Prisma ORM]
|
|
Fastify[Fastify Media API :4100<br/>Drizzle ORM]
|
|
end
|
|
|
|
subgraph "Data Layer"
|
|
DB[(PostgreSQL 16<br/>v2_changemaker)]
|
|
FS[/media/local/library/<br/>Video Files]
|
|
end
|
|
|
|
subgraph "Processing"
|
|
FFprobe[FFprobe Service<br/>Metadata Extraction]
|
|
end
|
|
|
|
Admin -->|Media Requests| Fastify
|
|
Admin -->|Other Requests| Express
|
|
Public -->|View Videos| Fastify
|
|
|
|
Fastify -->|Drizzle Queries| DB
|
|
Express -->|Prisma Queries| DB
|
|
|
|
Fastify -->|Read/Write| FS
|
|
Fastify -->|Extract Metadata| FFprobe
|
|
FFprobe -->|Analyze| FS
|
|
|
|
style Fastify fill:#e74c3c
|
|
style Express fill:#3498db
|
|
style DB fill:#2ecc71
|
|
style FS fill:#f39c12
|
|
```
|
|
|
|
**Architecture Highlights:**
|
|
|
|
1. **Port Separation** — Media API on 4100, Main API on 4000
|
|
2. **ORM Independence** — Drizzle for media, Prisma for everything else
|
|
3. **Shared Database** — Both APIs access same PostgreSQL instance
|
|
4. **File System Access** — Media API has direct volume mount to `/media/local/library`
|
|
5. **Nginx Routing** — `media.cmlite.org` routes to port 4100
|
|
|
|
**Why Dual API?**
|
|
|
|
The media system was added after V2 launch as a self-contained enhancement. Keeping it as a separate Fastify microservice:
|
|
|
|
- Avoids disrupting the stable Express API
|
|
- Allows independent scaling and deployment
|
|
- Provides testing ground for Drizzle ORM migration
|
|
- Isolates video processing workloads from core application logic
|
|
|
|
---
|
|
|
|
## Database Model (Drizzle)
|
|
|
|
### Videos Table Schema
|
|
|
|
```typescript
|
|
// api/src/modules/media/db/schema.ts
|
|
import { pgTable, uuid, text, integer, timestamp, boolean, jsonb } from 'drizzle-orm/pg-core';
|
|
|
|
export const videos = pgTable('videos', {
|
|
id: uuid('id').primaryKey().defaultRandom(),
|
|
|
|
// File Information
|
|
path: text('path').notNull().unique(), // Relative path from library root
|
|
filename: text('filename').notNull(),
|
|
originalFilename: text('original_filename'), // User-uploaded filename
|
|
directoryType: text('directory_type').notNull(), // studios|gifs|private|inbox|curated|playback|compilations|videos|highlights
|
|
|
|
// Metadata
|
|
producer: text('producer'),
|
|
creator: text('creator'),
|
|
title: text('title'),
|
|
tags: jsonb('tags').$type<string[]>().default([]),
|
|
|
|
// Video Properties
|
|
durationSeconds: integer('duration_seconds'),
|
|
quality: text('quality'), // SD|HD|FHD|UHD
|
|
orientation: text('orientation'), // portrait|landscape|square
|
|
hasAudio: boolean('has_audio').default(false),
|
|
width: integer('width'),
|
|
height: integer('height'),
|
|
|
|
// File Details
|
|
fileSize: integer('file_size'), // Bytes
|
|
fileHash: text('file_hash'), // SHA-256 for duplicate detection
|
|
|
|
// Validation
|
|
isValid: boolean('is_valid').default(true),
|
|
lastValidated: timestamp('last_validated'),
|
|
standardizedAt: timestamp('standardized_at'), // When file was moved to standard location
|
|
|
|
// Thumbnail
|
|
thumbnailPath: text('thumbnail_path'),
|
|
|
|
// Public Sharing
|
|
publicViewCount: integer('public_view_count').default(0),
|
|
publicUpvoteCount: integer('public_upvote_count').default(0),
|
|
movedFromPublicAt: timestamp('moved_from_public_at'), // When video was unlocked from public
|
|
|
|
// Timestamps
|
|
createdAt: timestamp('created_at').defaultNow(),
|
|
updatedAt: timestamp('updated_at').defaultNow(),
|
|
});
|
|
```
|
|
|
|
### Directory Types Enum
|
|
|
|
| Directory Type | Purpose | Public Eligible |
|
|
|---------------|---------|-----------------|
|
|
| `studios` | Studio-organized content | ✅ |
|
|
| `gifs` | Short looping videos | ✅ |
|
|
| `private` | Private/unreleased content | ❌ |
|
|
| `inbox` | Upload staging area | ❌ |
|
|
| `curated` | Hand-picked highlights | ✅ |
|
|
| `playback` | Playback-optimized encodes | ✅ |
|
|
| `compilations` | Multi-video compilations | ✅ |
|
|
| `videos` | General video library | ✅ |
|
|
| `highlights` | Auto-generated highlights | ✅ |
|
|
|
|
### Quality Classifications
|
|
|
|
| Quality | Height Range | Typical Resolution |
|
|
|---------|-------------|-------------------|
|
|
| `SD` | < 720px | 480p, 576p |
|
|
| `HD` | 720px - 1079px | 720p |
|
|
| `FHD` | 1080px - 2159px | 1080p |
|
|
| `UHD` | ≥ 2160px | 4K, 8K |
|
|
|
|
### Orientation Detection
|
|
|
|
```typescript
|
|
const detectOrientation = (width: number, height: number): string => {
|
|
const ratio = width / height;
|
|
if (ratio > 1.1) return 'landscape';
|
|
if (ratio < 0.9) return 'portrait';
|
|
return 'square';
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
## API Endpoints
|
|
|
|
All endpoints require authentication with `SUPER_ADMIN` role unless marked as public.
|
|
|
|
### List Videos
|
|
|
|
```http
|
|
GET /api/media/videos
|
|
```
|
|
|
|
**Query Parameters:**
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `page` | number | 1 | Page number for pagination |
|
|
| `limit` | number | 20 | Results per page (max 100) |
|
|
| `directoryType` | string | - | Filter by directory (studios, gifs, etc.) |
|
|
| `orientation` | string | - | Filter by orientation (portrait, landscape, square) |
|
|
| `producer` | string | - | Filter by producer (partial match) |
|
|
| `creator` | string | - | Filter by creator (partial match) |
|
|
| `quality` | string | - | Filter by quality (SD, HD, FHD, UHD) |
|
|
| `hasAudio` | boolean | - | Filter by audio presence |
|
|
| `isValid` | boolean | true | Filter by validation status |
|
|
| `search` | string | - | Search in title, producer, creator |
|
|
|
|
**Response:**
|
|
|
|
```json
|
|
{
|
|
"data": [
|
|
{
|
|
"id": "550e8400-e29b-41d4-a716-446655440000",
|
|
"path": "videos/sample.mp4",
|
|
"filename": "sample.mp4",
|
|
"directoryType": "videos",
|
|
"producer": "Studio A",
|
|
"creator": "Director B",
|
|
"title": "Sample Video",
|
|
"durationSeconds": 180,
|
|
"quality": "FHD",
|
|
"orientation": "landscape",
|
|
"hasAudio": true,
|
|
"width": 1920,
|
|
"height": 1080,
|
|
"fileSize": 52428800,
|
|
"isValid": true,
|
|
"createdAt": "2026-02-10T12:00:00Z"
|
|
}
|
|
],
|
|
"pagination": {
|
|
"page": 1,
|
|
"limit": 20,
|
|
"total": 156,
|
|
"totalPages": 8
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Get Video Details
|
|
|
|
```http
|
|
GET /api/media/videos/:id
|
|
```
|
|
|
|
**Response:**
|
|
|
|
```json
|
|
{
|
|
"id": "550e8400-e29b-41d4-a716-446655440000",
|
|
"path": "videos/sample.mp4",
|
|
"filename": "sample.mp4",
|
|
"originalFilename": "my-video.mp4",
|
|
"directoryType": "videos",
|
|
"producer": "Studio A",
|
|
"creator": "Director B",
|
|
"title": "Sample Video",
|
|
"tags": ["action", "sports", "highlight"],
|
|
"durationSeconds": 180,
|
|
"quality": "FHD",
|
|
"orientation": "landscape",
|
|
"hasAudio": true,
|
|
"width": 1920,
|
|
"height": 1080,
|
|
"fileSize": 52428800,
|
|
"fileHash": "a3d2f1e8b9c7...",
|
|
"isValid": true,
|
|
"lastValidated": "2026-02-10T12:00:00Z",
|
|
"thumbnailPath": "thumbnails/550e8400.jpg",
|
|
"publicViewCount": 1250,
|
|
"publicUpvoteCount": 85,
|
|
"createdAt": "2026-02-10T12:00:00Z",
|
|
"updatedAt": "2026-02-10T12:00:00Z"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Create Video Record
|
|
|
|
```http
|
|
POST /api/media/videos
|
|
```
|
|
|
|
**Request Body:**
|
|
|
|
```json
|
|
{
|
|
"path": "videos/new-video.mp4",
|
|
"filename": "new-video.mp4",
|
|
"directoryType": "videos",
|
|
"producer": "Studio A",
|
|
"creator": "Director B",
|
|
"title": "New Video",
|
|
"tags": ["action", "sports"]
|
|
}
|
|
```
|
|
|
|
**Notes:**
|
|
|
|
- File must already exist at specified path on filesystem
|
|
- FFprobe metadata extraction runs automatically after creation
|
|
- Use `/api/media/upload/single` for file upload + record creation
|
|
|
|
**Response:**
|
|
|
|
```json
|
|
{
|
|
"id": "660e8400-e29b-41d4-a716-446655440000",
|
|
"path": "videos/new-video.mp4",
|
|
"filename": "new-video.mp4",
|
|
"directoryType": "videos",
|
|
"isValid": true,
|
|
"createdAt": "2026-02-13T10:30:00Z"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Update Video Metadata
|
|
|
|
```http
|
|
PUT /api/media/videos/:id
|
|
```
|
|
|
|
**Request Body:**
|
|
|
|
```json
|
|
{
|
|
"producer": "Updated Studio",
|
|
"creator": "New Director",
|
|
"title": "Updated Title",
|
|
"tags": ["updated", "tags"]
|
|
}
|
|
```
|
|
|
|
**Updatable Fields:**
|
|
|
|
- `producer` — Video producer/studio
|
|
- `creator` — Director/creator name
|
|
- `title` — Display title
|
|
- `tags` — Array of tag strings
|
|
- `thumbnailPath` — Custom thumbnail path
|
|
|
|
**Response:**
|
|
|
|
```json
|
|
{
|
|
"id": "550e8400-e29b-41d4-a716-446655440000",
|
|
"producer": "Updated Studio",
|
|
"creator": "New Director",
|
|
"title": "Updated Title",
|
|
"tags": ["updated", "tags"],
|
|
"updatedAt": "2026-02-13T10:35:00Z"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Delete Video
|
|
|
|
```http
|
|
DELETE /api/media/videos/:id
|
|
```
|
|
|
|
**Behavior:**
|
|
|
|
- **Soft Delete** — Sets `isValid = false` instead of removing record
|
|
- File remains on filesystem (manual cleanup required)
|
|
- Video no longer appears in default listings
|
|
- Can be restored by setting `isValid = true` via database
|
|
|
|
**Response:**
|
|
|
|
```json
|
|
{
|
|
"success": true,
|
|
"message": "Video marked as invalid"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Scan Directory
|
|
|
|
```http
|
|
POST /api/media/videos/scan
|
|
```
|
|
|
|
**Request Body:**
|
|
|
|
```json
|
|
{
|
|
"directoryType": "videos",
|
|
"skipExisting": true
|
|
}
|
|
```
|
|
|
|
**Parameters:**
|
|
|
|
| Field | Type | Required | Description |
|
|
|-------|------|----------|-------------|
|
|
| `directoryType` | string | ✅ | Directory to scan (videos, studios, etc.) |
|
|
| `skipExisting` | boolean | - | Skip files already in database (default: true) |
|
|
|
|
**Process:**
|
|
|
|
1. Reads filesystem directory `/media/local/library/{directoryType}/`
|
|
2. Filters for video extensions (`.mp4`, `.mov`, `.avi`, `.mkv`, `.webm`, `.m4v`, `.flv`)
|
|
3. Checks each file against database (by path)
|
|
4. Creates records for new files
|
|
5. Runs FFprobe metadata extraction on new records
|
|
|
|
**Response:**
|
|
|
|
```json
|
|
{
|
|
"scanned": 45,
|
|
"created": 12,
|
|
"skipped": 33,
|
|
"failed": 0,
|
|
"errors": []
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Validate Video
|
|
|
|
```http
|
|
POST /api/media/videos/:id/validate
|
|
```
|
|
|
|
**Purpose:**
|
|
|
|
- Re-run FFprobe metadata extraction
|
|
- Update video properties (duration, dimensions, etc.)
|
|
- Verify file still exists and is readable
|
|
- Refresh file size and hash
|
|
|
|
**Response:**
|
|
|
|
```json
|
|
{
|
|
"id": "550e8400-e29b-41d4-a716-446655440000",
|
|
"isValid": true,
|
|
"lastValidated": "2026-02-13T10:40:00Z",
|
|
"metadata": {
|
|
"durationSeconds": 180,
|
|
"width": 1920,
|
|
"height": 1080,
|
|
"quality": "FHD",
|
|
"orientation": "landscape",
|
|
"hasAudio": true
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
|
|
```bash
|
|
# Media API Server
|
|
MEDIA_API_PORT=4100
|
|
MEDIA_API_HOST=0.0.0.0
|
|
|
|
# File Paths
|
|
MEDIA_LIBRARY_PATH=/media/local/library
|
|
MEDIA_INBOX_PATH=/media/local/inbox
|
|
|
|
# Feature Flags
|
|
ENABLE_MEDIA_FEATURES=true
|
|
|
|
# Database (shared with main API)
|
|
DATABASE_URL=postgresql://user:pass@v2-postgres:5432/v2_changemaker
|
|
|
|
# FFprobe
|
|
FFPROBE_TIMEOUT=30000 # milliseconds
|
|
FFPROBE_PATH=/usr/bin/ffprobe # Auto-detected if not set
|
|
```
|
|
|
|
### Docker Volume Mounts
|
|
|
|
```yaml
|
|
# docker-compose.yml
|
|
services:
|
|
media-api:
|
|
volumes:
|
|
- /media/local/library:/media/local/library:ro # Read-only library
|
|
- /media/local/inbox:/media/local/inbox:rw # Read-write inbox
|
|
```
|
|
|
|
**Important:** Inbox requires `:rw` (read-write) for uploads. Library can be `:ro` (read-only) for security.
|
|
|
|
### Site Settings
|
|
|
|
The media system respects the global `ENABLE_MEDIA_FEATURES` flag in Site Settings:
|
|
|
|
```sql
|
|
SELECT * FROM settings WHERE key = 'ENABLE_MEDIA_FEATURES';
|
|
```
|
|
|
|
When disabled:
|
|
|
|
- Media API still runs but returns 503 Service Unavailable
|
|
- Admin GUI hides Media menu items
|
|
- Public gallery shows maintenance message
|
|
|
|
---
|
|
|
|
## Admin Workflow
|
|
|
|
### Viewing the Video Library
|
|
|
|
1. Navigate to **Media → Library** in admin sidebar
|
|
2. Table displays all videos with:
|
|
- Thumbnail preview
|
|
- Title, producer, creator
|
|
- Duration, quality, orientation
|
|
- Directory type
|
|
- File size
|
|
- Created date
|
|
3. Use filters at top:
|
|
- **Directory Type** dropdown
|
|
- **Orientation** radio buttons (All / Portrait / Landscape / Square)
|
|
- **Quality** checkboxes (SD, HD, FHD, UHD)
|
|
- **Search** input (searches title, producer, creator)
|
|
|
|
### Scanning a Directory
|
|
|
|
**When to Use:**
|
|
|
|
- After manually copying videos to library directory
|
|
- After video processing jobs complete
|
|
- When videos exist on filesystem but not in database
|
|
|
|
**Steps:**
|
|
|
|
1. Click **"Scan Directory"** button in Library page toolbar
|
|
2. Select directory type from dropdown
|
|
3. Toggle **"Skip Existing"** (recommended for large libraries)
|
|
4. Click **"Start Scan"**
|
|
5. Progress modal shows:
|
|
- Files scanned
|
|
- New records created
|
|
- Skipped (already in DB)
|
|
- Failed (with error messages)
|
|
6. Click **"Close"** when complete
|
|
7. Table refreshes with new videos
|
|
|
|
**Example Output:**
|
|
|
|
```
|
|
Scanning /media/local/library/videos...
|
|
Found 45 video files
|
|
- Created 12 new records
|
|
- Skipped 33 existing records
|
|
- Failed 0 files
|
|
Scan complete in 8.3 seconds
|
|
```
|
|
|
|
### Editing Video Metadata
|
|
|
|
1. Click **pencil icon** in video row
|
|
2. Edit modal opens with fields:
|
|
- **Producer** — Studio or production company
|
|
- **Creator** — Director or primary creator
|
|
- **Title** — Display title
|
|
- **Tags** — Comma-separated tags (auto-suggests existing tags)
|
|
3. Click **"Save"** to update
|
|
4. Metadata changes immediately visible in table
|
|
|
|
**Bulk Editing:**
|
|
|
|
1. Select multiple videos using checkboxes
|
|
2. Click **"Bulk Edit"** button
|
|
3. Set common fields (producer, tags, etc.)
|
|
4. Click **"Apply to Selected"**
|
|
|
|
### Validating Videos
|
|
|
|
**Purpose:** Refresh metadata and verify file integrity
|
|
|
|
**Steps:**
|
|
|
|
1. Click **"Validate"** button in video row (or Actions dropdown)
|
|
2. FFprobe re-analyzes video file
|
|
3. Database updates with fresh metadata:
|
|
- Duration (may have changed if file was re-encoded)
|
|
- Dimensions
|
|
- Audio detection
|
|
- File size and hash
|
|
4. `lastValidated` timestamp updates
|
|
5. If file missing or corrupt, `isValid` set to `false`
|
|
|
|
**Bulk Validation:**
|
|
|
|
1. Select multiple videos
|
|
2. Click **"Validate Selected"**
|
|
3. Progress modal shows validation results
|
|
4. Failed validations highlighted in red
|
|
|
|
### Deleting Videos
|
|
|
|
**Soft Delete (Default):**
|
|
|
|
1. Click **trash icon** in video row
|
|
2. Confirm deletion dialog
|
|
3. Video marked `isValid = false`
|
|
4. Video disappears from default view
|
|
5. File remains on filesystem
|
|
6. Record preserved in database
|
|
|
|
**Viewing Deleted Videos:**
|
|
|
|
1. Toggle **"Show Invalid"** filter
|
|
2. Deleted videos appear with strikethrough
|
|
3. Can restore by clicking **"Restore"** button
|
|
|
|
**Hard Delete (Database Only):**
|
|
|
|
1. Filter for invalid videos
|
|
2. Select video(s)
|
|
3. Click **"Permanently Delete"**
|
|
4. Removes database record
|
|
5. File still on filesystem (manual cleanup required)
|
|
|
|
**File System Cleanup:**
|
|
|
|
Deleted video files must be manually removed from filesystem:
|
|
|
|
```bash
|
|
# SSH into media-api container
|
|
docker compose exec media-api sh
|
|
|
|
# Navigate to library
|
|
cd /media/local/library/videos
|
|
|
|
# Remove specific file
|
|
rm deleted-video.mp4
|
|
|
|
# Or find and remove all invalid videos (BE CAREFUL)
|
|
# (requires database query to get invalid file paths)
|
|
```
|
|
|
|
---
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
/media/local/library/
|
|
├── studios/ # Studio-organized content
|
|
│ ├── studio-a/
|
|
│ │ ├── video-001.mp4
|
|
│ │ └── video-002.mp4
|
|
│ └── studio-b/
|
|
│ └── video-003.mp4
|
|
│
|
|
├── gifs/ # Short looping videos
|
|
│ ├── loop-001.mp4
|
|
│ └── loop-002.webm
|
|
│
|
|
├── private/ # Private/unreleased content
|
|
│ └── unreleased.mp4
|
|
│
|
|
├── inbox/ # Upload staging area (READ-WRITE)
|
|
│ ├── uuid-123.mp4 # Temp uploads
|
|
│ └── uuid-456.mov
|
|
│
|
|
├── curated/ # Hand-picked highlights
|
|
│ ├── best-of-2025.mp4
|
|
│ └── top-plays.mp4
|
|
│
|
|
├── playback/ # Playback-optimized encodes
|
|
│ ├── streaming-001.mp4
|
|
│ └── streaming-002.mp4
|
|
│
|
|
├── compilations/ # Multi-video compilations
|
|
│ ├── compilation-001.mp4
|
|
│ └── mega-compilation.mp4
|
|
│
|
|
├── videos/ # General video library
|
|
│ ├── video-001.mp4
|
|
│ ├── video-002.mp4
|
|
│ └── ... (thousands of videos)
|
|
│
|
|
└── highlights/ # Auto-generated highlights
|
|
├── highlight-001.mp4
|
|
└── highlight-002.mp4
|
|
```
|
|
|
|
**Directory Guidelines:**
|
|
|
|
- **studios/** — Organize by producer/studio name (subfolder structure allowed)
|
|
- **gifs/** — Short videos under 15 seconds, suitable for looping
|
|
- **private/** — Never shared publicly, admin-only access
|
|
- **inbox/** — Temporary upload location, files moved after processing
|
|
- **curated/** — High-quality selections for public gallery homepage
|
|
- **playback/** — Web-optimized encodes (H.264, web-friendly profiles)
|
|
- **compilations/** — Merged videos created by compilation jobs
|
|
- **videos/** — Main library, all-purpose storage
|
|
- **highlights/** — AI-generated or manually created highlight reels
|
|
|
|
---
|
|
|
|
## Code Examples
|
|
|
|
### List Videos with Filters (Fastify Route)
|
|
|
|
```typescript
|
|
// api/src/modules/media/routes/videos.routes.ts
|
|
import { FastifyInstance } from 'fastify';
|
|
import { eq, and, like, desc, sql } from 'drizzle-orm';
|
|
import { videos } from '@/modules/media/db/schema';
|
|
import { db } from '@/modules/media/db';
|
|
|
|
export default async function (app: FastifyInstance) {
|
|
app.get('/api/media/videos', async (req, reply) => {
|
|
const {
|
|
page = 1,
|
|
limit = 20,
|
|
directoryType,
|
|
orientation,
|
|
producer,
|
|
creator,
|
|
quality,
|
|
hasAudio,
|
|
isValid = true,
|
|
search,
|
|
} = req.query as any;
|
|
|
|
// Build filters
|
|
const filters = [];
|
|
|
|
if (directoryType) {
|
|
filters.push(eq(videos.directoryType, directoryType));
|
|
}
|
|
|
|
if (orientation) {
|
|
filters.push(eq(videos.orientation, orientation));
|
|
}
|
|
|
|
if (producer) {
|
|
filters.push(like(videos.producer, `%${producer}%`));
|
|
}
|
|
|
|
if (creator) {
|
|
filters.push(like(videos.creator, `%${creator}%`));
|
|
}
|
|
|
|
if (quality) {
|
|
filters.push(eq(videos.quality, quality));
|
|
}
|
|
|
|
if (typeof hasAudio === 'boolean') {
|
|
filters.push(eq(videos.hasAudio, hasAudio));
|
|
}
|
|
|
|
if (typeof isValid === 'boolean') {
|
|
filters.push(eq(videos.isValid, isValid));
|
|
}
|
|
|
|
if (search) {
|
|
filters.push(
|
|
sql`(
|
|
${videos.title} ILIKE ${'%' + search + '%'} OR
|
|
${videos.producer} ILIKE ${'%' + search + '%'} OR
|
|
${videos.creator} ILIKE ${'%' + search + '%'}
|
|
)`
|
|
);
|
|
}
|
|
|
|
// Count total
|
|
const [{ count }] = await db
|
|
.select({ count: sql<number>`count(*)` })
|
|
.from(videos)
|
|
.where(and(...filters));
|
|
|
|
// Fetch paginated results
|
|
const results = await db
|
|
.select()
|
|
.from(videos)
|
|
.where(and(...filters))
|
|
.limit(Number(limit))
|
|
.offset((Number(page) - 1) * Number(limit))
|
|
.orderBy(desc(videos.createdAt));
|
|
|
|
reply.send({
|
|
data: results,
|
|
pagination: {
|
|
page: Number(page),
|
|
limit: Number(limit),
|
|
total: Number(count),
|
|
totalPages: Math.ceil(Number(count) / Number(limit)),
|
|
},
|
|
});
|
|
});
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Scan Directory for Videos
|
|
|
|
```typescript
|
|
// api/src/modules/media/routes/videos.routes.ts
|
|
import fs from 'fs/promises';
|
|
import path from 'path';
|
|
import { eq } from 'drizzle-orm';
|
|
import { videos } from '@/modules/media/db/schema';
|
|
import { ffprobeService } from '@/modules/media/services/ffprobe.service';
|
|
|
|
app.post('/api/media/videos/scan', async (req, reply) => {
|
|
const { directoryType, skipExisting = true } = req.body as any;
|
|
|
|
if (!directoryType) {
|
|
return reply.code(400).send({ error: 'directoryType required' });
|
|
}
|
|
|
|
const dirPath = path.join(process.env.MEDIA_LIBRARY_PATH!, directoryType);
|
|
|
|
try {
|
|
// Check directory exists
|
|
await fs.access(dirPath);
|
|
} catch {
|
|
return reply.code(400).send({ error: `Directory not found: ${directoryType}` });
|
|
}
|
|
|
|
// Read directory
|
|
const files = await fs.readdir(dirPath, { recursive: true });
|
|
|
|
// Filter for video files
|
|
const videoExtensions = ['.mp4', '.mov', '.avi', '.mkv', '.webm', '.m4v', '.flv'];
|
|
const videoFiles = files.filter((f) =>
|
|
videoExtensions.some((ext) => f.toLowerCase().endsWith(ext))
|
|
);
|
|
|
|
const results = {
|
|
scanned: videoFiles.length,
|
|
created: 0,
|
|
skipped: 0,
|
|
failed: 0,
|
|
errors: [] as string[],
|
|
};
|
|
|
|
for (const filename of videoFiles) {
|
|
try {
|
|
const relativePath = path.join(directoryType, filename);
|
|
|
|
// Check if already exists
|
|
if (skipExisting) {
|
|
const existing = await db
|
|
.select()
|
|
.from(videos)
|
|
.where(eq(videos.path, relativePath))
|
|
.limit(1);
|
|
|
|
if (existing.length > 0) {
|
|
results.skipped++;
|
|
continue;
|
|
}
|
|
}
|
|
|
|
// Extract metadata
|
|
const fullPath = path.join(dirPath, filename);
|
|
const metadata = await ffprobeService.extract(fullPath);
|
|
|
|
// Create record
|
|
await db.insert(videos).values({
|
|
path: relativePath,
|
|
filename: path.basename(filename),
|
|
directoryType,
|
|
durationSeconds: metadata.duration,
|
|
width: metadata.width,
|
|
height: metadata.height,
|
|
orientation: metadata.orientation,
|
|
quality: metadata.quality,
|
|
hasAudio: metadata.hasAudio,
|
|
fileSize: metadata.fileSize,
|
|
isValid: true,
|
|
});
|
|
|
|
results.created++;
|
|
} catch (error: any) {
|
|
results.failed++;
|
|
results.errors.push(`${filename}: ${error.message}`);
|
|
}
|
|
}
|
|
|
|
reply.send(results);
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
### Validate Video Metadata
|
|
|
|
```typescript
|
|
// api/src/modules/media/routes/videos.routes.ts
|
|
import { eq } from 'drizzle-orm';
|
|
import { videos } from '@/modules/media/db/schema';
|
|
import { ffprobeService } from '@/modules/media/services/ffprobe.service';
|
|
|
|
app.post('/api/media/videos/:id/validate', async (req, reply) => {
|
|
const { id } = req.params as { id: string };
|
|
|
|
// Fetch video record
|
|
const [video] = await db
|
|
.select()
|
|
.from(videos)
|
|
.where(eq(videos.id, id))
|
|
.limit(1);
|
|
|
|
if (!video) {
|
|
return reply.code(404).send({ error: 'Video not found' });
|
|
}
|
|
|
|
try {
|
|
// Build full file path
|
|
const fullPath = path.join(process.env.MEDIA_LIBRARY_PATH!, video.path);
|
|
|
|
// Extract fresh metadata
|
|
const metadata = await ffprobeService.extract(fullPath);
|
|
|
|
// Update database
|
|
const [updated] = await db
|
|
.update(videos)
|
|
.set({
|
|
durationSeconds: metadata.duration,
|
|
width: metadata.width,
|
|
height: metadata.height,
|
|
orientation: metadata.orientation,
|
|
quality: metadata.quality,
|
|
hasAudio: metadata.hasAudio,
|
|
fileSize: metadata.fileSize,
|
|
fileHash: metadata.fileHash,
|
|
isValid: true,
|
|
lastValidated: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(videos.id, id))
|
|
.returning();
|
|
|
|
reply.send({
|
|
id: updated.id,
|
|
isValid: updated.isValid,
|
|
lastValidated: updated.lastValidated,
|
|
metadata: {
|
|
durationSeconds: updated.durationSeconds,
|
|
width: updated.width,
|
|
height: updated.height,
|
|
quality: updated.quality,
|
|
orientation: updated.orientation,
|
|
hasAudio: updated.hasAudio,
|
|
},
|
|
});
|
|
} catch (error: any) {
|
|
// Mark as invalid if validation fails
|
|
await db
|
|
.update(videos)
|
|
.set({
|
|
isValid: false,
|
|
lastValidated: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(videos.id, id));
|
|
|
|
reply.code(500).send({
|
|
error: 'Validation failed',
|
|
message: error.message,
|
|
isValid: false,
|
|
});
|
|
}
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
### Frontend: Library Page Table
|
|
|
|
```typescript
|
|
// admin/src/pages/media/LibraryPage.tsx
|
|
import { Table, Button, Select, Input, Tag, Space } from 'antd';
|
|
import { useEffect, useState } from 'react';
|
|
import { mediaApi } from '@/lib/media-api';
|
|
|
|
export default function LibraryPage() {
|
|
const [videos, setVideos] = useState([]);
|
|
const [loading, setLoading] = useState(false);
|
|
const [pagination, setPagination] = useState({ page: 1, limit: 20, total: 0 });
|
|
const [filters, setFilters] = useState({
|
|
directoryType: undefined,
|
|
orientation: undefined,
|
|
search: '',
|
|
});
|
|
|
|
const fetchVideos = async () => {
|
|
setLoading(true);
|
|
try {
|
|
const { data } = await mediaApi.get('/api/media/videos', {
|
|
params: {
|
|
page: pagination.page,
|
|
limit: pagination.limit,
|
|
...filters,
|
|
},
|
|
});
|
|
setVideos(data.data);
|
|
setPagination((prev) => ({ ...prev, total: data.pagination.total }));
|
|
} catch (error) {
|
|
console.error('Failed to fetch videos:', error);
|
|
} finally {
|
|
setLoading(false);
|
|
}
|
|
};
|
|
|
|
useEffect(() => {
|
|
fetchVideos();
|
|
}, [pagination.page, filters]);
|
|
|
|
const columns = [
|
|
{
|
|
title: 'Preview',
|
|
dataIndex: 'thumbnailPath',
|
|
width: 100,
|
|
render: (path: string) => (
|
|
<img
|
|
src={path || '/placeholder.jpg'}
|
|
alt="Thumbnail"
|
|
style={{ width: 80, height: 60, objectFit: 'cover' }}
|
|
/>
|
|
),
|
|
},
|
|
{
|
|
title: 'Title',
|
|
dataIndex: 'title',
|
|
render: (text: string, record: any) => (
|
|
<div>
|
|
<div style={{ fontWeight: 600 }}>{text || record.filename}</div>
|
|
<div style={{ fontSize: 12, color: '#888' }}>
|
|
{record.producer} • {record.creator}
|
|
</div>
|
|
</div>
|
|
),
|
|
},
|
|
{
|
|
title: 'Duration',
|
|
dataIndex: 'durationSeconds',
|
|
width: 100,
|
|
render: (seconds: number) => {
|
|
const mins = Math.floor(seconds / 60);
|
|
const secs = seconds % 60;
|
|
return `${mins}:${secs.toString().padStart(2, '0')}`;
|
|
},
|
|
},
|
|
{
|
|
title: 'Quality',
|
|
dataIndex: 'quality',
|
|
width: 80,
|
|
render: (quality: string) => {
|
|
const colors: Record<string, string> = {
|
|
SD: 'default',
|
|
HD: 'blue',
|
|
FHD: 'green',
|
|
UHD: 'purple',
|
|
};
|
|
return <Tag color={colors[quality]}>{quality}</Tag>;
|
|
},
|
|
},
|
|
{
|
|
title: 'Orientation',
|
|
dataIndex: 'orientation',
|
|
width: 100,
|
|
},
|
|
{
|
|
title: 'Directory',
|
|
dataIndex: 'directoryType',
|
|
width: 120,
|
|
},
|
|
{
|
|
title: 'Actions',
|
|
width: 150,
|
|
render: (_: any, record: any) => (
|
|
<Space>
|
|
<Button size="small" onClick={() => handleEdit(record.id)}>
|
|
Edit
|
|
</Button>
|
|
<Button size="small" onClick={() => handleValidate(record.id)}>
|
|
Validate
|
|
</Button>
|
|
<Button size="small" danger onClick={() => handleDelete(record.id)}>
|
|
Delete
|
|
</Button>
|
|
</Space>
|
|
),
|
|
},
|
|
];
|
|
|
|
return (
|
|
<div>
|
|
<Space style={{ marginBottom: 16 }}>
|
|
<Select
|
|
placeholder="Directory Type"
|
|
style={{ width: 200 }}
|
|
onChange={(value) => setFilters({ ...filters, directoryType: value })}
|
|
allowClear
|
|
>
|
|
<Select.Option value="videos">Videos</Select.Option>
|
|
<Select.Option value="studios">Studios</Select.Option>
|
|
<Select.Option value="gifs">GIFs</Select.Option>
|
|
<Select.Option value="curated">Curated</Select.Option>
|
|
</Select>
|
|
|
|
<Input.Search
|
|
placeholder="Search title, producer, creator"
|
|
style={{ width: 300 }}
|
|
onSearch={(value) => setFilters({ ...filters, search: value })}
|
|
allowClear
|
|
/>
|
|
|
|
<Button type="primary" onClick={handleScanDirectory}>
|
|
Scan Directory
|
|
</Button>
|
|
</Space>
|
|
|
|
<Table
|
|
columns={columns}
|
|
dataSource={videos}
|
|
loading={loading}
|
|
rowKey="id"
|
|
pagination={{
|
|
current: pagination.page,
|
|
pageSize: pagination.limit,
|
|
total: pagination.total,
|
|
onChange: (page) => setPagination({ ...pagination, page }),
|
|
}}
|
|
/>
|
|
</div>
|
|
);
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Problem: Media API Not Accessible
|
|
|
|
**Symptoms:**
|
|
|
|
- Admin GUI shows "Cannot connect to media API"
|
|
- Browser console shows CORS errors or network failures
|
|
- Public gallery doesn't load
|
|
|
|
**Solutions:**
|
|
|
|
1. **Check Fastify server running:**
|
|
|
|
```bash
|
|
docker compose ps media-api
|
|
# Should show "Up" status
|
|
|
|
docker compose logs media-api
|
|
# Look for "Fastify server listening on port 4100"
|
|
```
|
|
|
|
2. **Verify port 4100 not in use:**
|
|
|
|
```bash
|
|
lsof -i :4100
|
|
# Should show only media-api container
|
|
|
|
# If another process using port, stop it or change MEDIA_API_PORT in .env
|
|
```
|
|
|
|
3. **Check nginx proxy configuration:**
|
|
|
|
```nginx
|
|
# nginx/conf.d/api.conf
|
|
# Media API block must come BEFORE general API block
|
|
|
|
server {
|
|
listen 80;
|
|
server_name media.cmlite.org;
|
|
|
|
location / {
|
|
proxy_pass http://localhost:4100;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Upgrade $http_upgrade;
|
|
proxy_set_header Connection 'upgrade';
|
|
proxy_set_header Host $host;
|
|
proxy_cache_bypass $http_upgrade;
|
|
}
|
|
}
|
|
```
|
|
|
|
4. **Test direct API access:**
|
|
|
|
```bash
|
|
# From host machine
|
|
curl http://localhost:4100/api/media/videos
|
|
|
|
# From inside container
|
|
docker compose exec media-api curl http://localhost:4100/api/media/videos
|
|
```
|
|
|
|
5. **Check Docker networking:**
|
|
|
|
```bash
|
|
docker network inspect changemaker-lite
|
|
# Verify media-api container connected
|
|
```
|
|
|
|
---
|
|
|
|
### Problem: Scan Finds No Videos
|
|
|
|
**Symptoms:**
|
|
|
|
- Scan completes with "Created 0 new records"
|
|
- Directory known to contain video files
|
|
- Scan reports 0 files scanned
|
|
|
|
**Solutions:**
|
|
|
|
1. **Verify MEDIA_LIBRARY_PATH correct:**
|
|
|
|
```bash
|
|
# Check environment variable
|
|
docker compose exec media-api printenv MEDIA_LIBRARY_PATH
|
|
# Should output: /media/local/library
|
|
|
|
# List directory contents
|
|
docker compose exec media-api ls -la /media/local/library/videos
|
|
# Should show video files
|
|
```
|
|
|
|
2. **Check directory exists:**
|
|
|
|
```bash
|
|
# Create missing directory
|
|
docker compose exec media-api mkdir -p /media/local/library/videos
|
|
|
|
# Copy test videos
|
|
docker cp test.mp4 $(docker compose ps -q media-api):/media/local/library/videos/
|
|
```
|
|
|
|
3. **Verify Docker volume mounted:**
|
|
|
|
```yaml
|
|
# docker-compose.yml
|
|
services:
|
|
media-api:
|
|
volumes:
|
|
- /media/local/library:/media/local/library:ro # Check path correct
|
|
```
|
|
|
|
```bash
|
|
# Inspect volume mounts
|
|
docker compose config | grep -A 5 media-api
|
|
```
|
|
|
|
4. **Check file extensions supported:**
|
|
|
|
Only these extensions scanned:
|
|
|
|
- `.mp4`
|
|
- `.mov`
|
|
- `.avi`
|
|
- `.mkv`
|
|
- `.webm`
|
|
- `.m4v`
|
|
- `.flv`
|
|
|
|
Rename files if using other extensions:
|
|
|
|
```bash
|
|
# Rename .MP4 to .mp4 (case-sensitive)
|
|
docker compose exec media-api sh -c 'cd /media/local/library/videos && rename "s/.MP4$/.mp4/" *.MP4'
|
|
```
|
|
|
|
5. **Check file permissions:**
|
|
|
|
```bash
|
|
# Verify readable by container user
|
|
docker compose exec media-api ls -la /media/local/library/videos
|
|
|
|
# Fix permissions if needed (on host)
|
|
sudo chmod -R 755 /media/local/library
|
|
```
|
|
|
|
---
|
|
|
|
### Problem: FFprobe Validation Fails
|
|
|
|
**Symptoms:**
|
|
|
|
- Validation returns error "FFprobe command failed"
|
|
- Videos marked `isValid = false`
|
|
- Timeout errors after 30 seconds
|
|
|
|
**Solutions:**
|
|
|
|
1. **Check FFmpeg installed in container:**
|
|
|
|
```bash
|
|
# Verify FFprobe available
|
|
docker compose exec media-api which ffprobe
|
|
# Should output: /usr/bin/ffprobe
|
|
|
|
docker compose exec media-api ffprobe -version
|
|
# Should show FFmpeg version info
|
|
```
|
|
|
|
2. **Install FFmpeg if missing:**
|
|
|
|
```dockerfile
|
|
# api/Dockerfile.media
|
|
FROM node:20-alpine
|
|
|
|
# Install FFmpeg (both dev and production stages)
|
|
RUN apk add --no-cache ffmpeg
|
|
|
|
# ... rest of Dockerfile
|
|
```
|
|
|
|
```bash
|
|
# Rebuild container
|
|
docker compose build media-api
|
|
docker compose up -d media-api
|
|
```
|
|
|
|
3. **Test FFprobe directly on video:**
|
|
|
|
```bash
|
|
# Run FFprobe manually
|
|
docker compose exec media-api ffprobe -v quiet -print_format json -show_streams -show_format /media/local/library/videos/test.mp4
|
|
|
|
# If this fails, video file corrupt or unsupported
|
|
```
|
|
|
|
4. **Check timeout not exceeded:**
|
|
|
|
Default timeout: 30 seconds
|
|
|
|
```bash
|
|
# For very large files (>5GB), increase timeout
|
|
# api/src/modules/media/services/ffprobe.service.ts
|
|
const FFPROBE_TIMEOUT = 60000; // 60 seconds
|
|
```
|
|
|
|
5. **Verify video file not corrupt:**
|
|
|
|
```bash
|
|
# Test playback
|
|
docker compose exec media-api ffplay /media/local/library/videos/test.mp4
|
|
|
|
# Or copy to host and test in VLC
|
|
docker cp $(docker compose ps -q media-api):/media/local/library/videos/test.mp4 ./test.mp4
|
|
vlc test.mp4
|
|
```
|
|
|
|
6. **Check for special characters in filename:**
|
|
|
|
```bash
|
|
# Rename files with spaces or special chars
|
|
docker compose exec media-api sh -c 'cd /media/local/library/videos && rename "s/ /_/g" *.mp4'
|
|
```
|
|
|
|
---
|
|
|
|
### Problem: Drizzle Schema Changes Not Applied
|
|
|
|
**Symptoms:**
|
|
|
|
- Code references new column but database doesn't have it
|
|
- Error: "column does not exist"
|
|
- Schema changes made but not reflected
|
|
|
|
**Solutions:**
|
|
|
|
1. **Push schema changes:**
|
|
|
|
```bash
|
|
# Drizzle uses push (not migrations)
|
|
cd api
|
|
npx drizzle-kit push
|
|
|
|
# Confirm changes
|
|
```
|
|
|
|
2. **Verify connection:**
|
|
|
|
```bash
|
|
# Check DATABASE_URL correct
|
|
docker compose exec media-api printenv DATABASE_URL
|
|
|
|
# Test connection
|
|
docker compose exec media-api npx drizzle-kit studio
|
|
# Opens DB browser on http://localhost:4983
|
|
```
|
|
|
|
3. **Compare with Prisma migrations:**
|
|
|
|
Media tables exist in same database as Prisma tables. If conflict:
|
|
|
|
```bash
|
|
# Check both schemas
|
|
npx prisma db pull # Prisma introspection
|
|
npx drizzle-kit introspect # Drizzle introspection
|
|
|
|
# Resolve conflicts manually
|
|
```
|
|
|
|
---
|
|
|
|
### Problem: Large Library Performance
|
|
|
|
**Symptoms:**
|
|
|
|
- Library page loads slowly (5+ seconds)
|
|
- Pagination sluggish
|
|
- Scan operations timeout
|
|
|
|
**Solutions:**
|
|
|
|
1. **Add database indexes:**
|
|
|
|
```sql
|
|
-- Index for common filters
|
|
CREATE INDEX idx_videos_directory_type ON videos(directory_type);
|
|
CREATE INDEX idx_videos_orientation ON videos(orientation);
|
|
CREATE INDEX idx_videos_quality ON videos(quality);
|
|
CREATE INDEX idx_videos_is_valid ON videos(is_valid);
|
|
CREATE INDEX idx_videos_created_at ON videos(created_at DESC);
|
|
|
|
-- Composite index for filtered queries
|
|
CREATE INDEX idx_videos_filters ON videos(directory_type, is_valid, created_at DESC);
|
|
|
|
-- Full-text search index
|
|
CREATE INDEX idx_videos_search ON videos USING gin(to_tsvector('english', coalesce(title, '') || ' ' || coalesce(producer, '') || ' ' || coalesce(creator, '')));
|
|
```
|
|
|
|
2. **Reduce page size:**
|
|
|
|
```typescript
|
|
// admin/src/pages/media/LibraryPage.tsx
|
|
const [pagination, setPagination] = useState({ page: 1, limit: 10, total: 0 });
|
|
// Reduced from 20 to 10
|
|
```
|
|
|
|
3. **Enable query caching:**
|
|
|
|
```typescript
|
|
// api/src/modules/media/routes/videos.routes.ts
|
|
import { redisClient } from '@/config/redis';
|
|
|
|
app.get('/api/media/videos', async (req, reply) => {
|
|
const cacheKey = `videos:list:${JSON.stringify(req.query)}`;
|
|
|
|
// Check cache
|
|
const cached = await redisClient.get(cacheKey);
|
|
if (cached) {
|
|
return reply.send(JSON.parse(cached));
|
|
}
|
|
|
|
// Fetch from database
|
|
const results = await db.select()...;
|
|
|
|
// Cache for 5 minutes
|
|
await redisClient.setex(cacheKey, 300, JSON.stringify(results));
|
|
|
|
reply.send(results);
|
|
});
|
|
```
|
|
|
|
4. **Use virtual scrolling:**
|
|
|
|
```typescript
|
|
// Replace Ant Design Table with react-window for large datasets
|
|
import { FixedSizeList } from 'react-window';
|
|
```
|
|
|
|
---
|
|
|
|
## Performance Considerations
|
|
|
|
### Directory Scans
|
|
|
|
**Scaling Factors:**
|
|
|
|
- 100 files: ~2 seconds
|
|
- 1,000 files: ~15 seconds
|
|
- 10,000 files: ~2.5 minutes
|
|
|
|
**Optimization Strategies:**
|
|
|
|
1. **Incremental Scans** — Use `skipExisting: true` to only process new files
|
|
2. **Parallel Processing** — Scan multiple directories simultaneously
|
|
3. **Background Jobs** — Queue scans as async jobs instead of synchronous requests
|
|
4. **Caching** — Cache directory listings in Redis
|
|
|
|
### FFprobe Extraction
|
|
|
|
**Timing:**
|
|
|
|
- Small video (<100MB): ~50-100ms
|
|
- Medium video (500MB): ~150-250ms
|
|
- Large video (2GB+): ~500ms-1s
|
|
|
|
**Batch Processing:**
|
|
|
|
For 100 videos: ~10-20 seconds total
|
|
|
|
**Optimization:**
|
|
|
|
```typescript
|
|
// Parallel extraction (limit concurrency)
|
|
import pLimit from 'p-limit';
|
|
|
|
const limit = pLimit(5); // Max 5 concurrent FFprobe calls
|
|
|
|
const results = await Promise.all(
|
|
videoFiles.map((file) =>
|
|
limit(() => ffprobeService.extract(file))
|
|
)
|
|
);
|
|
```
|
|
|
|
### Database Queries
|
|
|
|
**Query Performance:**
|
|
|
|
- List 20 videos (no filters): ~5-10ms
|
|
- List 20 videos (with filters): ~10-20ms
|
|
- Full-text search: ~20-50ms
|
|
- Count total videos: ~5ms (with index)
|
|
|
|
**Optimization:**
|
|
|
|
1. **Always use pagination** — Never fetch all records
|
|
2. **Index heavily filtered columns** — directoryType, orientation, quality, isValid
|
|
3. **Use SELECT only needed columns** — Avoid `SELECT *` for large tables
|
|
4. **Cache counts** — Total video count changes infrequently, cache in Redis
|
|
|
|
### Thumbnail Generation
|
|
|
|
**Deferred Loading:**
|
|
|
|
Don't generate thumbnails during scan. Instead:
|
|
|
|
1. Create video record without thumbnail
|
|
2. Queue thumbnail generation job
|
|
3. Worker processes job asynchronously
|
|
4. Update record with `thumbnailPath`
|
|
|
|
**Lazy Loading:**
|
|
|
|
Frontend requests thumbnails only when visible (IntersectionObserver).
|
|
|
|
---
|
|
|
|
## Dual API Architecture
|
|
|
|
### Why Separate Fastify API?
|
|
|
|
The media system was introduced as a **Phase 14 enhancement** after V2 core functionality stabilized. A separate Fastify microservice was chosen to:
|
|
|
|
1. **Avoid Disrupting Stable Express API** — V2 Express API battle-tested with 30+ models, introducing media directly risked regressions
|
|
2. **Test Drizzle ORM Migration** — Fastify+Drizzle serves as proof-of-concept for potential future Prisma→Drizzle migration
|
|
3. **Isolate Video Processing** — CPU/GPU-intensive FFprobe, encoding jobs isolated from main API request handling
|
|
4. **Independent Scaling** — Media API can be horizontally scaled separately based on video processing load
|
|
5. **Technology Experimentation** — Fastify's performance benefits evaluated for potential broader adoption
|
|
|
|
### Database Sharing Strategy
|
|
|
|
**Same PostgreSQL, Different ORMs:**
|
|
|
|
```
|
|
┌─────────────────┐
|
|
│ PostgreSQL 16 │
|
|
│ v2_changemaker │
|
|
└─────────────────┘
|
|
↑
|
|
┌────┴────┐
|
|
│ │
|
|
┌───┴───┐ ┌──┴────┐
|
|
│Prisma │ │Drizzle│
|
|
│ ORM │ │ ORM │
|
|
└───┬───┘ └──┬────┘
|
|
│ │
|
|
┌───┴────┐ ┌─┴─────┐
|
|
│Express │ │Fastify│
|
|
│ API │ │ Media │
|
|
│ :4000 │ │ API │
|
|
│ │ │ :4100 │
|
|
└────────┘ └───────┘
|
|
```
|
|
|
|
**Benefits:**
|
|
|
|
- **Single Source of Truth** — All data in one database
|
|
- **Cross-API Queries** — Main API can query media tables via Prisma raw queries
|
|
- **Unified Backups** — One PostgreSQL dump includes both APIs
|
|
- **Shared Connections** — Connection pooling optimizations benefit both
|
|
|
|
**Challenges:**
|
|
|
|
- **Schema Coordination** — Must manually sync schema changes between Prisma migrations and Drizzle pushes
|
|
- **Type Conflicts** — Same table, different type definitions (Prisma vs Drizzle types)
|
|
- **Migration Complexity** — Prisma generates migrations, Drizzle uses push (no migration files)
|
|
|
|
### Migration Strategy Roadmap
|
|
|
|
**Short Term (Current):**
|
|
|
|
- Keep dual API architecture
|
|
- Synchronize schemas manually
|
|
- Document shared tables in both ORMs
|
|
|
|
**Medium Term (6-12 months):**
|
|
|
|
- Evaluate Fastify+Drizzle performance vs Express+Prisma
|
|
- If Fastify superior, migrate select Express routes to Fastify
|
|
- If no significant benefit, consolidate media into Express+Prisma
|
|
|
|
**Long Term (12+ months):**
|
|
|
|
- Unified API (either all Express or all Fastify)
|
|
- Single ORM (either all Prisma or all Drizzle)
|
|
- Deprecate less performant stack
|
|
|
|
**Migration Effort Estimate:**
|
|
|
|
- **Media to Express+Prisma:** 3-5 days (convert Drizzle queries to Prisma, merge Fastify routes into Express)
|
|
- **All to Fastify+Drizzle:** 2-3 weeks (convert 30+ Prisma models to Drizzle, rewrite Express routes for Fastify)
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
### Backend Documentation
|
|
|
|
- **API Server:** `backend/api/media-server.md` — Fastify server setup, middleware, error handling
|
|
- **Videos Module:** `backend/modules/media/videos.md` — Video routes, service layer, business logic
|
|
- **FFprobe Service:** `backend/modules/media/ffprobe.md` — Metadata extraction implementation
|
|
- **Jobs System:** `backend/modules/media/jobs.md` — Job queue architecture, worker processes
|
|
|
|
### Frontend Documentation
|
|
|
|
- **Library Page:** `frontend/pages/media/library.md` — Video library management UI
|
|
- **Shared Media Page:** `frontend/pages/media/shared.md` — Public gallery admin UI
|
|
- **Media Components:** `frontend/components/media.md` — Reusable video components
|
|
|
|
### Database Documentation
|
|
|
|
- **Media Models:** `database/models/media.md` — Drizzle schema definitions for videos, compilations, jobs
|
|
- **Drizzle Setup:** `database/drizzle.md` — Drizzle ORM configuration, connection management
|
|
|
|
### Feature Documentation
|
|
|
|
- **Video Upload:** `features/media/upload.md` — Upload system workflow, FFprobe integration
|
|
- **Media Jobs:** `features/media/jobs.md` — Job queue system, processing pipeline
|
|
- **Public Gallery:** `features/media/public-gallery.md` — Public video sharing system
|
|
|
|
### Integration Documentation
|
|
|
|
- **Dual API Architecture:** `architecture/dual-api.md` — Express+Prisma vs Fastify+Drizzle comparison
|
|
- **Nginx Routing:** `deployment/nginx.md` — Reverse proxy configuration for media.cmlite.org
|
|
- **Docker Setup:** `deployment/docker.md` — Media API container, volume mounts, healthchecks
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
After mastering video library management:
|
|
|
|
1. **Upload System** — Read `features/media/upload.md` to understand video upload workflow
|
|
2. **Jobs Queue** — Review `features/media/jobs.md` for video processing automation
|
|
3. **Public Gallery** — Explore `features/media/public-gallery.md` for sharing videos publicly
|
|
4. **Custom Integrations** — Use Media API endpoints to build custom video features
|
|
|
|
For hands-on practice, try:
|
|
|
|
```bash
|
|
# 1. Upload test videos
|
|
curl -X POST http://localhost:4100/api/media/upload/single \
|
|
-H "Authorization: Bearer YOUR_TOKEN" \
|
|
-F "video=@test.mp4" \
|
|
-F "producer=Test Studio" \
|
|
-F "title=Test Video"
|
|
|
|
# 2. Scan directory
|
|
curl -X POST http://localhost:4100/api/media/videos/scan \
|
|
-H "Authorization: Bearer YOUR_TOKEN" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"directoryType": "videos"}'
|
|
|
|
# 3. List videos
|
|
curl http://localhost:4100/api/media/videos?page=1&limit=10
|
|
|
|
# 4. Validate video
|
|
curl -X POST http://localhost:4100/api/media/videos/VIDEO_ID/validate \
|
|
-H "Authorization: Bearer YOUR_TOKEN"
|
|
```
|
|
|
|
---
|
|
|
|
**Last Updated:** 2026-02-13
|
|
**Version:** V2.0
|
|
**Maintainer:** Changemaker Lite Team
|