bunker-admin 7895ce683e Tonne of debugging - getting ready for the production builds

2026-02-16 10:44:18 -07:00

21 KiB

Raw Blame History

MkDocs Export Integration

Export landing pages to MkDocs Material theme with Jinja2 template wrapping, front matter configuration, and synchronized stub files.

Overview

The MkDocs Export system bridges the Page Builder and static documentation site. Administrators can publish landing pages to the main MkDocs site, where they benefit from Material theme styling, navigation, and SEO features.

Key Features

Two Export Modes: THEMED (extends Material theme) vs STANDALONE (full HTML document)
Jinja2 Template Wrapping: Integrates with MkDocs Material theme inheritance
Front Matter Configuration: Control navigation, table of contents visibility
Dual-File Output: HTML override + Markdown stub
Automatic Sync: Export triggered on publish, cleanup on unpublish
Path Validation: Prevents directory traversal attacks
Stub Backfill: Repair missing files via "Validate Exports"

Architecture

graph TD
    A[Admin] -->|Publish Page| B[PUT /api/pages/:id]
    B --> C[pages.service.update]
    C --> D{published && !skipExport?}
    D -->|Yes| E[exportToMkDocs]
    E --> F[wrapInMaterialOverride]
    E --> G[generateMdStub]
    F --> H[Write .html to overrides/]
    G --> I[Write .md to docs/]

    J[MkDocs Build] --> K[Read .md stub]
    K --> L[Front matter: template]
    L --> H
    H --> M[Render with Material theme]
    M --> N[Public site]

    style E fill:#9d4edd
    style H fill:#3498db
    style N fill:#2ecc71

Flow:

Trigger: Admin publishes page (or updates published page)
Service: pages.service.update() checks publish status
Export: Calls exportToMkDocs() with page data
Wrap: HTML wrapped in Jinja2 {% extends "main.html" %}
Write: Two files created:
- mkdocs/docs/overrides/{slug}.html — HTML override
- mkdocs/docs/{slug}.md — Markdown stub
Build: MkDocs rebuild (mkdocs build)
Render: Stub references override, Material theme applies
Serve: Page accessible at https://cmlite.org/pages/{slug}/

Export Modes

THEMED Mode (Default)

Purpose: Integrate page with MkDocs Material theme (header, footer, navigation)

Jinja2 Template:

{% extends "main.html" %}
{% block content %}
<style>
section { padding: 40px; }
</style>
<section>
  <h1>Welcome</h1>
  <p>Page content here.</p>
</section>
{% endblock %}

Features:

Uses Material theme header/footer
Respects site navigation
Table of contents auto-generated
Search integration works
Responsive design inherited

Use Cases:

Documentation pages
Campaign info pages
Community guidelines

STANDALONE Mode

Purpose: Full control over HTML (no MkDocs chrome)

HTML Document:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>About Us | Campaign 2026</title>
    <meta name="description" content="Join our movement for change.">
    <style>
    section { padding: 40px; }
    </style>
</head>
<body>
<section>
  <h1>Welcome</h1>
  <p>Page content here.</p>
</section>
</body>
</html>

Features:

No Material theme elements
Custom head section (meta tags, styles)
Independent from site navigation
Full design freedom

Use Cases:

Marketing landing pages (like lander.html)
Event registration pages
Embedded pages (iframes)

File Outputs

Override File (.html)

Location: mkdocs/docs/overrides/{slug}.html

Example: mkdocs/docs/overrides/about-us.html

Content (THEMED mode):

{% extends "main.html" %}
{% block content %}
<style>
/* Page CSS */
</style>
<!-- Page HTML -->
{% endblock %}

Content (STANDALONE mode):

<!DOCTYPE html>
<html lang="en">
<head>
  <title>About Us</title>
  <style>/* Page CSS */</style>
</head>
<body>
  <!-- Page HTML -->
</body>
</html>

Access Control:

Readable by MkDocs build process
Not directly served (accessed via stub)

Stub File (.md)

Location: mkdocs/docs/{slug}.md

Example: mkdocs/docs/about-us.md

Content:

---
template: about-us.html
title: "About Us | Campaign 2026"
description: "Join our movement for change."
hide:
  - navigation
  - toc
---

Front Matter Fields:

Field	Type	Description
`template`	string	Override filename (relative to `custom_dir`)
`title`	string	Page title (from `seoTitle` or `title`)
`description`	string	Meta description (from `seoDescription`)
`hide`	array	Hide navigation/toc elements

Important: Template path is relative to custom_dir (mkdocs/overrides/). Use about-us.html, NOT overrides/about-us.html (causes TemplateNotFound error).

Database Fields

LandingPage Export Configuration

Fields:

Field	Type	Default	Description
`mkdocsPath`	String?	`{slug}.html`	Override filename (auto-generated from slug)
`mkdocsStubPath`	String?	`{slug}.md`	Stub filename (derived from mkdocsPath)
`mkdocsExportMode`	Enum	`THEMED`	`THEMED` or `STANDALONE`
`mkdocsHideNav`	Boolean	`false`	Hide navigation sidebar (THEMED only)
`mkdocsHideToc`	Boolean	`false`	Hide table of contents (THEMED only)
`mkdocsSkipExport`	Boolean	`false`	Skip MkDocs export entirely

Behavior:

On publish (published=true):
- If mkdocsSkipExport=false: Export files
- If mkdocsSkipExport=true: No export (page only at /p/:slug)
On unpublish (published=false): Remove export files
On title change: Regenerate slug, update mkdocsPath, clean up old files

Admin Workflow

Exporting a Page

Automatic Export (on publish):

Admin → Pages → Click "Publish" button
API updates published=true
Service checks mkdocsSkipExport
If false: Calls exportToMkDocs()
Files written to disk
Database updated with mkdocsStubPath

Manual Export Trigger:

Edit page settings
Change mkdocsExportMode or mkdocsHideNav
Save settings
If published: Auto re-exports

Configuring Export Options

Location: Page Settings modal → MkDocs Integration section

Steps:

Admin → Pages → Click gear icon (Settings)
Scroll to "MkDocs Integration"
Configure options:
- Skip MkDocs Export: ☐ (unchecked)
- Override Path: about.html (auto-filled)
- Full page MkDocs: ☐ (THEMED mode)
- Hide navigation sidebar: ☑ (checked)
- Hide table of contents: ☑ (checked)
Click "Save"
If published: Files re-exported immediately

Rebuilding MkDocs Site

Trigger: After exporting pages

Methods:

Option 1: Admin UI

Admin → Pages → "Build Site" button (SUPER_ADMIN only)
Confirmation modal appears
Click "Confirm"
API executes docker compose exec mkdocs mkdocs build
Success notification

Option 2: Command Line

docker compose exec mkdocs mkdocs build
# Rebuilds site from mkdocs/docs/ directory
# Output: mkdocs/site/ (static HTML)

Auto-rebuild: Not implemented (manual trigger required)

Syncing Overrides

Purpose: Import hand-coded .html files from overrides/ directory

Workflow:

Place .html file in mkdocs/docs/overrides/custom.html
Admin → Pages → "Sync Overrides" button
API scans directory:
- Untracked files → Create CODE-mode page
- Tracked CODE-mode pages → Update htmlOutput from disk
- VISUAL pages → Skip (managed by GrapesJS)
Backfills missing .md stubs
Shows result: Synced: 2 imported, 1 updated, 3 stubs created

Use Cases:

Migrate legacy templates
Import designer-created HTML
Restore after file system corruption

Validating Exports

Purpose: Verify files exist on disk, repair if missing

Workflow:

Admin → Pages → "Validate Exports" button
API queries all published, non-skipped pages
For each page:
- Check .html override exists
- Check .md stub exists
- If either missing: Re-export
Shows result: Validated 10 pages: 2 repaired, 0 errors

Use Cases:

Recover from accidental deletion
Fix state after container restarts
Audit before production deploy

Code Examples

Themed Mode Export

// From pages.service.ts

function wrapInMaterialOverride(html: string, css: string | null): string {
  const styleBlock = css ? `<style>\n${css}\n</style>` : '';
  return `{% extends "main.html" %}
{% block content %}
${styleBlock}
${html}
{% endblock %}
`;
}

// Usage
const content = wrapInMaterialOverride(
  '<section><h1>About Us</h1></section>',
  'section { padding: 40px; }'
);

// Result:
// {% extends "main.html" %}
// {% block content %}
// <style>
// section { padding: 40px; }
// </style>
// <section><h1>About Us</h1></section>
// {% endblock %}

Standalone Mode Export

function wrapInStandaloneDocument(
  html: string,
  css: string | null,
  title: string,
  description: string | null
): string {
  const metaDesc = description
    ? `\n    <meta name="description" content="${description.replace(/"/g, '&quot;')}">`
    : '';
  const styleBlock = css ? `\n    <style>\n${css}\n    </style>` : '';

  return `<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>${title.replace(/</g, '&lt;')}</title>${metaDesc}${styleBlock}
</head>
<body>
${html}
</body>
</html>
`;
}

// Usage
const content = wrapInStandaloneDocument(
  '<section><h1>About Us</h1></section>',
  'section { padding: 40px; }',
  'About Us | Campaign 2026',
  'Join our movement for change.'
);

Markdown Stub Generation

interface StubOptions {
  overrideFilename: string;
  title: string;
  description: string | null;
  hideNav: boolean;
  hideToc: boolean;
}

function generateMdStub(opts: StubOptions): string {
  const hideItems: string[] = [];
  if (opts.hideNav) hideItems.push('  - navigation');
  if (opts.hideToc) hideItems.push('  - toc');

  const hideBlock = hideItems.length > 0
    ? `hide:\n${hideItems.join('\n')}\n`
    : '';

  const descLine = opts.description
    ? `description: "${opts.description.replace(/"/g, '\\"')}"\n`
    : '';

  return `---
template: ${opts.overrideFilename}
${hideBlock}title: "${opts.title.replace(/"/g, '\\"')}"
${descLine}---
`;
}

// Usage
const stub = generateMdStub({
  overrideFilename: 'about-us.html',
  title: 'About Us | Campaign 2026',
  description: 'Join our movement.',
  hideNav: true,
  hideToc: true,
});

// Result:
// ---
// template: about-us.html
// hide:
//   - navigation
//   - toc
// title: "About Us | Campaign 2026"
// description: "Join our movement."
// ---

Export Orchestration

// From pages.service.update()

// After updating page in database
if (page.published && !page.mkdocsSkipExport && page.mkdocsPath && page.htmlOutput) {
  const stubPath = await exportToMkDocs({
    mkdocsPath: page.mkdocsPath,
    html: page.htmlOutput,
    css: page.cssOutput,
    editorMode: page.editorMode,
    exportMode: page.mkdocsExportMode,
    title: page.title,
    seoTitle: page.seoTitle,
    seoDescription: page.seoDescription,
    hideNav: page.mkdocsHideNav,
    hideToc: page.mkdocsHideToc,
  });

  // Store stubPath if changed
  if (stubPath !== page.mkdocsStubPath) {
    await prisma.landingPage.update({
      where: { id },
      data: { mkdocsStubPath: stubPath },
    });
  }
} else if ((!page.published || page.mkdocsSkipExport) && existing.mkdocsPath) {
  // Clean up exports on unpublish or skip
  await removeFromMkDocs(existing.mkdocsPath, existing.mkdocsStubPath);

  if (existing.mkdocsStubPath) {
    await prisma.landingPage.update({
      where: { id },
      data: { mkdocsStubPath: null },
    });
  }
}

Path Validation

function validateMkdocsPath(mkdocsPath: string): void {
  // Check for null bytes
  if (mkdocsPath.includes('\0')) {
    throw new AppError(400, 'Invalid path: null byte detected', 'INVALID_MKDOCS_PATH');
  }

  // Normalize and check for traversal
  const normalized = path.normalize(mkdocsPath);
  if (normalized.includes('..') || path.isAbsolute(normalized)) {
    throw new AppError(400, 'Path traversal not allowed', 'INVALID_MKDOCS_PATH');
  }

  // Check for encoded traversal sequences
  if (mkdocsPath.includes('%2e') || mkdocsPath.includes('%2E')) {
    throw new AppError(400, 'Encoded path traversal not allowed', 'INVALID_MKDOCS_PATH');
  }

  if (!mkdocsPath.endsWith('.html')) {
    throw new AppError(400, 'Path must end with .html', 'INVALID_MKDOCS_PATH');
  }
}

// Safe paths
validateMkdocsPath('about.html'); // ✓
validateMkdocsPath('pages/contact.html'); // ✓

// Rejected paths
validateMkdocsPath('../etc/passwd.html'); // ✗ Path traversal
validateMkdocsPath('/etc/shadow.html'); // ✗ Absolute path
validateMkdocsPath('admin%2e%2e/config.html'); // ✗ Encoded traversal
validateMkdocsPath('about.md'); // ✗ Missing .html extension

MkDocs Configuration

mkdocs.yml Settings

Required Configuration:

site_name: Changemaker Lite
theme:
  name: material
  custom_dir: overrides  # Points to mkdocs/docs/overrides/

nav:
  - Home: index.md
  - Pages:
    - About: about-us.md
    - Contact: contact.md

Key Points:

custom_dir: overrides — Enables template overrides
Stub files must be listed in nav to appear in navigation
Unlisted stubs still accessible via direct URL

Template Search Paths

MkDocs Material searches:

mkdocs/overrides/ (custom_dir)
Material theme templates
MkDocs core templates

Resolution:

# In stub front matter
template: about-us.html

# MkDocs searches:
# 1. mkdocs/overrides/about-us.html ✓ (found here)
# 2. material/templates/about-us.html
# 3. mkdocs/templates/about-us.html

Common Mistake:

# WRONG - causes TemplateNotFound
template: overrides/about-us.html

# MkDocs searches:
# 1. mkdocs/overrides/overrides/about-us.html ✗ (not found)

Solution: Use filename only, not path with overrides/.

Troubleshooting

Problem: Template Not Found Error

Symptoms:

MkDocs build fails
Error: jinja2.exceptions.TemplateNotFound: overrides/about-us.html

Causes:

Stub uses template: overrides/about-us.html (incorrect path)
custom_dir not configured in mkdocs.yml
Override file doesn't exist

Solutions:

Fix stub front matter:

# Before (wrong)
template: overrides/about-us.html

# After (correct)
template: about-us.html

Verify custom_dir:

# In mkdocs.yml
theme:
  name: material
  custom_dir: overrides

Check file exists:

ls -la mkdocs/docs/overrides/about-us.html
# Should exist if page published

Validate exports:
- Admin → Pages → "Validate Exports"
- Repairs missing files

Problem: Export Files Missing After Restart

Symptoms:

Pages were published before restart
After docker compose restart: Files gone
MkDocs build fails

Causes:

Volume mount not configured
Files written to container filesystem (not host)
Container recreated (ephemeral storage lost)

Solutions:

Check volume mount:

# In docker-compose.yml
services:
  api:
    volumes:
      - ./mkdocs:/mkdocs:rw  # Must have :rw for write access

Verify host files:

ls -la mkdocs/docs/overrides/
# Files should persist on host filesystem

Re-export all pages:
- Admin → Pages → "Validate Exports"
- Regenerates all missing files

Problem: Page Not Appearing in MkDocs Site

Symptoms:

Page published, files exist
MkDocs builds successfully
Page shows 404 on site

Causes:

Stub not listed in mkdocs.yml nav
MkDocs not rebuilt after export
Nginx cache serving old version

Solutions:

Add to nav (optional):

nav:
  - Pages:
    - About: about-us.md  # Stub filename

Rebuild MkDocs:

docker compose exec mkdocs mkdocs build
# Or Admin → Pages → "Build Site"

Clear Nginx cache:

docker compose exec nginx nginx -s reload

Test direct access:

curl http://localhost:4001/pages/about-us/
# Should return HTML, not 404

Problem: Styles Not Applying in MkDocs

Symptoms:

Page renders in GrapesJS editor
MkDocs site shows unstyled content

Causes:

CSS not exported (CODE mode without cssOutput)
Material theme CSS conflicts
Inline styles overridden

Solutions:

Check cssOutput field:

SELECT css_output FROM landing_pages WHERE slug = 'about-us';
-- Should contain CSS, not NULL

Inspect rendered HTML:

curl http://localhost:4001/pages/about-us/ | grep '<style>'
# Should include page CSS

Use !important for overrides:

/* In page CSS */
section {
  padding: 40px !important;
}

Test STANDALONE mode:
- Settings → Full page MkDocs (checked)
- Bypasses Material theme CSS

Symptoms:

Page settings: mkdocsHideNav=true
Navigation sidebar still shows

Causes:

Stub front matter not updated
MkDocs cache not cleared
STANDALONE mode enabled (hide options ignored)

Solutions:

Check stub front matter:

cat mkdocs/docs/about-us.md
# Should have:
# hide:
#   - navigation

Re-export:
- Edit page settings → Save
- Triggers stub regeneration

Clear MkDocs cache:

rm -rf mkdocs/site/
docker compose exec mkdocs mkdocs build

Verify not STANDALONE:
- Settings → Full page MkDocs (unchecked)
- STANDALONE ignores hide options

Performance Considerations

File System I/O

Export operation: Writes 2 files per page (~1ms each)

Bottleneck: Synchronous file writes in API request handler

Impact:

Publish operation: +2ms overhead
Batch operations: Linear scaling (10 pages = +20ms)

Optimization (future):

// Current: Synchronous writes in request
await fs.writeFile(path, content);

// Future: Background job queue
await queue.add('export-page', { pageId });

MkDocs Build Time

Build duration: Proportional to page count

10 pages: ~2 seconds
100 pages: ~10 seconds
1000 pages: ~90 seconds

Optimization:

Use mkdocs serve --dirtyreload in dev (incremental builds)
Production builds: Full rebuild recommended

Security Considerations

Path Traversal Protection

Validation:

Null byte check: Prevents about\0.html attacks
Normalization: path.normalize() resolves ../
Absolute path check: Rejects /etc/passwd.html
Encoded traversal: Blocks %2e%2e/admin.html
Extension validation: Must end with .html

Rejected Paths:

../../../etc/passwd.html
/var/www/config.html
admin%2e%2e%2fconfig.html
about.md (wrong extension)

File Permission Isolation

Docker Volume Mount:

volumes:
  - ./mkdocs:/mkdocs:rw

Permissions:

API container writes as node user (UID 1000)
Host user must have write access to mkdocs/docs/
MkDocs container reads as mkdocs user (UID 1001)

Risk: Container escape could write arbitrary files

Mitigation:

API container runs as non-root user
Volume mount scoped to /mkdocs only (no host root access)

Template Injection

Risk: Malicious admin injects Jinja2 code

Example:

<!-- Malicious HTML in editor -->
<h1>{{ config.site_name }}</h1>

Rendering:

THEMED mode: Jinja2 processes {{ }} expressions
Could expose MkDocs config or Material theme internals

Mitigation:

Accepted risk: Admins are trusted users
Template code only renders in MkDocs (isolated from main app)
Public users cannot edit landing pages

Frontend Components

LandingPagesPage — Export buttons + validation
PageEditorPage — Auto-export on publish

Backend Modules

pages.service — Export logic (exportToMkDocs, validateExports, syncOverrides)
pages-admin.routes — /sync and /validate endpoints

Features

Page Builder — Landing page system overview
GrapesJS Editor — Editor integration
Block Library — Reusable blocks

MkDocs Resources

MkDocs Material Templates — Theme customization
Jinja2 Documentation — Template syntax
MkDocs Configuration — mkdocs.yml reference

Deployment

Docker Setup — Volume mounts + permissions
MkDocs Service — Container configuration

21 KiB Raw Blame History

MkDocs Export Integration

Overview

Key Features

Architecture

Export Modes

THEMED Mode (Default)

STANDALONE Mode

File Outputs

Override File (.html)

Stub File (.md)

Database Fields

LandingPage Export Configuration

Admin Workflow

Exporting a Page

Configuring Export Options

Rebuilding MkDocs Site

Syncing Overrides

Validating Exports

Code Examples

Themed Mode Export

Standalone Mode Export

Markdown Stub Generation

Export Orchestration

Path Validation

MkDocs Configuration

mkdocs.yml Settings

Template Search Paths

Troubleshooting

Problem: Template Not Found Error

Problem: Export Files Missing After Restart

Problem: Page Not Appearing in MkDocs Site

Problem: Styles Not Applying in MkDocs

Problem: Hide Navigation Not Working

Performance Considerations

File System I/O

MkDocs Build Time

Security Considerations

Path Traversal Protection

File Permission Isolation

Template Injection

Related Documentation

Frontend Components

Backend Modules

Features

MkDocs Resources

Deployment

21 KiB

Raw Blame History