admin/changemaker.lite
1
0
Fork 0
Code Issues Packages Projects Releases 31 Wiki Activity
changemaker.lite/mkdocs/docs/v2/backend/modules/locations.md

36 KiB
Raw Blame History

Locations Module

Overview

The Locations module manages geographic locations for organizing campaigns, mapping volunteers, and tracking supporter data. It features multi-provider geocoding, NAR (National Address Register) bulk import with 2025 format support, CSV import/export, location history tracking, and comprehensive filtering with spatial queries.

Key Features:

  • Location CRUD with automatic geocoding
  • Multi-provider geocoding (Nominatim, Mapbox, ArcGIS, Photon, Google, LocationIQ)
  • Batch geocoding with BullMQ queue integration
  • NAR 2025 bulk import (Canadian electoral data with Lambert projection support)
  • CSV import/export with flexible column mapping
  • Location history tracking (audit trail for all changes)
  • Reverse geocoding (lat/lng → address)
  • Spatial filtering (cut polygons, bounding boxes, postal codes)
  • Deduplication (coordinate-based with configurable radius)
  • Support level tracking (LEVEL_1 through LEVEL_4)
  • Sign tracking (lawn signs, sizes)
  • Public map API (PII-filtered)
  • Statistics dashboard (geocoding quality, provider distribution, confidence levels)

File Paths

File Purpose
api/src/modules/map/locations/locations.routes.ts 2 routers (admin + public) with 20 endpoints
api/src/modules/map/locations/locations.service.ts Location business logic + geocoding + NAR import (1,100 lines)
api/src/modules/map/locations/locations.schemas.ts Zod validation schemas
api/src/modules/map/locations/nar-import.service.ts NAR import service (server-side streaming, legacy support)
api/src/modules/map/locations/nar-import.routes.ts NAR import admin routes
api/src/modules/map/locations/bulk-geocode.routes.ts Bulk geocoding queue routes
api/src/modules/map/locations/bulk-geocode.schemas.ts Bulk geocoding schemas

Database Models

Location

model Location {
  id                  String         @id @default(cuid())
  address             String
  unitNumber          String?
  firstName           String?
  lastName            String?
  email               String?
  phone               String?
  supportLevel        SupportLevel?
  sign                Boolean        @default(false)
  signSize            String?
  notes               String?        @db.Text
  buildingNotes       String?        @db.Text

  // Geocoding
  latitude            Float?
  longitude           Float?
  geocodeConfidence   Int?
  geocodeProvider     GeocodeProvider?

  // NAR fields (2025 format support)
  postalCode          String?
  province            String?
  federalDistrict     String?
  buildingUse         Int?           // 1=Residential, 2=Commercial, 3=Mixed

  // Audit
  createdByUserId     String?
  updatedByUserId     String?
  createdAt           DateTime       @default(now())
  updatedAt           DateTime       @updatedAt

  // Relations
  createdByUser       User?          @relation("LocationCreator", fields: [createdByUserId], references: [id], onDelete: SetNull)
  updatedByUser       User?          @relation("LocationUpdater", fields: [updatedByUserId], references: [id], onDelete: SetNull)
  history             LocationHistory[]

  @@index([latitude, longitude])
  @@index([supportLevel])
  @@index([sign])
  @@index([geocodeConfidence])
  @@map("locations")
}

enum SupportLevel {
  LEVEL_1  // Strong support
  LEVEL_2  // Moderate support
  LEVEL_3  // Undecided
  LEVEL_4  // Opposed
}

enum GeocodeProvider {
  NOMINATIM
  MAPBOX
  ARCGIS
  PHOTON
  GOOGLE
  LOCATIONIQ
  UNKNOWN
}

LocationHistory

model LocationHistory {
  id         String                @id @default(cuid())
  locationId String
  location   Location              @relation(fields: [locationId], references: [id], onDelete: Cascade)
  userId     String?
  user       User?                 @relation(fields: [userId], references: [id], onDelete: SetNull)
  action     LocationHistoryAction
  field      String?
  oldValue   String?
  newValue   String?
  metadata   Json?
  createdAt  DateTime              @default(now())

  @@index([locationId])
  @@index([userId])
  @@index([action])
  @@map("location_history")
}

enum LocationHistoryAction {
  CREATED
  UPDATED
  GEOCODED
  MOVED_ON_MAP
  DELETED
}

History Tracking:

  • All location changes recorded with before/after values
  • CREATED — Location created (manual or import)
  • UPDATED — Field changed
  • GEOCODED — Address geocoded (auto or bulk geocoding)
  • MOVED_ON_MAP — Lat/lng changed via map drag
  • DELETED — Location deleted

API Endpoints

Admin Endpoints (Authentication Required)

Method Path Description
GET /api/map/locations List locations (paginated, filtered)
GET /api/map/locations/stats Location statistics
GET /api/map/locations/export-csv Export CSV download
GET /api/map/locations/all All geocoded locations for map (admin, 5000 limit)
GET /api/map/locations/:id Get single location
GET /api/map/locations/:id/history Get location edit history
POST /api/map/locations Create location (auto-geocodes if no lat/lng)
POST /api/map/locations/geocode Geocode single address
POST /api/map/locations/geocode-missing Geocode all ungeocoded locations
POST /api/map/locations/import-csv Upload + import CSV (10MB limit)
POST /api/map/locations/import-bulk Bulk import NAR or CSV (100MB limit, 5min timeout)
POST /api/map/locations/reverse-geocode Reverse geocode lat/lng to address
POST /api/map/locations/bulk-delete Delete multiple locations
PUT /api/map/locations/:id Update location
DELETE /api/map/locations/:id Delete location

Admin Roles: SUPER_ADMIN, MAP_ADMIN

Public Endpoints (No Authentication)

Method Path Description
GET /api/map/locations/public Public locations for map (PII-filtered, 5000 limit)

Admin Endpoint Details

GET /api/map/locations

List locations with pagination, search, and filtering.

Query Parameters:

Parameter Type Required Default Description
page number No 1 Page number
limit number No 20 Results per page (max 100)
search string No - Search address, first/last name, email
supportLevel SupportLevel No - Filter by support level
hasSign boolean No - Filter by sign presence
confidenceLevel string No - Filter by geocode confidence: high (85+), medium (60-84), low (<60), none (0 or null)
sortBy string No createdAt Sort field: createdAt, address, supportLevel
sortOrder string No desc Sort order: asc, desc

Example Request:

curl -H "Authorization: Bearer <token>" \
  "http://api.cmlite.org/api/map/locations?page=1&limit=20&supportLevel=LEVEL_1&hasSign=true&confidenceLevel=high"

Response (200 OK):

{
  "locations": [
    {
      "id": "clx1234567890",
      "address": "123 Main St, Toronto, ON",
      "unitNumber": "Apt 4",
      "firstName": "John",
      "lastName": "Doe",
      "email": "john@example.com",
      "phone": "416-555-1234",
      "supportLevel": "LEVEL_1",
      "sign": true,
      "signSize": "Large",
      "notes": "Willing to volunteer",
      "buildingNotes": "Apartment building, intercom required",
      "latitude": 43.6532,
      "longitude": -79.3832,
      "geocodeConfidence": 95,
      "geocodeProvider": "NOMINATIM",
      "postalCode": "M5H 2N2",
      "province": "ON",
      "federalDistrict": "Toronto Centre",
      "buildingUse": 1,
      "createdByUserId": "clxUser123",
      "updatedByUserId": null,
      "createdAt": "2026-02-08T12:00:00.000Z",
      "updatedAt": "2026-02-08T12:00:00.000Z"
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 20,
    "total": 342,
    "totalPages": 18
  }
}

Search Logic:

if (search) {
  where.OR = [
    { address: { contains: search, mode: 'insensitive' } },
    { firstName: { contains: search, mode: 'insensitive' } },
    { lastName: { contains: search, mode: 'insensitive' } },
    { email: { contains: search, mode: 'insensitive' } },
  ];
}

Confidence Level Filtering:

if (confidenceLevel === 'high') {
  where.geocodeConfidence = { gte: 85 };
} else if (confidenceLevel === 'medium') {
  where.geocodeConfidence = { gte: 60, lt: 85 };
} else if (confidenceLevel === 'low') {
  where.geocodeConfidence = { lt: 60, gt: 0 };
} else if (confidenceLevel === 'none') {
  where.OR = [{ geocodeConfidence: null }, { geocodeConfidence: 0 }];
}

GET /api/map/locations/stats

Get aggregate statistics for locations.

Example Request:

curl -H "Authorization: Bearer <token>" \
  "http://api.cmlite.org/api/map/locations/stats"

Response (200 OK):

{
  "total": 1247,
  "supportLevels": {
    "LEVEL_1": 342,
    "LEVEL_2": 189,
    "LEVEL_3": 276,
    "LEVEL_4": 98,
    "NONE": 342
  },
  "signs": 142,
  "geocoded": 1189,
  "ungeocoded": 58,
  "confidence": {
    "high": 892,
    "medium": 213,
    "low": 84,
    "none": 58,
    "average": 87
  },
  "providers": {
    "nominatim": 654,
    "mapbox": 312,
    "arcgis": 98,
    "photon": 76,
    "google": 34,
    "locationiq": 15,
    "manual": 58
  }
}

Field Descriptions:

  • total — Total location count
  • supportLevels — Breakdown by support level
  • signs — Locations with sign=true
  • geocoded — Locations with lat/lng
  • ungeocoded — Locations without lat/lng
  • confidence.high — Geocode confidence ≥ 85
  • confidence.medium — Geocode confidence 60-84
  • confidence.low — Geocode confidence < 60
  • confidence.none — No geocode confidence (0 or null)
  • confidence.average — Average geocode confidence (excludes 0/null)
  • providers — Breakdown by geocode provider

POST /api/map/locations

Create new location with automatic geocoding.

Request Body:

{
  "address": "123 Main St, Toronto, ON",
  "unitNumber": "Apt 4",
  "firstName": "John",
  "lastName": "Doe",
  "email": "john@example.com",
  "phone": "416-555-1234",
  "supportLevel": "LEVEL_1",
  "sign": true,
  "signSize": "Large",
  "notes": "Willing to volunteer",
  "buildingNotes": "Apartment building, intercom required"
}

Response (201 Created):

Returns created location object.

Auto-Geocoding:

If address provided and no latitude/longitude, automatically geocodes:

if (data.address && data.latitude == null && data.longitude == null) {
  const result = await geocodingService.geocode(data.address);
  if (result) {
    createData.latitude = result.latitude;
    createData.longitude = result.longitude;
    createData.geocodeConfidence = result.confidence;
    createData.geocodeProvider = result.provider;
  }
}

History Tracking:

Creates LocationHistory record with action GEOCODED (if geocoded) or CREATED (if manual coordinates).

PUT /api/map/locations/:id

Update location. Re-geocodes if address changes without explicit lat/lng.

Request Body (Partial):

{
  "address": "456 Oak St, Toronto, ON",
  "supportLevel": "LEVEL_2"
}

Response (200 OK):

Returns updated location object.

Smart Geocoding:

  • If address changes and no explicit lat/lng provided: re-geocode automatically
  • If lat/lng provided: use provided coordinates (manual override)

History Tracking:

Records field changes with before/after values:

// Track changes
const changes: { field: string; oldValue: unknown; newValue: unknown }[] = [];

if (data.address && data.address !== existing.address) {
  changes.push({ field: 'address', oldValue: existing.address, newValue: data.address });
}

// Determine action based on changes
let action: LocationHistoryAction = LocationHistoryAction.UPDATED;

if (data.latitude !== undefined && data.latitude !== existing.latitude) {
  action = LocationHistoryAction.MOVED_ON_MAP; // Explicit coordinate change (map drag)
}

if (address changed && auto-geocoded) {
  action = LocationHistoryAction.GEOCODED;
}

POST /api/map/locations/import-csv

Upload and import CSV file with flexible column mapping.

Multipart Form Data:

  • file (required): CSV file (max 10MB)

Supported Column Names (Case-Insensitive):

Field Column Names
address address, street, street address
firstName first name, firstname, first
lastName last name, lastname, last
email email, e-mail
phone phone, telephone, tel, phone number
unitNumber unit, unit number, apt, apartment, suite
supportLevel support level, supportlevel, support, level
sign sign, lawn sign
signSize sign size, signsize
notes notes, note, comments
latitude latitude, lat
longitude longitude, lng, lon

Example CSV:

address,first name,last name,email,phone,support level,sign
"123 Main St, Toronto, ON",John,Doe,john@example.com,416-555-1234,LEVEL_1,true
"456 Oak St, Toronto, ON",Jane,Smith,jane@example.com,416-555-5678,LEVEL_2,false

Example Request:

curl -X POST -H "Authorization: Bearer <token>" \
  -F "file=@locations.csv" \
  "http://api.cmlite.org/api/map/locations/import-csv"

Response (200 OK):

{
  "total": 1000,
  "success": 942,
  "warnings": 34,
  "failed": 24,
  "errors": [
    "Row 12: Missing address",
    "Row 45: Invalid email format",
    "Row 89: Geocoding failed"
  ]
}

Field Descriptions:

  • total — Total rows in CSV
  • success — Successfully created locations
  • warnings — Created but geocoding failed (no lat/lng)
  • failed — Failed to create (validation errors)
  • errors — First 50 error messages (row numbers 1-indexed)

Geocoding:

  • If CSV has latitude/longitude columns: uses provided coordinates
  • Otherwise: auto-geocodes each address (slow for large files, consider NAR import for bulk)

POST /api/map/locations/import-bulk

Bulk import NAR (National Address Register) or standard CSV with advanced filtering.

Multipart Form Data:

  • file (required): CSV file (max 100MB)
  • format (required): nar or standard
  • filterType (optional): none, cut, mapArea, city, province
  • cutId (optional): Cut ID for filterType=cut
  • filterCity (optional): City name for filterType=city
  • filterProvince (optional): Province code for filterType=province (e.g., ON, BC)
  • residentialOnly (optional, default: false): Skip non-residential buildings (NAR only)
  • deduplicateRadius (optional, default: 5): Coordinate deduplication radius in meters
  • skipGeocoding (optional, default: true): Skip geocoding (NAR files have coordinates)
  • batchSize (optional, default: 1000): Database batch insert size

Request Timeout: 5 minutes (extended for large files)

Example Request (NAR Import with Cut Filter):

curl -X POST -H "Authorization: Bearer <token>" \
  -F "file=@Address_24_part_1.csv" \
  -F "format=nar" \
  -F "filterType=cut" \
  -F "cutId=clxCut123" \
  -F "residentialOnly=true" \
  -F "deduplicateRadius=5" \
  "http://api.cmlite.org/api/map/locations/import-bulk"

Response (200 OK):

{
  "total": 50000,
  "created": 12847,
  "skippedDuplicate": 1243,
  "skippedOutOfBounds": 34892,
  "skippedInvalid": 1018,
  "errors": [
    "Row 234: Invalid coordinates",
    "Row 1892: Missing civic number"
  ]
}

NAR Format Support:

2025 NAR Format (Recommended):

  • Address File Columns: CIVIC_NO, CIVIC_NO_SUFFIX, OFFICIAL_STREET_NAME, OFFICIAL_STREET_TYPE, OFFICIAL_STREET_DIR, APT_NO_LABEL, BG_X, BG_Y, MAIL_MUN_NAME, MAIL_PROV_ABVN, MAIL_POSTAL_CODE, FED_ENG_NAME, BU_USE
  • Location File Columns: BG_LATITUDE, BG_LONGITUDE (WGS84), LOC_GUID
  • Coordinate Systems:
    • BG_X/BG_Y — EPSG:3347 Lambert Conformal Conic (converted to WGS84)
    • BG_LATITUDE/BG_LONGITUDE — WGS84 (used directly)

Legacy NAR Format (Backward Compatible):

  • Columns: STR_NBR, STR_NME, STR_TYP, STR_DIR, LAT, LNG, MUN_NME, PRV_NME

Auto-Detection:

If 3+ NAR-specific columns detected, automatically treats as NAR format.

Lambert Projection Conversion:

import proj4 from 'proj4';

// Define EPSG:3347 (Statistics Canada Lambert Conformal Conic)
proj4.defs('EPSG:3347', '+proj=lcc +lat_1=49 +lat_2=77 +lat_0=63.390675 +lon_0=-91.86666666666666 +x_0=6200000 +y_0=3000000 +ellps=GRS80 +units=m +no_defs');

function lambertToLatLng(bgX: number, bgY: number): [number, number] {
  const [lng, lat] = proj4('EPSG:3347', 'EPSG:4326', [bgX, bgY]);
  return [lat, lng];
}

Filtering Options:

  1. Cut Filter (filterType=cut):

    • Only imports locations inside specified cut polygon
    • Uses point-in-polygon ray-casting algorithm

  2. Map Area Filter (filterType=mapArea):

    • Imports locations visible on current map view
    • Calculates bounding box from MapSettings (center, zoom)

  3. City Filter (filterType=city):

    • Imports locations matching city name (case-insensitive)

  4. Province Filter (filterType=province):

    • Imports locations matching province code (e.g., ON, BC)

Deduplication:

Prevents duplicate locations at same coordinates:

const coordKey = `${roundCoord(lat, 5)}:${roundCoord(lng, 5)}`; // 5 decimal places = ~1.1m precision

if (existingCoords.has(coordKey) || inFileCoords.has(coordKey)) {
  skippedDuplicate++;
  continue;
}

Batch Processing:

Inserts locations in batches (default 1000) for performance:

const batch: Prisma.LocationCreateManyInput[] = [];

// ... collect locations ...

if (batch.length >= options.batchSize) {
  await prisma.location.createMany({ data: batch, skipDuplicates: true });
  batch.length = 0;
}

GET /api/map/locations/export-csv

Export locations as CSV download.

Example Request:

curl -H "Authorization: Bearer <token>" \
  "http://api.cmlite.org/api/map/locations/export-csv" \
  -o locations.csv

Response (200 OK):

CSV file with headers:

address,firstName,lastName,email,phone,unitNumber,supportLevel,sign,signSize,notes,latitude,longitude,geocodeConfidence,geocodeProvider,createdAt
"123 Main St, Toronto, ON",John,Doe,john@example.com,416-555-1234,Apt 4,LEVEL_1,Yes,Large,Willing to volunteer,43.6532,-79.3832,95,NOMINATIM,2026-02-08T12:00:00.000Z

POST /api/map/locations/reverse-geocode

Reverse geocode coordinates to address.

Request Body:

{
  "latitude": 43.6532,
  "longitude": -79.3832
}

Response (200 OK):

{
  "address": "123 Main St, Toronto, ON M5H 2N2, Canada",
  "provider": "NOMINATIM",
  "confidence": 85
}

Use Cases:

  • Click-to-add location on map (get address from coordinates)
  • Move location on map (update address after drag)
  • Verify coordinates match expected address

GET /api/map/locations/all

Get all geocoded locations for admin map view.

Query Parameters:

Parameter Type Description
minLat number Minimum latitude (bounding box)
maxLat number Maximum latitude
minLng number Minimum longitude
maxLng number Maximum longitude

Example Request:

# All locations
curl -H "Authorization: Bearer <token>" \
  "http://api.cmlite.org/api/map/locations/all"

# Bounding box (visible map area)
curl -H "Authorization: Bearer <token>" \
  "http://api.cmlite.org/api/map/locations/all?minLat=43.6&maxLat=43.7&minLng=-79.4&maxLng=-79.3"

Response (200 OK):

Returns array of location objects (max 5000).

Safety Limit:

If result hits 5000 locations, adds header X-Location-Limit-Hit: true to warn client.

GET /api/map/locations/:id/history

Get location edit history with audit trail.

Query Parameters:

  • page (optional, default: 1): Page number
  • limit (optional, default: 20): Results per page

Example Request:

curl -H "Authorization: Bearer <token>" \
  "http://api.cmlite.org/api/map/locations/clx1234567890/history?page=1&limit=20"

Response (200 OK):

{
  "history": [
    {
      "id": "clxHistory123",
      "locationId": "clx1234567890",
      "userId": "clxUser123",
      "user": {
        "id": "clxUser123",
        "email": "admin@example.com",
        "name": "Admin User",
        "role": "SUPER_ADMIN"
      },
      "action": "MOVED_ON_MAP",
      "field": "latitude",
      "oldValue": "43.6532",
      "newValue": "43.6540",
      "metadata": null,
      "createdAt": "2026-02-11T12:00:00.000Z"
    },
    {
      "id": "clxHistory124",
      "locationId": "clx1234567890",
      "userId": "clxUser123",
      "user": {...},
      "action": "GEOCODED",
      "field": "latitude",
      "oldValue": null,
      "newValue": "43.6532",
      "metadata": {
        "provider": "NOMINATIM",
        "confidence": 95,
        "geocoded": true
      },
      "createdAt": "2026-02-08T12:00:00.000Z"
    },
    {
      "id": "clxHistory125",
      "locationId": "clx1234567890",
      "userId": "clxUser123",
      "user": {...},
      "action": "CREATED",
      "field": null,
      "oldValue": null,
      "newValue": null,
      "metadata": null,
      "createdAt": "2026-02-08T12:00:00.000Z"
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 20,
    "total": 7,
    "totalPages": 1
  }
}

History Actions:

  • CREATED — Location created
  • UPDATED — Field changed (address, name, email, etc.)
  • GEOCODED — Auto-geocoded (address → lat/lng)
  • MOVED_ON_MAP — Coordinates changed via map drag
  • DELETED — Location deleted (orphaned history records)

Public Endpoint Details

GET /api/map/locations/public

Get locations for public map (PII-filtered).

Query Parameters:

  • minLat, maxLat, minLng, maxLng (optional): Bounding box

Example Request:

curl "http://api.cmlite.org/api/public/map/locations?minLat=43.6&maxLat=43.7&minLng=-79.4&maxLng=-79.3"

Response (200 OK):

[
  {
    "id": "clx1234567890",
    "latitude": 43.6532,
    "longitude": -79.3832,
    "supportLevel": "LEVEL_1",
    "sign": true,
    "signSize": "Large",
    "unitNumber": "Apt 4",
    "address": "123 Main St, Toronto, ON"
  }
]

PII Filtering:

Only returns non-sensitive fields:

  • Included: id, latitude, longitude, supportLevel, sign, signSize, unitNumber, address
  • Excluded: firstName, lastName, email, phone, notes, buildingNotes, geocodeConfidence, geocodeProvider, createdByUserId, postalCode, province, federalDistrict, buildingUse

Service Functions

locationsService.create(data, userId)

Create location with auto-geocoding.

Auto-Geocoding Logic:

if (data.address && data.latitude == null && data.longitude == null) {
  const result = await geocodingService.geocode(data.address);
  if (result) {
    createData.latitude = result.latitude;
    createData.longitude = result.longitude;
    createData.geocodeConfidence = result.confidence;
    createData.geocodeProvider = result.provider;
  }
}

History Recording:

Creates history record in transaction:

const location = await prisma.$transaction(async (tx) => {
  const newLocation = await tx.location.create({ data: createData });

  await tx.locationHistory.create({
    data: {
      locationId: newLocation.id,
      userId,
      action: geocodeMetadata ? LocationHistoryAction.GEOCODED : LocationHistoryAction.CREATED,
      metadata: geocodeMetadata,
    },
  });

  return newLocation;
});

locationsService.update(id, data, userId)

Update location with smart geocoding and history tracking.

Smart Geocoding:

  • If address changes and no explicit lat/lng: re-geocode
  • If lat/lng provided: use provided coordinates (manual override)

Action Detection:

let action: LocationHistoryAction = LocationHistoryAction.UPDATED;

// Explicit coordinate change (map drag)
if (data.latitude !== undefined && data.latitude !== existing.latitude) {
  action = LocationHistoryAction.MOVED_ON_MAP;
}

// Auto-geocode on address change
if (data.address && data.address !== existing.address && !data.latitude && !data.longitude) {
  const result = await geocodingService.geocode(data.address);
  if (result) {
    updateData.latitude = result.latitude;
    updateData.longitude = result.longitude;
    action = LocationHistoryAction.GEOCODED;
  }
}

Change Tracking:

const changes: { field: string; oldValue: unknown; newValue: unknown }[] = [];

const fieldsToTrack = ['address', 'firstName', 'lastName', 'email', 'phone', 'unitNumber', 'supportLevel', 'sign', 'signSize', 'notes'];

for (const field of fieldsToTrack) {
  if (data[field] !== undefined && data[field] !== existing[field]) {
    changes.push({ field, oldValue: existing[field], newValue: data[field] });
  }
}

// Record all changes in transaction
await tx.locationHistory.createMany({ data: historyRecords });

locationsService.importFromCsv(buffer, userId)

Import CSV with flexible column mapping.

Column Mapping:

const CSV_HEADER_MAP: Record<string, keyof CsvRow> = {
  'address': 'address',
  'street': 'address',
  'street address': 'address',
  'first name': 'firstName',
  'firstname': 'firstName',
  // ... 50+ mappings
};

Processing:

  1. Parse CSV with csv-parse library
  2. Detect column mapping from headers
  3. For each row:
    • Validate required fields (address)
    • Parse support level, sign boolean
    • Use provided lat/lng or geocode address
    • Create location in database
  4. Return summary statistics

locationsService.importBulk(buffer, userId, options, filters)

Bulk import NAR or standard CSV with advanced filtering.

NAR Format Detection:

function detectNarFormat(headers: string[]): boolean {
  const NAR_DETECT_COLUMNS = [
    'CIVIC_NO', 'OFFICIAL_STREET_NAME', 'BG_X', 'BG_Y', // 2025 format
    'STR_NBR', 'STR_NME', 'LAT', 'LNG',                 // Legacy format
  ];

  const normalizedHeaders = headers.map((h) => h.trim().toUpperCase());
  let matchCount = 0;

  for (const col of NAR_DETECT_COLUMNS) {
    if (normalizedHeaders.includes(col)) matchCount++;
  }

  return matchCount >= 3; // At least 3 NAR columns
}

3-Phase Processing:

Phase 1: Parse & Filter

// Parse all records
for (const record of records) {
  // Build address from NAR fields
  const civicNo = getValue('CIVIC_NO');
  const streetName = getValue('STREET_NAME');
  const address = [civicNo, streetName, ...].join(' ');

  // Apply filters
  if (filters?.city && !matchesCity(address, filters.city)) {
    skippedOutOfBounds++;
    continue;
  }

  // Residential filter
  if (options.residentialOnly && buildingUse === 3) {
    skippedOutOfBounds++;
    continue;
  }

  parsedRecords.push({ address, lat, lng, needsGeocoding });
}

Phase 2: Batch Geocode

// Collect addresses needing geocoding
const addressesToGeocode: string[] = parsedRecords
  .filter(r => r.needsGeocoding)
  .map(r => r.address);

// Batch geocode (parallel)
const geocodeResults = await geocodingService.geocodeBatch(addressesToGeocode);

Phase 3: Create Records

const batch: Prisma.LocationCreateManyInput[] = [];

for (const parsed of parsedRecords) {
  // Apply geocoding result
  if (parsed.needsGeocoding) {
    const result = geocodeResults[geocodeIndex];
    if (result) {
      lat = result.latitude;
      lng = result.longitude;
    }
  }

  // Cut polygon filter
  if (filters?.cutPolygon) {
    if (!isPointInPolygon(lat, lng, cutPolygon)) {
      skippedOutOfBounds++;
      continue;
    }
  }

  // Deduplication
  if (existingCoords.has(coordKey)) {
    skippedDuplicate++;
    continue;
  }

  batch.push({ address, lat, lng, ... });

  // Flush batch
  if (batch.length >= options.batchSize) {
    await prisma.location.createMany({ data: batch });
    batch.length = 0;
  }
}

locationsService.exportToCsv(filters?)

Export locations as CSV.

CSV Generation:

import { stringify } from 'csv-stringify/sync';

const rows = locations.map((loc) => ({
  address: loc.address || '',
  firstName: loc.firstName || '',
  lastName: loc.lastName || '',
  email: loc.email || '',
  phone: loc.phone || '',
  unitNumber: loc.unitNumber || '',
  supportLevel: loc.supportLevel || '',
  sign: loc.sign ? 'Yes' : 'No',
  signSize: loc.signSize || '',
  notes: loc.notes || '',
  latitude: loc.latitude?.toString() || '',
  longitude: loc.longitude?.toString() || '',
  geocodeConfidence: loc.geocodeConfidence?.toString() || '',
  geocodeProvider: loc.geocodeProvider || '',
  createdAt: loc.createdAt.toISOString(),
}));

return stringify(rows, { header: true });

Validation Schemas

Create Location Schema

export const createLocationSchema = z.object({
  address: z.string().min(1, 'Address is required'),
  firstName: z.string().optional(),
  lastName: z.string().optional(),
  email: z.string().email().optional().or(z.literal('')),
  phone: z.string().optional(),
  unitNumber: z.string().optional(),
  supportLevel: z.nativeEnum(SupportLevel).optional(),
  sign: z.boolean().optional().default(false),
  signSize: z.string().optional(),
  notes: z.string().optional(),
  buildingNotes: z.string().max(2000).optional(),
  latitude: z.number().min(-90).max(90).optional(),
  longitude: z.number().min(-180).max(180).optional(),
});

Bulk Import Schema

export const bulkImportSchema = z.object({
  format: z.enum(['standard', 'nar']).default('standard'),
  filterType: z.enum(['none', 'cut', 'mapArea', 'city', 'province']).default('none'),
  cutId: z.string().optional(),
  filterCity: z.string().optional(),
  filterProvince: z.string().optional(),
  residentialOnly: z.coerce.boolean().default(false),
  deduplicateRadius: z.coerce.number().min(0).max(100).default(5),
  skipGeocoding: z.coerce.boolean().default(true),
  batchSize: z.coerce.number().int().min(100).max(5000).default(1000),
});

Code Examples

Admin: Create Location with Auto-Geocoding

import { api } from '@/lib/api';
import { message } from 'antd';

const createLocation = async () => {
  try {
    const { data } = await api.post('/api/map/locations', {
      address: '123 Main St, Toronto, ON',
      firstName: 'John',
      lastName: 'Doe',
      email: 'john@example.com',
      supportLevel: 'LEVEL_1',
      sign: true,
    });

    message.success('Location created and geocoded');
    console.log(`Created at: ${data.latitude}, ${data.longitude}`);
    console.log(`Confidence: ${data.geocodeConfidence}%`);
  } catch (error) {
    message.error('Failed to create location');
  }
};

Admin: Import NAR File with Cut Filter

import { api } from '@/lib/api';
import { message } from 'antd';

const importNAR = async (file: File, cutId: string) => {
  const formData = new FormData();
  formData.append('file', file);
  formData.append('format', 'nar');
  formData.append('filterType', 'cut');
  formData.append('cutId', cutId);
  formData.append('residentialOnly', 'true');
  formData.append('deduplicateRadius', '5');

  try {
    const { data } = await api.post('/api/map/locations/import-bulk', formData, {
      headers: { 'Content-Type': 'multipart/form-data' },
      timeout: 300000, // 5 minutes
    });

    message.success(`Created ${data.created} locations`);
    console.log(`Skipped ${data.skippedDuplicate} duplicates`);
    console.log(`Skipped ${data.skippedOutOfBounds} out of bounds`);
  } catch (error) {
    message.error('NAR import failed');
  }
};

Admin: Export Locations

import { api } from '@/lib/api';
import { message } from 'antd';

const exportLocations = async () => {
  try {
    const { data } = await api.get('/api/map/locations/export-csv', {
      responseType: 'blob',
    });

    const url = window.URL.createObjectURL(new Blob([data]));
    const link = document.createElement('a');
    link.href = url;
    link.setAttribute('download', 'locations.csv');
    document.body.appendChild(link);
    link.click();
    link.remove();

    message.success('Locations exported');
  } catch (error) {
    message.error('Export failed');
  }
};

Frontend Integration

The LocationsPage component (admin/src/pages/LocationsPage.tsx) provides:

  • Location table with pagination (20 results/page)
  • Search (address, name, email)
  • Filters (support level, sign, confidence level)
  • Sorting (createdAt, address, supportLevel)
  • Statistics dashboard (total, support levels, signs, geocoded, confidence breakdown, provider distribution)
  • Create location modal (form with auto-geocoding preview)
  • Edit location modal (pre-populated form)
  • Delete location action
  • Bulk delete (select multiple rows)
  • CSV import (10MB limit)
  • NAR bulk import (100MB limit, cut/city/province filters)
  • CSV export (download button)
  • Geocode missing button (batch geocodes all ungeocoded)
  • Location history drawer (audit trail with user, action, field changes)
  • Map integration (shows all geocoded locations, click-to-add, drag-to-move)

State Management:

const [locations, setLocations] = useState<Location[]>([]);
const [pagination, setPagination] = useState({ page: 1, limit: 20, total: 0 });
const [filters, setFilters] = useState({ search: '', supportLevel: null, hasSign: null, confidenceLevel: null });
const [stats, setStats] = useState({ total: 0, supportLevels: {}, signs: 0, geocoded: 0, ungeocoded: 0, confidence: {}, providers: {} });

Performance Considerations

Batch Processing:

  • NAR import uses 1000-record batches (configurable)
  • Reduces transaction overhead
  • Improves import speed (10,000+ locations/minute)

Deduplication:

  • Coordinate-based (5 decimal places = ~1.1m precision)
  • In-memory Set for fast lookups
  • Prevents duplicate imports within same file

Indexing:

  • @@index([latitude, longitude]) — Fast map bounds queries
  • @@index([supportLevel]) — Fast filtering by support level
  • @@index([sign]) — Fast sign filtering
  • @@index([geocodeConfidence]) — Fast confidence filtering

Safety Limits:

  • Map queries limited to 5000 locations
  • CSV import limited to 10MB
  • Bulk import limited to 100MB (5-minute timeout)
  • Bulk import warning header when limit hit

Geocoding:

  • Auto-geocodes on create/update (individual addresses)
  • Batch geocoding for bulk imports (parallel processing)
  • Uses BullMQ queue for background geocoding (separate service)

Troubleshooting

Issue: CSV import fails with "Invalid CSV file format"

Cause: CSV not UTF-8 encoded or has malformed rows

Solution:

  • Save CSV as UTF-8 in Excel/LibreOffice
  • Ensure no missing quote delimiters
  • Remove empty rows at end of file

Issue: NAR import skips all records (skippedOutOfBounds = total)

Cause: Cut/city/province filter doesn't match any records

Solution:

  • Verify cut ID is correct
  • Check city/province spelling matches NAR data (case-insensitive)
  • Try without filters first to verify file format

Issue: Geocoding confidence is low (<60) for many locations

Cause: Incomplete addresses or geocoding provider limitations

Solution:

  • Use NAR import (has pre-geocoded coordinates)
  • Add city/province to addresses
  • Try different geocoding provider (see settings)
  • Use "Geocode Missing" button to retry with fallback providers

Issue: Bulk import times out after 5 minutes

Cause: File too large or too many locations to geocode

Solution:

  • Set skipGeocoding=true for NAR imports (coordinates included)
  • Split large files into smaller batches
  • Use cut filter to reduce import size
  • Increase batchSize parameter (1000 → 2000)

Related Documentation