| 1 | # Sightline - ALPR / Flock Intelligence Platform |
| 2 | |
| 3 | A private intelligence aggregation platform built around Automatic License Plate Recognition |
| 4 | (ALPR) data. Sightline ingests Flock-style ALPR exports, normalizes and deduplicates them, |
| 5 | maps camera networks, correlates vehicle sightings into investigation timelines, reconstructs |
| 6 | movement and detects anomalies, manages watchlists and alerts, generates analyst reports, and |
| 7 | enriches investigations with passive OSINT. |
| 8 | |
| 9 | **ALPR is the core product. OSINT enrichment is a supporting module.** |
| 10 | |
| 11 | > Responsible-use note: ALPR data is privacy-sensitive. Sightline assumes the operator holds |
| 12 | > the data lawfully and is accountable for its use. Access control, full audit logging, source |
| 13 | > attribution, and configurable retention are first-class features so that use is traceable. |
| 14 | |
| 15 | --- |
| 16 | |
| 17 | ## What works today (verified end-to-end against real PostgreSQL + Redis + Celery) |
| 18 | |
| 19 | | Capability | Status | |
| 20 | |---|---| |
| 21 | | JWT auth + role-based access control (admin/analyst/viewer) | ✅ verified | |
| 22 | | **API-key auth** (X-API-Key) for machine-to-machine, with revoke | ✅ verified | |
| 23 | | **OIDC / SSO** login (config-gated, standard auth-code flow) | ✅ implemented | |
| 24 | | CSV + JSON + **PDF** Flock-style ingest with column auto-mapping | ✅ verified | |
| 25 | | Normalization (plate/timestamp/direction/confidence/geo) | ✅ unit-tested | |
| 26 | | Deterministic dedup (idempotent re-import, sub-second burst collapse) | ✅ verified | |
| 27 | | **Async ingest** via Celery worker (+ synchronous path) | ✅ verified | |
| 28 | | **Real-time stream ingest** (push reads, no file) | ✅ verified | |
| 29 | | **API connectors** (generic JSON-REST poller, webhook) | ✅ verified | |
| 30 | | Plate / partial-plate / unified search | ✅ verified | |
| 31 | | Movement intelligence: route reconstruction, haversine speed, **impossible-travel anomaly detection**, dwell, frequency, co-travel, camera transitions | ✅ unit-tested + live | |
| 32 | | Watchlists + inline hit generation + history backfill | ✅ verified | |
| 33 | | Investigation timeline + entity graph | ✅ verified | |
| 34 | | **Evidence** with SHA-256 chain-of-custody + integrity-checked download | ✅ verified | |
| 35 | | Report generation (Markdown/JSON/CSV/**PDF**) with auto key-findings | ✅ verified | |
| 36 | | OSINT enrichment plugins (DNS, RDAP, crt.sh, IP/ASN, security.txt, robots, GitHub, TLS) | ✅ verified | |
| 37 | | **Pluggable storage** (local filesystem / S3-compatible) | ✅ verified | |
| 38 | | **Retention enforcement** (admin trigger + scheduled Celery beat, cascade-safe) | ✅ verified | |
| 39 | | Immutable audit log on every mutation (entity-linked) | ✅ verified | |
| 40 | | In-process rate limiting | ✅ | |
| 41 | |
| 42 | Frontend is a complete Next.js app (17 routes) wired to the live API: dashboard, investigations |
| 43 | list/detail with evidence + report export, plate search, camera map + detail, vehicle profile, |
| 44 | entity graph, watchlist manager, alerts, ALPR import, OSINT panel, audit log, and an admin |
| 45 | console (settings, retention, API keys, connectors, users, sources). |
| 46 | |
| 47 | ## Quick start (Docker) |
| 48 | |
| 49 | ```bash |
| 50 | cp .env.example .env |
| 51 | docker compose up --build |
| 52 | # API → http://localhost:8000/docs (seeded automatically) |
| 53 | # UI → http://localhost:3000 |
| 54 | # login → admin@sightline.app / changeme123 |
| 55 | ``` |
| 56 | |
| 57 | ## Quick start (local backend) |
| 58 | |
| 59 | ```bash |
| 60 | cd backend |
| 61 | python -m venv .venv && . .venv/bin/activate |
| 62 | pip install -r requirements.txt |
| 63 | |
| 64 | # point at a Postgres instance |
| 65 | export SIGHTLINE_DATABASE_URL="postgresql+psycopg2://sightline:sightline@localhost:5432/sightline" |
| 66 | alembic upgrade head |
| 67 | python seed.py # admin@sightline.app / changeme123 + sample data |
| 68 | uvicorn app.main:app --reload # http://localhost:8000/docs |
| 69 | |
| 70 | pytest # 16 tests: pipeline + analytics, no DB required |
| 71 | ``` |
| 72 | |
| 73 | ## Architecture |
| 74 | |
| 75 | ``` |
| 76 | Next.js (TS) ──REST+JWT──▶ FastAPI ──▶ PostgreSQL (pg_trgm partial-plate search, FTS-ready) |
| 77 | │ └──▶ Redis + Celery (ingest, enrichment, retention jobs) |
| 78 | └──────────▶ Object store (evidence + raw import blobs) |
| 79 | Parser pipeline: CSV/JSON/PDF → normalize → dedup → persist → watchlist eval → analytics → audit |
| 80 | ``` |
| 81 | |
| 82 | Full design - schema, API routes, parser pipeline, analytics, frontend plan, file tree - is in |
| 83 | [`docs/DESIGN.md`](docs/DESIGN.md). |
| 84 | |
| 85 | ## Demonstration highlight: impossible-travel detection |
| 86 | |
| 87 | The seeded data includes a plate (`GHI5566`) observed at two cameras ~13.6 km apart only |
| 88 | 60 seconds apart. The movement engine computes the implied speed (~818 km/h) and flags it: |
| 89 | |
| 90 | ``` |
| 91 | GET /api/v1/vehicles/GHI5566/movement |
| 92 | → "anomalies": [{ "reason": "impossible_travel_speed", "speed_kmh": 817.8, ... }] |
| 93 | ``` |
| 94 | |
| 95 | This surfaces likely plate misreads or cloned plates - a core ALPR analytic. |
| 96 | |
| 97 | ## Project layout |
| 98 | |
| 99 | ``` |
| 100 | Sightline/ |
| 101 | ├── docker-compose.yml # postgres, redis, api, worker, beat, frontend |
| 102 | ├── docs/DESIGN.md # full architecture & schema |
| 103 | ├── backend/ # FastAPI + SQLAlchemy + Alembic + Celery |
| 104 | │ ├── app/ |
| 105 | │ │ ├── core/ # config, security, storage, rate limit, permissions |
| 106 | │ │ ├── api/routes/ # auth, oidc, investigations, imports, live, evidence, |
| 107 | │ │ │ # search, cameras, watchlists, enrichment, api-keys, |
| 108 | │ │ │ # reports, admin |
| 109 | │ │ ├── services/ # normalize, dedup, ingest, parsers (csv/json/pdf), |
| 110 | │ │ │ # analytics, watchlist, reporting, report_pdf, |
| 111 | │ │ │ # connectors, retention, enrichment plugins |
| 112 | │ │ └── workers/ # celery app + tasks (ingest, enrichment, retention) |
| 113 | │ ├── alembic/ # migrations |
| 114 | │ ├── seed.py # idempotent sample-data loader |
| 115 | │ ├── sample_data/ # Flock-style CSV + JSON + PDF exports |
| 116 | │ └── tests/ # pytest: pipeline + analytics (17 tests) |
| 117 | └── frontend/ # Next.js App Router (TypeScript) - 17 routes |
| 118 | ``` |
| 119 | |
| 120 | ## Security & operations |
| 121 | |
| 122 | - **RBAC**: viewer (read), analyst (read/write/ingest/enrich/export), admin (all). Enforced per |
| 123 | route; verified by an automated permission matrix. |
| 124 | - **Audit**: every mutation writes an immutable `audit_logs` row (actor, action, entity, metadata). |
| 125 | - **Idempotent ingest**: content SHA-256 short-circuits re-uploads; a per-read dedup hash prevents |
| 126 | double-counting. |
| 127 | - **API keys**: per-user keys, SHA-256 hashed at rest, plaintext shown once, revocable; |
| 128 | authenticate machine clients via the `X-API-Key` header. |
| 129 | - **Evidence chain-of-custody**: every file is SHA-256 hashed on upload; downloads re-verify the |
| 130 | hash and the audit trail records upload/download events per artifact. |
| 131 | - **Retention**: cascade-safe purge (reads + dependent sightings/hits) via admin trigger or the |
| 132 | scheduled `daily-retention` Celery beat task. |
| 133 | - **Pluggable storage**: local filesystem in dev, S3-compatible (MinIO/R2/Wasabi/AWS) in prod, |
| 134 | selected by `SIGHTLINE_STORAGE_BACKEND` - evidence and raw import blobs both flow through it. |
| 135 | - **Input safety**: upload size caps; parsers are row-resilient (a bad row is logged, never |
| 136 | fatal); enrichers and connectors degrade to clean errors instead of crashing ingest. |
| 137 | |
| 138 | ## Tech stack |
| 139 | |
| 140 | FastAPI · SQLAlchemy 2.0 · Alembic · PostgreSQL · Redis · Celery · Pydantic · python-jose · |
| 141 | bcrypt · pdfminer.six · reportlab · boto3 · Authlib · Next.js 14 · React 18 · TypeScript · Leaflet. |