# DropHawk - Master Build Specification

> **Working name:** DropHawk (rename freely)

> **Purpose:** Always-on, multi-source ticket drop monitor with real-time UI, instant push notifications, and shock-drop detection across concerts, festivals, sports (NFL/NBA/NHL/MLB/tennis/F1/soccer), and exclusive events (FIFA World Cup, etc.).

> **Audience:** This document is the **architectural source of truth** for any executing agent (Claude, Codex, etc.).

>

> **⚠️ COMPANION FILE IS BINDING:** `CONTRACTS.md` lives alongside this file and contains the **exact pinned versions, exact API schemas, exact SQL DDL, exact design tokens, exact algorithms, and exact failure runbooks**. Where CONTRACTS.md and this file disagree, **CONTRACTS.md wins**. Read both before writing any code.

>

> **Loading order for any agent:**

> 1. `CLAUDE.md` (this repo) - bootstrap

> 2. `SPEC.md` (this file) - architecture & intent

> 3. `CONTRACTS.md` - binding implementation contracts

>

> When something is ambiguous, the answer is in CONTRACTS.md first, then this file. If both are silent, **ask the operator** - do not improvise on security, data integrity, or notification correctness.

---

## 0. How to use this spec

- Read sections 1-6 before writing any code. They are non-negotiable.

- Section 7 (Source Adapters) defines an interface; implement v1 sources (7.1, 7.2) fully, then expand per the roadmap (§14).

- Section 15 (Coding Standards) lists hard rules. Violations are bugs.

- Section 17 (Execution Checklist) is the per-phase verification gate. Do not merge a phase until its checklist passes.

- File paths in this doc are **absolute within the repo root** (e.g. `backend/Cargo.toml`).

---

## 1. Project Overview & Vision

### 1.1 What DropHawk is

A self-hosted service that continuously polls dozens of primary and resale ticket sources, detects the instant an event's status flips to "on sale" (or a new event appears, or a resale price drops), and pushes a notification to the user's phone within seconds - with a one-tap link to buy.

### 1.2 What DropHawk is NOT

- **Not a bulk auto-purchaser.** It surfaces drops; the human taps the link and buys. This keeps the tool legal under the US BOTS Act and equivalent EU/UK laws.

- **Not a scalping bot.** Personal-use monitoring + manual purchase only. (See §16 for legal guardrails.)

- **Not a ticket marketplace.** No transactions occur in DropHawk.

### 1.3 User

- Single user (the operator). No auth system, no multi-tenancy. The "watchlist" is global.

- Runs on the operator's homelab, accessed via LAN or Tailscale.

### 1.4 Quality bar

- **Sub-15-second detection-to-notification latency** for adaptive-poll sources.

- **Zero false-positive notifications** - every alert must correspond to a real status change.

- **Visually stunning UI** - dark, dense, animated, real-time. Treat the dashboard like a Bloomberg terminal for tickets, not a CRUD app.

---

## 2. Tech Stack (LOCKED - do not substitute)

### 2.1 Rationale

The operator asked for "fastest and most cutting edge." This means: Rust for the hot path (polling, scheduling, API), Python for scraping (because Playwright + anti-bot libraries only exist maturely in Python), TypeScript for the UI.

### 2.2 Stack table

| Layer | Technology | Version | Why |

|---|---|---|---|

| Core engine + API | **Rust** | 1.96+ | Fastest, lowest memory, async-native |

| Rust web framework | **Axum** | 0.7 | Tower middleware, typed handlers, de-facto standard |

| Rust async runtime | **Tokio** | 1.x | Required by Axum |

| Rust HTTP client | **reqwest** + **rquest** (for TLS fingerprinting) | latest | rquest impersonates Chrome TLS JA3 |

| Rust DB | **sqlx** (compile-time checked) | 0.8 | No ORM, type-safe queries |

| Rust messaging | **async-nats** | latest | JetStream for durable event bus |

| Rust scheduler | Custom Tokio task per source (no cron crate) | - | Each source owns its poll loop; clean backoff |

| Rust serialization | **serde** + **serde_json** | 1.x | Standard |

| Rust tracing | **tracing** + **tracing-subscriber** + **opentelemetry-otlp** | latest | Structured logs to Loki |

| Scraper workers | **Python** | 3.11 | Required for Playwright |

| Python HTTP | **httpx** + **curl_cffi** | latest | curl_cffi for browser TLS impersonation |

| Python browser | **Playwright** | 1.40+ | Headless Chromium for JS-rendered sources |

| Python HTML | **selectolax** (fast) + **BeautifulSoup** (fallback) | latest | Selectolax is 10x faster |

| Python task runner | **Celery** + **Redis broker** | 5.3 | Distributed scraper workers |

| Frontend framework | **Next.js** | 14.2+ (App Router) | RSC, streaming, edge-ready |

| Frontend lang | **TypeScript** | 5.4+ | Strict mode |

| Styling | **Tailwind CSS** | 3.4+ | Utility-first |

| Component lib | **shadcn/ui** | latest | Owned, composable, beautiful |

| Animation | **Framer Motion** | 11+ | Spring physics for drop alerts |

| Dashboard widgets | **Tremor** | latest | Charts, KPIs, sparklines |

| Data fetching | **TanStack Query** | 5+ | Cache + invalidation |

| Client state | **Zustand** | 5+ | Minimal, no boilerplate |

| Forms | **React Hook Form** + **Zod** | latest | Type-safe validation |

| Real-time push (server→client) | **Server-Sent Events (SSE)** via Axum `axum::response::sse` | - | Simpler than WebSocket, auto-reconnects |

| Database | **PostgreSQL** | 16 | Primary store |

| Time-series | **TimescaleDB** (Postgres extension) | 2.14+ | hypertables for price/source-health history |

| Cache + queue | **Redis** | 7.2+ | Hot cache, Celery broker, rate-limit tokens |

| Search | **Meilisearch** | 1.6+ | Fuzzy event/artist search, typo-tolerant |

| Message bus | **NATS** + **JetStream** | 2.10+ | Durable, fast pub/sub between Rust core and Python workers |

| Notifications | **ntfy** (self-hosted) | 2.x | Free push to iPhone (ntfy iOS app), self-hosted |

| Object storage | **MinIO** | latest | Screenshots of scraped pages for debug |

| Reverse proxy | **Caddy** | 2.7+ | Auto-HTTPS, simple config |

| Observability | **Prometheus** + **Grafana** + **Loki** + **OpenTelemetry** | latest | Metrics, dashboards, logs |

| Containerization | **Docker** + **Docker Compose** | 24+ / v2 | Homelab deploy |

| Python package mgr | **uv** | latest | 10x faster than pip |

| Node package mgr | **pnpm** | 9+ | Disk-efficient, deterministic |

### 2.3 Hard "do not use" list

- ❌ Go (not available in build env, no benefit over Rust here)

- ❌ MongoDB (use Postgres)

- ❌ Django/Flask for scrapers (use Celery workers; no web server in Python land)

- ❌ create-react-app (use Next.js App Router)

- ❌ Redux (use Zustand)

- ❌ Material-UI / Chakra (use shadcn/ui)

- ❌ Bootstrap (use Tailwind)

- ❌ Prisma on backend (use sqlx raw)

- ❌ GraphQL (YAGNI for single-user)

- ❌ WebSockets for server→client push (use SSE; simpler, auto-reconnect)

---

## 3. High-Level Architecture

### 3.1 Service map

```

                       ┌─────────────────────────────────────────────┐

                       │                HOMELAB HOST                  │

                       │                                             │

   Phone (ntfy app) ◄──┤  ntfy  ◄──┐                                 │

                       │           │                                 │

   Browser ◄── Caddy ──┤  ┌────────┴────────┐                        │

                       │  │  Next.js (UI)   │                        │

                       │  └────────┬────────┘                        │

                       │           │ SSE                              │

                       │  ┌────────▼────────────────────────┐        │

                       │  │      Rust core (Axum API)        │        │

                       │  │  - REST handlers                 │        │

                       │  │  - SSE broadcaster               │        │

                       │  │  - Scheduler (per-source tasks)  │        │

                       │  │  - Drop detector (hash diff)     │        │

                       │  └──┬──────────┬───────────┬────────┘        │

                       │     │          │           │                 │

                       │  ┌──▼───┐  ┌───▼────┐  ┌───▼─────────┐      │

                       │  │ NATS │  │ Redis  │  │ PostgreSQL  │      │

                       │  │ JetS │  │        │  │ +Timescale  │      │

                       │  │  tr  │  │        │  │             │      │

                       │  └──┬───┘  └────────┘  └─────────────┘      │

                       │     │ publish scraped_payload                  │

                       │  ┌──▼──────────────────────┐                 │

                       │  │  Python scraper workers  │                 │

                       │  │  (Celery, Playwright)     │                 │

                       │  │  - ticketmaster           │                 │

                       │  │  - seatgeek               │                 │

                       │  │  - fifa                   │                 │

                       │  │  - axs, seetickets        │                 │

                       │  │  - stubhub, vividseats    │                 │

                       │  │  - twitter shock-drop     │                 │

                       │  │  - sports (nfl/nba/nhl…)   │                 │

                       │  │  - festivals              │                 │

                       │  └───────────┬──────────────┘                 │

                       │              │                                 │

                       │  ┌───────────▼───────┐  ┌─────────────┐      │

                       │  │ Meilisearch index │  │ MinIO (screenshot)│ │

                       │  └───────────────────┘  └─────────────┘      │

                       │                                             │

                       │  Observability: Prometheus / Grafana / Loki │

                       └─────────────────────────────────────────────┘

```

### 3.2 Data flow - happy path for a drop detection

1. Rust scheduler spawns one Tokio task per source adapter.

2. Each task polls on its adaptive interval (see §11) - for API-backed sources (Ticketmaster, SeatGeek), the Rust task calls directly. For scrape-backed sources (FIFA, AXS, StubHub), the Rust task publishes a `scrape_request` to NATS subject `scrape.{source_id}`.

3. A Python Celery worker subscribed to `scrape.{source_id}` runs Playwright/httpx, returns a `ScrapedPayload` JSON to NATS subject `scrape.{source_id}.result`.

4. Rust task ingests payload, normalizes into the `Event` shape, computes a `content_hash`.

5. Rust compares hash against `events.content_hash` in Postgres.

   - If new event → insert, emit `event.new` on NATS, broadcast `drop` via SSE, push ntfy.

   - If hash changed (status flip, price change, inventory change) → update row, emit appropriate event.

6. SSE broadcaster fans out to all connected browser clients.

7. ntfy publisher POSTs to local ntfy server → user's iPhone buzzes.

**Latency budget:** source-poll-response → hash-diff → SSE+ntfy must complete in <2 seconds. End-to-end notification (poll → phone) target <15 seconds, dominated by poll interval.

### 3.3 Bounded context boundaries

| Service | Owns | Does NOT touch |

|---|---|---|

| Rust core | Scheduling, DB, hash-diff, SSE, ntfy, REST API | Browser automation, HTML parsing |

| Python scrapers | HTTP fetch, JS rendering, parsing, anti-bot | DB, scheduling, notifications |

| Next.js | UI rendering, watchlist forms, SSE consumption | Direct DB access, business logic |

| NATS | Message delivery only | Persistence beyond JetStream retention |

| Redis | Cache, rate-limit tokens, Celery broker | Source of truth (that's Postgres) |

---

## 4. Repository Structure

```

drophawk/

├── SPEC.md                       ← this file

├── CLAUDE.md                     ← condensed agent instructions (extract of this)

├── README.md                     ← operator quickstart

├── docker-compose.yml            ← all services

├── docker-compose.prod.yml       ← homelab prod overrides

├── .env.example                  ← all required env vars (no secrets)

├── .gitignore

│

├── backend/                      ← Rust core

│   ├── Cargo.toml

│   ├── Cargo.lock

│   ├── Dockerfile

│   ├── migrations/               ← sqlx migrations (timestamped, .sql files)

│   │   ├── 20260101000000_init.sql

│   │   ├── 20260101000001_sources.sql

│   │   ├── 20260101000002_events.sql

│   │   ├── 20260101000003_drops.sql

│   │   ├── 20260101000004_watchlist.sql

│   │   ├── 20260101000005_timescale.sql

│   │   └── 20260101000006_seed_sources.sql

│   └── src/

│       ├── main.rs               ← tokio entry, service wiring

│       ├── config.rs             ← env config (figment or config crate)

│       ├── error.rs              ← AppError, thiserror

│       ├── state.rs              ← AppState (DB pool, NATS, Redis, ntfy handle)

│       ├── model/

│       │   ├── mod.rs

│       │   ├── source.rs         ← Source struct

│       │   ├── event.rs          ← Event struct (the normalized shape)

│       │   ├── drop.rs           ← DropEvent struct

│       │   └── watchlist.rs

│       ├── store/                ← DB access layer (sqlx)

│       │   ├── mod.rs

│       │   ├── source.rs

│       │   ├── event.rs

│       │   ├── drop.rs

│       │   └── watchlist.rs

│       ├── api/                  ← Axum routes

│       │   ├── mod.rs            ← router assembly

│       │   ├── events.rs         ← GET /api/events, filters

│       │   ├── drops.rs          ← GET /api/drops (paginated)

│       │   ├── watchlist.rs      ← CRUD

│       │   ├── sources.rs        ← GET /api/sources (health)

│       │   ├── stats.rs          ← GET /api/stats (dashboard KPIs)

│       │   └── sse.rs            ← GET /api/stream (SSE broadcaster)

│       ├── scheduler/

│       │   ├── mod.rs            ← spawns one task per source

│       │   └── poller.rs         ← the poll loop (adaptive interval)

│       ├── sources/              ← adapters - Rust-side, for API sources

│       │   ├── mod.rs            ← SourceAdapter trait

│       │   ├── ticketmaster.rs

│       │   └── seatgeek.rs

│       ├── scrapers/             ← bridge to Python workers via NATS

│       │   ├── mod.rs            ← ScraperBridge (publishes, awaits reply)

│       │   └── req.rs            ← ScrapeRequest / ScrapedPayload types

│       ├── detect/               ← change detection

│       │   ├── mod.rs

│       │   ├── hash.rs           ← content_hash computation

│       │   └── classify.rs       ← NewEvent / StatusFlip / PriceDrop / InventoryDrop

│       ├── notify/

│       │   ├── mod.rs

│       │   ├── ntfy.rs           ← ntfy HTTP publish

│       │   └── filter.rs         ← watchlist filter, dedupe, quiet hours

│       ├── bus/                  ← NATS wrapper

│       │   ├── mod.rs

│       │   └── subjects.rs       ← constants

│       ├── cache/                ← Redis wrapper

│       │   └── mod.rs

│       └── metrics.rs            ← Prometheus counter/histogram defs

│

├── scrapers/                     ← Python workers

│   ├── pyproject.toml            ← uv-managed

│   ├── uv.lock

│   ├── Dockerfile

│   ├── celery_app.py             ← Celery instance, NATS broker config

│   ├── worker.py                 ← task dispatcher

│   ├── base.py                   ← BaseScraper ABC + anti-bot helpers

│   ├── anti_bot/

│   │   ├── __init__.py

│   │   ├── fingerprints.py       ← UA, sec-ch-ua, JA3 rotation

│   │   ├── proxies.py            ← proxy pool client

│   │   └── retry.py              ← exponential backoff + jitter

│   ├── sources/

│   │   ├── __init__.py

│   │   ├── fifa.py

│   │   ├── axs.py

│   │   ├── seetickets.py

│   │   ├── stubhub.py

│   │   ├── vividseats.py

│   │   ├── twitter.py            ← shock-drop monitor

│   │   ├── nfl.py

│   │   ├── nba.py

│   │   ├── nhl.py

│   │   ├── mlb.py

│   │   ├── atp_wta.py            ← tennis

│   │   ├── f1.py

│   │   ├── uefa.py               ← Champions League, Euros

│   │   ├── premier_league.py

│   │   └── festivals/

│   │       ├── __init__.py

│   │       ├── coachella.py

│   │       ├── glastonbury.py

│   │       ├── tomorrowland.py

│   │       ├── edc.py

│   │       └── lollapalooza.py

│   └── tests/

│       ├── conftest.py           ← VCR cassettes for replay

│       └── test_*.py

│

├── frontend/                     ← Next.js

│   ├── package.json

│   ├── pnpm-lock.yaml

│   ├── next.config.mjs

│   ├── tailwind.config.ts

│   ├── tsconfig.json

│   ├── Dockerfile

│   ├── app/

│   │   ├── layout.tsx            ← root layout, theme provider, fonts

│   │   ├── globals.css           ← tailwind directives, CSS vars

│   │   ├── page.tsx              ← Dashboard / live drop feed

│   │   ├── drops/

│   │   │   └── page.tsx          ← searchable drop history

│   │   ├── watchlist/

│   │   │   └── page.tsx          ← manage tracked entities

│   │   ├── sources/

│   │   │   └── page.tsx          ← source health grid

│   │   ├── explore/

│   │   │   └── page.tsx          ← Meilisearch-powered browse

│   │   ├── settings/

│   │   │   └── page.tsx          ← ntfy topic, quiet hours

│   │   └── api/                  ← route handlers (proxy to backend, optional)

│   ├── components/

│   │   ├── ui/                   ← shadcn primitives (button, card, sheet, etc.)

│   │   ├── layout/

│   │   │   ├── Sidebar.tsx

│   │   │   ├── Topbar.tsx

│   │   │   └── CommandPalette.tsx ← Cmd+K, fuzzy jump

│   │   ├── dashboard/

│   │   │   ├── LiveDropFeed.tsx   ← SSE consumer, animated cards

│   │   │   ├── DropCard.tsx       ← per-drop, shimmer-in animation

│   │   │   ├── StatsRow.tsx       ← Tremor KPI cards

│   │   │   ├── SourceGrid.tsx     ← health/latency per source

│   │   │   ├── DropHeatmap.tsx    ← calendar of drop density

│   │   │   └── PriceSparkline.tsx

│   │   ├── drops/

│   │   │   ├── DropTable.tsx

│   │   │   └── DropFilters.tsx

│   │   ├── watchlist/

│   │   │   ├── WatchlistForm.tsx

│   │   │   └── WatchlistItem.tsx

│   │   └── shared/

│   │       ├── ThemeToggle.tsx

│   │       ├── RelativeTime.tsx

│   │       └── EmptyState.tsx

│   ├── hooks/

│   │   ├── useDropsStream.ts     ← SSE subscription

│   │   ├── useWatchlist.ts

│   │   └── useSources.ts

│   ├── lib/

│   │   ├── api.ts                ← typed fetch wrappers (ky or fetch)

│   │   ├── sse.ts                ← SSE client

│   │   ├── types.ts              ← TS types mirroring Rust models

│   │   ├── utils.ts              ← cn(), formatters

│   │   └── query.ts              ← TanStack Query client

│   ├── stores/

│   │   └── ui.ts                 ← Zustand: sidebar collapsed, filters

│   └── public/

│       └── icons/

│

├── infra/                        ← homelab deploy artifacts

│   ├── Caddyfile

│   ├── ntfy.server.yml           ← ntfy config (auth, attachment size)

│   ├── prometheus.yml

│   ├── grafana/

│   │   ├── dashboards/

│   │   │   ├── drophawk-overview.json

│   │   │   └── source-health.json

│   │   └── datasources.yml

│   ├── loki/

│   │   └── loki-config.yml

│   └── backup/

│       └── pg-backup.sh          ← nightly pg_dump cron

│

└── scripts/

    ├── seed-watchlist.ts         ← bootstrap common artists/teams

    ├── healthcheck.sh            ← curl all services, exit 0 if green

    └── dev-up.sh                 ← tmux split: backend logs, frontend, nats

```

---

## 5. Backend Services (Rust core) - detailed contracts

### 5.1 `SourceAdapter` trait (`backend/src/sources/mod.rs`)

```rust

#[async_trait]

pub trait SourceAdapter: Send + Sync {

    /// Stable identifier, matches `sources.slug` in DB.

    fn id(&self) -> &str;

    /// Human label, e.g. "Ticketmaster Discovery API".

    fn name(&self) -> &str;

    /// Category for UI grouping: "primary", "resale", "sports", "festival".

    fn category(&self) -> SourceCategory;

    /// Poll once. Returns a vec of normalized `Event` candidates.

    /// Errors must be retried by the scheduler's backoff, not propagated up.

    async fn poll(&self, ctx: &PollContext) -> Result<Vec<EventCandidate>, SourceError>;

    /// Adaptive interval hint based on `ctx.now` and any known drop window.

    /// Default: 5 minutes. Tighten to 10s inside ±15min of known drop.

    fn next_poll_hint(&self, ctx: &PollContext) -> Duration {

        Duration::from_secs(300)

    }

}

pub struct PollContext {

    pub now: DateTime<Utc>,

    pub http: reqwest::Client,         // shared, pre-configured (TLS impersonation)

    pub scraper_bridge: ScraperHandle, // for sources that need Python

    pub cache: RedisHandle,

    pub metrics: MetricsHandle,

}

pub struct EventCandidate {

    pub source_id: String,

    pub source_event_id: String,    // ID from the source's namespace

    pub title: String,

    pub subtitle: Option<String>,   // tour name, support act, etc.

    pub category: EventCategory,    // Concert, Sports, Festival, Theater, Other

    pub sport: Option<Sport>,       // NFL/NBA/NHL/MLB/Tennis/F1/Soccer/...

    pub starts_at: Option<DateTime<Utc>>,

    pub venue_name: Option<String>,

    pub city: Option<String>,

    pub country: Option<String>,

    pub latitude: Option<f64>,

    pub longitude: Option<f64>,

    pub image_url: Option<String>,

    pub on_sale_at: Option<DateTime<Utc>>,    // announced sale start

    pub presale_on_sale_at: Option<DateTime<Utc>>,

    pub status: SaleStatus,                   // Announced, Presale, OnSale, SoldOut, Cancelled

    pub price_min: Option<i64>,    // cents (USD or source currency)

    pub price_max: Option<i64>,

    pub currency: Option<String>,  // ISO 4217

    pub inventory_remaining: Option<u32>,

    pub buy_url: String,           // canonical, with affiliate tags where allowed

    pub raw: serde_json::Value,    // full original payload, for debug

}

```

### 5.2 Scheduler (`backend/src/scheduler/`)

- One `tokio::task` per source.

- Loop body: `sleep(next_poll_hint) → poll() → detect::diff() → store + emit`.

- On error: exponential backoff capped at 10 minutes; emit `source.unhealthy` after 3 consecutive failures.

- Use `tokio::select!` to also listen on a `shutdown` channel for clean exit.

- Persist `last_poll_at`, `last_success_at`, `last_error`, `consecutive_failures` in `sources` table.

### 5.3 Change detector (`backend/src/detect/`)

- `content_hash` = SHA-256 of canonical JSON of `(source_event_id, status, on_sale_at, presale_on_sale_at, price_min, price_max, inventory_remaining)`. Fields are normalized (None → null, datetimes in RFC3339 UTC, prices in minor units). Render with `serde_json::to_string` after sorting keys (BTreeMap).

- Diff outcomes → emit different NATS subjects + different SSE event types:

  - **New** (source_event_id never seen) → `event.new` → SSE `drop.new`

  - **StatusFlip** (status changed) → `event.status_changed` → SSE `drop.status`

    - Most-wanted transition: `Announced → OnSale`

  - **PriceDrop** (price_min decreased ≥10%) → `event.price_drop` → SSE `drop.price`

  - **InventoryDrop** (inventory_remaining decreased ≥25%) → `event.inventory` → SSE `drop.inventory`

  - **Unchanged** (hash same) → no-op

### 5.4 SSE broadcaster (`backend/src/api/sse.rs`)

- Maintain a `tokio::sync::broadcast::Sender<ServerEvent>` in `AppState`.

- Each `/api/stream` connection spawns a task that subscribes and writes `event: <type>\ndata: <json>\n\n` lines.

- Heartbeat every 15s (comment line `:\n\n`) to keep proxies from closing idle conns.

- Backpressure: if a client's receiver falls >50 events behind, drop oldest.

### 5.5 Notification pipeline (`backend/src/notify/`)

Order of operations on a new `DropEvent`:

1. **Filter** against watchlist (`notify::filter`):

   - Does the event match any watchlist rule? (artist/team/venue/sport/category keywords, regex)

   - Inside quiet hours? (configurable; default: never quiet - operator wants to wake up)

   - Already notified for this `source_event_id` in last 10 min? (dedupe)

2. If pass → build notification payload (title, body with markdown, click URL = `buy_url`, tags = `[source_id, category]`).

3. POST to ntfy at `http://ntfy:80/internal-{topic}` with `Title`, `Message`, `Click`, `Tags`, `Priority` (elevated for status flips), `Actions` (a "Buy" button via ntfy action URLs).

4. Insert row into `notifications_log`.

### 5.6 REST API surface (all under `/api`)

| Method | Path | Purpose |

|---|---|---|

| GET | `/api/events` | Paginated, filterable (category, sport, source, status, price range, date range). Default sort: `on_sale_at asc nulls last`. |

| GET | `/api/events/:id` | Single event with price history. |

| GET | `/api/drops` | Drop history, paginated, filterable by type/category/source. |

| GET | `/api/drops/stream` | **SSE** - live drops. |

| GET | `/api/watchlist` | List rules. |

| POST | `/api/watchlist` | Create rule (validated by Zod-equivalent: serde + validator crate). |

| PATCH | `/api/watchlist/:id` | Update rule (toggle active, edit pattern). |

| DELETE | `/api/watchlist/:id` | Delete rule. |

| GET | `/api/sources` | Source health grid (status, latency p50/p95, last_success_at, consecutive_failures). |

| GET | `/api/stats` | Dashboard KPIs (drops last 24h, sources healthy, watchlist matches, etc.). |

| GET | `/api/explore?q=` | Proxy to Meilisearch for fuzzy event/artist search. |

| GET | `/api/health` | Liveness (200). |

| GET | `/api/ready` | Readiness (DB + NATS + Redis reachable). |

**Response shape (consistent):**

```json

{ "data": [...], "meta": { "page": 1, "per_page": 50, "total": 1234 } }

```

Errors: `{ "error": { "code": "STRING", "message": "human-readable" } }` with appropriate HTTP status.

---

## 6. Scraper Workers (Python) - detailed contracts

### 6.1 Worker model

- One Celery worker process per Docker replica. Replica count per source is configurable.

- Tasks are published by Rust via NATS → a small bridge (the Celery worker subscribes to NATS and enqueues a Celery task) OR - simpler - Rust publishes scrape_request to NATS, a single Python "dispatcher" subscribes and calls Celery tasks; results come back via NATS reply subject.

- **Preferred pattern (simpler):** Use NATS directly as the Celery-less RPC bus. The Python process is a long-running asyncio loop subscribing to `scrape.>` subjects; each message is a `ScrapeRequest`, each reply is a `ScrapedPayload`. This avoids Celery's complexity for v1. (Migrate to Celery in v2 if scale demands.)

**Decision: use asyncio + NATS, NOT Celery, for v1.** Simpler, fewer services, faster cold-path.

### 6.2 `BaseScraper` (`scrapers/base.py`)

```python

from typing import Protocol, Sequence

from .types import ScrapeRequest, ScrapedPayload

class BaseScraper(Protocol):

    source_id: str          # matches Rust SourceAdapter id

    async def scrape(self, req: ScrapeRequest) -> ScrapedPayload: ...

    async def health(self) -> ScraperHealth: ...

```

- Each scraper lives in `scrapers/sources/<name>.py` and is registered in a registry keyed by `source_id`.

- The dispatcher reads NATS messages, looks up the scraper, calls `scrape()`, publishes reply.

- Use `httpx.AsyncClient` for HTML sources; `playwright.async_api` for JS-rendered sources (lazy-import per source to keep memory down).

### 6.3 Anti-bot module (`scrapers/anti_bot/`)

- `fingerprints.py`: rotating pool of (User-Agent, sec-ch-ua, accept-language, JA3) tuples, sourced from a checked-in JSON. Update monthly.

- `proxies.py`: optional proxy rotation. For v1, support a static list of `http://user:pass@host:port` from env. (Future: integrate with rotating residential proxy.)

- `retry.py`: exponential backoff with jitter; honor `Retry-After` on 429/503; circuit-open after 5 consecutive failures in 60s.

- `playwright_stealth`: use `playwright-stealth` plugin to defeat basic bot detection.

- **Never** solve CAPTCHAs. If a source starts requiring CAPTCHA, mark it unhealthy and alert the operator.

### 6.4 ScrapedPayload schema (must match Rust)

```python

@dataclass

class ScrapedPayload:

    source_id: str

    fetched_at: str          # RFC3339 UTC

    ok: bool

    error: Optional[str]

    events: List[EventCandidateDict]  # same shape as Rust EventCandidate

    raw_html_sha256: Optional[str]    # for fast "page unchanged" skip

    latency_ms: int

```

---

## 7. Source Adapters - full spec for v1 + interface for v2

Each adapter section specifies: auth, endpoint, rate limit, parser approach, gotchas.

### 7.1 Ticketmaster Discovery API ✅ v1 (Rust)

- **Auth:** API key via `apikey` query param. Free, register at `developer.ticketmaster.com`.

- **Base:** `https://app.ticketmaster.com/discovery/v2/`

- **Endpoints:**

  - `GET /events.json?keyword={artist}&countryCode={CC}&size=200&page={n}`

  - `GET /events.json?segmentName=Music|Sports|Arts&countryCode={CC}`

  - `GET /events.json?attractionId={id}` (after resolving attraction IDs)

- **Rate limit:** 5000 req/day per key. With multiple keys (allowed up to 5 free keys/account), rotate.

- **Parsing notes:**

  - `sales.public.startDateTime` → `on_sale_at`

  - `sales.presale.startDateTime` → `presale_on_sale_at`

  - `dates.start.dateTime` → `starts_at`

  - `_embedded.venues[0]` → venue, city, country, lat/lon

  - `priceRanges[0].min`/`.max` → prices (already in major units, multiply by 100 for cents)

  - `url` → `buy_url`

  - Status inference: if `on_sale_at` is past and no `sold_out` flag → `OnSale`; if `dates.status.code == "sold"` → `SoldOut`.

- **Gotchas:**

  - Pagination: cap `size=200`, follow `page.total` to know how many pages.

  - Some events have null `priceRanges` - leave prices as None.

  - `_embedded.attractions` gives canonical artist/team IDs - store these for watchlist resolution.

### 7.2 SeatGeek API ✅ v1 (Rust)

- **Auth:** Client ID via `client_id` query param. Free at `seatgeek.com/developers`.

- **Base:** `https://api.seatgeek.com/2/`

- **Endpoints:**

  - `GET /events?client_id={id}&performers.slug={slug}&type={concert|sports|...}`

  - `GET /events?client_id={id}&listing_count.gt=0&q={query}`

  - `GET /performers?client_id={id}&slug={slug}` to resolve IDs

- **Rate limit:** not officially stated; stay under 1 req/sec per endpoint.

- **Parsing notes:**

  - `time_utc` → `starts_at`

  - `score` → relevance score (use to rank)

  - `stats.lowest_price` / `.highest_price` / `.average_price` → prices (in dollars → cents)

  - `url` → `buy_url` (SeatGeek affiliate handles resale listings too)

  - `venue.city`, `venue.country`, `venue.location.lat`/`.lon`

  - Status inference: `announced` field; `visible_at_utc` → `on_sale_at`.

- **Gotchas:**

  - SeatGeek aggregates resale - `listing_count` flips >0 when tickets hit secondary market.

### 7.3 FIFA World Cup portal ✅ v1 (Python scraper)

- **URL:** `https://tickets.fifa.com` (rotates per tournament - make this configurable).

- **Approach:** Playwright headless. Render the sales phases page, extract:

  - Active phase name (Random Selection Draw / First Come First Served / Last Minute)

  - Phase window (`start`, `end`)

  - Per-match availability indicator (often color-coded)

- **Detection:** When phase transitions from "closed/coming" → "open" → emit drop. When a match goes from "sold out" → "available" (last-minute sales) → emit `drop.status`.

- **Gotchas:**

  - FIFA uses heavy anti-bot (Cloudflare). Use `playwright-stealth` + realistic TLS (curl_cffi for any XHR you can identify).

  - The site is localized; parameterize language to `en`.

  - Phase dates are announced but actual opening minute varies - poll every 30s within ±1h of announced start.

- **Legal:** Display only. Link to official portal for purchase. Do not attempt to auto-submit applications.

### 7.4 AXS ✅ v2 (Python scraper)

- `https://www.axs.com/events` - JSON behind the search endpoint; reverse-engineer XHR. Use `curl_cffi` with browser impersonation.

- Look for `onSaleDate` field.

### 7.5 See Tickets ✅ v2 (Python scraper)

- `https://www.seetickets.com/events` - server-rendered HTML; use selectolax.

### 7.6 StubHub ✅ v2 (Python scraper, resale)

- Target: detect price drops. StubHub has GraphQL; reverse-engineer carefully.

- Track `lowestPrice` per event; emit `drop.price` when it drops ≥10%.

### 7.7 Vivid Seats ✅ v2 (Python scraper, resale)

- HTML scrape; track lowest listing price.

### 7.8 Twitter/X shock-drop monitor ✅ v2 (Python)

- Use the official X API v2 filtered stream (paid) OR scrape via Nitter instances (fragile) OR RSS of artist/team accounts via rsshub.

- Track a curated list of accounts (configured in watchlist with `kind: twitter_account`).

- Pattern match tweets for: "tickets", "on sale", "presale code", "shock drop", "available now", "link in bio".

- Emit `drop.announcement` with the tweet URL.

### 7.9 NFL ✅ v1 via Ticketmaster segment + SeatGeek

- Use Ticketmaster `segmentName=Sports&sport=NFL` (or genre ID). Cross-reference with SeatGeek `performers.slug` for known teams.

### 7.10 NBA ✅ v1 via Ticketmaster + SeatGeek - same pattern as NFL.

### 7.11 NHL ✅ v1 - same pattern.

### 7.12 MLB ✅ v1 - same pattern (note: MLB uses Tickets.com / SeatGeek heavily).

### 7.13 ATP/WTA Tennis ✅ v2

- Scrape `https://www.atptour.com/en/tournaments` and `https://www.wtatennis.com/tournaments`.

- Cross-reference Ticketmaster for slams (US Open, Wimbledon via official + Ticketmaster).

### 7.14 F1 ✅ v2

- Scrape `https://www.formula1.com/en/tickets.html` - Playwright.

- Grand Prix tickets are sold in waves; detect wave open.

### 7.15 UEFA (Champions League, Europa, Euros) ✅ v2

- `https://www.uefa.com/tickets/` - Playwright.

- UEFA uses ballot + FCFS phases.

### 7.16 Premier League ✅ v2

- Each club has its own ticket portal. v2 implements a per-club adapter interface; v2 ships with the Big 6 (Man City, Man Utd, Liverpool, Chelsea, Arsenal, Tottenham).

### 7.17 Festivals ✅ v2-v3 (one adapter each)

- Coachella, Glastonbury, Tomorrowland, EDC, Lollapalooza, Austin City Limits, Bonnaroo, Rolling Loud, Ultra, Boom, Burning Man (lottery).

- Each festival has unique sales mechanics (lottery, registration, FCFS waves). Each adapter documents its phase model.

---

## 8. Data Model & Schema (PostgreSQL + TimescaleDB)

### 8.1 Tables

```sql

-- sources: one row per registered source adapter

CREATE TABLE sources (

    id              TEXT PRIMARY KEY,            -- e.g. "ticketmaster", "fifa"

    name            TEXT NOT NULL,

    category        TEXT NOT NULL,               -- primary|resale|sports|festival|social

    enabled         BOOLEAN NOT NULL DEFAULT TRUE,

    poll_interval_s INTEGER NOT NULL DEFAULT 300,

    tight_interval_s INTEGER NOT NULL DEFAULT 10,

    last_poll_at    TIMESTAMPTZ,

    last_success_at TIMESTAMPTZ,

    last_error      TEXT,

    consecutive_failures INTEGER NOT NULL DEFAULT 0,

    config          JSONB NOT NULL DEFAULT '{}', -- per-source: api keys live here? NO - env only

    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()

);

-- events: normalized events across all sources

CREATE TABLE events (

    id                  BIGSERIAL PRIMARY KEY,

    source_id           TEXT NOT NULL REFERENCES sources(id),

    source_event_id     TEXT NOT NULL,           -- ID in source's namespace

    title               TEXT NOT NULL,

    subtitle            TEXT,

    category            TEXT NOT NULL,           -- Concert|Sports|Festival|Theater|Other

    sport               TEXT,                    -- NFL|NBA|NHL|MLB|Tennis|F1|Soccer|...

    starts_at           TIMESTAMPTZ,

    venue_name          TEXT,

    city                TEXT,

    country             TEXT,

    latitude            DOUBLE PRECISION,

    longitude           DOUBLE PRECISION,

    image_url           TEXT,

    on_sale_at          TIMESTAMPTZ,

    presale_on_sale_at  TIMESTAMPTZ,

    status              TEXT NOT NULL,           -- Announced|Presale|OnSale|SoldOut|Cancelled

    price_min           INTEGER,                 -- cents

    price_max           INTEGER,

    currency            TEXT,

    inventory_remaining INTEGER,

    buy_url             TEXT NOT NULL,

    raw                 JSONB,

    content_hash        TEXT NOT NULL,           -- SHA-256, see §5.3

    first_seen_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    updated_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    UNIQUE (source_id, source_event_id)

);

CREATE INDEX idx_events_status ON events(status);

CREATE INDEX idx_events_on_sale ON events(on_sale_at);

CREATE INDEX idx_events_category ON events(category);

CREATE INDEX idx_events_starts_at ON events(starts_at);

CREATE INDEX idx_events_title_trgm ON events USING gin (title gin_trgm_ops);  -- needs pg_trgm

-- drops: every detected change. Append-only.

CREATE TABLE drops (

    id              BIGSERIAL PRIMARY KEY,

    event_id        BIGINT NOT NULL REFERENCES events(id),

    drop_type       TEXT NOT NULL,               -- new|status|price|inventory|announcement

    from_state      JSONB,                       -- prior values

    to_state        JSONB,                       -- new values

    detected_at     TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    notified        BOOLEAN NOT NULL DEFAULT FALSE,

    notified_at     TIMESTAMPTZ

);

CREATE INDEX idx_drops_detected ON drops(detected_at DESC);

CREATE INDEX idx_drops_event ON drops(event_id);

CREATE INDEX idx_drops_type ON drops(drop_type);

-- watchlist: user-defined rules

CREATE TABLE watchlist (

    id          BIGSERIAL PRIMARY KEY,

    kind        TEXT NOT NULL,                   -- artist|team|venue|sport|category|keyword|twitter_account

    pattern     TEXT NOT NULL,                   -- regex or glob

    label       TEXT NOT NULL,                   -- friendly name

    active      BOOLEAN NOT NULL DEFAULT TRUE,

    priority    TEXT NOT NULL DEFAULT 'normal',  -- low|normal|high|critical (drives ntfy priority)

    created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()

);

CREATE INDEX idx_watchlist_active ON watchlist(active);

-- notifications_log: every ntfy push

CREATE TABLE notifications_log (

    id              BIGSERIAL PRIMARY KEY,

    drop_id         BIGINT REFERENCES drops(id),

    channel         TEXT NOT NULL,               -- ntfy|telegram|email

    title           TEXT NOT NULL,

    body            TEXT NOT NULL,

    priority        TEXT NOT NULL,

    sent_at         TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    success         BOOLEAN NOT NULL,

    response        TEXT

);

CREATE INDEX idx_notif_sent ON notifications_log(sent_at DESC);

-- price_history (TimescaleDB hypertable)

CREATE TABLE price_history (

    event_id    BIGINT NOT NULL REFERENCES events(id),

    observed_at TIMESTAMPTZ NOT NULL,

    price_min   INTEGER,

    price_max   INTEGER,

    currency    TEXT,

    source      TEXT                              -- 'primary' | 'stubhub' | 'vividseats' | ...

);

SELECT create_hypertable('price_history', 'observed_at');

CREATE INDEX idx_price_event_time ON price_history(event_id, observed_at DESC);

-- source_health (TimescaleDB hypertable)

CREATE TABLE source_health (

    source_id       TEXT NOT NULL,

    observed_at     TIMESTAMPTZ NOT NULL,

    ok              BOOLEAN NOT NULL,

    latency_ms      INTEGER NOT NULL,

    events_returned INTEGER NOT NULL,

    error           TEXT

);

SELECT create_hypertable('source_health', 'observed_at');

CREATE INDEX idx_health_source_time ON source_health(source_id, observed_at DESC);

```

### 8.2 Extensions required in migrations

```sql

CREATE EXTENSION IF NOT EXISTS pg_trgm;

CREATE EXTENSION IF NOT EXISTS timescaledb;

```

### 8.3 Seed data

- Seed `sources` rows for every adapter listed in §7 (with `enabled = TRUE` for v1 sources, `FALSE` for v2-v3).

- Seed a starter `watchlist` with a few example rules the operator can edit (e.g. `"Taylor Swift"`, `"World Cup"`, `"Super Bowl"`, `"Wimbledon"`).

---

## 9. Notification System (ntfy, self-hosted, free iPhone push)

### 9.1 Why ntfy

- Free, self-hosted, no third-party relay.

- iOS app supports instant push via APNs (ntfy's public APNs key - no Apple Developer account needed by you).

- HTTP-only API: `POST http://ntfy.your_homelab/topic { Title, Message, Click, Priority, Tags, Actions }`.

### 9.2 Setup

- Run `ntfy` container in `docker-compose.yml`.

- Topic naming: `drophawk-drops` (high-priority), `drophawk-info` (low).

- **Auth:** generate an access token; persist in `.env` as `NTFY_TOKEN`. Subscribe the iOS app with the token.

- iOS app: add subscription to `https://ntfy.your_homelab/drophawk-drops`.

### 9.3 Notification payload spec

```

POST /drophawk-drops HTTP/1.1

Host: ntfy

Authorization: Bearer {NTFY_TOKEN}

Title: 🎟️ On Sale: Taylor Swift - Miami

Tags: ticket,concert,concerts,ticketmaster

Priority: high

Click: https://www.ticketmaster.com/event/...

Actions: view, Buy now, https://www.ticketmaster.com/event/...

Eras Tour floor seats from $49.50. 15,000+ remaining.

Venue: Hard Rock Stadium · Jun 22 2026

```

### 9.4 Priority mapping

- `drop.new` for a watchlist match → `urgent`

- `drop.status` (Announced→OnSale) → `high`

- `drop.price` ≥25% drop → `high`

- `drop.price` <25% drop → `default`

- `drop.inventory` → `default`

- Non-watchlist events → not notified (still visible in UI).

### 9.5 Fallback channels (optional, future)

- Telegram bot (set `TELEGRAM_BOT_TOKEN` to enable).

- Email via Postfix relay.

---

## 10. Frontend - detailed UI spec

### 10.1 Design language

- **Theme:** Dark by default (light available). Use OKLCH colors.

- **Type:** Inter for UI, JetBrains Mono for numbers/timestamps.

- **Density:** High. Card-based grid for drops, no whitespace waste.

- **Motion:** Spring physics on drop cards (slide-in + shimmer), gentle parallax on stats. Respect `prefers-reduced-motion`.

- **Identity accent:** electric teal (#14b8a6) + amber for warnings + red for sold out.

### 10.2 Layout

- **Sidebar (left, collapsible):** Dashboard, Live Drops, Explore, Watchlist, Sources, Settings.

- **Topbar:** global Cmd+K palette, theme toggle, source health dot, "watchlist active" badge.

- **Main area:** per-route content.

### 10.3 Routes & components

**`/` (Dashboard) - the wow page**

- Row 1: Tremor KPI cards - Drops (24h), Active Sources, Watchlist Matches, Avg Detection Latency.

- Row 2 (3/4 width): **LiveDropFeed** - SSE-driven, animated cards. Each `DropCard` shows: source icon, title, status pill, price range, time-since-detected (RelativeTime), "Buy" button (opens `buy_url` in new tab), category tag. New cards slide in from top; flash accent border for 2s.

- Row 2 (1/4 width): **SourceGrid** - health dots (green/amber/red), latency sparkline per source, click → sources page filtered.

- Row 3: **DropHeatmap** - calendar of last 90 days, color intensity = drop count per day. Click a day → drops page filtered.

- Row 4: **Top Drops** - table of highest-signal drops this week.

**`/drops` - Drop history**

- Filters: type (new/status/price/inventory), category, source, date range, search (Meilisearch-backed).

- Virtualized table (TanStack Virtual) - handle 100k+ rows.

- Each row expandable → event detail + price history sparkline.

**`/explore` - Browse**

- Search box (debounced 200ms → `/api/explore?q=`).

- Results grouped by category. Click → modal with detail + "Add to watchlist" CTA.

**`/watchlist` - Manage**

- Form (React Hook Form + Zod): kind (select), pattern (input), label, priority.

- List of rules with toggle, edit, delete, "matched N events" count.

- Bulk import: paste a list of artist/team names → creates `keyword` rules in batch.

**`/sources` - Source health**

- Grid of cards, one per source. Shows: status, latency p50/p95 (last hour), events seen (24h), last error, poll interval.

- Toggle enable/disable per source (writes to `sources.enabled`).

**`/settings` - Settings**

- ntfy topic + token (test button sends a push).

- Quiet hours (time range, days of week).

- Theme + density preferences (persisted in localStorage).

### 10.4 SSE handling (`hooks/useDropsStream.ts`)

- Open EventSource on `/api/drops/stream`.

- On message: parse, push to Zustand store (cap buffer at 200, FIFO).

- On error: close, exponential backoff reconnect (max 30s).

- On `drop.new` for a watchlist match: trigger browser notification (request permission on first dashboard mount).

### 10.5 Accessibility

- All interactive elements keyboard-navigable.

- ARIA labels on icon-only buttons.

- Color contrast ≥ 4.5:1.

- Status pills have text, not color alone.

### 10.6 PWA

- `manifest.json` + service worker (via `next-pwa` or `serwist`).

- Installable to home screen; works offline for cached views.

---

## 11. Polling & Anti-Ban Strategy

### 11.1 Adaptive intervals (per-source, in Rust scheduler)

- **Baseline:** `poll_interval_s` from `sources` table (default 300s = 5min).

- **Tight window:** from `T-15min` to `T+15min` around a known `on_sale_at`, switch to `tight_interval_s` (default 10s).

- **Pre-announcement:** for sources where the next drop time is unknown (e.g. FIFA last-minute sales), stay at 30s baseline.

- Source adapters can override `next_poll_hint` for source-specific intelligence (e.g. F1 wave patterns).

### 11.2 Polite scraping rules

- Honor `robots.txt`. If a path is disallowed, do not scrape it; mark source as `disabled` and alert.

- Identify DropHawk with a stable User-Agent: `DropHawk/1.0 (+https://your-homelab/drophawk)`. Do not impersonate browsers for *API* sources (only for sources that require it, e.g. FIFA behind Cloudflare).

- Default `Accept-Encoding: gzip, br`.

- Cache aggressively: if `raw_html_sha256` unchanged, skip parsing entirely.

### 11.3 Rate limiting (per-source token bucket in Redis)

- Key: `ratelimit:{source_id}`. Refill rate from source config.

- Acquire token before each poll. If empty, wait (don't skip).

- Honors 429s: on 429 with `Retry-After`, set bucket to empty for that duration.

### 11.4 Circuit breaker (per source)

- 3 consecutive failures → status `degraded` (still try, but at half rate).

- 10 consecutive failures → status `down` (pause, alert operator).

- First success after degraded/down → status `healthy`.

### 11.5 Anti-bot resilience (Python scrapers)

- TLS fingerprint rotation via `curl_cffi` (impersonates Chrome 120, Firefox 121, Safari 17).

- Header rotation: matching (UA + sec-ch-ua + accept-language) tuples.

- Optional residential proxy pool (env: `PROXY_POOL_URL`). For v1, no proxy - start clean.

- `playwright-stealth` for JS-rendered sources.

- **Never** attempt CAPTCHA bypass. Fail loudly instead.

### 11.6 Distributed polling

- v1 is single-instance (one Rust process). The architecture supports horizontal scaling later by sharding sources across instances via a Redis lease (`source:{id}:owner` with TTL).

---

## 12. Real-Time Pipeline

```

[source poll] → Rust detect::diff → Postgres write →

  ├→ NATS publish "drop.{type}" (durable, JetStream)

  ├→ SSE broadcast to browsers (in-process broadcast channel)

  └→ ntfy publish (if watchlist match)

```

- JetStream retention: `drop.*` subjects retained 7 days (for replay on restart).

- On Rust startup: replay any unacked drops newer than `MAX(last_browser_seen, now-1h)`. (Browsers also reconnect via SSE and resync from `/api/drops?since=`.)

---

## 13. Docker Compose Layout

### 13.1 Services (`docker-compose.yml`)

| Service | Image / Build | Ports | Depends on |

|---|---|---|---|

| `caddy` | `caddy:2.7` | 80, 443 | `frontend`, `backend`, `ntfy` |

| `frontend` | build `./frontend` | internal | `backend` |

| `backend` | build `./backend` | internal | `postgres`, `redis`, `nats` |

| `scrapers` | build `./scrapers` | internal | `nats`, `redis` |

| `postgres` | `postgres:16` + Timescale | internal | - |

| `redis` | `redis:7.2-alpine` | internal | - |

| `nats` | `nats:2.10` (with JetStream) | internal | - |

| `meilisearch` | `getmeili/meilisearch:v1.6` | internal | - |

| `ntfy` | `binwiederhier/ntfy:2` | internal | - |

| `minio` | `minio/minio` | internal | - |

| `prometheus` | `prom/prometheus` | internal | - |

| `grafana` | `grafana/grafana` | 3000 (LAN only) | `prometheus`, `loki` |

| `loki` | `grafana/loki` | internal | - |

### 13.2 Networking

- One internal network `drophawk-net`.

- Only `caddy` exposes ports to the host. Everything else is internal.

- Caddy routes:

  - `drophawk.lan` → `frontend:3000`

  - `drophawk.lan/api/*` → `backend:8080`

  - `drophawk.lan/ntfy/*` → `ntfy:80`

- For homelab DNS: operator adds `drophawk.lan` to their Pi-hole/AdGuard (or uses Tailscale MagicDNS).

### 13.3 Volumes

- `pg-data`, `redis-data`, `nats-data`, `meili-data`, `ntfy-data`, `minio-data`, `grafana-data`, `loki-data`, `prometheus-data`.

### 13.4 Resource guidance

- Postgres: 2GB memory ceiling. Backend: 512MB. Scrapers: 1GB (Playwright is hungry). Frontend (SSR): 256MB.

### 13.5 `.env.example` (template - copy to `.env`, fill in)

```

# Postgres

POSTGRES_PASSWORD=change_me

POSTGRES_DB=drophawk

DATABASE_URL=postgres://drophawk:change_me@postgres:5432/drophawk

# Redis

REDIS_URL=redis://redis:6379/0

# NATS

NATS_URL=nats://nats:4222

# Sources - Ticketmaster

TICKETMASTER_API_KEYS=key1,key2,key3

# Sources - SeatGeek

SEATGEEK_CLIENT_ID=xxx

# ntfy

NTFY_BASE_URL=http://ntfy:80

NTFY_TOPIC=drophawk-drops

NTFY_TOKEN=

# Meilisearch

MEILI_URL=http://meilisearch:7700

MEILI_MASTER_KEY=change_me

# Optional

PROXY_POOL_URL=

TELEGRAM_BOT_TOKEN=

```

**Secrets never go in the repo.** `.env` is gitignored. Document keys in `.env.example` only.

---

## 14. Phased Delivery Roadmap

Each phase ends with the §17 checklist passing.

### Phase 1 - Vertical slice (1 source end-to-end)

Scope:

- Repo scaffold per §4.

- Rust backend: config, DB pool, NATS connect, Axum server, `health`/`ready`, scheduler skeleton with **Ticketmaster adapter only**.

- Postgres + Timescale + migrations through §8.

- ntfy container + publisher.

- Next.js: layout, dashboard, LiveDropFeed (real SSE), DropCard, StatsRow (basic), watchlist page (CRUD).

- Caddy + Docker Compose for this subset.

- Deliverable: operator can add "Taylor Swift" to watchlist, get an ntfy push when a Ticketmaster event flips on-sale, see it in the UI.

### Phase 2 - Second source + resale + Meilisearch

- SeatGeek adapter (Rust).

- Meilisearch integration, `/explore` page.

- `price_history` writes, PriceSparkline component.

- SourceGrid + `/sources` page.

- DropHeatmap.

- Deliverable: two primary sources feeding the UI; search works.

### Phase 3 - Python scraper bridge

- NATS-based RPC to Python workers.

- FIFA adapter (Python, Playwright).

- AXS adapter (Python).

- Scraper health surfacing in `/sources`.

- MinIO screenshot capture on parse failure (debug).

- Deliverable: FIFA + AXS drops flow through the same pipeline.

### Phase 4 - Sports expansion

- NFL, NBA, NHL, MLB via Ticketmaster segment + SeatGeek cross-reference.

- Sport-aware UI badges.

- Watchlist `kind: team` with team slug resolution.

- Deliverable: major sports covered.

### Phase 5 - Resale price tracking

- StubHub + Vivid Seats Python scrapers.

- Price-drop detection (≥10% / ≥25% thresholds).

- Resale price sparkline overlay on event detail.

- Deliverable: alerts when resale prices drop.

### Phase 6 - Shock drops + extras

- Twitter/X monitor.

- Tennis (ATP/WTA), F1, UEFA, Premier League (Big 6).

- Festival adapters (Coachella, Glastonbury, Tomorrowland, EDC, Lollapalooza first).

- Command palette (Cmd+K).

- PWA manifest.

- Deliverable: full breadth.

### Phase 7 - Polish & resilience

- Grafana dashboards + alerts (Prometheus alertmanager).

- Backup automation.

- Source sharding for horizontal scale.

- i18n (en first; structure for more).

- Accessibility audit.

- Deliverable: production-grade.

---

## 15. Coding Standards (HARD RULES)

### 15.1 Rust

- `#![forbid(unsafe_code)]` at crate root.

- `clippy::pedantic` enabled; fix all warnings before merge.

- `rustfmt` with edition 2021.

- Errors: `thiserror` for error enums, `?` propagation, no `unwrap()`/`expect()` outside tests and main startup.

- Async: `tokio` runtime only. No `async_std`.

- DB: sqlx with `query!` macros (compile-time checked). Migrations under `migrations/`, never edit applied.

- Logging: `tracing::{info, warn, error, debug}` with structured fields. No `println!`.

- Config: load from env via `figment` or hand-rolled; fail fast on missing required keys.

- Tests: `#[tokio::test]` for async; use `wiremock` for HTTP mocking.

### 15.2 Python

- Python 3.11+.

- `ruff` (lint + format). `mypy --strict` passes.

- `uv` for env + lockfile.

- All scrapers typed (Protocol return types).

- Async-first (`async def` everywhere in hot path).

- No global mutable state.

- Tests: `pytest` + `vcr.py` cassettes for HTTP replay.

### 15.3 TypeScript / Next.js

- `"strict": true` in tsconfig.

- `eslint` + `prettier`.

- App Router conventions: server components by default, `"use client"` only where needed (SSE consumers, forms with state).

- All API calls go through `lib/api.ts` - no raw `fetch` in components.

- All server actions / route handlers validate input with Zod.

- No `any`. No `// @ts-ignore`.

- Tailwind only for styling - no inline `style={}` except dynamic values.

- Tests: Vitest for unit, Playwright for e2e (one happy-path per phase).

### 15.4 Cross-cutting

- **No secrets in repo.** `.env` gitignored. CI fails if a secret-looking string is committed (use `gitleaks`).

- **Every PR runs:** `cargo clippy`, `cargo test`, `ruff`, `mypy`, `pytest`, `tsc --noEmit`, `eslint`, `prettier --check`. Wire these into a root `Makefile` target `make check`.