# CTI Detection Automation

Pulls indicators from live threat-intel feeds, deduplicates them, extracts the

MITRE ATT&CK techniques behind them, turns the result into Wazuh detection rules

and CDB lists, and emails an analyst for sign-off before anything goes live. No

rule reaches the SIEM without a human approving it.

This is the piece that feeds the watchlists used by the

[SOC automation lab](../soc-automation-lab) - the `cti-malicious-ip`,

`cti-malicious-domain`, and `cti-malware-hash` lists that the Wazuh rules look up

against are generated here.

![Approval email](docs/screenshots/01-approval-email.png)

## What it does

- **Collects** from ThreatFox, Feodo Tracker, URLhaus, AlienVault OTX, and OpenPhish,

  covering IPs, domains, URLs, file hashes, phishing, exploit-kit and malware-download

  URLs, C2 infrastructure, and (optionally) leaked credentials.

- **Normalizes** every source into one indicator model and **deduplicates** across

  feeds - an IP seen in both ThreatFox and Feodo becomes one indicator carrying both

  sources, the higher confidence, and the union of techniques.

- **Extracts and dedups TTPs**: ATT&CK technique IDs come straight from OTX pulses and

  are inferred from malware family and threat type for the feeds that don't carry them,

  then collapsed into a ranked coverage list.

- **Generates Wazuh rules**: CDB lists per indicator type plus an XML ruleset whose

  list-lookup rules are tagged with the dominant ATT&CK techniques for that bucket.

- **Gates on approval**: every run produces a candidate bundle and emails the analyst a

  signed, time-limited review link. Rules are written to Wazuh only when the analyst

  approves.

```mermaid

flowchart LR

    F[ThreatFox / Feodo / URLhaus<br/>OTX / OpenPhish] --> N[normalize]

    N --> D[deduplicate]

    D --> C{confidence<br/>>= threshold}

    C --> T[extract + dedup TTPs]

    T --> G[generate CDB lists<br/>+ Wazuh rules]

    G --> M[email analyst<br/>signed review link]

    M -->|approve| W[promote to Wazuh]

    M -->|reject| X[discard]

```

## The approval gate

The pipeline never deploys on its own. It writes the candidate to a staging

directory and emails a review link. The analyst sees the diff against the last

approved bundle, the indicator counts, and the ATT&CK coverage, then approves or

rejects.

| Review page | Bundle history |

|---|---|

| ![Review](docs/screenshots/02-review-page.png) | ![Dashboard](docs/screenshots/03-dashboard.png) |

Short walkthrough of the flow: [docs/media/cti-approval-walkthrough.mp4](docs/media/cti-approval-walkthrough.mp4)

The review link is an `itsdangerous` signed token with a TTL, so it can't be

forged or replayed after it expires.

## Run the demo

No API keys needed - the demo runs against the bundled fixtures:

```bash

make install

make demo      # generates a bundle and writes the approval email to output/emails/

make serve     # approval console on http://localhost:8080

```

Open the email under `output/emails/`, click through to the review page, and

approve. The promoted lists and rules land in `output/active/`.

## Run it for real

```bash

cp .env.example .env          # set CTI_APPROVAL_SECRET and feed API keys

cp config.example.yaml config.yaml

python -m cti.cli run -c config.yaml

```

ThreatFox and OTX need free API keys (`THREATFOX_AUTH_KEY`, `OTX_API_KEY`); Feodo,

URLhaus, and OpenPhish are keyless. Set `email.backend: smtp` and the SMTP settings

in `config.yaml` to send the approval mail to a real inbox. Point `wazuh_etc_dir` at

the manager's `/var/ossec/etc` and approval will write the lists and rules straight

into place.

Schedule it with the systemd timer in `deploy/` (hourly by default) or any cron.

## Layout

```

src/cti/feeds/     one connector per source, each with a pure parse()

src/cti/dedup.py   cross-feed merge

src/cti/ttp.py     ATT&CK extraction + coverage report

src/cti/rules.py   CDB list + Wazuh XML generation

src/cti/approval.py signed tokens + email rendering + SMTP

src/cti/pipeline.py orchestration, candidate staging, promotion

src/cti/web.py     Flask approval console

fixtures/          sample feed payloads for the demo and tests

tests/             pytest suite (feeds, dedup, ttp, rules, approval, web)

```

## Tests

```bash

make test

```

Connectors are split into `fetch` and `parse`, so the suite runs the real parsing

logic against fixtures with no network. Coverage includes cross-feed dedup, TTP

extraction, rule generation well-formedness, token signing/expiry, and the full

approve/reject path through the web app.

See [docs/architecture.md](docs/architecture.md) for the data model and

[docs/approval-flow.md](docs/approval-flow.md) for the gate in detail.