pitch-tracker-cv

A real-time computer-vision aim-assist for offline, single-player MLB The Show 26. Built as an accessibility aid for players with motor disabilities who cannot reliably execute the small left-stick corrections that the game's Zone hitting interface demands, or the precise gesture inputs that Pinpoint pitching requires.

The system reads the game through a capture card, runs a vision pipeline that tracks the ball and the PCI (Plate Coverage Indicator), predicts where the pitch will cross the plate, and drives a Titan Two adapter to nudge the controller stick so that the user can play vs-CPU modes they otherwise could not.

This is not a cheat. The tool only operates in offline modes (Diamond Dynasty vs CPU, Conquest, Moments, Showdown offline, Road to the Show, March to October, Franchise). It refuses to arm if any online UI element is on screen. See ACCESSIBILITY_STATEMENT.md and NOTICE.md.

What it is, in one sentence

A 1080p60 capture-card video stream goes in, controller stick deflections come out, with a YOLO-trained ball detector and classical CV PCI tracker in between.

Architecture

+----------------+        +-----------+      +--------------+      +-----------+
| Capture card   |  USB3  | ingest.py |  ZMQ | ball_tracker |  ZMQ | bridge.py |
| (1920x1080@60) +------->|           +----->|              +----->|           |
+----------------+        +-----------+      | pci_tracker  |      +-----+-----+
                                             +--------------+            |
                                                                         v
                                                                +-----------------+
                                                                | Titan Two (GPC) |
                                                                +--------+--------+
                                                                         |
                                                                         v
                                                                +-----------------+
                                                                | Xbox controller |
                                                                +-----------------+

Three independent processes connected by ZMQ on localhost:

1. capture/ingest.py opens the capture card via OpenCV/DirectShow and publishes raw frames as multipart [meta_json, jpeg] messages on tcp://127.0.0.1:5555. Measured at 59.88 FPS @ 1920x1080 with mean read latency 16.3 ms (p95 17.4 ms) on the reference hardware.
2. cv/ball_tracker.py subscribes to capture frames, applies an HSV + circularity filter (or an ONNX YOLO detector for harder lighting), fits a 2D parabolic trajectory across a rolling 0.8 s window, and emits pitch_pred events with predicted plate-crossing (x, eta_ms).
3. cv/pci_tracker.py template-matches or HSV-segments the PCI circle and emits pci_track events with the current PCI position.
4. io_titan/bridge.py subscribes to both tracker streams, runs the decision logic (aim error → stick deflection, residual error + count → swing decision), packs a 20-byte fixed-layout command frame, and writes it to the Titan Two via Gtuner IV's GCV (Game Computer Vision) interface.
5. io_titan/mlb26_bridge.gpc is the GPC script that runs on the Titan Two itself, reads the command frame from GCV memory, and applies the stick / button input to the Xbox controller passthrough.

Hardware required

• Xbox Series X/S
• Monster 4K USB 3.0 capture card (or any 1080p60 DirectShow-compatible card)
• Titan Two adapter with Gtuner IV installed
• Wired Xbox controller
• Windows PC for the vision pipeline (Python 3.11+)

Quick start

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

python -m capture.diagnose
python -m capture.ingest --smoke

python -m cv.ball_tracker
python -m cv.pci_tracker
python -m tools.viewer

YOLO ball detection (optional, more robust under HDR tone-mapping):

pip install -r requirements-yolo.txt
python -m tools.yolo_train_ball --data configs/ball_yolo.yaml
python -m io_titan.mlb26_gcv_yolo

Project layout

capture/         capture card ingest, device probe
cv/              ball tracker (HSV + parabolic fit), PCI tracker
io_titan/        Python + GPC bridge to Titan Two
configs/         runtime tunables, YOLO data config, templates
tools/           one-shot utilities: viewer, snapshot, hsv_probe,
                 detect_on_frame, YOLO frame collection/labeling/training
docs/            architecture notes, ball-detector reference

Safety model

The pipeline is designed to fail safe:

• safety.online_mode_abort in configs/runtime.yaml instructs the bridge to disarm if any online-mode UI element is detected.
• safety.abort_on_menu_detected disarms when the game leaves play.
• safety.abort_on_capture_loss_ms disarms if the capture stream stalls.
• A hardware deadman hotkey (default F12) forces full passthrough.
• The Titan Two script always passes raw controller input through. Aim assist is additive only and clamped to small stick deflections.

License

MIT, with an explicit additional clause forbidding use in any online or multiplayer context. See LICENSE.