GameSet

GameSet turns one ordinary recording of an amateur match (a phone on a tripod, or a fixed club camera) into per-player stats, highlights and a replay of the game. Players don’t wear sensors, nobody tags the footage, and no special camera is needed. I’m building it as a startup, and I’ve done the computer vision pipeline, the cloud processing and the product myself. It went live at gameset.gg in September 2026, and volleyball is the first sport.

+23%

tracking accuracy (HOTA) from one evidence-led detector swap

0.04 m

court registration error, with zero wrong registrations

0.998

identity purity on a 5-minute volleyball test

2.1×

less GPU time per minute of footage after the cloud redesign

The problem

Amateur and social sport is filmed constantly, but almost none of it turns into anything useful. The systems that extract stats from video are built for professional broadcasts with multiple calibrated cameras, analysts tagging events by hand, or both. A social volleyball team has one phone and no analyst.

The hard part isn’t detecting people. It’s knowing who each person is for the whole match. Players leave the frame, cross in front of each other and wear identical kits. A stat is only worth something if it’s attached to the right person, so identity ended up being the centre of the whole system.

The pipeline

GameLab (the processing side of GameSet) is a modular tracking-by-detection pipeline rather than one end-to-end model. Each of its 13 stages is a slot in a config file, so a new detector, tracker or calibrator can be swapped in and compared against the incumbent on the same footage without touching the rest of the system.

video → rally state → detect → track → team → calibrate → associate → identity
      → fuse → possession → events → spotting → attribute → annotate → emit

The key design decision is that identity is resolved offline, per tracklet, over the whole match, never frame by frame. A tracklet (a short run of frames where the tracker is confident it’s following one person) is cheap to produce and usually correct. Joining tracklets into people is where the evidence from the full match can be brought to bear.

Measure the ceiling before tuning

The first version of the tracker swapped player identities far too often to be useful. The obvious next step was to tune the tracker. Before doing that, I ran an oracle test: I replaced the detector’s output with the ground-truth boxes and kept everything else the same.

Tracking accuracy (HOTA) jumped from 0.49 to 0.84 on SoccerNet and from 0.17 to 0.86 on SportsMOT, and 63 to 75% of identity switches disappeared. In other words, most of the tracking errors were detection errors in disguise. A round of tracker tuning confirmed it, recovering only about 4% of the gap.

That changed where the effort went. I set up a detector bake-off of more than 20 checkpoints with a pass/fail bar fixed before any results came in. Two public SoccerNet checkpoints had to be thrown out after I found they’d been trained on the benchmark’s test split. The winner lifted HOTA from 0.502 to 0.618, cut identity switches by 59% and ran 3.3 times faster. That one swap was worth roughly eight times the entire tuning phase.

This shows the value of measuring the ceiling of each stage before optimising it. It’s now a standing tool in the pipeline: every identity switch is automatically blamed on the layer that caused it.

Identity that can say “I don’t know”

Players on the same team wear the same kit, so appearance alone can’t tell them apart. When I measured it, the similarity between two crops of the same player overlapped heavily with the similarity between two different teammates. Jersey numbers help, but on their own they’re hard to read reliably from a phone camera at a distance.

Rather than trusting any one signal, the associator combines several weak ones as calibrated evidence: appearance embeddings, jersey readings, team, motion and court position. Each pair of tracklets gets a single likelihood score for “same person”. When the evidence is weak, the system abstains rather than guessing. On football, this recovered 50 tracklet merges with zero wrong merges. For volleyball, I used a vote between a small 2B-parameter vision-language model and a dedicated scene-text model to read jersey numbers, which reached 97.6% per crop (against 82% for the best dedicated OCR model).

The same principle carries through to the product. When a player’s identity across different matches is uncertain, the app asks the player to confirm from a ranked shortlist instead of assigning it silently. In testing, the cross-match matcher assigned 10 of 30 players and got none wrong, leaving the rest for the player to confirm. A wrong stat on someone’s profile does more damage than a missing one.

Sport-specific geometry beats generic models

In September I moved the first release from football to volleyball. Volleyball has strong structure that a general model doesn’t know about: a fixed court, a net that splits the teams, a rotation order and about twelve players. It’s also the sport where the target users actually play weekly competitions.

Several general-purpose models fell over on gym footage. A learned court-keypoint network had a median error of 53 pixels, and a specialist ball tracker that was best at low resolution found under 3% of balls at 1080p. Replacing them with approaches that used the sport’s structure worked much better:

• Court registration: a model-free search for the court lines that refuses to register when its evidence disagrees. Across 13 test sources it registered 7 with a median error of 0.04 m, got none wrong, and declined the other 6.
• Team assignment: which side of the net a player is on, at 98.1% accuracy.
• Ball detection: a general detector’s own ball class reached 96.8% recall at full resolution.
• Rallies: a fine-tuned video model found all 159 rallies in the test footage, and floor bounces were located to a median of 0.14 m, with all 119 in/out calls correct.
• Who touched the ball: pose-based contact attribution moved from 8 correct, 42 wrong and 46 abstained to 71 correct, 14 wrong and 11 abstained.

This highlights a pattern I rely on: when a general model fails in a narrow domain, it’s usually cheaper to encode what you know about the domain than to collect more data for the general model.

Shipping null

Not every measurement made it. Jump height was accurate on average, but when I checked whether its error bars could be trusted, a nominal 95% interval only contained the true value 60% of the time. Rather than ship a number that looked precise and wasn’t, jump height ships as not measured.

This is enforced all the way through. In the results contract between the pipeline and the app, null means “not measured” and is never rendered as zero. A metric’s meaning is pinned to the contract version, so changing how something is calculated means a new version, not a silent change. The contract went through six versions in about six weeks without the app and the pipeline drifting apart, because both sides test against the same schema.

Making a match affordable to process

The first full half of football I ran end to end took 22.7 GPU-hours on paper. Moving pose estimation to the GPU made it 3 times faster, but the run also exposed problems that only show up at full length. One of them was 280 GB of image crops being kept on disk.

Moving to the cloud exposed the next problem. When I costed a match on AWS, moving video frames through shared storage cost more than the GPUs themselves. I redesigned processing around that:

• Pixels never leave the worker that decoded them. Only JSON goes to shared storage, which cut the run directory from 950 MB to 35 MB per minute of footage.
• The match is split into chunks that are tracked in parallel and stitched back together at the overlaps.
• Each match launches its own fleet of spot GPU workers through an event, and the fleet shuts down as soon as the work runs out.

GPU time per minute of footage dropped from 615 seconds to about 290 on test clips, and the part of the run that can’t be parallelised dropped from 78 seconds to about 17. Chunking has a cost, which I measured rather than assumed: it costs about 7 points of identity completeness on one team, which is the trade I’m currently working to close.

Where it’s at

GameSet launched in production at gameset.gg in September 2026, with volleyball first. The cloud processing fleet is deployed to staging, and I’m now working on identity through substitutions and libero swaps, and on getting a full match processed in under 30 minutes.

Stack

Computer vision: YOLO-family detectors, BoT-SORT and other multi-object trackers, OSNet and part-based re-identification models, RTMPose, PARSeq and Qwen3-VL for jersey numbers, VideoMAE and action-spotting models, and custom court calibration.

Evaluation: TrackEval (HOTA, IDF1), oracle ablations, held-out splits with leakage checks, and success criteria fixed before results.

Infrastructure: Python and FastAPI, a job table with leases and heartbeats, AWS ECS on spot GPUs, EventBridge, Aurora Serverless, Pulumi.

Product: Next.js, React, Postgres with Drizzle, ffmpeg (HLS streaming and clip export), and a typed contract shared between Python and TypeScript.

I’m happy to walk through the code and the decisions in more detail on a call.