I realize, that now once a month someones shares this kind of project, but still wanted to share :)
I wanted to make it work on regular club footage, have it working on 3 courts now. It's mostly general by now, but still requires manual calibration (10 points) on each court, or if camera have moved.
It's my first time dabling with CV, so just playing around. All of it is vibecoded, but with my direction, which is sometimes limited, because.. well, no experience with CV. I have over 10 years of dev experience in other fields, so that helps a lot.
This is an excerpt of my marking/diagnostics app (also vibecoded), two more examples here:
https://streamable.com/au93be
https://streamable.com/axz0pg
All the footage here has gone trough my review (so hits and point endings fixed, don't remember the actual state of these points, but you can see it in the bottom graph (with little square boxes) - the bottom one is the actual label, and the middle one is decoded one. I only now realized, that this might not be in the spirit of this subredit to show "fixed" footage, but it's mostly identical, and I don't have any other examples at the moment)
Some info about the project (AI generated, sorry), if you care about the tech, or want to roast it (please do!):
The pipeline takes a fixed club-camera recording of a padel match and turns it into a list of ball contacts (who hit, when) grouped into points. Every stage feeds the next.
Stages and models:
Court calibration (court_calibrate.py) — no model. A homography plus lens distortion fitted from ~10 clicked court points, so pixel positions can be converted to court metres.
Player tracking (track_players.py) — YOLO detector at 1280 px. Finds the four players each frame and assigns them to court slots (near-left, near-right, far-left, far-right).
Ball detection (track_ball_wasb.py) — WASB, a small heatmap network fine-tuned on padel footage. Run twice: a strict "event" track for kinematics and a looser "state" track only for coverage features.
Player pose (track_pose.py + far-crop pass) — YOLO11-pose, 17 keypoints. Gives wrist/body positions; far players get an upscaled crop pass because they are tiny.
Candidate generation (detect_hits_from_ball.py) — no model. Speed reversals and gaps in the ball track propose possible contact frames and attribute them to a player slot.
Strike head (strike_scores.py, strike_dense.py lattice) — a CNN on player crops around the candidate frame, trained to say "this player is striking now". Scores candidates, re-times them to the true contact frame, and recovers missed hits.
Strike-withhold mask — no model. Suppresses candidates whose attributed player is a stale, frame-clipped box (out of view).
Point decode (decode_points.py) — a Viterbi-style dynamic program over candidates. Uses two gradient-boosted-tree heads: an emission head (is this candidate a real hit) and a bounce head (is this a bounce, not a hit), plus the strike score and ~15 hand-set costs, to pick the hit sequence and open/close points.
Merge stages — join point fragments across short gaps; a small formation MLP vetoes point starts when players are not in a serving formation.
Eval (eval_hits.py) — scores against hand labels with a 6-frame hit tolerance and 12-frame point-start tolerance.
Bottom line: five learned models (WASB ball, YOLO players, YOLO11-pose, strike CNN, two GBT heads + formation MLP) produce evidence; one deterministic decoder turns that evidence into hits and points. Current in-sample pool F1 is 0.9507 across six cameras.
It has around 95% accuracy (detects 95% of hits correctly, within a specific margin). It detects and opens points with around 90% accuracy using serve formation. The problem is the point endings. I realise that it's likely where this project dies - currently I'm using the "quitness" of the ball and the players, and it works decently. But I'm fixing those manually for now, as well as the winners, so I'm able to generate extended report about the match and mistakes.
Other problem I have is long processing time. I already optimized it a bit, but I feel like I'm probobly using to much stuff. The pipeline grown naturally, by using different things to improve the decode. But I feel like I can try to delete some stuff from it. For example, skipping player pose and using strike head only, things like that. Although, I'll want pose for detecting shot types later, so.. yeah..
My current pipeline timings:
| stage |
wall |
| phase A (ball dual + players + base pose, concurrent) |
1h13m01s |
| far-crop pose |
36m07s |
| candidates (C3) |
4m13s |
| strike scores (fp16) |
5m32s |
| strike dense scan (fp16) |
16m14s |
| lattice / withhold / decode+merges / sidecar |
1m59s / 8s / 28s / 7s |
| total |
2h17m56s (1.63x realtime) |
Another problem is the ball. Would love to have some suggestions, how I could get the actual 3d position of it, but my research came with nothing valuable for my case (1 camera angle).
Would love to hear any advice, directions or any other feedback. Thank you!
