Everyframe · computer vision
One swing, eight moments, one picture.
Segmentation is the engine's second model family after pose. It cuts the golfer out of every frame. The strobe pass stacks eight moments of one swing onto the address frame, the four the phase detector named and four between them, and draws the path the hands took. A still and a clip, from one phone clip.
The strobe, built up
Left, the build-up: the cutouts land one per beat on the address frame, and the hands' path grows with them. Right, the still the pass writes. Every cutout is the engine's mask on that frame; the path is the pose the tempo read already had.
Eight cutouts on the address frame, later moments on top
Later moments sit on top, and the trail fades in from the first moment after address to full at the finish, so the pile of bodies reads as motion. The moments are spaced by how far the hands travelled, not by time, so the downswing gets its share of a clip that spends most of its frames standing still.
Where the prompt sits
SAM2 takes one point and returns what is under it. The engine's prompt is the frame centre, so the pass cuts a square around a point on the golfer's body and segments that. No single point survives a whole swing: the arms cross the belly at impact, the pelvis sways. Three anchors, three renders, and per moment the pass keeps the mask that looks like a golfer.
| Anchor | Square | Frames caught |
|---|
Frames caught is the share of the swing's frames whose mask is a plausible golfer: an area within a band around the typical mask. A frame no anchor caught takes its nearest caught neighbour, and says so.
Pipeline
Two passes on the engine and one in numpy. The pose pass is the tempo read's; the cutout pass is new; the strobe pass stacks.
Phases from the pose golf_phases.py
The tempo read already knows address, top, impact and finish, and the hands' path in pixels. The strobe pass borrows both: the four phases are always moments, and the hands' travel spaces the rest.
Cut the golfer out golf_cutout.rs
A square around an anchor on the body, then
segment
with SAM2 and one centred prompt, then
mask_composite
painting everything outside the mask magenta. The engine returns the cutouts as a video.
let masks = segment_with(&framed, "sam2-tiny", SegmentParams {
points_per_side: Some(1), // one prompt, the frame centre
..Default::default()
});
let keyed = framed.try_pipe(mask_composite_with(masks, MaskCompositeParams {
mode: Some(CompositingMode::Replace),
background: Some("#FF00FF".into()), // nothing on a course is magenta
opacity: Some(1.0),
}))?;
sink_video(output, keyed.pipe(format("yuv420p")))?.execute().await?;Stack the moments golf_strobe.py
Key the cutouts back out, drop the specks, erode two pixels and feather. Then paste them onto the address frame in order, later on top, opacity rising to the finish, and draw the hands' path. A still, a poster and the build-up clip.
moments = pick_moments(hands_path(kps), fps, phases, n=8) # phases + 2 / 1 / 1 by travel
for m in moments[1:]:
k = best(m["frame"]) # the anchor whose mask is a plausible golfer
canvas = paste(canvas, src[f], masks[(k, f)] * m["opacity"], origins[k])
still = draw_path(canvas, path, moments)What it does not do
Honest edges of a first outing for segmentation.
SAM2's mask is 256 pixels a side. A shaft a few pixels wide falls between its cells, so most cutouts carry the hands and not the club. The address frame keeps its club because it is the frame, not a cutout.
One prompt, one mask. A second golfer in the square would be cut out with the first, or instead of them.
The engine used to stretch the mask onto the frame with nearest-neighbour, which drew ten-pixel steps at 1440p. It now resamples bilinearly and thresholds; the mask is still coarse, only rounder.
Nothing here is measured. The tempo and the stroke reads measure; the strobe shows.