Everyframe · computer vision

Counting exercise reps from motion.

A pose-driven pipeline that watches a person exercise, finds the repeating movement, and labels each rep i / N. It is built on the Everyframe vision stack and was validated on eight real workplace-ergonomics clips.

7 exercise clips counted pipeline pose → signal → label 17-keypoint RTMPose 100% on-device exercise- agnostic counting

The result, live

A box-lift clip with the detected rep count burned in, and the underlying motion signal tracking beneath it. The counter and the playhead are driven by the same per-frame data the algorithm counts on.

0 / 0

Dominant-motion signal · rep peaks marked

The waveform is the projection of the body's motion onto its single most-active axis (see the mathematics ). Each amber mark is one detected repetition; the static intro and outro carry no oscillation, so they are never counted.

What the machine sees

The Everyframe engine never reasons about pixels. Every frame it runs a person detector (D-FINE) and top-down pose estimation (RTMPose), reducing the body to 17 tracked joints, with no sensor or marker worn. The two sides of the body are drawn in contrasting colours, so the alternation the counter keys on is visible directly.

march : the raised knee reads identically in the frame and the skeleton.

box lift : overhead reach and wide stance, tracked joint-for-joint.

The pipeline

Three stages. Perception and labelling run inside the Everyframe engine; the counting is pure signal processing on the extracted joints.

Perceive Everyframe

Person detection → per-person pose → a joint stream written to disk. One new SDK sink, sink_pose_jsonl , emits one skeleton record per frame.

// examples/ai/rep_extract_pose.rs
let dets    = detect_with(&v, "dfine_l_coco", …);   // person boxes
let persons = filter_detections(dets, "person");
let poses   = estimate_pose_detections(&v, persons,
                  "rtmpose-m", …)?;                // 17 COCO keypoints
sink_pose_jsonl("pose.jsonl", poses);            // ← new sink

Measure signal processing

Normalise the joints, project onto the dominant motion axis to get one clean 1-D signal, and read the rep period off its autocorrelation. The full derivation is in the mathematics .

Mark Everyframe

Back in the Everyframe engine: one time-gated drawtext per rep, each visible only across its own window, burns the running i / N onto the original full-resolution video.

// examples/ai/rep_label.rs
for r in reps {                       // Everyframe drawtext, per rep
  v = v.pipe(drawtext(DrawText {
    text:   format!("{} / {}", r.index, n),
    enable: between(t, r.start, next),   // only during this rep
  }));
}

Results

Every clip, counted end to end. Cadence and period fall straight out of the autocorrelation; PC1 % is how much of the body's motion the single measured axis captures: high numbers mean an unambiguous, clean rep signal.

Clip	Exercise	Reps	Period	Cadence	PC1

The mathematics

Once the joints are extracted, the counting is four steps of signal processing: no learned model, no per-exercise rules.

Normalise the pose

p̃ _i = ( p _i − c ) ∕ L

Every joint p _i is re-expressed relative to the hip centre c = ½(hip _L + hip _R ) and divided by the torso length L = median ‖shoulder _c − c ‖. Camera zoom, framing and where the person stands all cancel, only the shape of the motion survives.

Find the axis of motion: PCA

X = U Σ V ^⊤

s = X v ₁

Stack every frame's normalised joint coordinates into a matrix X and take its SVD. The first right-singular vector v ₁ is the single direction the body varies along most; projecting onto it collapses two dozen wobbling coordinates into one clean signal s ( t ). This is why the counter is exercise-agnostic : PCA reads the axis off the data, so knees drive a march and the torso a lift with no code change. The share σ ₁ ² ∕ Σ σ ² is the PC1 % in the results table.

Measure the period: autocorrelation

r ( τ ) = Σ _t s ( t ) s ( t + τ ) ∕ Σ _t s ( t ) ²

Compare the signal with time-shifted copies of itself. The first strong peak of r ( τ ) after its initial dip, searched only in a plausible 0.33–6 s band, is the rep period T . If nothing clears the bar there is no periodicity, and the pipeline reports zero rather than inventing reps.

Count the cycles

N = #{ local maxima of s , spaced ≥ 0.6 T , above a prominence gate }

On the smoothed signal, each qualifying peak is one repetition; the troughs between peaks bound each rep in time, exactly the windows the i / N label is gated on. One rep is one full period: for an alternating move that is a left-plus-right cycle, so per-side counting simply doubles it.

Exercise-agnostic by construction

No per-exercise rules. PCA discovers whichever body part actually moves (knees for a march, the whole torso for a lift), so the same code counted marches and box-lifts without a single parameter change.

It refuses to guess

An eighth clip is a talking intro with no exercise. Autocorrelation found no periodic signal, so the pipeline counted zero . No invented reps on non-repetitive footage.

Everyframe vision stack · per stage:
  detect_with · filter_detections · estimate_pose_detections  → pose
  sink_pose_jsonl (added)  →  pose.jsonl  → host signal processing
  drawtext (time-gated, between(t,a,b))  → labelled mp4 + audio
models · D-FINE-L (person, Apache-2.0)  +  RTMPose-m (17-kpt, Apache-2.0)