Everyframe · TypeScript SDK reference

ai: analysis ops

Sort-changing ML analysis (detect: Video to Detection, transcribe: Audio to Transcript) and typed post-processing on analysis streams (track, zone, diarize).

@jtdigital/renderbox-sdk on npm →

aggregateTracks

Rust

Detection → Detection

DetectionStream.aggregateTracks(opts?: AggregateTracksOpts): DetectionStream

Aggregate per-track attribute predictions across frames

Param	Type	Description
attribute	string	Attribute to aggregate across each track.
method	'majority_vote' \| 'weighted_avg' \| 'last_n'	Aggregation strategy applied per track.
window	number	Number of trailing frames considered. Range: greater than 0

classifyAudioEvent

ONNX

Audio → AudioEvent

AudioStream.classifyAudioEvent(opts?: ClassifyAudioEventOpts): AudioEventStream

Classify audio events (gunshot, siren, animal call, etc.)

Param	Type	Description
model	string	Audio event classification model id.
threshold	number	Minimum event confidence. Range: [0, 1]
window_seconds	number	Analysis window length in seconds. Range: greater than 0
hop_seconds	number	Stride between analysis windows in seconds. Range: greater than 0

classifyScene

ONNX

Video → Scene

VideoStream.classifyScene(opts?: ClassifySceneOpts): SceneStream

Classify scene content

Param	Type	Description
model	string	Scene classification model id.
threshold	number	Minimum class confidence. Range: [0, 1]
top_k	number	Number of top classes to report per frame. Range: greater than 0
batchSize	number	Frames per inference batch.

computeAngles

Rust

Pose → Pose

PoseStream.computeAngles(opts?: ComputeAnglesOpts): PoseStream

Compute joint angles between keypoint triplets

Param	Type	Description
angles	string

count

Rust

Detection → Detection

DetectionStream.count(opts?: CountOpts): DetectionStream

Count detected objects per frame

Param	Type	Description
aggregate	'sum' \| 'avg' \| 'max' \| 'min'	How per-frame counts are reduced over time.

detect

ONNX

Video → Detection

VideoStream.detect(opts: DetectOpts): DetectionStream

Detect objects in video

Param	Type	Description
model required	string	Detection model id (e.g. 'retinaface_mv2' for faces, permissively licensed). See `renderbox models list`.
threshold	number	Minimum confidence for a detection to be kept. Range: [0, 1]
classes	string	Comma-separated class names the model should report.
filter_classes	string	Comma-separated class names to drop from the output.
inputSize	string	Model input resolution override, e.g. '640x640'.
stride	number	Run inference every N frames; detections are interpolated between.
batchSize	number	Frames per inference batch.

diarize

Rust

Transcript → Transcript

TranscriptStream.diarize(opts?: DiarizeOpts): TranscriptStream

Speaker diarization on transcript

Param	Type	Description
max_speakers	number	Upper bound on distinct speakers. Range: greater than 0
min_duration	number	Minimum segment length in seconds. Range: at least 0

estimateDensity

ONNX

Video → Density

VideoStream.estimateDensity(opts?: EstimateDensityOpts): DensityStream

Estimate crowd density

Param	Type	Description
model	string	Crowd-density model id.

estimateDepth

ONNX

Video → Depth

VideoStream.estimateDepth(opts?: EstimateDepthOpts): DepthStream

Estimate depth map

Param	Type	Description
model	string	Monocular depth model id.

estimatePose

ONNX

Video → Pose

VideoStream.estimatePose(opts?: EstimatePoseOpts): PoseStream

Estimate human poses

Param	Type	Description
model	string	Pose estimation model id.
threshold	number	Minimum keypoint confidence. Range: [0, 1]
max_persons	number	Cap on the number of tracked persons per frame. Range: greater than 0

filterAudioEvents

Rust

AudioEvent → AudioEvent

AudioEventStream.filterAudioEvents(opts?: FilterAudioEventsOpts): AudioEventStream

Filter audio events by class or confidence

Param	Type	Description
include	string
exclude	string
min_confidence	number	Range: [0, 1]
min_duration	number	Range: at least 0

filterDetections

Rust

Detection → Detection

DetectionStream.filterDetections(opts?: FilterDetectionsOpts): DetectionStream

Filter detections by class, attribute, or track_id (assigned by track())

Param	Type	Description
include	string	Comma-separated class names to keep.
exclude	string	Comma-separated class names to drop.
min_confidence	number	Drop detections below this confidence. Range: [0, 1]
min_area	number	Drop detections smaller than this fraction of the frame. Range: [0, 1]
min_track_frames	number	Drop tracks shorter than this many frames. Range: at least 0
attribute	string	Keep only detections carrying this attribute.
attribute_value	string	Required value for `attribute`.
attribute_absent	string	Keep only detections missing this attribute.
track_id	number	Keep only this track id. Range: at least 0
exclude_track_id	number	Drop this track id. Range: at least 0

filterPoses

Rust

Pose → Pose

PoseStream.filterPoses(opts?: FilterPosesOpts): PoseStream

Filter poses by score

Param	Type	Description
min_score	number	Range: [0, 1]
max_persons	number	Range: greater than 0

filterScenes

Rust

Scene → Scene

SceneStream.filterScenes(opts?: FilterScenesOpts): SceneStream

Filter scenes by confidence

Param	Type	Description
min_confidence	number	Range: [0, 1]

filterSegmentation

Rust

Segmentation → Segmentation

SegmentationStream.filterSegmentation(opts?: FilterSegmentationOpts): SegmentationStream

Filter segmentation masks

Param	Type	Description
classes	string
min_area	number	Range: at least 0

filterText

Rust

Text → Text

TextStream.filterText(opts?: FilterTextOpts): TextStream

Filter text regions by confidence or pattern

Param	Type	Description
min_confidence	number	Range: [0, 1]
pattern	string

insertChapters

Rust

Scene → Scene

SceneStream.insertChapters(opts?: InsertChaptersOpts): SceneStream

Insert chapter markers from scene boundaries

Param	Type	Description
min_duration	number	Range: at least 0
format	string

recognizeText

ONNX

Video → Text

VideoStream.recognizeText(opts?: RecognizeTextOpts): TextStream

OCR text recognition

Param	Type	Description
model	string	OCR model id.
threshold	number	Minimum text-region confidence. Range: [0, 1]

segment

ONNX

Video → Segmentation

VideoStream.segment(opts?: SegmentOpts): SegmentationStream

Segment image regions

Param	Type	Description
model	string	Segmentation model id.
threshold	number	Minimum mask confidence. Range: [0, 1]
pointsPerSide	number	Grid density for automatic mask generation (SAM-style models).

track

Rust

Detection → Detection

DetectionStream.track(opts: TrackOpts): DetectionStream

Track detected objects across frames

Param	Type	Description
algorithm required	'bytetrack' \| 'sort' \| 'deepsort'	Multi-object tracking algorithm to assign stable track ids.
persistFrames	number	Range: [0, 30]

transcribe

ONNX

Audio → Transcript

AudioStream.transcribe(opts?: TranscribeOpts): TranscriptStream

Transcribe audio to text

Param	Type	Description
model	string	Speech-to-text model id.
language	string	ISO language hint (e.g. 'en', 'sl'); auto-detected when omitted.

zone

Rust

Detection → Detection

DetectionStream.zone(opts?: ZoneOpts): DetectionStream

Filter detections by spatial zone (ROI geofencing)

Param	Type	Description
polygon	string	Zone polygon as 'x1,y1 x2,y2 …' in normalized coordinates.
anchor	'centroid' \| 'bottom_center' \| 'bbox'	Which point of the bounding box must fall inside the zone.
invert	boolean	Keep detections outside the zone instead.
attribute	string	Attribute name stamped onto matching detections.