Auto annotate tracks

[vykhovanets]

This is more like a question, I suppose it is possible with existing functionality, but I do not know how. If not, then this is a feature request :)

The workflow is as follows: I select the initial bounding box, give it a track id as an attribute, and then ask model to use that bounding box to predict/track on subsequent frames untill it fails.

I know that SAM2 kinda can do that, and because it is big enough model, I can rely on its predicitons. But the simpler solution would be just to use the actual tracker model.

[ryouchinsa]

Thanks for your feature request.
Reading this document, we will learn how to implement BoT-SORT or ByteTrack on RectLabel.

Apple's document will be easier for us to implement.
Tracking Multiple Objects or Rectangles in Video.

We will focus on the ByteTrack ONNX model.
https://github.com/FoundationVision/ByteTrack/tree/main/deploy/ONNXRuntime

[vykhovanets]

Ultralytics can have license issues.

I am currently evaluating small transformer-based trackers, and the interface is pretty simple, also, I suppose, all of them can be ported to either coreML or onnx.

model.init(bbox, frame)
model.predict(new_frame) → new_bbox, confidence

the simplest tracking implementation would be:

1. draw bbox and select "tracking mode" for it
3. on → (right arrow) click we initialize tracker with the current frame and tracked box, go to next frame, predict new box, and make it tracked.

(time for user to update box if it is not super precise)

5. repeat

that way tracker will work one next frame, preventing it from drifting too much, keeping user in control, and also making annotation of moving objects way faster.

[ryouchinsa]

The new version 2025.10.25 was released.

We added "Create box using Tracking" on Edit menu.
Using the labels on the previous image, the tracking model predicts bounding boxes on the current image.

Currently we are using the tracking model of Apple.

We will investigate on the ByteTrack model soon.

To refine the predicted boxes.

• Change the edit mode to Create polygon using SAM and start preprocessing.
• Select all bounding boxes, right click on them, and select “Convert to box using SAM” to tighten selected boxes.

tracking.mp4

[vykhovanets]

Hi!

Thank you so much for adding this!

Regarding SAM box correction, can the action Convert box using SAM be added to the shortcuts list/menu items?

[ryouchinsa]

Thanks for your further request.
The new version 2025.10.30 was released.

Added "Convert to box using SAM" hotkey on Edit menu.
Please let us know your opinion.

box_sam.mp4

[vykhovanets]

Thank you for this update, I will give the opinion as soon as I return to the dataset creation!