Open Road Code

Documentation

OpenRoadCode Computer Vision

Source: controllers/computer_vision/README.md

The computer-vision layer consumes CameraFrame objects and publishes presentation-neutral object detections. Camera ownership remains in hardware_io.camera; perception does not open video devices directly.

Debian/Ubuntu setup

Use the project setup script rather than installing dependencies by hand:

./development/debian/setup_camera_perception.sh

The script installs the Linux camera tooling (v4l-utils, OpenCV, Python venv support), finds the active OpenRoadCode virtual environment or the repository venv, creates venv if needed, and installs Ultralytics there. It finishes with import checks and reports whether /dev/video0 is present.

Activate the environment when needed:

source venv/bin/activate

VM perception preview

Run the live preview with the default USB camera:

python -m controllers.computer_vision.component_test.perception_preview

The default configuration uses /dev/video0, 1920x1080 MJPEG capture at 30 FPS, the small yolo11n.pt model, 640-pixel inference input, and a confidence threshold of 0.35. Ultralytics may download the model weights on the first run.

The preview displays detections for road-relevant COCO classes: person, bicycle, motorcycle, car, bus, and truck. It also displays camera FPS, AI FPS, inference latency, and the current object count. Press q or Escape to exit.

Useful overrides:

python -m controllers.computer_vision.component_test.perception_preview \
  --device /dev/video0 \
  --model yolo11n.pt \
  --confidence 0.40 \
  --imgsz 640

Concurrency model

PerceptionWorker runs inference on a background thread. Camera capture and preview remain on the caller’s thread. Submitting a newer frame replaces any older pending frame that inference has not started processing, so the detector prefers fresh road-scene data over building latency in a frame queue.

Bounding boxes are normalized to [0, 1], allowing inference resolution and preview resolution to differ without coupling the UI to model input geometry.

orcUi VISION evaluation screen

The integrated orcUi shell exposes an isolated VISION navigation destination for camera/perception evaluation. It intentionally does not feed HOME, NAVIGATION, VEHICLE, or other screens while the feature is being developed.

Start orcUi normally:

python -m apps.orcUi

Select VISION from the side navigation. Entering the screen sends a semantic activation request to VisionController. The controller opens /dev/video0, starts YOLO inference, and publishes immutable VisionUiState snapshots with RGB image data and normalized object overlays. Leaving the screen sends a deactivation request; the controller invalidates queued callbacks, stops inference, restores daytime camera defaults, and releases the camera.

The Tk screen depends only on contracts under ui/vision. It renders complete state snapshots and emits explicit target-state requests for camera mode and AI enablement. Camera, V4L2, OpenCV, YOLO, ByteTrack, worker, and lifecycle ownership remain outside the widget. State uses seconds, hertz, normalized luminance, and normalized image coordinates; the frontend owns display labels and conversions such as seconds to milliseconds.

The screen provides three processing profiles:

  • AUTO: select low-light enhancement when mean scene luminance is low.
  • DAY: show and analyze the unmodified camera frame.
  • LOW LIGHT: apply CLAHE to the LAB lightness channel before preview and inference.

The VISION screen now combines software and hardware profiles. DAY restores the camera’s conservative auto-exposure profile. LOW LIGHT keeps the camera in its supported auto-exposure mode, applies modest gain/gamma/backlight adjustment, and then applies CLAHE. The validated camera advertises absolute exposure but reports that control inactive and rejects writes while streaming, so ORC does not force it. AUTO uses scene luminance with hysteresis: it enters low-light below a mean luma of 65 and returns to day above 85, avoiding rapid mode flapping near the threshold.

The VISION status panel reports the measured scene luminance and the active hardware profile so profile transitions can be evaluated directly on the VM before road testing.

Persistent object tracking

VISION passes YOLO detections through an ObjectTrackerIf abstraction backed initially by ByteTrack. Detection remains independent of tracking: YOLO reports frame-local observations while the tracker assigns persistent identities over time.

The VISION overlay renders tracked objects as LABEL #ID confidence age, and the status panel reports active track count and oldest-track age. YOLO uses a lower candidate threshold in VISION so ByteTrack can use weaker detections to maintain an existing identity through brief confidence drops.

Tracking state is owned by the perception worker and reset whenever VISION is started, so IDs are session-local rather than permanent object identities. Session generations prevent a frame or inference completion queued by a hidden or closed session from updating the current screen.

Supervision 0.28 deprecated its bundled ByteTrack class in favor of ByteTrackTracker from the separate trackers distribution. ORC narrowly suppresses that specific warning while retaining the current tested adapter. Moving to the replacement requires an explicit dependency and adapter migration before Supervision 0.31 removes the bundled class.