I'm Victor,
ML & Robotics Engineer

Overview

This demo shows Image-Based Visual Servoing (IBVS) controlling a quadrotor to track a target using only camera feedback. The drone tilts to move, demonstrating underactuated dynamics.

IBVS Controller

The controller minimizes feature error in image space:

L is the interaction matrix relating feature motion to camera velocity. L⁺ is its pseudo-inverse.

Interaction Matrix

For each image point (x, y) at depth Z:

Cyan: translation terms, Yellow: rotation terms

Differential Flatness Control

Direct computation of thrust & attitude from desired acceleration:

Pinhole Camera

Projects 3D world points to 2D image coordinates:

f: focal length, (c_x, c_y): principal point

Parameters

Overview

A LLaMA-style transformer with ternary-quantized weights running entirely in your browser. Inspired by BitNet b1.58, linear layer weights are constrained to {-1, 0, +1} and packed at 2 bits per weight, achieving ~8× compression relative to FP16 (excluding per-channel scale overhead).

This implementation targets inference efficiency under browser constraints, prioritizing memory bandwidth reduction over raw arithmetic throughput.

Architecture

Modern transformer with LLaMA-style components:

The architecture is conventional; the novelty lies in the weight representation and execution strategy rather than the model design itself.

Weight Encoding

2-bit ternary packing (4 weights per byte) with per-channel FP16 scales:

Output: y = scale ⊙ (W_ternary · x). Weights remain packed end-to-end: on disk, in memory, and in GPU buffers.

On-Demand Decoding

Weights are never pre-expanded to INT8 or FP16. Decoding occurs per-weight inside the matrix multiply using branch-free arithmetic:

Avoids control-flow divergence. Memory traffic stays proportional to packed size.

WebGPU Execution

Inference runs entirely on GPU via compute shaders:

Weights remain resident in GPU buffers. No per-token weight transfers between CPU and GPU.

CPU Fallback

Browsers without WebGPU use a JavaScript fallback with the same packed format:

Single-threaded JS. Significantly slower than WebGPU but functionally equivalent.

Why This Is Efficient

Performance gains come from reduced data movement, not from eliminating multiplications.

Model Details

Data Flow

No expanded weight representations at any stage:

Overview

This demo implements Model Predictive Control (MPC) for autonomous drone racing using Sequential Quadratic Programming (SQP). The controller predicts 500ms into the future and optimizes thrust and angular rate commands in real-time.

Coordinate system: Y-up (Three.js convention)

1. System Model

State Vector (14-dimensional):

Input Vector (4-dimensional):

2. Continuous Dynamics

Position: ṗ = v

Velocity (with drag):

Quaternion kinematics:

Actuator dynamics (first-order):

3. MPC Formulation

Optimal control problem solved at each timestep:

u^ref from feedforward (hover thrust + trajectory acceleration)

Subject to:

4. MPC Configuration

Input Bounds:

5. Sequential Quadratic Programming

Linearize discrete dynamics at operating point (x̄_k, ū_k):

Condensed QP formulation eliminates states:

6. QP Solver

Projected Gradient with Nesterov Acceleration:

Algorithm per iteration:

Step size α = 1/λ_max via Gershgorin circles. QP dim: 40×40.

7. Control Pipeline

8. Potential Improvements

Controls

Code References

Overview

This demo shows real-time monocular depth estimation using MiDaS v2.1 running entirely in the browser via ONNX Runtime Web. The model predicts relative depth from a single RGB image, with optional 3D point cloud visualization.

Note: Depth values are relative (ordinal), not metric distances.

Model Architecture

MiDaS v2.1 small uses an EfficientNet-Lite backbone:

3D Point Cloud

Back-projects depth to 3D using approximate camera intrinsics:

Uses turbo colormap for depth visualization and background removal via depth thresholding.

Browser Deployment

ONNX Runtime Web with multi-threaded WASM:

Limitations

What one robot knows

Not the map. Not anyone's plan. Two 5×5 grids:

The goal is almost never inside the window, so it is projected onto the rim: the robot is told a direction, not a position.

The network

A is the graph you see drawn: an edge to each of the four nearest robots within 4 cells. Weights are shared by every robot and the fleet size appears nowhere in them — which is why 10 robots' worth of training runs 200.

How it was trained

Step 3 is the interesting one. Cloning only ever shows the policy states the expert visits; the states it actually gets itself into are exactly the ones it has no data for.

Why sampling, not argmax

Shared weights mean two robots facing symmetric situations make the same choice. They collide, both revert, and repeat — forever. Of 200 failed episodes, 98% were one joint state repeating. No horizon escapes a fixed point; only noise does.

Uniform random scores zero, so the policy is doing the work — the noise only breaks the tie. Too much of it and the policy’s own preference is drowned out. Drag the slider to either end and both failure modes are one gesture apart.

What communication is worth

Nothing, until suddenly everything. A reactive policy plus a tie-break really is enough at small scale — set the fleet to 20 and the two panels are the same demo twice. That is the finding, not a bug.

How far should the radio reach?

The obvious guess is “further is better”. It is wrong, and the slider lets you watch it be wrong. Measured at 100 robots:

Only the four nearest neighbours are kept, so a wider radio does not add links — it swaps close ones for distant ones. A message from a robot too far away to collide with you is worse than no message, and the aggregate drifts away from anything the network was trained on.

More hops do not help either, for the same reason: at 1.4 links per robot, K=3 already reaches 2.6 robots and K=5 would reach 3.1 — two thirds more parameters to hear half a robot more.