diff --git a/.gitignore b/.gitignore index aa836db..3602d5e 100644 --- a/.gitignore +++ b/.gitignore @@ -28,3 +28,7 @@ settings.json .venv/ venv/ env/ + +# Local TLS certs for the VR /vr page (mkcert-generated, per-machine, +# not secret in the cryptographic sense but no reason to ship them) +*.pem diff --git a/README.md b/README.md index 392e572..4efd197 100644 --- a/README.md +++ b/README.md @@ -56,8 +56,29 @@ openmuscle predict -m data/models/random_forest_*/model.pkl # Test without hardware openmuscle simulate --device-type flexgrid + +# Test the whole VR pipeline without hardware (pair with `openmuscle web`): +# synthetic flexgrid + synthetic Quest hand driven by the same finger curls +openmuscle simulate --device-type combo +``` + +### VR companion (Meta Quest 3) + +A WebXR client lets you use a Quest 3's hand tracking as ML ground truth (richer than the LASK5 4-piston labeler) and visualize the model's predictions live as a ghost hand overlaid on your real hand. + +```bash +# Windows one-click launcher: starts the server, sets up adb-reverse, +# opens Quest Browser to /vr automatically +pc/start-vr.bat # right arm (default) +pc/start-vr.bat left # left arm + +# Or manually: openmuscle web, then in Quest Browser go to +# http://localhost:8000/vr (via `adb reverse tcp:8000 tcp:8000`) +# https://:8000/vr (via `openmuscle web --ssl-certfile cert.pem --ssl-keyfile key.pem`) ``` +Full setup + per-session walkthrough: [`docs/vr-setup.md`](docs/vr-setup.md). + ### Firmware (ESP32-S3 + MicroPython) ```bash @@ -80,8 +101,9 @@ mpremote cp embedded/devices/flexgrid_v1/config/defaults.json :/config/ | Command | Description | |---------|-------------| | `openmuscle receive` | Live heatmap of sensor data (matplotlib) | -| `openmuscle web` | Browser UI: live heatmap, LASK5 piston bars, ML inference panel, recording, captures management — see [`pc/src/openmuscle/web/README.md`](pc/src/openmuscle/web/README.md) | +| `openmuscle web` | Browser UI: live heatmap, LASK5 piston bars, ML inference panel, recording, captures management. Also serves the VR companion at `/vr` (see [`docs/vr-setup.md`](docs/vr-setup.md)). Full docs: [`pc/src/openmuscle/web/README.md`](pc/src/openmuscle/web/README.md) | | `openmuscle web --model M.pkl --hand IP` | Same UI plus live inference, with optional UDP forwarding of predictions to an OpenHand device | +| `openmuscle web --ssl-certfile cert.pem --ssl-keyfile key.pem` | Same UI over HTTPS (required for the VR `/vr` page over LAN, since Quest Browser refuses WebXR hand-tracking on plain HTTP) | | `openmuscle record -o file.csv` | Record paired data to CSV | | `openmuscle train data.csv` | Train ML model (RandomForest) | | `openmuscle predict -m model.pkl` | Real-time inference (matplotlib) | @@ -91,9 +113,11 @@ mpremote cp embedded/devices/flexgrid_v1/config/defaults.json :/config/ ## Documentation - [Architecture Overview](docs/architecture.md) -- [Packet Protocol Spec](docs/protocol.md) +- [Packet Protocol Spec](docs/protocol.md) — includes the `quest_hand` device type - [Adding a New Device](docs/adding-a-device.md) - [CLI Usage Guide](docs/pc-cli.md) +- [VR Setup & Operation](docs/vr-setup.md) — mkcert + Quest cert install + per-session walkthrough +- [VR Testing Scenarios](docs/vr-testing-scenarios.md) — bring-up order, 2-minute smoke test, per-feature test runbook ## Adding a New Device @@ -114,6 +138,9 @@ See [docs/adding-a-device.md](docs/adding-a-device.md) for details. **Standalone firmware (promoted from this repo's `embedded/devices/`):** - [FlexGridV3-Firmware](https://github.com/Open-Muscle/FlexGridV3-Firmware) — current FlexGrid revision, with the sensor-scan techniques writeup +**AR / VR:** +- [OpenMuscle-AR](https://github.com/Open-Muscle/OpenMuscle-AR) — AR/VR companion. The current WebXR client lives here in `pc/src/openmuscle/web/static/vr/` (tight coupling to the FastAPI server), but the AR repo is the discoverability anchor and the future home for the planned native Quest APK / BLE-direct work. See its [ROADMAP](https://github.com/Open-Muscle/OpenMuscle-AR/blob/main/ROADMAP.md). + **Coordination / docs:** - [OpenMuscle-Hub](https://github.com/Open-Muscle/OpenMuscle-Hub) — central docs and roadmap diff --git a/docs/pc-cli.md b/docs/pc-cli.md index 558557e..dd3cfab 100644 --- a/docs/pc-cli.md +++ b/docs/pc-cli.md @@ -57,6 +57,16 @@ Send synthetic data for testing without hardware. openmuscle simulate --device-type flexgrid openmuscle simulate --device-type lask5 --target-ip 192.168.1.100 openmuscle simulate --replay data/raw/legacy/capture_45.txt + +# Synthetic WebXR hand: streams 25-joint frames to the running +# `openmuscle web` server's /ws/quest WebSocket (no headset needed). +openmuscle simulate --device-type quest_hand + +# Combo: the same latent finger-curl signal drives BOTH a flexgrid UDP +# device and the quest hand, so the capture is learnable and the whole +# record -> train -> predict pipeline can be exercised end to end with +# zero hardware. Use --web-port if the web server is not on 8000. +openmuscle simulate --device-type combo ``` ### `openmuscle models` diff --git a/docs/protocol.md b/docs/protocol.md index c500381..7231af3 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -56,10 +56,44 @@ All OpenMuscle devices communicate over **UDP port 3141** using **JSON-encoded U 4 piston sensor values. Joystick is optional. +### Quest hand tracking (`type: "quest_hand"`) + +Synthesized server-side from WebSocket frames sent by the WebXR client at `/vr` (browsers can't speak UDP). The payload represents one tracked hand sampled from `XRHand` each XRFrame: + +```json +{ + "values": [px,py,pz, rx,ry,rz,rw, ...] // flat, 7 floats per joint + "handedness": "left" | "right", + "joint_names": ["wrist", "thumb-metacarpal", ...], + "hands": { + "handedness": "left" | "right", + "joints": [ + {"name": "wrist", "pos": [x,y,z], "rot": [x,y,z,w], "radius": 0.02, "valid": true}, + ... + ] + } +} +``` + +- `values` follows the same convention as LASK5 — flat, in canonical joint × channel order — so the recorder and matcher pair `quest_hand` frames with FlexGrid frames identically to LASK5. The order is `[px, py, pz, rx, ry, rz, rw] × N joints`. +- **Fixed-length contract.** A well-behaved client MUST send all joints in canonical `joint_names` order every frame, so the `values` length and per-slot meaning are stable across frames. When an individual joint pose is momentarily unavailable, send a zero position + identity quaternion with `"valid": false` rather than omitting the joint — omitting it would shift every later joint into the wrong CSV column and silently misalign the labels. The server defends against violations by locking the label width at the first label packet and padding/truncating later rows to keep the CSV rectangular (it counts and logs any mismatch), but clients should not rely on that. +- `valid` (per joint, in `hands.joints`) marks whether the pose was actually tracked this frame. It is preserved in the JSONL sidecar so offline analysis can filter zero-filled joints; the flat `values` / CSV carry only the numbers. +- `joint_names` lists the joints in the same order they're flattened into `values`. The standard set is the W3C WebXR Hand Input spec (25 joints: wrist + 4 thumb + 5 each for index/middle/ring/pinky). +- `hands` is the structured per-joint form, kept in the JSONL sidecar for offline analysis. It's redundant with `values` + `joint_names` but easier to diff by eye. +- Empty payloads (the headset reports tracking-lost for the whole hand this frame) are dropped silently by the server — they'd otherwise produce zero-rows that mislead training. + +A per-capture `.labels.schema.json` sidecar is written on the first `quest_hand` packet of a recording. It maps `label_0..label_N` columns in the CSV back to `(joint, channel)`, so the wide label vector is self-describing. + +**Future-proofing note:** v1 captures one hand per recording (`handedness` is a single `"left"` or `"right"` string). A future `handedness: "both"` extension — payload carries both hands in `data.values` with the schema sidecar growing a parallel `joint_names_left` / `joint_names_right` (or a per-column `hand` field) — would be backward-compatible. Old consumers see a wider but still-flat values vector; new consumers can split it via the schema. Don't accidentally close that door in any future refactor of `_flatten_quest_joint` / `_write_labels_schema`. + ### Adding a New Device Type Define a new `type` string and document the `data` shape. The PC-side parser auto-discovers devices by their `type` field. +## Versioning policy + +Adding a new `type` string (e.g. `quest_hand`) is **non-breaking under v1.0** — existing parsers ignore unknown types, the schema envelope is unchanged, and devices that don't speak the new type are unaffected. Bump the `"v"` field to `"1.1"` (or beyond) only when changing the envelope itself (required-field set, semantics of `ts`/`id`, etc.). + ## Backward Compatibility The PC parser (`openmuscle.protocol.parser`) auto-detects three formats: diff --git a/docs/vr-setup.md b/docs/vr-setup.md new file mode 100644 index 0000000..677f084 --- /dev/null +++ b/docs/vr-setup.md @@ -0,0 +1,171 @@ +# OpenMuscle VR — setup & operation + +Wear FlexGrid + a Meta Quest 3S together. The headset's hand tracking becomes the ground-truth label source for training a muscle-signal → finger-pose model. The desktop OpenMuscle UI is unchanged; this guide just adds the VR companion. + +Companion code: [`pc/src/openmuscle/web/static/vr/`](../pc/src/openmuscle/web/static/vr/) (client) and [`pc/src/openmuscle/web/app.py`](../pc/src/openmuscle/web/app.py) (server endpoints). The Quest is integrated as a synthetic device with `device_type: "quest_hand"` — see [`protocol.md`](protocol.md) for the wire format. + +## Why HTTPS is non-optional + +Quest Browser refuses to grant WebXR hand-tracking outside a secure context. You have two ways to get one: + +| Mode | Pros | Cons | +|---|---|---| +| **HTTPS over LAN** via mkcert | untethered, real use case | one-time cert install on the headset | +| **`adb reverse` over USB** (localhost) | zero certs, fast for debugging | cable on the headset distorts natural arm motion | + +Day-to-day capture work should use HTTPS. The USB fallback is for proving the loop end-to-end before you've finished cert setup. + +## One-time setup + +### 1. mkcert on the PC + +```powershell +# Windows (choco). On macOS: `brew install mkcert`. On Linux: distro package. +choco install mkcert + +mkcert -install # installs the mkcert root CA into Windows +mkcert -cert-file vr-cert.pem ` + -key-file vr-key.pem ` + 192.168.1.42 localhost # <-- replace with your PC's LAN IP +``` + +The two PEM files land in your current directory. Keep them next to the captures dir, or anywhere the `openmuscle web` command can read them. + +### 2. Install the mkcert root CA on the Quest + +**Heads up:** this is the painful step. Meta's Horizon OS consumer shell **hides** the standard Android "Settings → Security → Install a certificate" path. The Settings panel you see when tapping the gear icon in VR is Meta's Horizon Settings, not AOSP Settings — different surface, no cert-install button. You have to launch the AOSP Settings activity via ADB. + +Procedure validated on Quest 3S, Horizon OS as of May 2026: + +```powershell +# (a) Push the root CA from your PC to the headset's Downloads folder +adb push "$env:LOCALAPPDATA\mkcert\rootCA.pem" /sdcard/Download/openmuscle-vr-rootCA.pem + +# (b) Launch the AOSP Security Dashboard activity directly via ADB. +# The $ MUST be backslash-escaped through PowerShell single-quotes so it +# survives BOTH the host shell AND the Android shell. Without the escape, +# $ gets eaten as a variable expansion and the activity name truncates to +# just '.Settings' which is not exported (Permission Denial). +adb shell 'am start -n com.android.settings/.Settings\$SecurityDashboardActivity' +``` + +On the Quest after running those: + +1. **The 2D Settings panel may not appear immediately in your view.** Horizon shell aggressively suppresses unfamiliar 2D activities. **Press the Meta button** on your right controller, look for "Settings" (or similar) in the universal-menu app switcher, and click to bring it forward. +2. In the AOSP Security dashboard: **Trusted credentials** → **Encryption** section → **Install a certificate** → **CA certificate**. +3. **"Your data won't be private"** warning → tap **Install anyway** (your own root CA, the warning is generic Android boilerplate). +4. **If no screen-lock PIN is set**, Quest refuses and prompts you to set one. Set a PIN, then re-do step 2. +5. **File picker gotcha:** defaults to "Recent" view which is **empty** on a fresh install. Tap the **hamburger menu (≡)** at the top of the picker → navigate to **Internal Storage → Download** → tap **openmuscle-vr-rootCA.pem**. +6. Success: "CA certificate installed" or similar. + +You only do this once per headset. The CA stays trusted across Horizon OS updates. + +**Fallback if you can't find the AOSP Settings panel in the app switcher:** open the headset's **Files** app, navigate to **Download/openmuscle-vr-rootCA.pem**, tap it. Some Horizon versions trigger the install dialog from the file-open intent directly. + +**Activities that work on Horizon OS** (useful for future Quest dev when standard menus are hidden): +- `com.android.settings/.Settings$SecurityDashboardActivity` — security menu (the one you just used) +- `com.android.settings/.Settings$TrustedCredentialsSettingsActivity` — view installed certs (for verifying yours is there) +- `com.android.settings/.security.CredentialStorage` — direct install activity, but Horizon often suppresses its panel +- `com.android.settings/.Settings$NetworkDashboardActivity` +- `com.android.settings/.Settings$DevelopmentSettingsDashboardActivity` + +**Activities that DON'T exist on Horizon** (don't try — Error type 3): +- `com.android.settings/.Settings$SecuritySettingsActivity` (older Android name, removed) +- `com.android.settings/.Settings$EncryptionAndCredentialActivity` (renamed) + +**Non-exported, can't launch via ADB** (Permission Denial from null uid): +- `com.android.settings/.Settings` (the bare main activity) + +### 3. Start the server with HTTPS + +```powershell +cd D:\path\to\OpenMuscle-Software\pc +pip install -e . # first time only + +openmuscle web --ssl-certfile vr-cert.pem ` + --ssl-keyfile vr-key.pem +``` + +You'll see: + +``` +OpenMuscle web UI: https://localhost:8000 +Listening for devices on UDP 3141 +TLS: cert=vr-cert.pem key=vr-key.pem +WebXR URL for the Quest: https://:8000/vr +``` + +## Per-session flow + +1. **Boot FlexGrid** — it joins your Wi-Fi and starts streaming UDP to port 3141. Confirm in the desktop UI's Devices panel. +2. **Open `/vr` in Quest Browser** at `https://:8000/vr`. (The desktop UI lives at `/`; the VR companion at `/vr`.) +3. The landing page runs three preflight checks. All three should be green checkmarks: + - HTTPS / secure context + - WebXR + immersive-vr supported + - Server reachable +4. **Pick the FlexGrid arm** in the dropdown (`right` is default). Only the joints of this hand get pushed to `/ws/quest`. The other hand stays free for the record button. +5. Tap **Enter VR**. Grant hand-tracking when Quest asks. +6. A floating panel ~70cm in front shows the live FlexGrid heatmap. Header strip above it shows per-device Hz when idle, or `REC · N rows · match X%` when recording. +7. Hold your captured hand up; you'll see small blue spheres on each tracked joint. +8. **Start recording**: + - **Off-hand button**: tap the gray sphere below the heatmap with your other hand's index finger. It turns red. + - **Pinch gesture**: pinch index + thumb on the captured hand and hold for ~1 second. A yellow ring on the index tip fills during the hold; release triggers the toggle. +9. Do your gesture set. For the first FlexGrid V3 training run we focus on **index / middle / ring / pinky finger curls** — the thumb is driven by intrinsic hand muscles the FlexGrid can't see, so it's excluded. +10. **Stop recording** the same way. +11. Take the headset off. The capture is in `data/raw/merged/`: + + ``` + capture_.csv # paired (sensor + 182 joint floats) + capture_.sensor.jsonl # raw FlexGrid packets + capture_.label.jsonl # raw Quest packets + capture_.labels.schema.json # column -> (joint, channel) map + capture_.meta.json # auto.label_source="quest_hand" + your tags + ``` + +12. **Train**: in the desktop UI's Captures panel, tick your new capture(s) and hit **Train selected**. Or from CLI: `openmuscle train data/raw/merged/capture_.csv`. + +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| "Enter VR" button stays disabled | Page opened in the in-headset Library panel, not Quest Browser | Open Quest Browser app explicitly; navigate to the URL | +| "INSECURE (http:)" in the HTTPS checkmark | Reached the page over plain HTTP | Either switch to `https://` (LAN + mkcert) or use `adb reverse` + `http://localhost:8000/vr` | +| Cert install on Quest fails silently | Quest doesn't have a screen-lock PIN | Settings → Security → Lock screen → set a PIN, then re-try CA install | +| Heatmap stays "waiting for FlexGrid…" inside VR | FlexGrid not streaming, or wrong UDP port | Open the desktop UI; verify the device is in the Devices panel and packets > 0 | +| `match_rate` is very low (< 30%) in recording header | Quest WS connection dropping, or sensor and label streams have wildly different rates | Check WS reconnect logs in the desktop UI's Log panel; try increasing `window_ms` via the recording start body if needed (default for Quest is 175 ms) | +| Joints visible but no `quest-` device in the Devices panel | `/ws/quest` failed to connect (e.g. CORS, cert mismatch) | Check the Quest Browser dev-tools (chrome://inspect on a connected PC) for WS errors | + +## Architecture, in one diagram + +``` + ┌──────────────────────┐ + │ Meta Quest 3S │ + │ │ + ┌───────────┐ │ Quest Browser /vr │ + │ FlexGrid │ │ │ │ + │ (ESP32- │ UDP │ │ WebXR XRHand │ + │ S3 WiFi) │ ─────────→ │ │ sample joints │ + └───────────┘ :3141 │ ▼ │ + ▲ │ WS /ws/quest │ + │ │ │ │ + │ │ │ WS /ws/live │ + │ │ │ ◀──── heatmap │ + │ │ │ │ + │ └────┼─────────────────┘ + │ │ + │ ▼ + │ ┌──────────────────────────┐ + │ │ PC: openmuscle web │ + │ │ │ + │ UDP │ UDPListener ─→ ┐ │ + └────────────┤ │ │ + │ ingest_quest ─→ AppState ─→ paired CSV + │ (synthesizes a matcher + JSONL + + │ quest_hand recorder labels.schema.json + │ packet) + meta.json + │ │ + │ /ws/live ◀── snapshot ───────────┘ + └──────────────────────────┘ +``` + +The whole integration trick is that `ingest_quest_packet` synthesizes an `OpenMusclePacket` of type `quest_hand` in-process, then hands it to `_handle_packet` — the **same** function the UDP listener feeds into. From the recorder's view the Quest is just another device. Zero special-casing downstream. diff --git a/docs/vr-testing-scenarios.md b/docs/vr-testing-scenarios.md new file mode 100644 index 0000000..3e0c0ae --- /dev/null +++ b/docs/vr-testing-scenarios.md @@ -0,0 +1,202 @@ +# VR Testing Scenarios + Smoke-Test Checklist + +A practical runbook for validating the OpenMuscle VR app against the hardware. Use this when you sit down to test: it gives you a fixed bring-up order, a 2-minute smoke test to confirm the basics, and per-feature scenarios with expected results so you can validate each piece fast instead of rediscovering how it all works. + +Companion docs: +- [`vr-setup.md`](vr-setup.md) -- one-time setup (mkcert, Quest cert install, launchers) +- [OpenMuscle-AR wiki](https://github.com/Open-Muscle/OpenMuscle-AR/wiki) -- troubleshooting + community successes log +- [`pc/src/openmuscle/web/README.md`](../pc/src/openmuscle/web/README.md) -- architecture of the web/VR surface + +--- + +## Bring-up order (do this every session, in order) + +Order matters. Each step depends on the previous one being live. + +1. **Power the FlexGrid.** Wait for its OLED to show it joined Wi-Fi. It streams UDP to the PC on port 3141. +2. **Start the server on the PC.** Pick the path that matches how you'll test: + - **USB tethered** (fast iteration, no certs): `cd pc` then `start-vr.bat` (VR) or `start-vr.bat right ar` (AR). It auto-runs `adb reverse` and opens Quest Browser for you. + - **Cordless HTTPS** (walk around, real field capture): `cd pc` then `start-vr-https.bat`. It prints the headset URL. Requires the one-time mkcert + Quest cert install. +3. **Confirm the device shows up.** On the PC, open the desktop UI (`http://localhost:8000/` or `https://:8000/`). The Devices panel should list `flexgrid-v3-*` with a non-zero Hz and a live heatmap. +4. **Put the headset on. Set both controllers down** on a flat surface. Quest auto-switches to hand tracking within 2 to 5 seconds. (It stays in controller mode as long as it can see them being held.) +5. **Open `/vr` in Quest Browser** (the USB launcher does this for you; for HTTPS, type the printed URL). Confirm the three preflight checkmarks are green, then tap Enter VR / Start AR. + +If any step fails, the [troubleshooting wiki page](https://github.com/Open-Muscle/OpenMuscle-AR/wiki/Troubleshooting) lists the common causes per symptom. + +--- + +## Zero-hardware smoke test (no headset, no FlexGrid) + +Before any headset session, you can exercise the entire PC-side quest +pipeline with the simulator. Two terminals: + +```bash +cd pc +openmuscle web # terminal 1 +openmuscle simulate --device-type combo # terminal 2 +``` + +Then in the desktop UI (`http://localhost:8000/`): + +1. Devices panel lists `flexgrid-sim` (~12 Hz, live heatmap) and + `quest-sim` (~25 Hz, `quest_hand`). +2. The Live stage swaps the piston comparator for the 3D hand viewer; a + green articulated hand auto-rotates and its fingers visibly curl. +3. REC pairs frames (match rate goes green), STOP writes a CSV with 64 + sensor columns and 175 label columns plus the labels-schema sidecar. +4. Train on that capture; the combo simulator's sensor values are + derived from the same latent finger curls as the hand, so the model + should reach a clearly positive held-out R^2. Load the model and the + amber predicted hand tracks the green real hand in the viewer. + +If all of that works, a real-headset failure is in the headset/network +layer, not the PC pipeline. + +--- + +## 2-minute smoke test (no FlexGrid required) + +The fastest "is anything obviously broken" check. Only needs the headset + PC + server. Do this first whenever you pick the project back up after code changes. + +| # | Action | Expected | +|---|---|---| +| 1 | Open `/vr` (or `/vr?mode=ar`), look at the preflight page | Three green checkmarks: secure context, WebXR supported, server reachable | +| 2 | Tap Enter VR / Start AR | Scene loads. VR = dark background; AR = passthrough (you see your room) | +| 3 | Look forward | Heatmap panel (says "waiting for FlexGrid"), 3x2 menu below it, status strip below that | +| 4 | Hold up your captured arm (the FlexGrid arm) | Blue joint spheres on that hand | +| 5 | Hold up your other hand | Green joint spheres + a blue ray pointer from the index | +| 6 | Point the ray at a menu button | Ray turns amber, button highlights | +| 7 | Pinch (off-hand index + thumb) while hovering RECENTER | Brief white flash on the button; panels re-anchor in front of you | +| 8 | Point at a panel's drag handle (small cube, top-left corner) and pinch-hold | Handle turns green; panel follows your hand. Release: panel drops + faces you | +| 9 | Tap REC | Yellow SYNC slate flashes ~2.5s; menu collapses to one big red STOP button; header shows a live ms clock + filename + a capture-quality dot (gray while warming up) | +| 10 | Tap STOP | Menu returns to the full 3x2 grid; status strip shows "saved: ..." | + +If all 10 pass, the core interaction loop is healthy. If FlexGrid is connected, the heatmap in step 3 animates and the REC in step 9 actually pairs sensor frames. + +### Reading capture quality (live, while recording) + +The recording header is also a live data-quality gauge, so you can tell mid-capture whether the data is good without taking the headset off: + +- **Quality dot color** (tracks the sensor-to-label match rate): + - **gray** = warming up (fewer than ~10 sensor frames seen yet) + - **green** = match rate >= 70% (good) + - **amber** = 40 to 70% (marginal -- check that both FlexGrid and Quest are streaming) + - **red** = below 40% (poor pairing -- something is off; see Troubleshooting) +- **`NN%`** in the header is the live match rate. +- **`JOINTS DROPPING`** appears (and the dot goes amber) when your hand is partially out of the cameras' view and the headset is sending incomplete joint frames. Those frames get zero-filled joint columns -- keep your capture hand more fully in view if you see this a lot. The final count is also reported in the stop result (`label_width_mismatch`). + +--- + +## Full test scenarios + +Each scenario is independent. Run the ones relevant to what you changed. + +### Scenario A: Gesture-training capture (VR mode) + +The deliberate-gesture workflow for building training data. + +1. Launch VR mode: `start-vr.bat right` (or left). Enter VR. +2. Tap **SESSION** (turns red). All captures now group under one session. +3. Tap **REC**. Curl your index finger open/close ~10 to 15 times. Tap **STOP**. +4. Repeat step 3 for middle, ring, pinky. (Thumb is intentionally excluded -- FlexGrid can't see it.) +5. Tap **SESSION** again to end it. +6. Tap **TRAIN**. Wait ~10 to 30s. + +**Expected:** +- Each REC shows a rising row count + match rate in the header while recording. +- Status strip after each STOP: `saved: capture_.csv (N rows, match X%)`. Match rate should be well above 0 if both FlexGrid and Quest are streaming. +- TRAIN status: `trained: R2= ... model loaded ... predict ON`. +- On the PC, `pc/data/raw/merged/` has the capture CSVs + `.labels.schema.json` + `.meta.json` sidecars, and `pc/data/raw/sessions/` has the session JSON. + +**Watch for:** match rate near 0 (Quest WebSocket not connected, or hands out of camera FOV during capture). Very low R2 is normal for a first small dataset; record more and retrain. + +### Scenario B: Field capture (AR passthrough mode) + +The natural-activity workflow. You see your real workspace and do real tasks. + +1. Launch AR mode: `start-vr-https.bat`, open `https://:8000/vr?mode=ar` (cordless), or `start-vr.bat right ar` (tethered for a quick check). Start AR. +2. Confirm passthrough: you should see your real room behind the panels. (If it is black, the AR session did not grant passthrough -- check the browser console for the blend-mode log line.) +3. **Start the Quest's built-in screen recorder** (press and hold the Meta button, or use the camera shortcut) so the video is captured for later labeling. +4. Drag the panels out of your central view using the handles, so they sit in your periphery and do not block your work. +5. Tap **REC**. Do a real task with your hands in view (type, stir, assemble something). Tap **STOP** when done. +6. Stop the screen recorder. + +**Expected:** +- SYNC slate flashes at REC with `capture_.csv` + a Unix-ms timestamp. This is your video-to-CSV pairing anchor. +- Header strip shows a live `HH:MM:SS.mmm` clock + filename throughout the recording, visible in the screen recording. +- Captured-arm joints stream to the CSV whenever the hand is in the headset cameras' FOV; frames where the hand is out of view are dropped (gaps in the data, by design). + +**Watch for:** panels snapping back to center after you moved them -- if that happens on reload, the localStorage layout persistence regressed (see Scenario E). + +### Scenario C: Pause / boundary redraw recovery + +This is the v1.11 fix. It used to glitch the app. + +1. Enter either mode. Tap **REC** to start a recording. +2. Trigger a pause: press the **Meta button** (opens the universal menu) OR walk past your Guardian boundary so Quest prompts a redraw. +3. Observe the status strip, then close the menu / finish the redraw. + +**Expected:** +- During pause: status strip reads `session paused (visible-blurred) -- boundary redraw or system UI`. No joint frames are sent during this time (the recorder timeline stays clean). +- On resume: status reads `session resumed after Xs -- re-anchoring UI`, and the panels pop back in front of your current head position. +- If you were mid-drag when the pause hit, the panel is released cleanly (not stuck following a frozen controller). +- Tap **STOP**: the recording closes normally with all the valid pre/post-pause frames. + +### Scenario D: Collapse-to-STOP while recording + +1. Enter either mode. Confirm the full 3x2 menu is visible. +2. Tap **REC**. + +**Expected:** the menu collapses to a single large red STOP button at the same location. The status strip stays visible. Pointing + pinching STOP ends the recording and the full menu returns. + +**Why it matters:** during real field capture you do not want six action buttons cluttering your view -- just an obvious way to stop. + +### Scenario E: Panel layout persistence + +This is the v1.12 feature. + +1. Enter a mode. Drag the heatmap cluster and the menu to custom positions. +2. Exit VR (or just reload the Quest Browser tab) and re-enter the same mode. + +**Expected:** the panels come back where you left them. VR and AR remember their layouts independently (per-mode localStorage key). + +3. Now tap **RECENTER**. + +**Expected:** panels reset to defaults AND the saved layout is cleared, so a subsequent reload also starts from defaults until you drag again. + +### Scenario F: Predict + ghost hand + +1. Complete Scenario A at least once so a model is trained and loaded. +2. Confirm **PREDICT** is red (TRAIN auto-enables it; otherwise tap it). +3. Hold your captured hand up in view and curl fingers. + +**Expected:** +- The REAL-vs-PRED bar panel appears (four fingers, green = real curl, amber = predicted). +- An amber ghost hand overlays your real hand, anchored to your real wrist position + orientation, showing what the model predicts. +- As the model improves with more training data, the ghost should track your real hand more closely. + +--- + +## What "good" looks like (acceptance bar) + +Before calling the app "ready to demo or hand to a tester," these should all be true: + +- [ ] 2-minute smoke test passes end to end +- [ ] Scenario A produces a trainable CSV with match rate > 50% +- [ ] Scenario B passthrough is visible and the sync slate + header clock are legible in the screen recording +- [ ] Scenario C recovers gracefully from a Meta-button pause with no app glitch +- [ ] Scenario D collapses and restores the menu correctly +- [ ] Scenario E remembers panel layout across a reload and RECENTER clears it +- [ ] Scenario F shows the ghost hand tracking once a model is trained + +--- + +## Reporting a problem + +If a scenario fails, capture: +- Which scenario + step +- Mode (vr / ar) and transport (USB / HTTPS) +- Quest model + Horizon OS version +- Browser console output (via `chrome://inspect/#devices` with the Quest USB-connected) and the `openmuscle web` server console + +File it as an issue on [OpenMuscle-AR](https://github.com/Open-Muscle/OpenMuscle-AR/issues) (AR/VR-specific) or [OpenMuscle-Software](https://github.com/Open-Muscle/OpenMuscle-Software/issues) (server/pipeline). Add a [Troubleshooting wiki](https://github.com/Open-Muscle/OpenMuscle-AR/wiki/Troubleshooting) entry once you resolve it so the next person benefits. diff --git a/embedded/_upload_lask5.py b/embedded/_upload_lask5.py new file mode 100644 index 0000000..ef50e35 --- /dev/null +++ b/embedded/_upload_lask5.py @@ -0,0 +1,168 @@ +"""Direct pyserial uploader for the LASK5 device. + +mpremote keeps hitting "could not enter raw REPL" because the running labeler +floods the UART. This script speaks the MicroPython raw-REPL protocol over +COM24 directly: Ctrl-C to interrupt, Ctrl-A to enter raw REPL, write each +file by sending Python code that creates it, Ctrl-B to exit, then reset. + +Run from the embedded/ folder so the relative paths to lib/ and devices/ +resolve correctly. +""" +import os +import sys +import time +import serial + +PORT = "COM24" +BAUD = 115200 + +# (local_path, remote_path) tuples to push +PUSH_LIST = [ + ("lib/om_network.py", "/lib/om_network.py"), + ("lib/om_subscribers.py", "/lib/om_subscribers.py"), + ("lib/om_discovery.py", "/lib/om_discovery.py"), + ("lib/om_commands.py", "/lib/om_commands.py"), + ("devices/lask5_v2/labeler.py", "/labeler.py"), +] + + +def drain(s, ms=200): + """Read everything currently buffered on the serial port.""" + end = time.time() + ms / 1000.0 + buf = bytearray() + while time.time() < end: + b = s.read(s.in_waiting or 1) + if b: + buf.extend(b) + else: + time.sleep(0.01) + return bytes(buf) + + +def enter_raw_repl(s): + """Halt any running script and enter raw REPL mode.""" + # Two Ctrl-C: halt any running script + s.write(b"\r\x03\x03") + time.sleep(0.3) + drain(s, 300) + # Ctrl-A: enter raw REPL + s.write(b"\r\x01") + time.sleep(0.5) + resp = drain(s, 500) + if b"raw REPL" not in resp: + # Try one more time after a fresh interrupt + s.write(b"\x03\x03\r\x01") + time.sleep(0.7) + resp = drain(s, 700) + if b"raw REPL" not in resp: + raise RuntimeError("Could not enter raw REPL. Got: {!r}".format(resp[:200])) + print(" raw REPL active") + + +def exit_raw_repl(s): + s.write(b"\r\x02") + time.sleep(0.2) + drain(s, 200) + + +def exec_raw(s, code): + """Execute Python code in raw REPL; return (stdout, stderr) bytes.""" + s.write(code.encode("utf-8")) + s.write(b"\x04") # EOT: run + # Expect b"OK" + output + 0x04 + error + 0x04> + buf = bytearray() + end = time.time() + 8.0 + while time.time() < end: + b = s.read(s.in_waiting or 1) + if b: + buf.extend(b) + if buf.count(b"\x04") >= 2 and buf.endswith(b"\x04>"): + break + else: + time.sleep(0.01) + raw = bytes(buf) + if raw[:2] != b"OK": + raise RuntimeError("raw REPL did not OK: {!r}".format(raw[:80])) + # Strip leading OK, trailing > + body = raw[2:] + if body.endswith(b"\x04>"): + body = body[:-2] + parts = body.split(b"\x04") + out = parts[0] if len(parts) >= 1 else b"" + err = parts[1] if len(parts) >= 2 else b"" + return out, err + + +def ensure_dir(s, remote_path): + """If remote_path is in a subdirectory, mkdir the parent.""" + parts = remote_path.lstrip("/").split("/") + if len(parts) <= 1: + return + d = "/" + "/".join(parts[:-1]) + code = ( + "import uos\n" + "try:\n" + " uos.stat('{0}')\n" + "except OSError:\n" + " uos.mkdir('{0}')\n" + "print('dir ok')\n" + ).format(d) + out, err = exec_raw(s, code) + if err.strip(): + raise RuntimeError("mkdir failed for {}: {!r}".format(d, err[:120])) + + +def push_file(s, local_path, remote_path): + """Upload local file to remote path via raw REPL.""" + with open(local_path, "rb") as f: + data = f.read() + print(" pushing {} -> {} ({} bytes)".format(local_path, remote_path, len(data))) + ensure_dir(s, remote_path) + # Open file on device for writing + open_code = "f=open('{}','wb')\n".format(remote_path) + out, err = exec_raw(s, open_code) + if err.strip(): + raise RuntimeError("open() failed: {!r}".format(err[:120])) + # Write in chunks. Each chunk goes through raw REPL exec; small enough + # to fit comfortably under the stdin buffer (typ. ~1KB safe). + chunk = 256 + for i in range(0, len(data), chunk): + piece = data[i:i+chunk] + # Encode as repr() of bytes so embedded backslashes / newlines work + code = "f.write({!r})\n".format(piece) + out, err = exec_raw(s, code) + if err.strip(): + raise RuntimeError("write() failed at offset {}: {!r}".format(i, err[:120])) + out, err = exec_raw(s, "f.close()\nimport uos\nprint(uos.stat('{}')[6])\n".format(remote_path)) + if err.strip(): + raise RuntimeError("close()/stat() failed: {!r}".format(err[:120])) + written = int(out.strip()) + if written != len(data): + raise RuntimeError("size mismatch: wrote {} but stat says {}".format(len(data), written)) + print(" confirmed {} bytes on device".format(written)) + + +def hard_reset(s): + print(" triggering machine.reset()...") + s.write(b"\r\x01") # re-enter raw REPL just in case + time.sleep(0.3) + drain(s, 200) + s.write(b"import machine\nmachine.reset()\n\x04") + time.sleep(0.5) + + +def main(): + s = serial.Serial(PORT, BAUD, timeout=2) + print("Opened {}".format(PORT)) + try: + enter_raw_repl(s) + for local, remote in PUSH_LIST: + push_file(s, local, remote) + hard_reset(s) + print("Done; device is rebooting") + finally: + s.close() + + +if __name__ == "__main__": + main() diff --git a/embedded/devices/lask5_v2/labeler.py b/embedded/devices/lask5_v2/labeler.py index a5d0f60..9524218 100644 --- a/embedded/devices/lask5_v2/labeler.py +++ b/embedded/devices/lask5_v2/labeler.py @@ -14,10 +14,14 @@ # ADCs are used for the on-OLED bar chart and the Calibrate flow. import time +import machine from machine import Pin, ADC import uasyncio as asyncio import om_logger as log from om_device import BaseDevice +from om_subscribers import Subscribers +from om_discovery import Discovery +from om_commands import CommandServer, build_standard_handlers from sensor_pistons import SensorPistons from display_lask5 import draw_taskbar, play_splash @@ -50,6 +54,15 @@ class LASK5(BaseDevice): # "espnow" broadcasts directly to a paired bracelet/robot hand. # Persisted in settings.json -- can be toggled live from the menu. "stream_mode": "udp", + # Discovery + subscribe protocol additions (see spec section 4-6). + # No udp_target_ip is used for sensor frames in this mode; sources + # unicast to subscribed hubs only. ESP-NOW path is unchanged. + "fw_version": "v3.0.0", + "cmd_port": 8002, + "udp_sensor_port": 3141, # broadcast beacon port; hubs also pick this for default + "announce_interval_s": 1, + "max_subscribers": 4, + "heartbeat_timeout_s": 5, } # Menu UX: a single "Start" button both OPENS the menu (from live) and @@ -101,6 +114,51 @@ def __init__(self): # active, _display_loop yields and lets them write the OLED directly. self._display_owner = "live" # "live" | "menu" | "modal" + # Discovery + subscribe state (V3 protocol): + # subscribers - hubs currently receiving label frames + # _streaming - false halts UDP send_loop without removing subscribers + # _reboot_requested - flag polled by reboot_watcher + # Discovery + CommandServer instances are built in run() after the + # network is up and the Wi-Fi STA has an IP. + self.subscribers = Subscribers( + max_subscribers=self.settings.get("max_subscribers", 4), + heartbeat_timeout_s=self.settings.get("heartbeat_timeout_s", 5), + ) + self._streaming = True + self._reboot_requested = False + + # ----- DeviceState interface for om_commands.build_standard_handlers ----- + # The standard handler factory needs: subscribers, device_id, device_type, + # fw_version, caps, extra_info, start_stream(), stop_stream(), request_reboot() + + @property + def fw_version(self): + return self.settings.get("fw_version", "v3.0.0") + + @property + def device_type(self): + return self.DEVICE_TYPE + + @property + def caps(self): + return ["label", "status", "cmd"] + + @property + def extra_info(self): + return {"pistons": 4, "joystick": True, "espnow_paired": True} + + def start_stream(self): + self._streaming = True + log.info("Streaming started") + + def stop_stream(self): + self._streaming = False + log.info("Streaming stopped") + + def request_reboot(self): + self._reboot_requested = True + log.info("Reboot requested") + # ----- helpers ----- def blink(self, count=2, on_ms=300, off_ms=200): @@ -126,36 +184,94 @@ async def run(self): self.blink(2) self.network.init_espnow() - # Streaming + display + menu all run concurrently. UDP and ESPNow - # broadcast in parallel so both the web UI and any peer bracelet - # receive ground-truth labels with no menu gating. + # New in V3 firmware: discovery + subscribe protocol on Wi-Fi. + # ESP-NOW path to OpenHand is untouched. The cmd server binds before + # we spawn any other task so hubs hitting the device immediately + # after boot can connect. + self._discovery = Discovery( + self.settings, self.subscribers, self.network.sta, + device_type=self.DEVICE_TYPE, + services={ + "label": self.settings.get("udp_sensor_port", 3141), + "cmd": self.settings.get("cmd_port", 8002), + }, + caps=self.caps, + extra_fields={"pistons": 4, "joystick": True}, + beacon_port=self.settings.get("udp_sensor_port", 3141), + announce_interval_s=self.settings.get("announce_interval_s", 1), + fw_version=self.fw_version, + device_id=self.device_id, + ) + self._cmd_server = CommandServer( + port=self.settings.get("cmd_port", 8002), + handlers=build_standard_handlers(self), + ) + await self._cmd_server.start() + + # Streaming + display + menu + discovery + subscriber pruning + reboot + # watcher all run concurrently. UDP fans out to subscribers and ESPNow + # broadcasts to the paired bracelet in parallel; both kept independent + # so a Wi-Fi flap does not disturb the ESP-NOW link. asyncio.create_task(self._send_loop()) asyncio.create_task(self._espnow_loop()) asyncio.create_task(self._display_loop()) asyncio.create_task(self._menu_loop()) + asyncio.create_task(self._discovery.announce_loop()) + asyncio.create_task(self._subscriber_prune_loop()) + asyncio.create_task(self._reboot_watcher()) while True: await asyncio.sleep(1) + async def _subscriber_prune_loop(self): + """Drop subscribers whose last heartbeat aged past the configured + timeout. Self-cleans the list when a hub disappears.""" + while True: + try: + dropped = self.subscribers.prune_stale() + if dropped: + log.info("Pruned {} stale subscriber(s); remaining={}".format( + dropped, self.subscribers.count())) + except Exception as e: + log.warn("subscriber_prune_loop failed: {}".format(e)) + await asyncio.sleep(1) + + async def _reboot_watcher(self): + """Polls the reboot flag set by a `reboot` command verb.""" + while True: + if self._reboot_requested: + log.info("Soft-resetting in 500 ms...") + await asyncio.sleep_ms(500) + machine.reset() + await asyncio.sleep_ms(200) + # ----- streaming ----- def _stream_mode(self): return self.settings.get("stream_mode", "udp") async def _send_loop(self): - """UDP send loop. Only transmits when stream_mode == 'udp'. Keeps - running (and reading sensors) even when idle so the on-OLED taskbar - stays responsive immediately after a mode toggle.""" + """UDP send loop. Only transmits when stream_mode == 'udp' AND at + least one hub is subscribed. Keeps reading sensors regardless so + the on-OLED taskbar stays responsive immediately after a mode toggle. + + V3 protocol change: replaces send_udp() to a hardcoded udp_target_ip + with send_udp_to_subscribers() that fans out to every entry in the + live subscriber list. The list is populated by hubs talking to the + cmd server (see om_commands).""" interval = 1.0 / self.settings.get("sample_rate_hz", 25) while True: - if self._stream_mode() == "udp": + if self._stream_mode() == "udp" and self._streaming: data = self.sensor.read() # {"values": [0..1, ...]} data["joystick"] = { "x": self.joystick_x.read(), "y": self.joystick_y.read(), } packet = self.make_packet(data) - await self.network.send_udp(packet) + # send_udp_to_subscribers no-ops when the list is empty, so + # the work above is wasted when idle; we still do it so the + # very first frame after a subscribe is ready to go. + await self.network.send_udp_to_subscribers(packet, self.subscribers) await asyncio.sleep(interval) async def _espnow_loop(self): diff --git a/embedded/devices/openhand_v2/boot.py b/embedded/devices/openhand_v2/boot.py index ee3f002..6aeb1db 100644 --- a/embedded/devices/openhand_v2/boot.py +++ b/embedded/devices/openhand_v2/boot.py @@ -348,22 +348,38 @@ def espnow_listen(): pass frint('ESP-NOW ready') frint('SEL=exit') - while True: - # Non-blocking recv with 100ms timeout so buttons stay responsive - msg = e.recv(100) - if msg and msg[1]: - print(f'msg[1]:{msg[1]}') - result = parse_packet(msg[1]) - if result: - device_id, values = result - apply_packet(device_id, values) - if select_btn.value() == 0: - time.sleep(0.2) # debounce - break - e.active(False) + try: + while True: + # Non-blocking recv with 100ms timeout so buttons stay responsive + msg = e.recv(100) + if msg and msg[1]: + print('msg[1]:' + str(msg[1])) + result = parse_packet(msg[1]) + if result: + device_id, values = result + apply_packet(device_id, values) + if select_btn.value() == 0: + time.sleep(0.2) # debounce + break + except KeyboardInterrupt: + frint('ESPNow -> REPL') + try: release_all() + except Exception: pass + raise + finally: + try: e.active(False) + except Exception: pass frint('ESP-NOW stopped') +# Idle-sleep threshold: after this many seconds with no incoming UDP packet +# we release all servos. They stop humming/holding torque, and the OLED +# shows "Sleeping..." so the operator knows the hand is intentionally +# limp (not crashed). The next packet wakes it up — set_finger() implicitly +# re-energizes the servo, so wake is just "go back to applying packets". +UDP_IDLE_SLEEP_S = 30 + + def udp_listen(): # Reuse the boot-time STA connection. If for some reason we don't have # one yet (auto_mode=menu user navigated here manually before boot @@ -382,20 +398,62 @@ def udp_listen(): s.setblocking(False) frint('UDP :' + str(UDP_PORT)) frint('SEL=exit') - while True: - try: - data, addr = s.recvfrom(256) - if data: - result = parse_packet(data) - if result: - device_id, values = result - apply_packet(device_id, values) - except OSError: - pass # no data available (non-blocking) - if select_btn.value() == 0: - time.sleep(0.2) # debounce - break - s.close() + + last_packet_t = time.time() # grace period: 30s before first sleep + is_asleep = False + + # KeyboardInterrupt-aware loop: a Ctrl-C from the REPL (mpremote, Thonny, + # serial monitor) now exits this loop cleanly so the operator can land + # in the REPL and edit settings / firmware without having to power-cycle. + try: + while True: + got_packet = False + try: + data, addr = s.recvfrom(256) + if data: + got_packet = True + result = parse_packet(data) + if result: + device_id, values = result + apply_packet(device_id, values) + except OSError: + pass # no data available (non-blocking) + + if got_packet: + last_packet_t = time.time() + if is_asleep: + # Wake transition: brief OLED note, then resume listening. + # apply_packet() above already drove this packet's values, + # which implicitly re-energized the servos -- no extra work. + is_asleep = False + frint('Awake') + else: + # No packet this iteration -- check idle timeout. + if not is_asleep and (time.time() - last_packet_t) > UDP_IDLE_SLEEP_S: + release_all() # stop all 16 channels (servos go limp) + is_asleep = True + frint('Sleeping') + + if select_btn.value() == 0: + time.sleep(0.2) # debounce + break + + # Small yield so the non-blocking recv loop doesn't peg the CPU + # (also lets the REPL Ctrl-C have a chance to interrupt). + if is_asleep: + time.sleep(0.05) # asleep: be lazy, 20 Hz wake-check + else: + time.sleep(0.002) # awake: tight loop, ~500 Hz + except KeyboardInterrupt: + frint('UDP -> REPL') + try: release_all() + except Exception: pass + # Re-raise so the outer boot-sequence handler can also drop cleanly. + raise + finally: + try: s.close() + except Exception: pass + # Keep STA connected on exit so ESPNow still has its channel locked. frint('UDP stopped') @@ -485,20 +543,26 @@ def run_menu(): selected = 0 n_items = len(MENU_ITEMS) draw_menu(selected) - while True: - if up_btn.value() == 0: - selected = (selected - 1) % n_items - draw_menu(selected) - time.sleep(0.25) - if down_btn.value() == 0: - selected = (selected + 1) % n_items - draw_menu(selected) - time.sleep(0.25) - if start_btn.value() == 0: - time.sleep(0.2) # debounce - MENU_ACTIONS[selected]() - draw_menu(selected) - time.sleep(0.05) + try: + while True: + if up_btn.value() == 0: + selected = (selected - 1) % n_items + draw_menu(selected) + time.sleep(0.25) + if down_btn.value() == 0: + selected = (selected + 1) % n_items + draw_menu(selected) + time.sleep(0.25) + if start_btn.value() == 0: + time.sleep(0.2) # debounce + MENU_ACTIONS[selected]() + draw_menu(selected) + time.sleep(0.05) + except KeyboardInterrupt: + frint('Menu -> REPL') + try: release_all() + except Exception: pass + raise # ============================================================================= @@ -518,19 +582,33 @@ def run_menu(): # Hold Select at boot to force the menu, regardless of auto_mode -- escape # hatch in case auto-mode is set to something that crashes or hangs. -if select_btn.value() == 0: - frint('Auto skipped') - time.sleep(0.5) -else: - mode = settings.get('auto_mode', 'menu') - if mode == 'espnow': - frint('Auto: ESP-NOW') - time.sleep(0.4) - espnow_listen() # returns when Select pressed - elif mode == 'udp': - frint('Auto: UDP') - time.sleep(0.4) - udp_listen() # returns when Select pressed - -blink(2) -run_menu() +# +# The whole runtime is wrapped in try/except KeyboardInterrupt so a Ctrl-C +# from the REPL (mpremote, Thonny, serial monitor) drops back to an +# interactive prompt cleanly. Without this, the device locks out the host +# while a listen loop is running and we can't edit firmware/settings +# without a physical power-cycle. Servos are released on the way out so +# fingers don't hold torque while you're hacking. +try: + if select_btn.value() == 0: + frint('Auto skipped') + time.sleep(0.5) + else: + mode = settings.get('auto_mode', 'menu') + if mode == 'espnow': + frint('Auto: ESP-NOW') + time.sleep(0.4) + espnow_listen() # returns when Select pressed + elif mode == 'udp': + frint('Auto: UDP') + time.sleep(0.4) + udp_listen() # returns when Select pressed + + blink(2) + run_menu() +except KeyboardInterrupt: + try: release_all() + except Exception: pass + frint('REPL') + print('\nCtrl-C received -- dropped to REPL.') + print('To resume: run_menu() or exec(open("boot.py").read())') diff --git a/embedded/lib/om_commands.py b/embedded/lib/om_commands.py new file mode 100644 index 0000000..a9e0cbb --- /dev/null +++ b/embedded/lib/om_commands.py @@ -0,0 +1,167 @@ +# om_commands.py - TCP command server for source-role devices. +# +# Hubs connect to this device's cmd_port and send JSON command messages, +# one per line. The server replies with a JSON ack per line. See spec +# section 6.1 for the standard verb taxonomy. +# +# This module provides: +# - CommandServer: the asyncio TCP server + dispatch loop +# - build_standard_handlers(state): factory for the verbs that every +# source device needs (subscribe / unsubscribe / heartbeat / get_info / +# start_stream / stop_stream / reboot). Device-specific verbs can be +# added on top by the caller. +# +# `state` is an object the device provides. Required attributes: +# subscribers om_subscribers.Subscribers +# device_id str +# device_type str ("flexgrid"|"lask5"|...) +# fw_version str +# caps list[str] +# extra_info dict (merged into get_info responses) +# start_stream() method +# stop_stream() method +# request_reboot() method + +import uasyncio as asyncio +import ujson +import om_logger as log + + +_OK = "ok" +_ERR = "error" + + +class CommandServer: + def __init__(self, port, handlers): + """ + port: TCP port to bind + handlers: dict mapping verb -> async fn(data, peer) -> dict + Each handler returns the ack's `data` field. + Raising any exception turns the ack into an error. + """ + self.port = port + self.handlers = handlers + self._server = None + + async def start(self): + """Bind and start accepting connections in the background.""" + self._server = await asyncio.start_server(self._handle_client, "0.0.0.0", self.port) + log.info("Command server listening on TCP {}".format(self.port)) + + async def _handle_client(self, reader, writer): + peer = writer.get_extra_info("peername") or ("?", 0) + log.info("Command client connected: {}".format(peer)) + try: + while True: + line = await reader.readline() + if not line: + break + line = line.strip() + if not line: + continue + ack = await self._handle_one(line, peer) + writer.write(ujson.dumps(ack).encode("utf-8") + b"\n") + await writer.drain() + except Exception as e: + log.warn("Command client {} errored: {}".format(peer, e)) + finally: + try: + writer.close() + await writer.wait_closed() + except Exception: + pass + log.info("Command client disconnected: {}".format(peer)) + + async def _handle_one(self, raw_line, peer): + try: + pkt = ujson.loads(raw_line.decode("utf-8")) + except Exception as e: + return {"v": "1.0", "type": "ack", "status": _ERR, + "msg_id": None, "data": {"message": "invalid_json: {}".format(e)}} + + msg_id = pkt.get("msg_id") + data = pkt.get("data") or {} + verb = data.get("verb") + if not verb: + return {"v": "1.0", "type": "ack", "status": _ERR, + "msg_id": msg_id, "data": {"message": "missing verb in data"}} + handler = self.handlers.get(verb) + if handler is None: + return {"v": "1.0", "type": "ack", "status": _ERR, + "msg_id": msg_id, "data": {"message": "unknown_verb: " + str(verb)}} + + try: + ack_data = await handler(data, peer) + except Exception as e: + log.warn("Command verb={} from {} raised: {}".format(verb, peer, e)) + return {"v": "1.0", "type": "ack", "status": _ERR, + "msg_id": msg_id, "data": {"verb": verb, "message": str(e)}} + return {"v": "1.0", "type": "ack", "status": _OK, + "msg_id": msg_id, "data": dict({"verb": verb}, **(ack_data or {}))} + + +def build_standard_handlers(state): + """Return a dict of handlers for the verbs every source device needs. + Devices can extend this dict with their own verbs before passing it + to CommandServer.""" + + async def subscribe(data, peer): + host = data.get("host") or peer[0] + port = int(data["port"]) + transport = data.get("transport", "wifi") + hub_id = data.get("hub_id") + ok = state.subscribers.add(host, port, transport, hub_id) + return { + "accepted": ok, + "subscriber_count": state.subscribers.count(), + "max_subscribers": state.subscribers.max_subscribers, + } + + async def unsubscribe(data, peer): + host = data.get("host") or peer[0] + port = int(data["port"]) + transport = data.get("transport", "wifi") + removed = state.subscribers.remove(host, port, transport) + return {"removed": removed, "subscriber_count": state.subscribers.count()} + + async def heartbeat(data, peer): + host = data.get("host") or peer[0] + port = int(data["port"]) + transport = data.get("transport", "wifi") + refreshed = state.subscribers.heartbeat(host, port, transport) + return {"refreshed": refreshed} + + async def get_info(data, peer): + info = { + "id": state.device_id, + "dev": state.device_type, + "fw": state.fw_version, + "caps": list(state.caps), + "subscribers": state.subscribers.snapshot(), + } + # Device-specific extras (matrix dims for FlexGrid, piston count for LASK5, etc.) + for k, v in getattr(state, "extra_info", {}).items(): + info[k] = v + return info + + async def start_stream(data, peer): + state.start_stream() + return {"streaming": True} + + async def stop_stream(data, peer): + state.stop_stream() + return {"streaming": False} + + async def reboot(data, peer): + state.request_reboot() + return {"rebooting": True} + + return { + "subscribe": subscribe, + "unsubscribe": unsubscribe, + "heartbeat": heartbeat, + "get_info": get_info, + "start_stream": start_stream, + "stop_stream": stop_stream, + "reboot": reboot, + } diff --git a/embedded/lib/om_discovery.py b/embedded/lib/om_discovery.py new file mode 100644 index 0000000..bef4e7d --- /dev/null +++ b/embedded/lib/om_discovery.py @@ -0,0 +1,127 @@ +# om_discovery.py - Wi-Fi discovery for source-role devices. +# +# Two parallel mechanisms per spec section 4: +# 1. mDNS hostname registration (best-effort; uses sta.config(hostname=...) +# which some MicroPython builds publish as an mDNS A record). The +# `_openmuscle._udp` service type with TXT records is not registered +# via this path; the UDP broadcast beacon below covers the gap. +# 2. UDP broadcast beacon to 255.255.255.255:beacon_port (default 3141) +# carrying the announce JSON. Reliable; works across consumer routers +# that disable mDNS / multicast. +# +# Cadence: ~1 Hz while no hub is subscribed. Once subscribers.has_any() +# returns True the beacon stops to keep the channel quiet, and resumes +# the moment the subscriber list empties. + +import uasyncio as asyncio +import socket +import ujson +import time +import om_logger as log + + +class Discovery: + def __init__(self, settings, subscribers, sta, + device_type, services, caps, + extra_fields=None, + beacon_port=3141, announce_interval_s=1, + fw_version=None, + device_id=None, + mdns_service="_openmuscle._udp"): + """ + settings: om_settings.Settings instance + subscribers: om_subscribers.Subscribers instance + sta: network.WLAN(STA_IF), used for hostname + IP lookup + device_type: "flexgrid" | "lask5" | "openhand" | ... + services: dict mapping capability name -> port + e.g. {"label": 3141, "cmd": 8002} + caps: list of capability strings, e.g. ["label","status","cmd"] + extra_fields: optional dict merged into the announce payload + (e.g. {"matrix": [15,4]} for FlexGrid) + beacon_port: UDP port to broadcast announces to (default 3141) + device_id: optional override; defaults to settings["device_id"] + fw_version: optional override; defaults to settings.get("fw_version","unknown") + mdns_service: mDNS service type string for documentation + """ + self.settings = settings + self.subscribers = subscribers + self.sta = sta + self.device_type = device_type + self.services = services + self.caps = caps + self.extra_fields = extra_fields or {} + self.beacon_port = beacon_port + self.announce_interval_s = announce_interval_s + self.device_id = device_id or settings.get("device_id") + self.fw_version = fw_version or settings.get("fw_version", "unknown") + self.mdns_service = mdns_service + + self._beacon_sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + self._beacon_sock.setblocking(False) + try: + self._beacon_sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) + except Exception as e: + log.warn("SO_BROADCAST setsockopt failed: {}".format(e)) + + self._mdns_registered = False + + def _announce_payload(self): + """Build the announce JSON. The IP is NOT in the payload; consumers + take it from the broadcast packet source or the mDNS A record.""" + pkt = { + "v": "1.0", + "type": "announce", + "id": self.device_id, + "role": "source", + "dev": self.device_type, + "fw": self.fw_version, + "transports": ["wifi"], # phase 4 appends "ble" + "caps": list(self.caps), + "services": dict(self.services), + "ts": time.ticks_ms(), + } + for k, v in self.extra_fields.items(): + pkt[k] = v + return pkt + + def register_mdns(self): + """Best-effort mDNS hostname registration. Some MicroPython builds + publish the STA hostname as an mDNS A record (so `.local` + resolves); others do not. Either way the broadcast beacon is the + reliable discovery path.""" + if self._mdns_registered: + return + try: + self.sta.config(hostname=self.device_id) + except Exception as e: + log.warn("mDNS hostname set failed (beacon will cover it): {}".format(e)) + self._mdns_registered = True + log.info("mDNS hostname best-effort registered as {}".format(self.device_id)) + + async def announce_loop(self): + """Periodic broadcast beacon. Goes quiet once at least one hub is + subscribed; resumes when the subscriber list empties.""" + try: + self.register_mdns() + except Exception: + pass + + while True: + try: + if not self.subscribers.has_any(): + self._send_beacon() + except Exception as e: + log.warn("announce_loop iter failed: {}".format(e)) + await asyncio.sleep(self.announce_interval_s) + + def _send_beacon(self): + try: + payload = ujson.dumps(self._announce_payload()).encode("utf-8") + self._beacon_sock.sendto(payload, ("255.255.255.255", self.beacon_port)) + except OSError as e: + errno = getattr(e, "errno", None) or (e.args[0] if e.args else None) + # ENOMEM (12) and EAGAIN (11) are non-fatal; next tick retries. + if errno not in (11, 12): + log.warn("Beacon send failed: errno={} {}".format(errno, e)) + except Exception as e: + log.warn("Beacon send unexpected error: {}".format(e)) diff --git a/embedded/lib/om_network.py b/embedded/lib/om_network.py index 2a4e747..175b5c6 100644 --- a/embedded/lib/om_network.py +++ b/embedded/lib/om_network.py @@ -72,6 +72,32 @@ def send_udp_sync(self, payload_bytes): except Exception as e: log.error("UDP send error: " + str(e)) + async def send_udp_to_subscribers(self, payload_bytes, subscribers): + """Fan out one UDP payload to every Wi-Fi subscriber in the list. + + New in the discovery + subscribe protocol (spec section 5.1). The + existing send_udp() to a hardcoded target is kept for backward + compat with FlexGrid V1 and any device that has not yet been + migrated to discovery. + + subscribers: om_subscribers.Subscribers instance. + """ + if not self._sock: + return + targets = subscribers.wifi_targets() + if not targets: + return + for host, port in targets: + try: + self._sock.sendto(payload_bytes, (host, port)) + except OSError as e: + errno = getattr(e, "errno", None) or (e.args[0] if e.args else None) + # ENOMEM (12) and EAGAIN (11) are non-fatal; next iter retries. + if errno not in (11, 12): + log.warn("UDP send to {}:{} failed errno={} {}".format(host, port, errno, e)) + except Exception as e: + log.warn("UDP send to {}:{} unexpected: {}".format(host, port, e)) + # --- ESPNOW (optional, for devices that need P2P communication) --- def init_espnow(self): diff --git a/embedded/lib/om_subscribers.py b/embedded/lib/om_subscribers.py new file mode 100644 index 0000000..1deef2c --- /dev/null +++ b/embedded/lib/om_subscribers.py @@ -0,0 +1,113 @@ +# om_subscribers.py - Subscriber list for source-role devices. +# +# A hub (phone, PC) discovers this device via om_discovery, opens the command +# channel exposed by om_commands, and sends a `subscribe` message with its +# (host, port, transport). This module stores the resulting subscriber list, +# fields heartbeats, and prunes entries whose last heartbeat aged past the +# configured timeout (default 5 s per spec section 5.1). +# +# Devices fan out frames to wifi_targets() each scan. + +import time +import om_logger as log + + +class Subscribers: + def __init__(self, max_subscribers=4, heartbeat_timeout_s=5): + self.max_subscribers = max_subscribers + self.heartbeat_timeout_ms = int(heartbeat_timeout_s * 1000) + # Each entry: dict {host, port, transport, hub_id, last_heartbeat_ms} + self._entries = [] + + def add(self, host, port, transport="wifi", hub_id=None): + """Add or refresh a subscriber. Returns True if accepted, False if the + list is full. Refreshing an existing (host, port, transport) tuple + does not consume a new slot.""" + now = time.ticks_ms() + for e in self._entries: + if e["host"] == host and e["port"] == port and e["transport"] == transport: + e["last_heartbeat_ms"] = now + if hub_id: + e["hub_id"] = hub_id + log.info("Subscriber refreshed: {} {}:{} (hub_id={})".format( + transport, host, port, hub_id)) + return True + if len(self._entries) >= self.max_subscribers: + log.warn("Subscriber list full ({}/{}); rejecting {}:{}".format( + len(self._entries), self.max_subscribers, host, port)) + return False + self._entries.append({ + "host": host, + "port": port, + "transport": transport, + "hub_id": hub_id, + "last_heartbeat_ms": now, + }) + log.info("Subscriber added: {} {}:{} (hub_id={}, {}/{})".format( + transport, host, port, hub_id, len(self._entries), self.max_subscribers)) + return True + + def remove(self, host, port, transport="wifi"): + """Explicit unsubscribe. Returns True if found+removed, False if not.""" + for i, e in enumerate(self._entries): + if e["host"] == host and e["port"] == port and e["transport"] == transport: + self._entries.pop(i) + log.info("Subscriber removed: {} {}:{}".format(transport, host, port)) + return True + return False + + def heartbeat(self, host, port, transport="wifi"): + """Refresh a subscriber's heartbeat. If the (host,port,transport) is + not currently subscribed this is a no-op + warning.""" + now = time.ticks_ms() + for e in self._entries: + if e["host"] == host and e["port"] == port and e["transport"] == transport: + e["last_heartbeat_ms"] = now + return True + log.warn("Heartbeat from unknown subscriber {}:{} ({})".format(host, port, transport)) + return False + + def prune_stale(self): + """Drop subscribers whose last heartbeat is older than the timeout. + Returns the number of entries dropped.""" + now = time.ticks_ms() + kept = [] + dropped = 0 + for e in self._entries: + age_ms = time.ticks_diff(now, e["last_heartbeat_ms"]) + if age_ms > self.heartbeat_timeout_ms: + log.info("Subscriber stale, dropping: {}:{} age={}ms".format( + e["host"], e["port"], age_ms)) + dropped += 1 + else: + kept.append(e) + self._entries = kept + return dropped + + def wifi_targets(self): + """List of (host, port) for every active Wi-Fi subscriber.""" + return [(e["host"], e["port"]) for e in self._entries if e["transport"] == "wifi"] + + def ble_targets(self): + """List of hub identifiers for every active BLE subscriber (phase 4).""" + return [e.get("hub_id") for e in self._entries if e["transport"] == "ble"] + + def count(self): + return len(self._entries) + + def has_any(self): + return len(self._entries) > 0 + + def snapshot(self): + """List of subscriber summaries for diagnostics / UI / get_info.""" + now = time.ticks_ms() + return [ + { + "host": e["host"], + "port": e["port"], + "transport": e["transport"], + "hub_id": e.get("hub_id"), + "age_ms": time.ticks_diff(now, e["last_heartbeat_ms"]), + } + for e in self._entries + ] diff --git a/pc/bridge_subscriber.py b/pc/bridge_subscriber.py new file mode 100644 index 0000000..a3ac750 --- /dev/null +++ b/pc/bridge_subscriber.py @@ -0,0 +1,207 @@ +"""Bridge subscriber: keeps the PC subscribed to V4-protocol sources. + +The PC app (`openmuscle web`) does not yet implement the V4 discovery + +subscribe protocol natively (that work lives in the OpenMuscle-Lab-Discovery +workspace). This helper bridges the gap: for each device IP you pass on the +command line, it opens a TCP command channel, sends `subscribe` with the PC's +own address, and sends a `heartbeat` once per second. Devices then unicast +sensor / label frames to the PC's UDP 3141 listener where openmuscle web +picks them up. + +This script does NOT listen on UDP 3141 itself, so it never conflicts with +openmuscle web for the port. Both can run side by side. + +Usage: + python bridge_subscriber.py 10.0.0.23 + python bridge_subscriber.py 10.0.0.23 10.0.0.232 10.0.0.192:8002 + +Default cmd ports: 8001 for FlexGrid, 8002 for LASK5. Append `:` to a +target to override. Auto-detect of device type happens by trying 8001 first +then 8002. + +Press Ctrl-C to unsubscribe cleanly and exit. +""" + +import argparse +import json +import socket +import sys +import threading +import time + + +PROTO_PORT = 3141 # UDP port openmuscle web listens on for sensor frames +HEARTBEAT_S = 1.0 +HUB_ID = "bridge-subscriber" + + +def get_default_pc_ip(test_target=("10.0.0.1", 80)): + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + try: + s.connect(test_target) + return s.getsockname()[0] + except Exception: + try: + return socket.gethostbyname(socket.gethostname()) + except Exception: + return "127.0.0.1" + finally: + s.close() + + +def try_subscribe(ip, port, pc_ip): + """Open TCP, send subscribe, return (tcp_socket, ack_dict) on success or + (None, None) on failure.""" + try: + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.settimeout(3.0) + s.connect((ip, port)) + msg = json.dumps({ + "v": "1.0", "type": "cmd", "msg_id": 1, + "data": { + "verb": "subscribe", + "host": pc_ip, "port": PROTO_PORT, + "transport": "wifi", "hub_id": HUB_ID, + }, + }) + "\n" + s.send(msg.encode()) + ack = s.recv(2048).decode().strip() + return s, json.loads(ack) + except Exception: + try: + s.close() + except Exception: + pass + return None, None + + +class Keeper: + """One subscription + heartbeat keeper running on its own daemon thread.""" + + def __init__(self, ip, port, pc_ip): + self.ip = ip + self.port = port + self.pc_ip = pc_ip + self.tcp = None + self.alive = True + self.thread = None + + def start(self): + self.tcp, ack = try_subscribe(self.ip, self.port, self.pc_ip) + if not self.tcp: + return False + verb_data = (ack or {}).get("data") or {} + accepted = verb_data.get("accepted") + print(" [{}:{}] subscribe ack: accepted={}, count={}/{}".format( + self.ip, self.port, accepted, + verb_data.get("subscriber_count"), + verb_data.get("max_subscribers"), + )) + if not accepted: + print(" (device rejected subscribe; subscriber list may be full)") + self.tcp.close() + return False + self.thread = threading.Thread(target=self._loop, daemon=True) + self.thread.start() + return True + + def _loop(self): + msg_id = 100 + while self.alive: + time.sleep(HEARTBEAT_S) + if not self.alive: + break + msg = json.dumps({ + "v": "1.0", "type": "cmd", "msg_id": msg_id, + "data": { + "verb": "heartbeat", + "host": self.pc_ip, "port": PROTO_PORT, + "transport": "wifi", + }, + }) + "\n" + try: + self.tcp.send(msg.encode()) + self.tcp.recv(2048) + msg_id += 1 + except Exception as e: + print(" [{}:{}] heartbeat failed: {}; restarting subscribe".format( + self.ip, self.port, e)) + try: + self.tcp.close() + except Exception: + pass + # Try to re-subscribe + self.tcp, _ack = try_subscribe(self.ip, self.port, self.pc_ip) + if not self.tcp: + print(" [{}:{}] re-subscribe failed; giving up".format(self.ip, self.port)) + self.alive = False + break + msg_id = 100 + + def stop(self): + self.alive = False + if not self.tcp: + return + try: + self.tcp.send((json.dumps({ + "v": "1.0", "type": "cmd", "msg_id": 999, + "data": {"verb": "unsubscribe", "host": self.pc_ip, "port": PROTO_PORT}, + }) + "\n").encode()) + self.tcp.recv(2048) + except Exception: + pass + try: + self.tcp.close() + except Exception: + pass + + +def parse_target(target_str): + """`'1.2.3.4'` -> [(1.2.3.4, 8001), (1.2.3.4, 8002)] for autodetect. + `'1.2.3.4:8002'` -> [(1.2.3.4, 8002)] exact.""" + if ":" in target_str: + ip, port = target_str.split(":", 1) + return [(ip, int(port))] + return [(target_str, 8001), (target_str, 8002)] + + +def main(): + p = argparse.ArgumentParser(description="Keep PC subscribed to V4 sources") + p.add_argument("targets", nargs="+", help="Device IPs (default ports 8001/8002 tried)") + p.add_argument("--pc-ip", default=None, help="Override PC's own LAN IP") + args = p.parse_args() + + pc_ip = args.pc_ip or get_default_pc_ip() + print("PC IP: {} (sensor/label frames will arrive at this address, UDP {})".format( + pc_ip, PROTO_PORT)) + + keepers = [] + for t in args.targets: + for ip, port in parse_target(t): + print("trying {}:{}".format(ip, port)) + k = Keeper(ip, port, pc_ip) + if k.start(): + keepers.append(k) + break + else: + print(" no responsive cmd server at {}".format(t)) + + if not keepers: + print("\nNo subscriptions established. Check device IPs + cmd ports.") + sys.exit(1) + + print("\n{} subscription(s) active. Heartbeating every {}s.".format(len(keepers), HEARTBEAT_S)) + print("Open http://localhost:8000 (openmuscle web) to see the live data.") + print("Press Ctrl-C to unsubscribe + exit.") + try: + while True: + time.sleep(1) + except KeyboardInterrupt: + print("\nshutting down. unsubscribing...") + for k in keepers: + k.stop() + print("done.") + + +if __name__ == "__main__": + main() diff --git a/pc/src/openmuscle/cli.py b/pc/src/openmuscle/cli.py index 7967c6d..1eaf4ba 100644 --- a/pc/src/openmuscle/cli.py +++ b/pc/src/openmuscle/cli.py @@ -43,11 +43,25 @@ def receive(port, save_dir): "10.0.0.55:3145). When set together with --model, inference " "outputs are forwarded over UDP as 'PC,a1,a2,a3,a4,a5'. " "Default port if omitted: 3145.") -def web(host, port, udp_port, captures_dir, model_path, hand): +@click.option("--ssl-certfile", default=None, type=click.Path(exists=True, dir_okay=False), + help="Path to TLS cert (PEM). Required for WebXR over LAN -- Quest " + "Browser refuses hand-tracking on plain HTTP. Generate with " + "`mkcert ` and install the mkcert root CA " + "on the headset (Settings -> Security -> Install a certificate).") +@click.option("--ssl-keyfile", default=None, type=click.Path(exists=True, dir_okay=False), + help="Path to TLS private key (PEM). Pair with --ssl-certfile.") +def web(host, port, udp_port, captures_dir, model_path, hand, ssl_certfile, ssl_keyfile): """Launch the browser-based UI with live heatmap, recording, and captures.""" from openmuscle.web.app import serve - click.echo(f"OpenMuscle web UI: http://{host if host != '0.0.0.0' else 'localhost'}:{port}") + # mkcert produces a cert AND key; we need both or neither. + if bool(ssl_certfile) != bool(ssl_keyfile): + raise click.BadParameter("--ssl-certfile and --ssl-keyfile must be used together") + scheme = "https" if ssl_certfile else "http" + click.echo(f"OpenMuscle web UI: {scheme}://{host if host != '0.0.0.0' else 'localhost'}:{port}") click.echo(f"Listening for devices on UDP {udp_port}") + if ssl_certfile: + click.echo(f"TLS: cert={ssl_certfile} key={ssl_keyfile}") + click.echo(f"WebXR URL for the Quest: {scheme}://:{port}/vr") if model_path: click.echo(f"Inference model: {model_path}") hand_target = None @@ -62,7 +76,8 @@ def web(host, port, udp_port, captures_dir, model_path, hand): hand_target = (hand, 3145) click.echo(f"Forwarding inference to robot hand at {hand_target[0]}:{hand_target[1]}") serve(host=host, port=port, udp_port=udp_port, captures_dir=captures_dir, - model_path=model_path, hand_target=hand_target) + model_path=model_path, hand_target=hand_target, + ssl_certfile=ssl_certfile, ssl_keyfile=ssl_keyfile) @main.command() @@ -175,15 +190,21 @@ def predict(model, port): @main.command() @click.option("--port", default=3141, help="UDP port to send to") @click.option("--device-type", default="flexgrid", - help="Device type to simulate (flexgrid, lask5)") + type=click.Choice(["flexgrid", "lask5", "quest_hand", "combo"]), + help="Device type to simulate. quest_hand streams synthetic " + "WebXR hand frames to the web server's /ws/quest; combo " + "adds a correlated flexgrid UDP device so record/train/" + "predict works end to end without hardware.") @click.option("--replay", type=click.Path(exists=True), help="Replay a capture file instead of generating synthetic data") @click.option("--target-ip", default="127.0.0.1", help="Target IP address") -def simulate(port, device_type, replay, target_ip): - """Send synthetic or replayed sensor data over UDP.""" +@click.option("--web-port", default=8000, + help="openmuscle web HTTP port (quest_hand/combo only)") +def simulate(port, device_type, replay, target_ip, web_port): + """Send synthetic or replayed sensor data.""" from openmuscle.simulate.transmitter import run_simulator run_simulator(port=port, device_type=device_type, - replay_file=replay, target_ip=target_ip) + replay_file=replay, target_ip=target_ip, web_port=web_port) @main.command() diff --git a/pc/src/openmuscle/data/converter.py b/pc/src/openmuscle/data/converter.py index f159d2b..c31c39c 100644 --- a/pc/src/openmuscle/data/converter.py +++ b/pc/src/openmuscle/data/converter.py @@ -1,16 +1,28 @@ -"""Convert legacy capture formats to standard CSV.""" +"""Convert legacy capture formats to standard CSV with packet matching.""" import ast import csv import os +from collections import deque from pathlib import Path +# Legacy device IDs for SensorBand (3 banks × 4 sensors = 12 total) +SENSOR_IDS = ['OM-SB-V1-C.0', 'OM-SB-V1-C.1', 'OM-SB-V1-C.2'] +LABEL_ID = 'OM-LASK5' + + def convert_legacy_capture(input_path: str, output_path: str) -> int: - """Convert a legacy capture_*.txt file to standard CSV format. + """Convert a legacy capture_*.txt file to matched CSV format. + + Matches sensor packets from 3 SensorBand banks with LASK5 label packets + by temporal proximity, producing rows with 12 sensor values + 4 labels. Legacy format: one Python dict repr per line, e.g.: - {'id': 'OM-LASK5', 'ticks': 164587, 'time': (2000, 1, 1, ...), 'data': [-30, -35, -30, -37]} + {'id': 'OM-SB-V1-C.0', 'ticks': 164587, 'data': [2686.1, 2926.1, 2519.7, 2653.1], ...} + + Output CSV columns: + Sensor_0..Sensor_11, Sensor_Timestamp, Label_0..Label_3, Label_Timestamp Args: input_path: path to legacy .txt capture file @@ -20,11 +32,19 @@ def convert_legacy_capture(input_path: str, output_path: str) -> int: Number of rows written """ Path(output_path).parent.mkdir(parents=True, exist_ok=True) + + buffers = {sid: deque(maxlen=5) for sid in SENSOR_IDS} + buffers[LABEL_ID] = deque(maxlen=5) + + header = ([f"Sensor_{i}" for i in range(12)] + ['Sensor_Timestamp'] + + [f"Label_{i}" for i in range(4)] + ['Label_Timestamp']) + rows_written = 0 + last_record = None with open(input_path, "r") as f_in, open(output_path, "w", newline="") as f_out: writer = csv.writer(f_out) - header_written = False + writer.writerow(header) for line in f_in: line = line.strip() @@ -38,17 +58,71 @@ def convert_legacy_capture(input_path: str, output_path: str) -> int: if not isinstance(pkt, dict) or "data" not in pkt: continue - device_id = pkt.get("id", "unknown") - ticks = pkt.get("ticks", 0) - data = pkt["data"] + pkt_id = pkt.get("id") + if pkt_id not in buffers: + continue + + buffers[pkt_id].append(pkt) + + # Try to match: need at least one packet from each sensor bank + labels + if not all(buffers[sid] for sid in SENSOR_IDS): + continue + if not buffers[LABEL_ID]: + continue + + # Combine latest from each sensor bank + sensor_values = [] + sensor_times = [] + for sid in SENSOR_IDS: + latest = buffers[sid][-1] + sensor_values.extend(latest.get('data', [])) + sensor_times.append(latest.get('rec_time', 0)) + + label_pkt = buffers[LABEL_ID][-1] + labels = label_pkt.get('data', []) + label_time = label_pkt.get('rec_time', 0) + sensor_time = min(sensor_times) - if not header_written: - n = len(data) - header = ["device_id", "ticks"] + [f"value_{i}" for i in range(n)] - writer.writerow(header) - header_written = True + record = sensor_values + [sensor_time] + labels + [label_time] - writer.writerow([device_id, ticks] + data) - rows_written += 1 + # Skip duplicate records (same data as last write) + if record == last_record: + continue + + if len(record) == 18: + writer.writerow(record) + rows_written += 1 + last_record = record return rows_written + + +def combine_csvs(csv_paths: list[str], output_path: str) -> int: + """Concatenate multiple CSVs (same schema) into one file. + + Args: + csv_paths: list of CSV file paths to combine + output_path: path for the combined output CSV + + Returns: + Total number of data rows written + """ + Path(output_path).parent.mkdir(parents=True, exist_ok=True) + total = 0 + header_written = False + + with open(output_path, "w", newline="") as f_out: + writer = csv.writer(f_out) + + for csv_path in csv_paths: + with open(csv_path, "r") as f_in: + reader = csv.reader(f_in) + header = next(reader) + if not header_written: + writer.writerow(header) + header_written = True + for row in reader: + writer.writerow(row) + total += 1 + + return total diff --git a/pc/src/openmuscle/data/dataset.py b/pc/src/openmuscle/data/dataset.py index 2b48fe9..b23d951 100644 --- a/pc/src/openmuscle/data/dataset.py +++ b/pc/src/openmuscle/data/dataset.py @@ -19,8 +19,12 @@ def detect_columns(df: pd.DataFrame) -> tuple[list[str], list[str]]: Returns: (sensor_columns, label_columns) """ - sensor_cols = [c for c in df.columns if c.startswith("R") or c.startswith("Sensor_")] - label_cols = [c for c in df.columns if c.startswith("label_") or c.startswith("Label_")] + sensor_cols = [c for c in df.columns + if (c.startswith("R") or c.startswith("Sensor_")) + and "Timestamp" not in c] + label_cols = [c for c in df.columns + if (c.startswith("label_") or c.startswith("Label_")) + and "Timestamp" not in c] if not sensor_cols: raise ValueError("No sensor columns found (expected R*C* or Sensor_* prefix)") diff --git a/pc/src/openmuscle/data/storage.py b/pc/src/openmuscle/data/storage.py index e8938ca..d488a74 100644 --- a/pc/src/openmuscle/data/storage.py +++ b/pc/src/openmuscle/data/storage.py @@ -4,17 +4,25 @@ import os import time from pathlib import Path +from typing import Optional class CaptureWriter: """Writes matched FlexGrid + label data to CSV files. The CSV format uses columns: - timestamp, R0C0, R0C1, ..., R3C15, label_0, label_1, label_2, label_3 + timestamp, R0C0, R0C1, ..., R3C15, label_0, label_1, ..., label_N + + Header is written lazily on the first `write_row` call so the number + of label columns can be inferred from that row's label_values length. + Callers that know the count up front can pass it via `label_count`; + callers that don't (e.g. Quest hand tracking, whose joint vector is + not known until the first label packet arrives) pass `label_count=None` + and the writer derives it from the first row. """ def __init__(self, output_path: str = None, matrix_rows: int = 4, - matrix_cols: int = 16, label_count: int = 4): + matrix_cols: int = 16, label_count: Optional[int] = 4): if output_path is None: os.makedirs("data/raw/merged", exist_ok=True) output_path = f"data/raw/merged/capture_{int(time.time())}.csv" @@ -22,15 +30,28 @@ def __init__(self, output_path: str = None, matrix_rows: int = 4, self.path = Path(output_path) self.path.parent.mkdir(parents=True, exist_ok=True) - sensor_cols = [f"R{r}C{c}" for r in range(matrix_rows) for c in range(matrix_cols)] - label_cols = [f"label_{i}" for i in range(label_count)] + self._matrix_rows = matrix_rows + self._matrix_cols = matrix_cols + self._label_count: Optional[int] = label_count # None = infer on first row self._file = open(self.path, "w", newline="") self._writer = csv.writer(self._file) - self._writer.writerow(["timestamp"] + sensor_cols + label_cols) + self._header_written = False self._count = 0 + def _write_header(self, label_count: int) -> None: + sensor_cols = [f"R{r}C{c}" for r in range(self._matrix_rows) + for c in range(self._matrix_cols)] + label_cols = [f"label_{i}" for i in range(label_count)] + self._writer.writerow(["timestamp"] + sensor_cols + label_cols) + self._label_count = label_count + self._header_written = True + def write_row(self, timestamp: float, sensor_values: list, label_values: list): + if not self._header_written: + count = (self._label_count if self._label_count is not None + else len(label_values)) + self._write_header(count) self._writer.writerow([timestamp] + sensor_values + label_values) self._count += 1 @@ -38,7 +59,19 @@ def write_row(self, timestamp: float, sensor_values: list, label_values: list): def row_count(self) -> int: return self._count + @property + def label_count(self) -> Optional[int]: + """How many label columns the CSV has. None until the first row + is written (or close() is called on an empty capture, which + falls back to the constructor hint or 0).""" + return self._label_count + def close(self): + if not self._header_written: + # No row was ever paired -- still emit a header so consumers + # don't trip on a zero-byte file. Use the constructor hint + # if given, else 0 label columns (sensor-only capture). + self._write_header(self._label_count if self._label_count is not None else 0) self._file.close() def __enter__(self): diff --git a/pc/src/openmuscle/discovery/__init__.py b/pc/src/openmuscle/discovery/__init__.py new file mode 100644 index 0000000..41466d1 --- /dev/null +++ b/pc/src/openmuscle/discovery/__init__.py @@ -0,0 +1,10 @@ +"""Native OpenMuscle V4 discovery + subscribe for the PC hub. + +Replaces the interim pc/bridge_subscriber.py: discovers V4 sources and keeps a +TCP subscription (1 Hz heartbeat) to each, so the existing UDP listener receives +their sensor frames with no other change. +""" + +from openmuscle.discovery.manager import DiscoveryManager, DiscoveredDevice + +__all__ = ["DiscoveryManager", "DiscoveredDevice"] diff --git a/pc/src/openmuscle/discovery/manager.py b/pc/src/openmuscle/discovery/manager.py new file mode 100644 index 0000000..b044602 --- /dev/null +++ b/pc/src/openmuscle/discovery/manager.py @@ -0,0 +1,433 @@ +"""Native V4 discovery + subscribe manager for the PC hub. + +Replaces the interim pc/bridge_subscriber.py. Discovers OpenMuscle V4 sources +and keeps a TCP subscription (with a 1 Hz heartbeat) to each, so the existing +UDP 3141 listener receives their sensor frames with no change to the parse or +state pipeline. + +Three discovery paths, because no single one is sufficient: + + 1. Passive beacons. V4 devices broadcast an `announce` to 255.255.255.255:3141 + at ~1 Hz WHILE THEY HAVE NO SUBSCRIBER. The moment any hub subscribes, the + beacon goes silent and resumes only when the subscriber list empties + (confirmed by the phone team). So passive listening alone misses any device + another hub (or our own prior run) already holds. + 2. Cache. Every device we have ever seen is persisted (id -> ip, cmd_port, + device_type). On startup we actively re-probe cached devices with the + `get_info` command, recovering them even when their beacon is silent. + 3. Manual probe. Probe an explicit ip:cmd_port (lab setup / UI "add device"). + +Discovered + reachable devices are auto-subscribed when auto_subscribe is set. +The contract this implements is documented in FlexGridV4-Firmware/lib/ +{discovery,commands,subscribers,network_manager}.py and confirmed on the +coordination board (vrpc #0008). +""" + +import json +import os +import socket +import threading +import time + +DEFAULT_UDP_PORT = 3141 # where we ask sources to unicast sensor frames +DEFAULT_CMD_PORTS = (8001, 8002) # flexgrid cmd, lask5 cmd (probe order) +HEARTBEAT_S = 1.0 # firmware drops a subscriber after ~5 s silence +TCP_TIMEOUT_S = 3.0 +RESUBSCRIBE_BACKOFF_S = 2.0 # wait between failed re-subscribe attempts +HUB_ID = "pc-native-discovery" + + +def default_lan_ip(test_target=("10.0.0.1", 80)): + """Best-effort local LAN IP (the address sources should unicast to). + + Opens a throwaway UDP socket toward the gateway and reads the chosen + source address; no packet is actually sent. Falls back to hostname + resolution, then loopback. + """ + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + try: + s.connect(test_target) + return s.getsockname()[0] + except Exception: + try: + return socket.gethostbyname(socket.gethostname()) + except Exception: + return "127.0.0.1" + finally: + s.close() + + +def default_cache_path(): + """~/.openmuscle/discovered_devices.json (override via DiscoveryManager).""" + return os.path.join(os.path.expanduser("~"), ".openmuscle", + "discovered_devices.json") + + +class DiscoveredDevice: + """A source we have discovered and may be subscribed to. + + Plain class (not a dataclass) so it stays import-light and easy to mutate + under the manager lock. + """ + + __slots__ = ("device_id", "device_type", "ip", "cmd_port", "sensor_port", + "fw", "caps", "matrix", "source", "last_seen", + "subscribed", "sub_error") + + def __init__(self, device_id, device_type, ip, cmd_port, + sensor_port=DEFAULT_UDP_PORT, fw="", caps=None, matrix=None, + source="beacon", last_seen=0.0): + self.device_id = device_id + self.device_type = device_type + self.ip = ip + self.cmd_port = int(cmd_port) + self.sensor_port = int(sensor_port) + self.fw = fw + self.caps = list(caps or []) + self.matrix = list(matrix or []) + self.source = source + self.last_seen = last_seen + self.subscribed = False + self.sub_error = "" + + def to_cache(self): + """Minimal persisted form (enough to re-probe next run).""" + return { + "device_id": self.device_id, + "device_type": self.device_type, + "ip": self.ip, + "cmd_port": self.cmd_port, + "sensor_port": self.sensor_port, + } + + def to_snapshot(self, now=None): + now = now if now is not None else time.time() + return { + "device_id": self.device_id, + "device_type": self.device_type, + "ip": self.ip, + "cmd_port": self.cmd_port, + "sensor_port": self.sensor_port, + "fw": self.fw, + "caps": list(self.caps), + "matrix": list(self.matrix), + "source": self.source, + "age_s": round(now - self.last_seen, 2) if self.last_seen else None, + "subscribed": self.subscribed, + "sub_error": self.sub_error, + } + + +def _send_cmd(sock, msg_id, verb, **fields): + """Encode + send one newline-delimited command, return the parsed ack dict. + + Raises on socket error or malformed ack so callers can react (re-subscribe). + """ + msg = {"v": "1.0", "type": "cmd", "msg_id": msg_id, + "data": dict({"verb": verb}, **fields)} + sock.sendall((json.dumps(msg) + "\n").encode("utf-8")) + raw = sock.recv(4096) + if not raw: + raise ConnectionError("command channel closed (empty read)") + # Firmware sends one ack JSON object per line; take the first line. + line = raw.split(b"\n", 1)[0].strip() + return json.loads(line.decode("utf-8")) + + +class _SubscriptionKeeper: + """Owns one device's TCP command channel: subscribe, then heartbeat at + 1 Hz, auto-resubscribing on failure. Sensor frames arrive out-of-band on + the manager's UDP port; this object never touches them. + """ + + def __init__(self, device, pc_host, udp_port, on_state): + self.device = device + self.pc_host = pc_host + self.udp_port = udp_port + self.on_state = on_state # callback(device) on any state change + self._stop = threading.Event() + self._tcp = None + self._thread = None + self._msg_id = 1 + + def start(self): + """Subscribe once synchronously; on success spawn the heartbeat thread. + Returns True if the device accepted the subscription.""" + if not self._connect_and_subscribe(): + return False + self._thread = threading.Thread( + target=self._loop, name="om-keeper-" + self.device.device_id, + daemon=True) + self._thread.start() + return True + + def _connect_and_subscribe(self): + try: + tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + tcp.settimeout(TCP_TIMEOUT_S) + tcp.connect((self.device.ip, self.device.cmd_port)) + ack = _send_cmd(tcp, self._next_id(), "subscribe", + host=self.pc_host, port=self.udp_port, + transport="wifi", hub_id=HUB_ID) + data = (ack or {}).get("data") or {} + if not data.get("accepted"): + tcp.close() + self.device.subscribed = False + self.device.sub_error = "rejected (list full?)" + self._emit() + return False + self._tcp = tcp + self.device.subscribed = True + self.device.sub_error = "" + self._emit() + return True + except Exception as e: + self.device.subscribed = False + self.device.sub_error = str(e) + self._emit() + return False + + def _loop(self): + while not self._stop.is_set(): + if self._stop.wait(HEARTBEAT_S): + break + try: + _send_cmd(self._tcp, self._next_id(), "heartbeat", + host=self.pc_host, port=self.udp_port, + transport="wifi") + except Exception as e: + # Connection dropped (device reboot, Wi-Fi blip). Try to + # re-establish; back off so we don't spin on a dead host. + self.device.subscribed = False + self.device.sub_error = "heartbeat failed: {}".format(e) + self._emit() + self._safe_close() + if self._stop.wait(RESUBSCRIBE_BACKOFF_S): + break + self._connect_and_subscribe() + + def stop(self): + self._stop.set() + if self._tcp is not None: + try: + _send_cmd(self._tcp, 999, "unsubscribe", + host=self.pc_host, port=self.udp_port, + transport="wifi") + except Exception: + pass + self._safe_close() + self.device.subscribed = False + self._emit() + + def _safe_close(self): + if self._tcp is not None: + try: + self._tcp.close() + except Exception: + pass + self._tcp = None + + def _next_id(self): + self._msg_id += 1 + return self._msg_id + + def _emit(self): + if self.on_state: + try: + self.on_state(self.device) + except Exception: + pass + + +class DiscoveryManager: + """Tracks discovered V4 sources and keeps subscriptions to them. + + Thread model: on_announce() runs on the UDP listener thread; probe() and + recover_cache() run on their own short-lived threads; snapshot() runs on + the web request thread. A single lock guards the device + keeper maps. + """ + + def __init__(self, pc_host=None, udp_port=DEFAULT_UDP_PORT, + cache_path=None, auto_subscribe=True): + self.pc_host = pc_host or default_lan_ip() + self.udp_port = udp_port + self.cache_path = cache_path if cache_path is not None else default_cache_path() + self.auto_subscribe = auto_subscribe + self._devices = {} # device_id -> DiscoveredDevice + self._keepers = {} # device_id -> _SubscriptionKeeper + self._lock = threading.Lock() + + # ---- discovery inputs ------------------------------------------------- + + def on_announce(self, obj, src_ip): + """Handle one announce beacon. `obj` is the decoded announce dict; + `src_ip` is the datagram source (the announce never carries its own IP). + """ + try: + device_id = obj["id"] + except (KeyError, TypeError): + return None + services = obj.get("services") or {} + cmd_port = int(services.get("cmd", DEFAULT_CMD_PORTS[0])) + sensor_port = int(services.get("sensor", self.udp_port)) + dev = DiscoveredDevice( + device_id=device_id, + device_type=obj.get("dev", "unknown"), + ip=src_ip, + cmd_port=cmd_port, + sensor_port=sensor_port, + fw=obj.get("fw", ""), + caps=obj.get("caps"), + matrix=obj.get("matrix"), + source="beacon", + last_seen=time.time(), + ) + return self._upsert(dev) + + def probe(self, ip, cmd_port=None): + """Actively query ip:cmd_port (or try the default ports) with get_info. + Returns the DiscoveredDevice on success, or None. Used for cache + recovery and manual add.""" + ports = [int(cmd_port)] if cmd_port else list(DEFAULT_CMD_PORTS) + for port in ports: + try: + tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + tcp.settimeout(TCP_TIMEOUT_S) + tcp.connect((ip, port)) + ack = _send_cmd(tcp, 1, "get_info") + tcp.close() + except Exception: + continue + data = (ack or {}).get("data") or {} + if not data.get("id"): + continue + dev = DiscoveredDevice( + device_id=data["id"], + device_type=data.get("dev", "unknown"), + ip=ip, + cmd_port=port, + sensor_port=self.udp_port, + fw=data.get("fw", ""), + caps=data.get("caps"), + matrix=data.get("matrix"), + source="probe", + last_seen=time.time(), + ) + return self._upsert(dev) + return None + + def recover_cache(self, background=True): + """Re-probe every cached device so we recover ones whose beacon is + silent (held by another hub). Non-blocking by default.""" + cached = self._load_cache() + if not cached: + return + + def _work(): + for entry in cached: + ip = entry.get("ip") + port = entry.get("cmd_port") + if ip: + self.probe(ip, port) + + if background: + threading.Thread(target=_work, name="om-cache-recover", + daemon=True).start() + else: + _work() + + # ---- subscription control -------------------------------------------- + + def subscribe(self, device_id): + """Start (or restart) a subscription keeper for a known device.""" + with self._lock: + dev = self._devices.get(device_id) + if dev is None: + return False + existing = self._keepers.get(device_id) + if existing is not None: + return dev.subscribed + keeper = _SubscriptionKeeper(dev, self.pc_host, self.udp_port, + self._on_keeper_state) + self._keepers[device_id] = keeper + ok = keeper.start() + if not ok: + with self._lock: + self._keepers.pop(device_id, None) + return ok + + def unsubscribe(self, device_id): + with self._lock: + keeper = self._keepers.pop(device_id, None) + if keeper is not None: + keeper.stop() + return True + return False + + def snapshot(self): + """List of device dicts for the web UI / API.""" + now = time.time() + with self._lock: + return [d.to_snapshot(now) for d in self._devices.values()] + + def stop(self): + with self._lock: + keepers = list(self._keepers.values()) + self._keepers.clear() + for k in keepers: + k.stop() + + # ---- internals -------------------------------------------------------- + + def _upsert(self, dev): + """Merge a freshly discovered device into the table, persist the cache, + and auto-subscribe if enabled and not already subscribed.""" + with self._lock: + existing = self._devices.get(dev.device_id) + if existing is None: + self._devices[dev.device_id] = dev + target = dev + else: + # Refresh address/metadata + last_seen; keep subscription state. + existing.ip = dev.ip + existing.cmd_port = dev.cmd_port + existing.sensor_port = dev.sensor_port + existing.device_type = dev.device_type + existing.fw = dev.fw or existing.fw + existing.caps = dev.caps or existing.caps + existing.matrix = dev.matrix or existing.matrix + existing.source = dev.source + existing.last_seen = dev.last_seen + target = existing + already_subscribed = target.device_id in self._keepers + self._save_cache() + if self.auto_subscribe and not already_subscribed: + self.subscribe(target.device_id) + return target + + def _on_keeper_state(self, device): + # Keeper mutated device.subscribed/sub_error in place; nothing else to + # do today, but this is where UI push / logging would hook in. + pass + + def _load_cache(self): + try: + with open(self.cache_path, "r", encoding="utf-8") as f: + data = json.load(f) + return data if isinstance(data, list) else [] + except (FileNotFoundError, ValueError, OSError): + return [] + + def _save_cache(self): + if not self.cache_path: + return + with self._lock: + entries = [d.to_cache() for d in self._devices.values()] + try: + os.makedirs(os.path.dirname(self.cache_path), exist_ok=True) + tmp = self.cache_path + ".tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump(entries, f, indent=2) + os.replace(tmp, self.cache_path) + except OSError: + pass diff --git a/pc/src/openmuscle/ml/export.py b/pc/src/openmuscle/ml/export.py new file mode 100644 index 0000000..b05657f --- /dev/null +++ b/pc/src/openmuscle/ml/export.py @@ -0,0 +1,107 @@ +"""Export predictions and ground truth to JSON for web visualization.""" + +import json +import numpy as np +import pandas as pd +from pathlib import Path + +from openmuscle.data.dataset import load_training_data +from openmuscle.ml.registry import ModelRegistry + + +def export_predictions(model_path: str, data_path: str, output_path: str) -> dict: + """Run a trained model on a capture CSV and export results as JSON. + + The JSON output contains sensor data, actual labels (ground truth), + and model predictions, suitable for web visualization. + + Args: + model_path: path to trained model .pkl file + data_path: path to CSV with sensor + label columns + output_path: path for output .json file + + Returns: + dict with summary info (n_samples, metrics, etc.) + """ + from sklearn.metrics import mean_squared_error, r2_score + + registry = ModelRegistry() + model = registry.load(model_path) + + X, y = load_training_data(data_path) + sensor_cols = list(X.columns) + label_cols = list(y.columns) + + # Load full dataframe for timestamps + df = pd.read_csv(data_path) + timestamps = None + for ts_col in ['Sensor_Timestamp', 'timestamp', 'time']: + if ts_col in df.columns: + timestamps = df[ts_col].tolist() + break + + # Run predictions + predictions = model.predict(X) + if predictions.ndim == 1: + predictions = predictions.reshape(-1, 1) + + actuals = y.values + + # Compute metrics + mse = float(mean_squared_error(actuals, predictions)) + r2 = float(r2_score(actuals, predictions)) + + # Per-label metrics + per_label = {} + for i, col in enumerate(label_cols): + per_label[col] = { + "mse": float(mean_squared_error(actuals[:, i], predictions[:, i])), + "r2": float(r2_score(actuals[:, i], predictions[:, i])), + } + + # Build output structure + n_samples = len(X) + output = { + "metadata": { + "model_path": model_path, + "data_path": data_path, + "n_samples": n_samples, + "sensor_columns": sensor_cols, + "label_columns": label_cols, + "metrics": { + "overall_mse": mse, + "overall_r2": r2, + "per_label": per_label, + }, + }, + "sensor_data": {}, + "labels": {}, + "predictions": {}, + } + + if timestamps: + output["timestamps"] = timestamps + + # Sensor values per channel + for col in sensor_cols: + output["sensor_data"][col] = X[col].tolist() + + # Actual labels per channel + for i, col in enumerate(label_cols): + output["labels"][col] = actuals[:, i].tolist() + + # Predicted labels per channel + for i, col in enumerate(label_cols): + output["predictions"][col] = predictions[:, i].tolist() + + # Write JSON + Path(output_path).parent.mkdir(parents=True, exist_ok=True) + with open(output_path, "w") as f: + json.dump(output, f, indent=2) + + print(f"Exported {n_samples} samples to {output_path}") + print(f" Overall MSE: {mse:.4f} R^2: {r2:.4f}") + for col, m in per_label.items(): + print(f" {col}: MSE={m['mse']:.4f} R^2={m['r2']:.4f}") + + return output["metadata"] diff --git a/pc/src/openmuscle/protocol/parser.py b/pc/src/openmuscle/protocol/parser.py index 91f9200..e79f2a1 100644 --- a/pc/src/openmuscle/protocol/parser.py +++ b/pc/src/openmuscle/protocol/parser.py @@ -34,6 +34,20 @@ def parse_packet(raw_bytes: bytes) -> OpenMusclePacket | None: # New protocol: JSON object with "v" field if isinstance(obj, dict) and "v" in obj: + # V4 discovery announce beacons share UDP 3141 with sensor frames. + # They are NOT sensor data: route the whole announce dict through .data + # so the listener can hand it to the DiscoveryManager and keep it out of + # the sensor pipeline (otherwise it becomes a phantom "announce" device). + if obj.get("type") == "announce": + return OpenMusclePacket( + version=obj["v"], + device_type="announce", + device_id=obj.get("id", ""), + timestamp_ms=obj.get("ts", 0), + data=obj, + metadata={}, + receive_time=recv_ts, + ) return OpenMusclePacket( version=obj["v"], device_type=obj["type"], diff --git a/pc/src/openmuscle/receiver/udp_listener.py b/pc/src/openmuscle/receiver/udp_listener.py index 534c9e1..e359b83 100644 --- a/pc/src/openmuscle/receiver/udp_listener.py +++ b/pc/src/openmuscle/receiver/udp_listener.py @@ -18,10 +18,15 @@ class UDPListener: # process pkt """ - def __init__(self, port: int = 3141, bind_ip: str = "0.0.0.0"): + def __init__(self, port: int = 3141, bind_ip: str = "0.0.0.0", + announce_handler=None): self.port = port self.bind_ip = bind_ip self.packet_queue: Queue = Queue() + # Optional callback(announce_dict, src_ip) for V4 discovery beacons. + # When set, announce packets are routed here and NOT enqueued as sensor + # frames. When None (CLI inference/heatmap tools), behavior is unchanged. + self.announce_handler = announce_handler self._running = False self._thread = None @@ -44,8 +49,18 @@ def _listen(self): try: data, addr = sock.recvfrom(8192) pkt = parse_packet(data) - if pkt: - self.packet_queue.put(pkt) + if pkt is None: + continue + # Divert V4 discovery beacons to the discovery handler; the + # announce carries no IP, so pass the datagram source address. + if pkt.device_type == "announce": + if self.announce_handler is not None: + try: + self.announce_handler(pkt.data, addr[0]) + except Exception as e: + print(f"announce handler error: {e}") + continue + self.packet_queue.put(pkt) except socket.timeout: continue except Exception as e: diff --git a/pc/src/openmuscle/simulate/quest_hand.py b/pc/src/openmuscle/simulate/quest_hand.py new file mode 100644 index 0000000..409504d --- /dev/null +++ b/pc/src/openmuscle/simulate/quest_hand.py @@ -0,0 +1,202 @@ +"""Synthetic WebXR hand frames for hardware-free pipeline testing. + +Generates the same JSON payload shape the Quest WebXR client pushes over +/ws/quest (see AppState.ingest_quest_packet), driven by a smooth latent +finger-curl signal. The companion flexgrid_matrix() derives correlated +pressure-sensor values from the SAME curls, so a capture recorded from +the simulator pair is actually learnable: train on it and the model's +predicted hand visibly tracks the real one in both viewers. + +Everything here is pure (t in, data out) so tests can call it directly +without sockets or timing. +""" + +import math + +# Canonical WebXR hand joint order, 25 joints. Must match what the VR +# client sends and what the desktop hand viewer assumes (wrist, 4 thumb +# joints, then 5 per finger). +JOINT_NAMES = ( + "wrist", + "thumb-metacarpal", "thumb-phalanx-proximal", "thumb-phalanx-distal", + "thumb-tip", + "index-finger-metacarpal", "index-finger-phalanx-proximal", + "index-finger-phalanx-intermediate", "index-finger-phalanx-distal", + "index-finger-tip", + "middle-finger-metacarpal", "middle-finger-phalanx-proximal", + "middle-finger-phalanx-intermediate", "middle-finger-phalanx-distal", + "middle-finger-tip", + "ring-finger-metacarpal", "ring-finger-phalanx-proximal", + "ring-finger-phalanx-intermediate", "ring-finger-phalanx-distal", + "ring-finger-tip", + "pinky-finger-metacarpal", "pinky-finger-phalanx-proximal", + "pinky-finger-phalanx-intermediate", "pinky-finger-phalanx-distal", + "pinky-finger-tip", +) + +N_JOINTS = len(JOINT_NAMES) + +# Per-finger latent curl oscillators (Hz, radians). Distinct frequencies +# so no two fingers move in lockstep and a model can't cheat by learning +# one shared phase. +_CURL_FREQ = (0.13, 0.21, 0.17, 0.26, 0.31) # thumb..pinky +_CURL_PHASE = (0.0, 1.3, 2.6, 3.9, 5.2) + +# Finger geometry for a plausible right hand, meters. The hand extends +# +y from the wrist with the palm facing -z, so curling rotates segment +# directions from +y toward -z. +# (lateral x offset of metacarpal head, metacarpal length, +# segment lengths..., max curl angle per segment in radians) +_FINGERS = { + "index": (0.030, 0.085, (0.042, 0.025, 0.020), 1.35), + "middle": (0.010, 0.090, (0.046, 0.028, 0.021), 1.40), + "ring": (-0.012, 0.085, (0.042, 0.026, 0.020), 1.40), + "pinky": (-0.032, 0.075, (0.033, 0.020, 0.017), 1.45), +} + + +def finger_curls(t: float) -> tuple: + """Latent curl signal per finger at time t, each in [0, 1]. + + Order: thumb, index, middle, ring, pinky. + """ + return tuple( + 0.5 + 0.5 * math.sin(2 * math.pi * f * t + p) + for f, p in zip(_CURL_FREQ, _CURL_PHASE) + ) + + +def _chain(start, direction_y, direction_z, lengths, curl, max_angle): + """Walk a finger's phalanx chain in the y/-z plane. + + Starts at `start` heading (0, direction_y, direction_z) (unit), bends + by curl*max_angle at each joint. Returns the list of joint positions + after each segment. + """ + out = [] + px, py, pz = start + angle = math.atan2(-direction_z, direction_y) + step = curl * max_angle / len(lengths) + for seg in lengths: + angle += step + py += seg * math.cos(angle) + pz -= seg * math.sin(angle) + out.append((px, py, pz)) + return out + + +def _yaw_quat(a: float) -> list: + """Quaternion [x,y,z,w] for a rotation of `a` radians about +y.""" + return [0.0, math.sin(a / 2), 0.0, math.cos(a / 2)] + + +def _yaw_rotate(p, origin, a): + """Rotate point p about a vertical axis through `origin` by angle a.""" + x, y, z = p[0] - origin[0], p[1], p[2] - origin[2] + ca, sa = math.cos(a), math.sin(a) + return (origin[0] + x * ca + z * sa, y, origin[2] - x * sa + z * ca) + + +def hand_pose(t: float): + """All 25 joint positions (world space, meters) plus the shared + orientation quaternion at time t. + + Returns (positions, quat) where positions is a list of 25 (x, y, z) + tuples in JOINT_NAMES order. + """ + curls = finger_curls(t) + + # Wrist wanders slowly so the wrist-local canonicalization in both + # viewers is exercised, not just fed a static origin. + wrist = ( + 0.10 * math.sin(2 * math.pi * 0.05 * t), + 1.00 + 0.05 * math.sin(2 * math.pi * 0.07 * t), + -0.30 + 0.08 * math.cos(2 * math.pi * 0.06 * t), + ) + + positions = [wrist] + + # Thumb: 4 joints, angled out +x, curling toward the palm. + tc = curls[0] + base = (wrist[0] + 0.025, wrist[1] + 0.020, wrist[2]) + positions.append(base) + px, py, pz = base + angle = 0.5 + tc * 0.9 # radians of accumulated flexion + for seg in (0.045, 0.032, 0.028): + px += seg * math.sin(0.6) * (1 - tc * 0.5) # keeps thumb fanned +x + py += seg * math.cos(angle) * 0.8 + pz -= seg * math.sin(angle) * 0.6 + angle += tc * 0.5 + positions.append((px, py, pz)) + + # Four fingers: metacarpal head, then a curling 3-segment chain plus tip. + for i, name in enumerate(("index", "middle", "ring", "pinky")): + dx, meta_len, segs, max_angle = _FINGERS[name] + curl = curls[i + 1] + meta = (wrist[0] + dx, wrist[1] + meta_len, wrist[2]) + positions.append(meta) + chain = _chain(meta, 1.0, 0.0, segs, curl, max_angle) + positions.extend(chain) + # Tip: short continuation of the last segment direction. + lx, ly, lz = chain[-1] + sx, sy, sz = chain[-2] if len(chain) > 1 else meta + norm = math.dist((lx, ly, lz), (sx, sy, sz)) or 1.0 + positions.append(( + lx + (lx - sx) / norm * 0.012, + ly + (ly - sy) / norm * 0.012, + lz + (lz - sz) / norm * 0.012, + )) + + # Slow global yaw about the wrist; every joint reports the same + # orientation so the viewers' inverse-wrist-rotation path gets real + # (non-identity) input. + yaw = 0.6 * math.sin(2 * math.pi * 0.04 * t) + positions = [wrist] + [_yaw_rotate(p, wrist, yaw) for p in positions[1:]] + return positions, _yaw_quat(yaw) + + +def hand_frame(t: float, device_id: str = "quest-sim", + handedness: str = "right") -> dict: + """One /ws/quest JSON payload for time t.""" + positions, quat = hand_pose(t) + return { + "device_id": device_id, + "ts": int(t * 1000) % 2**31, + "handedness": handedness, + "joints": [ + {"name": name, "pos": [round(c, 5) for c in pos], "rot": quat} + for name, pos in zip(JOINT_NAMES, positions) + ], + "meta": {"sim": True}, + } + + +# Flexgrid layout used by the existing simulator: 16 column-groups of 4. +_FG_COLS = 16 +_FG_ROWS = 4 + + +def flexgrid_matrix(curls, rng=None) -> list: + """A 16x4 flexgrid matrix correlated with the given finger curls. + + Each sensor responds to a weighted blend of nearby fingers (gaussian + falloff across the 16 columns, finger centers spread over them), plus + optional noise from `rng` (anything with .gauss). Values are ints in + [0, 4095] like real FlexGrid ADC counts. + """ + centers = (1.5, 5.0, 8.0, 11.0, 14.0) # thumb..pinky across 16 columns + matrix = [] + for col in range(_FG_COLS): + group = [] + for row in range(_FG_ROWS): + signal = 0.0 + for f, center in enumerate(centers): + w = math.exp(-((col - center) ** 2) / 6.0) + # Rows sample the muscle at slightly different gains. + signal += w * curls[f] * (0.7 + 0.1 * row) + v = 400 + 3000 * signal + if rng is not None: + v += rng.gauss(0, 60) + group.append(max(0, min(4095, int(v)))) + matrix.append(group) + return matrix diff --git a/pc/src/openmuscle/simulate/transmitter.py b/pc/src/openmuscle/simulate/transmitter.py index 3040fda..ad730a0 100644 --- a/pc/src/openmuscle/simulate/transmitter.py +++ b/pc/src/openmuscle/simulate/transmitter.py @@ -10,19 +10,29 @@ def run_simulator(port: int = 3141, device_type: str = "flexgrid", - replay_file: str = None, target_ip: str = "127.0.0.1"): - """Send synthetic or replayed sensor data over UDP. + replay_file: str = None, target_ip: str = "127.0.0.1", + web_port: int = 8000): + """Send synthetic or replayed sensor data. Args: port: UDP port to send to - device_type: device type to simulate ("flexgrid" or "lask5") + device_type: device type to simulate ("flexgrid", "lask5", + "quest_hand", or "combo"). quest_hand streams synthetic WebXR + hand frames to the web server's /ws/quest WebSocket (no UDP). + combo streams a flexgrid UDP device AND a quest hand driven by + the same latent finger curls, so a recorded capture is + actually learnable end to end without hardware. replay_file: path to a capture .txt file to replay (optional) target_ip: IP address to send to + web_port: HTTP port of the running `openmuscle web` server + (quest_hand/combo only) """ sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) if replay_file: _replay(sock, replay_file, target_ip, port) + elif device_type in ("quest_hand", "combo"): + _generate_quest(sock, device_type, target_ip, port, web_port) else: _generate(sock, device_type, target_ip, port) @@ -71,6 +81,66 @@ def _generate(sock, device_type: str, ip: str, port: int): print(f"\nStopped after {pkt_num} packets") +def _generate_quest(sock, device_type: str, ip: str, udp_port: int, + web_port: int): + """Stream synthetic WebXR hand frames to /ws/quest, and for combo mode + a correlated flexgrid device over UDP alongside. + + The hand goes over WebSocket because that's the only transport the + real Quest client has; reusing the same ingest path means everything + downstream (recorder, matcher, snapshot, both hand viewers) sees the + simulator exactly as it would see a headset. + """ + import random + from websockets.sync.client import connect + from openmuscle.simulate.quest_hand import finger_curls, hand_frame, \ + flexgrid_matrix + + url = f"ws://{ip}:{web_port}/ws/quest" + print(f"Streaming synthetic quest_hand frames -> {url}") + if device_type == "combo": + print(f" + correlated flexgrid UDP -> {ip}:{udp_port}") + print("Press Ctrl+C to stop (reconnects if the server restarts)") + + rng = random.Random(1138) + frame_num = 0 + t0 = time.time() + try: + while True: + try: + with connect(url) as ws: + while True: + t = time.time() - t0 + ws.send(json.dumps(hand_frame(t))) + frame_num += 1 + # Flexgrid at half the hand rate (~12.5 Hz vs 25 Hz), + # like the real rig where sensor and label rates differ. + if device_type == "combo" and frame_num % 2 == 0: + pkt = { + "v": "1.0", + "type": "flexgrid", + "id": "flexgrid-sim", + "ts": int(time.time() * 1000) % 2**31, + "data": { + "matrix": flexgrid_matrix(finger_curls(t), rng), + "rows": 4, "cols": 16, + }, + } + sock.sendto(json.dumps(pkt).encode("utf-8"), + (ip, udp_port)) + if frame_num % 250 == 0: + print(f"Sent {frame_num} hand frames") + time.sleep(0.04) # 25 Hz + except Exception as e: + # KeyboardInterrupt is a BaseException, so Ctrl+C still + # reaches the outer handler. + print(f"WebSocket dropped ({type(e).__name__}: {e}); " + f"retrying in 2s") + time.sleep(2) + except KeyboardInterrupt: + print(f"\nStopped after {frame_num} hand frames") + + def _replay(sock, filepath: str, ip: str, port: int): """Replay a capture file by re-sending each line as a UDP packet.""" import ast diff --git a/pc/src/openmuscle/web/README.md b/pc/src/openmuscle/web/README.md index 5b76a7f..dadce5c 100644 --- a/pc/src/openmuscle/web/README.md +++ b/pc/src/openmuscle/web/README.md @@ -207,12 +207,55 @@ The WS snapshot already exposes `matrix`, `values`, and `joystick` for any devic Today there's exactly one `UDPListener` per `AppState`, on `--udp-port`. If you need to listen on more (e.g. legacy LASK5 on 3145 while keeping FlexGrid on 3141), the cleanest extension is to instantiate multiple `UDPListener`s in `AppState.__init__` and merge their queues in `run_broadcaster`. As of v0.2.0 we've standardized everything on 3141 instead, so this hasn't been needed yet. +## VR companion (`/vr`) + +The same FastAPI process also serves a WebXR client that turns a Meta Quest 3 into a labeling rig and live demo for the muscle→finger model. Operator guide: [`docs/vr-setup.md`](../../../../../docs/vr-setup.md). Wire format for the new device type: [`docs/protocol.md`](../../../../../docs/protocol.md#quest-hand-tracking-type-quest_hand). + +### What's added vs the bare web UI + +| Surface | Purpose | +|---|---| +| `GET /vr` | Serves `static/vr/index.html` — landing page + WebXR client (Three.js + XRHand). | +| `WS /ws/quest` | Inbound — accepts XRHand joint frames from the headset. Each frame is synthesized in-process into an `OpenMusclePacket(device_type="quest_hand")` and fed through the same `_handle_packet` UDP devices use. From the recorder/matcher/snapshot's view the Quest is just another device. | +| `start-vr.bat` (in `pc/`) | One-click launcher: ADB sanity check → start server → poll until up → `adb reverse` → open Quest Browser to `/vr`. Optional arm arg (`right`\|`left`). | +| `--ssl-certfile` / `--ssl-keyfile` | Serve HTTPS so the headset can hit `/vr` over LAN. WebXR refuses hand-tracking on plain HTTP (localhost is the only exception, via `adb reverse`). | + +### Why a WebSocket inbound (when everything else is UDP) + +Browsers can't speak UDP. WebXR therefore can't run as a UDP-emitting device. The chosen workaround is one synthesizer method (`AppState.ingest_quest_packet`) that builds the same `OpenMusclePacket` shape the UDP listener emits, so the rest of the pipeline — `_handle_packet` → `DeviceInfo.update` → `_record_packet` → `TemporalMatcher` → `CaptureWriter` — is unchanged. The integration cost was one new endpoint plus one synthesizer; everything downstream rides for free. + +### `quest_hand` recording specifics + +- **Match window default** is 175 ms for `quest_hand` label sources vs 100 ms for `lask5`, set per-device-type in `AppState.DEFAULT_WINDOW_MS_BY_TYPE`. Quest WebXR has higher end-to-end latency than LASK5's ESP-NOW path, so a tighter window dropped too many sensor frames as unpaired. +- **`label_count` is None / lazy-inferred** for `quest_hand` — `CaptureWriter` defers the CSV header write until the first label packet so the column count is derived from `len(values)`. Quest 3 sends 25 joints × 7 floats = 175 floats per frame; the hardcoded LASK5 `label_count=4` doesn't fit. +- **Per-capture `.labels.schema.json` sidecar** lists joint names + channel order so the wide CSV is self-describing. Only emitted for label sources whose meaning isn't obvious from `device_type` alone (today: `quest_hand`). +- **`meta.json` `auto.label_source`** is tagged `"quest_hand"` so the Captures panel filter and any downstream training pipeline can cleanly separate Quest-labeled from LASK5-labeled datasets. +- **Auto-pick** in `start_recording` walks `AUTO_LABEL_TYPE_PREFERENCE = ("quest_hand", "lask5")` — Quest wins if both are connected, since it's the richer label source. + +### WebXR client structure (`static/vr/`) + +| File | Role | +|---|---| +| `index.html` | Landing page with a 3-checkmark preflight (HTTPS, WebXR support, server reachable), arm selector, VRButton mount. Script tag uses `?v=N` cache-buster (see gotcha below). | +| `app.js` | Scene + XR session lifecycle, per-frame joint capture → `/ws/quest`, real-time hand visualizer (blue captured-arm + green off-hand spheres), heatmap panel painted from `/ws/live`, 3×2 menu (REC / SESSION / PREDICT / TRAIN / RECENTER / EXIT VR), ray pointers + select-event button activation, REAL-vs-PRED finger-curl bars, ghost-hand overlay anchored at the real wrist when inference is on. | +| `styles.css` | Pre-VR landing page only (the XR session paints WebGL directly, CSS doesn't apply inside). | + +### Cache-bust contract + +The HTML references the JS as `app.js?v=N`. **Bump N every time `app.js` changes.** Quest Browser ignores `Cache-Control: no-store` for ES modules in some configurations — the querystring forces a fresh fetch. The no-cache middleware on `/`, `/vr`, and `/static/*` is correct, but the version-querystring is the actual cache-busting mechanism in practice. + +### Auto-enable inference on train (VR-only) + +The desktop UI's server-side default is **paused-on-load** for inference (see commit `bd1b68a` rationale). In VR there's no obvious second click to enable it, so `runTrain` in `app.js` POSTs `{enabled: true}` to `/api/inference/enabled` after a successful activate — pressing TRAIN implies "I want predictions running." The status strip surfaces `trained: R²=X · model loaded ✓ · predict ON` so the change is visible. + ## Known gotchas - **`from __future__ import annotations` breaks FastAPI body inference.** Don't add it back to `app.py` — the lazy-string annotations make FastAPI treat Pydantic-model parameters as query fields. (Bit us once, documented in `app.py` header.) - **`Pin.init(Pin.OUT, value=0)` quirks** are in the firmware, not here — but if you ever rewrite the matrix scan, see the firmware repo's "Sensor scan techniques" section first. -- **Browser cache during dev**: the no-cache middleware on `/` and `/static/*` makes JS/CSS edits land on plain F5. If you ever serve this off a CDN or behind a cache, remove or scope down that middleware. +- **Browser cache during dev**: the no-cache middleware on `/`, `/vr`, and `/static/*` makes JS/CSS edits land on plain F5 in desktop browsers. Quest Browser ignores it for ES modules — see the cache-bust contract in the VR section above (`?v=N` on the script src). +- **`/vr` was missing from the no-cache middleware** until commit `fb83f82`. If you add another HTML entry point, remember to whitelist it too. - **mpremote and the LASK5 don't coexist** on the same serial port — `openmuscle web` only uses UDP, but Thonny / PuTTY / another mpremote session will lock the COM port. Symptom: `mpremote: failed to access COMxx`. +- **WebXR requires a secure context.** localhost (`http://`) counts; LAN HTTPS via mkcert is the untethered path. Plain HTTP over LAN will silently refuse to grant hand-tracking and the user sees "WebXR not available" with no specific reason. The landing-page preflight surfaces this. ## v0.2.0 history (the LASK5 expansion) diff --git a/pc/src/openmuscle/web/app.py b/pc/src/openmuscle/web/app.py index 68f1c7f..b7a6128 100644 --- a/pc/src/openmuscle/web/app.py +++ b/pc/src/openmuscle/web/app.py @@ -9,6 +9,9 @@ # string and falls back to treating the param as a query string field). import asyncio +import shutil +import subprocess +import sys from contextlib import asynccontextmanager from pathlib import Path from typing import Optional @@ -24,6 +27,48 @@ STATIC_DIR = Path(__file__).parent / "static" +def _reveal_path_in_file_manager(path: Path, select_file: bool) -> None: + """Open `path` in the OS file manager. If `select_file=True` and the + platform supports it, highlight the file inside its parent folder + rather than just opening the folder. Raises RuntimeError on failure. + + Whitelist-guarded by the caller: this function does NOT verify that + `path` is inside captures_dir. That check happens in the route. + """ + if not path.exists(): + raise RuntimeError(f"Path does not exist: {path}") + + # Explorer / Finder / xdg-open all need *absolute* paths -- they don't + # inherit our CWD predictably and a relative path like + # "data/raw/merged/foo.csv" silently fails with "location not found". + path = path.resolve() + + try: + if sys.platform.startswith("win"): + if select_file and path.is_file(): + # explorer /select,"C:\full\path\file.csv" -- highlights file + subprocess.Popen(["explorer", f"/select,{path}"]) + else: + # Open the folder itself + folder = path if path.is_dir() else path.parent + subprocess.Popen(["explorer", str(folder)]) + elif sys.platform == "darwin": + if select_file and path.is_file(): + subprocess.Popen(["open", "-R", str(path)]) + else: + folder = path if path.is_dir() else path.parent + subprocess.Popen(["open", str(folder)]) + else: + # Linux / other -- xdg-open only opens directories cleanly + opener = shutil.which("xdg-open") or shutil.which("gio") + if opener is None: + raise RuntimeError("No file-manager opener found (xdg-open / gio)") + folder = path if path.is_dir() else path.parent + subprocess.Popen([opener, str(folder)]) + except Exception as e: + raise RuntimeError(f"Failed to open file manager: {e}") + + def create_app(udp_port: int = 3141, captures_dir: Optional[str] = None, model_path: Optional[str] = None, hand_target: Optional[tuple] = None) -> FastAPI: @@ -59,7 +104,11 @@ async def lifespan(_app: FastAPI): async def no_cache_static(request: Request, call_next): response = await call_next(request) path = request.url.path - if path == "/" or path.startswith("/static/"): + # Cache-bust the HTML entry points + every static asset. Without + # /vr in this list, Quest Browser cached the old VR HTML (which + # pointed at app.js without the version querystring), so refreshes + # kept loading stale JS even after the file changed on disk. + if path == "/" or path == "/vr" or path.startswith("/static/"): response.headers["Cache-Control"] = "no-store, no-cache, must-revalidate, max-age=0" response.headers["Pragma"] = "no-cache" response.headers["Expires"] = "0" @@ -69,6 +118,14 @@ async def no_cache_static(request: Request, call_next): async def index(): return FileResponse(STATIC_DIR / "index.html") + # WebXR companion page served at /vr. Quest Browser loads this URL, + # negotiates 'hand-tracking', opens /ws/quest, and streams XRHand frames. + # WebXR requires a secure context -- HTTPS over LAN (mkcert) or + # http://localhost via `adb reverse tcp:8000 tcp:8000` over USB. + @app.get("/vr") + async def vr_page(): + return FileResponse(STATIC_DIR / "vr" / "index.html") + app.mount("/static", StaticFiles(directory=STATIC_DIR), name="static") # ----- live data over WebSocket ----- @@ -89,12 +146,92 @@ async def ws_live(websocket: WebSocket): finally: state.ws_clients.discard(websocket) + # Inbound WS from the Quest headset. Browsers can't speak UDP so the + # WebXR client opens this socket and pushes XRHand joint frames as + # JSON. We feed each frame through ingest_quest_packet, which + # synthesizes a device_type="quest_hand" OpenMusclePacket and routes + # it through the same _handle_packet path as UDP devices. Net effect: + # the Quest looks like any other label-producing device to the + # recorder, matcher, snapshot, and meta-sidecar code. + @app.websocket("/ws/quest") + async def ws_quest(websocket: WebSocket): + await websocket.accept() + client = (f"{websocket.client.host}:{websocket.client.port}" + if websocket.client else "unknown") + state.log_buffer.info("quest", f"connected: {client}") + frame_count = 0 + try: + while True: + payload = await websocket.receive_json() + try: + state.ingest_quest_packet(payload) + frame_count += 1 + except Exception as e: + # Per-frame errors shouldn't kill the socket -- a single + # malformed frame happens; the next one is usually fine. + state.log_buffer.warn( + "quest", f"ingest failed at frame {frame_count}: " + f"{type(e).__name__}: {e}") + except WebSocketDisconnect: + state.log_buffer.info( + "quest", f"disconnected: {client} after {frame_count} frames") + except Exception as e: + state.log_buffer.error( + "quest", f"socket error from {client}: " + f"{type(e).__name__}: {e}") + # ----- REST: devices ----- @app.get("/api/devices") async def list_devices(): return JSONResponse(state._snapshot()["devices"]) + # ----- REST: native V4 discovery ----- + + @app.get("/api/discovery") + async def list_discovery(): + """Known V4 sources + subscription state (auto-discover replaces the + manual bridge_subscriber.py).""" + return JSONResponse(state.discovery.snapshot() if state.discovery else []) + + class ProbeBody(BaseModel): + ip: str + cmd_port: Optional[int] = None # try 8001 then 8002 when omitted + + @app.post("/api/discovery/probe") + async def discovery_probe(body: ProbeBody): + """Manually probe an address (lab setup / device not beaconing).""" + if not state.discovery: + raise HTTPException(status_code=409, detail="discovery disabled") + dev = await asyncio.to_thread(state.discovery.probe, body.ip, body.cmd_port) + if dev is None: + raise HTTPException( + status_code=404, + detail=f"no V4 command server responded at {body.ip}") + return dev.to_snapshot() + + class DeviceIdBody(BaseModel): + device_id: str + + @app.post("/api/discovery/subscribe") + async def discovery_subscribe(body: DeviceIdBody): + if not state.discovery: + raise HTTPException(status_code=409, detail="discovery disabled") + ok = await asyncio.to_thread(state.discovery.subscribe, body.device_id) + if not ok: + raise HTTPException( + status_code=400, + detail=f"could not subscribe to {body.device_id} " + f"(unknown, unreachable, or list full)") + return {"device_id": body.device_id, "subscribed": True} + + @app.post("/api/discovery/unsubscribe") + async def discovery_unsubscribe(body: DeviceIdBody): + if not state.discovery: + raise HTTPException(status_code=409, detail="discovery disabled") + removed = state.discovery.unsubscribe(body.device_id) + return {"device_id": body.device_id, "unsubscribed": bool(removed)} + # ----- REST: recording ----- class StartRecordingBody(BaseModel): @@ -104,7 +241,8 @@ class StartRecordingBody(BaseModel): sensor_device_id: Optional[str] = None label_device_id: Optional[str] = None filename: Optional[str] = None - window_ms: int = 100 + # If None, AppState picks per-device-type (lask5=100, quest_hand=175). + window_ms: Optional[int] = None @app.post("/api/recording") async def start_recording(body: StartRecordingBody): @@ -168,6 +306,31 @@ async def download_capture(name: str): raise HTTPException(status_code=404, detail="Capture not found") return FileResponse(p, filename=p.name, media_type="text/csv") + class RevealBody(BaseModel): + # If empty/None -> just open captures_dir. Otherwise must be a + # capture name whitelisted by state.capture_path(). + name: Optional[str] = None + + @app.post("/api/reveal") + async def reveal_in_folder(body: RevealBody): + """Open the captures folder (and optionally highlight a specific + capture) in the OS file manager. Local-only convenience; the server + is intended for localhost use.""" + if body.name: + p = state.capture_path(body.name) + if p is None: + raise HTTPException(status_code=404, detail="Capture not found") + target = p + select = True + else: + target = state.captures_dir + select = False + try: + _reveal_path_in_file_manager(target, select_file=select) + except RuntimeError as e: + raise HTTPException(status_code=500, detail=str(e)) + return {"opened": str(target), "selected": select} + @app.delete("/api/captures/{name}") async def delete_capture(name: str): ok = state.delete_capture(name) @@ -358,14 +521,98 @@ async def delete_session_endpoint(session_id: str, unlink_captures: bool = True) raise HTTPException(status_code=404, detail="Session not found") return {"deleted": session_id} + # ----- REST: retroactive session<->capture linking ----- + # + # A capture made *outside* an active session can be added to one + # afterwards, and vice versa removed. This is the "I forgot to start a + # session before recording" recovery path. We update both: + # 1. the session JSON's `captures` list (authoritative) + # 2. the capture's meta sidecar (tag `session:` + auto.session_id) + # so the captures-panel filter, the past-sessions expansion, and any + # future export all agree on which session a capture belongs to. + + class LinkCapturesBody(BaseModel): + capture_names: list[str] # bulk add + + @app.post("/api/sessions/{session_id}/captures") + async def add_captures_to_session(session_id: str, body: LinkCapturesBody): + s = state.get_session(session_id) + if s is None: + raise HTTPException(status_code=404, detail="Session not found") + + tag = "session:" + session_id + added, skipped = [], [] + for name in body.capture_names: + if state.capture_path(name) is None: + skipped.append({"name": name, "reason": "capture not found"}) + continue + if name in s.get("captures", []): + skipped.append({"name": name, "reason": "already in session"}) + continue + try: + state.link_capture_to_session(session_id, name) + # Update the capture's meta so the tag-based filter + any + # future export sees this capture as part of the session. + meta = state.read_capture_meta(name) or {} + tags = list(meta.get("tags") or []) + if tag not in tags: + tags.append(tag) + state.write_capture_meta(name, { + "tags": tags, + "auto": {"session_id": session_id}, + }) + added.append(name) + except Exception as e: + skipped.append({"name": name, "reason": str(e)}) + + return { + "added": added, + "skipped": skipped, + "session": state.get_session(session_id), + } + + @app.delete("/api/sessions/{session_id}/captures/{capture_name}") + async def remove_capture_from_session(session_id: str, capture_name: str): + s = state.get_session(session_id) + if s is None: + raise HTTPException(status_code=404, detail="Session not found") + if capture_name not in s.get("captures", []): + raise HTTPException(status_code=404, detail="Capture not in session") + try: + state.unlink_capture_from_session(session_id, capture_name) + # Strip the session tag + clear auto.session_id, but ONLY for this + # session (leave any other 'session:xxx' tags alone -- though by + # the data model a capture should only ever belong to one session). + tag = "session:" + session_id + meta = state.read_capture_meta(capture_name) or {} + new_tags = [t for t in (meta.get("tags") or []) if t != tag] + state.write_capture_meta(capture_name, { + "tags": new_tags, + "auto": {"session_id": None}, + }) + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) + return { + "removed": capture_name, + "session": state.get_session(session_id), + } + return app def serve(host: str = "0.0.0.0", port: int = 8000, udp_port: int = 3141, captures_dir: Optional[str] = None, model_path: Optional[str] = None, - hand_target: Optional[tuple] = None): - """Run the web UI server (blocks).""" + hand_target: Optional[tuple] = None, + ssl_certfile: Optional[str] = None, + ssl_keyfile: Optional[str] = None): + """Run the web UI server (blocks). + + Pass ssl_certfile + ssl_keyfile to serve HTTPS -- required for the + WebXR /vr page since Quest Browser refuses hand-tracking on plain + HTTP. Generate certs locally with mkcert and install the root CA + on the headset (see README). + """ import uvicorn app = create_app( udp_port=udp_port, @@ -373,4 +620,5 @@ def serve(host: str = "0.0.0.0", port: int = 8000, udp_port: int = 3141, model_path=model_path, hand_target=hand_target, ) - uvicorn.run(app, host=host, port=port, log_level="info") + uvicorn.run(app, host=host, port=port, log_level="info", + ssl_certfile=ssl_certfile, ssl_keyfile=ssl_keyfile) diff --git a/pc/src/openmuscle/web/state.py b/pc/src/openmuscle/web/state.py index 4dde98a..1794ac5 100644 --- a/pc/src/openmuscle/web/state.py +++ b/pc/src/openmuscle/web/state.py @@ -22,9 +22,41 @@ import socket from openmuscle.data.storage import CaptureWriter -from openmuscle.protocol.schema import OpenMusclePacket +from openmuscle.protocol.schema import CURRENT_VERSION, OpenMusclePacket + + +# Canonical channel ordering for a single quest_hand joint. This is the +# coupling point between two places: +# - `AppState.ingest_quest_packet` packs joints into `data.values` in +# this order via `_flatten_quest_joint`. +# - `AppState._write_labels_schema` emits the column->(joint, channel) +# map using `_quest_label_column` which assumes this same order. +# Changing one without the other makes the labels-schema sidecar lie +# about what's in the CSV. Both callers route through this tuple + the +# two helpers below to make the coupling structural rather than two +# aspirational comments. +QUEST_JOINT_CHANNEL_ORDER = ("px", "py", "pz", "rx", "ry", "rz", "rw") + + +def _flatten_quest_joint(pos, rot): + """Pack one joint's (pos, rot) into the canonical channel order. + + `pos` is [x, y, z]; `rot` is a unit quaternion [x, y, z, w]. Returns a + flat list of `len(QUEST_JOINT_CHANNEL_ORDER)` floats. + """ + components = {"px": pos[0], "py": pos[1], "pz": pos[2], + "rx": rot[0], "ry": rot[1], "rz": rot[2], "rw": rot[3]} + return [components[ch] for ch in QUEST_JOINT_CHANNEL_ORDER] + + +def _quest_label_column(joint_index: int, channel_index: int) -> int: + """Index of the CSV column corresponding to (joint, channel) for a + quest_hand recording. Inverse view of the layout produced by + `_flatten_quest_joint` applied N times.""" + return joint_index * len(QUEST_JOINT_CHANNEL_ORDER) + channel_index from openmuscle.receiver.matcher import TemporalMatcher from openmuscle.receiver.udp_listener import UDPListener +from openmuscle.discovery import DiscoveryManager from openmuscle.web.inference import InferenceEngine from openmuscle.web.log_buffer import LogBuffer, install as install_log_handler @@ -146,6 +178,22 @@ class ActiveCapture: matcher: TemporalMatcher sensor_jsonl: Optional[IO] = None label_jsonl: Optional[IO] = None + # Path for a per-capture labels-schema sidecar. Written on the first + # label packet (lazy, like the CSV header) so the schema reflects + # whatever the device actually sent rather than what we expected. + # Only populated for label sources whose label width / structure is + # not derivable from device_type alone -- e.g. quest_hand. + labels_schema_path: Optional[Path] = None + labels_schema_written: bool = False + # Label-width lock. quest_hand frames can vary in joint count when hand + # tracking is partial; writing those varying lengths straight to the CSV + # produces ragged rows (different column count per row) that break + # pandas.read_csv and corrupt the capture for training. We lock the + # expected width at the first label packet (same width the labels-schema + # sidecar describes) and pad/truncate every paired row to it, so the CSV + # stays rectangular and consistent with the schema. None until locked. + locked_label_count: Optional[int] = None + label_width_mismatch_count: int = 0 # Stats surfaced in the WS snapshot sensor_frames_seen: int = 0 label_packets_seen: int = 0 @@ -173,11 +221,15 @@ class AppState: def __init__(self, udp_port: int = 3141, captures_dir: Path | None = None, model_path: Optional[str] = None, - hand_target: Optional[tuple] = None): + hand_target: Optional[tuple] = None, + enable_discovery: bool = True): """ Args: udp_port: incoming device telemetry port (FlexGrid + LASK5) captures_dir: where recorded CSVs go + enable_discovery: run native V4 discovery + auto-subscribe (replaces + the interim bridge_subscriber.py). Set False to bind the + UDP port only and rely on an external subscriber/bridge. model_path: optional path to a trained model .pkl. When set, every FlexGrid packet is run through the model and its predictions populate the WS snapshot's `inference` @@ -199,7 +251,17 @@ def __init__(self, udp_port: int = 3141, captures_dir: Path | None = None, self.log_buffer.info("server", "AppState init udp_port={} captures_dir={}".format( udp_port, str(self.captures_dir))) - self.listener = UDPListener(port=udp_port) + # Native V4 discovery + subscribe. Discovers announcing sources, keeps a + # TCP subscription + 1 Hz heartbeat to each, and re-probes cached devices + # on startup (a device's beacon goes silent once any hub subscribes, so + # the cache is how we recover one another hub already holds). Replaces + # pc/bridge_subscriber.py. The listener hands announce beacons straight + # to it; subscribed devices' sensor frames flow through the normal queue. + self.discovery: Optional[DiscoveryManager] = ( + DiscoveryManager(udp_port=udp_port, auto_subscribe=True) + if enable_discovery else None) + announce_handler = self.discovery.on_announce if self.discovery else None + self.listener = UDPListener(port=udp_port, announce_handler=announce_handler) self.devices: dict[str, DeviceInfo] = {} self.recording: Optional[ActiveCapture] = None # At most one active session at a time. Recordings made while a @@ -241,10 +303,16 @@ def start(self): if self._started: return self.listener.start() + if self.discovery: + # Re-probe cached devices in the background so we recover sources + # whose beacon is silent (held by another hub) at startup. + self.discovery.recover_cache() self._started = True def stop(self): self.listener.stop() + if self.discovery: + self.discovery.stop() if self.recording: self.stop_recording() if self._hand_sock is not None: @@ -307,6 +375,77 @@ def _handle_packet(self, pkt: OpenMusclePacket): if self.hand_target: self._forward_to_hand(pred) + def ingest_quest_packet(self, payload: dict) -> None: + """Synthesize an OpenMusclePacket from a Quest WebSocket frame and + route it through the standard packet path. + + From the recorder's perspective the Quest is just another device: + once we build an OpenMusclePacket with `device_type="quest_hand"` + and hand it to `_handle_packet`, the DeviceInfo registry, the + TemporalMatcher, the JSONL sidecars, and the WS snapshot all + treat it identically to LASK5. This is why the Quest never needs + to learn UDP -- the JS in the headset only has WebSocket. + + Expected payload shape (one hand for v1, per the team's + "FlexGrid-arm only" decision): + + { + "device_id": "quest-01", # optional + "ts": 12345, # device-local ms (optional) + "handedness": "left" | "right", # which hand this frame is for + "joints": [ + {"name": "wrist", "pos": [x,y,z], "rot": [x,y,z,w], "radius": 0.02}, + {"name": "thumb-metacarpal", "pos": [...], "rot": [...]}, + ... 26 entries total, in OpenXR canonical order + ], + "meta": {...} # optional, e.g. tracking confidence + } + + We flatten joints into `data.values = [px,py,pz, rx,ry,rz,rw] * N` + so it matches LASK5's `data.values` convention (the recorder pulls + from this field). The structured per-joint form is preserved under + `data.hands` for the JSONL sidecar -- if you later want to know + WHICH joint a column corresponds to, the sidecar tells you, while + the trainable CSV stays a plain matrix of floats. + + Empty payloads (e.g. headset reports tracking lost this frame) + are silently dropped -- we want gaps in the data, not zero rows + that would mislead the model. + """ + joints = payload.get("joints") or [] + if not joints: + return + + flat: list[float] = [] + joint_names: list[str] = [] + for j in joints: + pos = j.get("pos") or [0.0, 0.0, 0.0] + rot = j.get("rot") or [0.0, 0.0, 0.0, 1.0] + # Route through _flatten_quest_joint so the channel order is + # taken from QUEST_JOINT_CHANNEL_ORDER -- same source-of-truth + # the labels-schema sidecar reads via _quest_label_column. + flat.extend(float(v) for v in _flatten_quest_joint(pos, rot)) + joint_names.append(j.get("name", "")) + + pkt = OpenMusclePacket( + version=CURRENT_VERSION, + device_type="quest_hand", + device_id=payload.get("device_id") or "quest-01", + timestamp_ms=int(payload.get("ts") or 0), + data={ + "values": flat, + "handedness": payload.get("handedness") or "unknown", + "joint_names": joint_names, + "hands": { + "handedness": payload.get("handedness") or "unknown", + "joints": joints, + }, + }, + metadata=payload.get("meta") or {}, + receive_time=time.time(), + ) + self._handle_packet(pkt) + # Flush JSONL sidecars every N frames to bound crash-loss to ~3 s of # data while keeping syscalls ~50× cheaper than line-buffered writes. # At 33 Hz sensor + 25 Hz label rates, 100 ≈ 3 s. @@ -323,6 +462,12 @@ def _record_packet(self, pkt: OpenMusclePacket): rec.matcher.add_label(pkt) rec.label_packets_seen += 1 self._write_jsonl(rec.label_jsonl, pkt) + # Lazy: emit the labels-schema sidecar on the first label packet + # for label sources whose column layout is opaque from device_type + # alone. v1: quest_hand only. The sidecar gives consumers a map + # from the CSV's label_0..label_N columns back to (joint, channel). + if rec.labels_schema_path is not None and not rec.labels_schema_written: + self._write_labels_schema(rec, pkt) # Bounded crash-loss flush if (rec.label_packets_seen % self.JSONL_FLUSH_EVERY == 0 and rec.label_jsonl is not None): @@ -361,6 +506,25 @@ def _record_packet(self, pkt: OpenMusclePacket): rec.matched_count += 1 label_values = list(matched.data.get("values", [])) + # Guarantee a rectangular CSV. If the label width was locked (quest_hand) + # and this matched label has a different length, pad with zeros or + # truncate to the locked width. Without this, variable-length payloads + # (partial hand tracking, or a misbehaving client) would write ragged + # rows that break pandas.read_csv and corrupt the whole capture. The + # locked width matches the labels-schema sidecar, so consumers can + # still map every column to a (joint, channel). NB: a well-behaved + # client sends a fixed-length array every frame, so this rarely fires; + # the mismatch counter surfaces it in the stop-recording stats + log + # if it does. + if rec.locked_label_count is not None and label_values: + n = rec.locked_label_count + if len(label_values) != n: + rec.label_width_mismatch_count += 1 + if len(label_values) < n: + label_values = list(label_values) + [0.0] * (n - len(label_values)) + else: + label_values = list(label_values[:n]) + # Flatten as-sent [cols][rows] matrix row-major. Header in CaptureWriter # is R0C0..R0Cn, R1C0.., so iterating rows-then-cols here keeps the # column meaning correct (cf. the col-major bug we fixed in 245cb8f). @@ -369,6 +533,78 @@ def _record_packet(self, pkt: OpenMusclePacket): flat = [mat[c][r] for r in range(rows) for c in range(cols)] rec.writer.write_row(pkt.receive_time, flat, label_values) + def _write_labels_schema(self, rec: "ActiveCapture", pkt: OpenMusclePacket) -> None: + """Emit the per-capture labels-schema sidecar. + + Maps the CSV's label_0..label_N columns back to the underlying + (joint, channel) coordinates for a quest_hand recording. Without + this a wide-label CSV is opaque -- you'd have to know the joint + ordering by convention. With it, any consumer can deserialize + label columns into named joint poses. + + TODO(wrist-relative-labels): joint positions are stored in absolute + world coordinates as captured by the headset, so a model trained + on them learns to predict positions where the recordings happened + to be -- generally not where the user is at inference time. The + VR ghost-hand viz works around this by anchoring predicted joints + to the real wrist each frame, but the right long-term fix is to + subtract the wrist position (and optionally rotate into the + wrist's frame) before writing, and reverse the transform at + inference. This changes the CSV semantics, so it deserves its + own scope. Track in the OpenMuscle-Software repo issue list + ("Wrist-relative label coordinates for portable quest_hand + models") before any model that needs to generalize across + capture-locations. + """ + if rec.labels_schema_path is None: + return + joint_names = list(pkt.data.get("joint_names") or []) + handedness = pkt.data.get("handedness") or "unknown" + # Pull the channel order from the module-level constant so the schema + # is guaranteed consistent with how ingest_quest_packet flattened + # the values into the CSV (see QUEST_JOINT_CHANNEL_ORDER docstring). + ordering = list(QUEST_JOINT_CHANNEL_ORDER) + n_floats = len(ordering) + # Build the explicit column->(joint, channel) map so consumers + # don't have to re-derive it from joint order. + columns = [] + for ji, jn in enumerate(joint_names): + for ci, ch in enumerate(ordering): + columns.append({ + "name": f"label_{_quest_label_column(ji, ci)}", + "joint": jn, + "channel": ch, + }) + n_label_columns = len(joint_names) * n_floats + # Lock the CSV label width to what this first label packet described, + # so _record_packet can pad/truncate every paired row to match the + # schema and keep the CSV rectangular. (Well-behaved clients send a + # fixed-length joint array every frame, so this almost never triggers + # a pad/truncate -- it's a safety net against partial-tracking frames + # and any client that emits a variable-length payload.) + rec.locked_label_count = n_label_columns + schema = { + "label_source": "quest_hand", + "handedness": handedness, + "ordering": ordering, + "floats_per_joint": n_floats, + "n_joints": len(joint_names), + "n_label_columns": n_label_columns, + "joint_names": joint_names, + "columns": columns, + } + try: + with open(rec.labels_schema_path, "w") as f: + json.dump(schema, f, indent=2) + rec.labels_schema_written = True + self.log_buffer.info( + "recording", + f"labels-schema written: {rec.labels_schema_path.name} " + f"({len(joint_names)} joints, {len(columns)} columns)") + except OSError as e: + self.log_buffer.warn( + "recording", f"labels-schema write failed: {e}") + @staticmethod def _write_jsonl(stream: Optional[IO], pkt: OpenMusclePacket): """Append one packet as a JSONL line. No-op if stream is None. @@ -400,24 +636,22 @@ def _write_jsonl(stream: Optional[IO], pkt: OpenMusclePacket): def _forward_to_hand(self, pred: list): """Send the prediction to the robot hand as a `PC,...` UDP datagram. - Builds 5 servo angles in 0..179 from the 4 piston predictions (assumed - normalized 0..1; clamped) plus the most recent LASK5 joystick X as the - 5th. The hand's `'PC'` device config uses linear 0..179 -> 0..179 - mapping, so values land directly on servo angles. + Builds 5 servo angles in 0..179. Channel order on the hand + (FINGER_CHANNELS = [1, 3, 5, 7, 9]) is anatomically: + channel 1 -> thumb + channel 3 -> index + channel 5 -> middle + channel 7 -> ring + channel 9 -> pinky + + The LASK5 has 4 pistons (the 4 closing fingers) and a joystick. + We map joystick X -> thumb, pistons 0..3 -> index..pinky. + The hand's 'PC' device config uses linear 0..179 -> 0..179, so values + land directly on servo angles. """ - # Pistons -> 0..179, assuming model output is normalized 0..1. - # Anything else gets clamped, which is the right failure mode -- - # bracelet finger goes to extreme rather than 4000-degree angle. - angles = [] - for v in pred[:4]: - try: - v = max(0.0, min(1.0, float(v))) - except Exception: - v = 0.0 - angles.append(int(v * 179)) - - # 5th finger = joystick X from the most recent LASK5 packet. Range - # 0..4095 -> 0..179. Default to center (90) if no LASK5 has been seen. + # Thumb (channel 1) = joystick X from the most recent LASK5 packet. + # Range 0..4095 -> 0..179. Default to center (90) if no LASK5 has been + # seen yet (so the thumb sits in a neutral pose instead of slamming open). joy_x = None for d in self.devices.values(): if d.device_type == "lask5" and d.last_joystick: @@ -425,23 +659,57 @@ def _forward_to_hand(self, pred: list): if isinstance(jx, (int, float)): joy_x = jx break - if joy_x is None: - angles.append(90) - else: - angles.append(max(0, min(179, int((joy_x / 4095.0) * 179)))) + thumb_angle = 90 if joy_x is None else max(0, min(179, int((joy_x / 4095.0) * 179))) + + # Index..pinky (channels 3, 5, 7, 9) from pistons 0..3. + # Model output is assumed normalized 0..1; anything else gets clamped, + # which is the right failure mode -- finger goes to extreme rather + # than commanding a 4000-degree servo angle. + finger_angles = [] + for v in pred[:4]: + try: + v = max(0.0, min(1.0, float(v))) + except Exception: + v = 0.0 + finger_angles.append(int(v * 179)) + + # The hand's 'PC' device config has reverse=False, but the LASK5's + # native ESPNow path uses the 'default' / 'L5' config (reverse=True) + # which flips the piston order before mapping to FINGER_CHANNELS. + # To match that mapping from our PC path, we reverse the pistons + # ourselves: P1 -> index, P2 -> middle, P3 -> ring, P4 -> pinky. + # (Documented in DEVICES of the hand firmware, archived 2026-05-14.) + angles = [thumb_angle] + finger_angles[::-1] # [thumb, P4, P3, P2, P1] # Build the CSV the hand expects: 'PC,a1,a2,a3,a4,a5' payload = ("PC," + ",".join(str(a) for a in angles)).encode("utf-8") + # Rate-limited log so we can SEE whether forwarding is working. + # First time + every 500th packet: log success/failure to the buffer + # so the operator can debug without strace. + self._hand_forward_count = getattr(self, "_hand_forward_count", 0) + 1 + log_now = (self._hand_forward_count == 1 + or self._hand_forward_count % 500 == 0) try: if self._hand_sock is None: self._hand_sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) self._hand_sock.setblocking(False) - self._hand_sock.sendto(payload, self.hand_target) - except Exception: - # Non-fatal: hand might be offline / on a different subnet. - # We don't spam logs since this fires per FlexGrid packet. - pass + n = self._hand_sock.sendto(payload, self.hand_target) + if log_now: + self.log_buffer.info("inference", + "hand forward #{}: sent {} bytes to {}:{} -> {!r}".format( + self._hand_forward_count, n, + self.hand_target[0], self.hand_target[1], + payload.decode("utf-8", errors="replace"))) + except Exception as e: + # Always log the first failure so the operator sees it; rate-limit + # subsequent ones (every 500) so we don't spam. + if log_now or not getattr(self, "_hand_forward_error_logged", False): + self.log_buffer.warn("inference", + "hand forward #{} FAILED: {} ({!r}) -> target={}".format( + self._hand_forward_count, type(e).__name__, str(e), + self.hand_target)) + self._hand_forward_error_logged = True async def _broadcast_latest_frames(self): """Push the latest frame for each device to all WS clients.""" @@ -498,6 +766,11 @@ def _snapshot(self) -> dict: "sensor_frames_seen": r.sensor_frames_seen, "label_packets_seen": r.label_packets_seen, "match_rate": round(r.match_rate, 3), + # Rows that had to be padded/truncated to the locked label + # width (variable-length label frames, e.g. partial hand + # tracking). Surfaced live so the in-VR header can warn the + # operator that joints are dropping mid-capture. + "label_width_mismatch": r.label_width_mismatch_count, } return { "type": "tick", @@ -505,6 +778,11 @@ def _snapshot(self) -> dict: "recording": rec, "inference": self._inference_snapshot(), "active_session": self.active_session, + # Native V4 discovery: known sources + subscription state. Empty + # list when discovery is disabled. Devices that are subscribed and + # streaming also appear in `devices` above (via their frames); this + # adds the ones that are known-but-silent plus per-device sub status. + "discovery": self.discovery.snapshot() if self.discovery else [], } def _inference_snapshot(self) -> dict: @@ -575,18 +853,34 @@ def _auto_pick_sensor(self) -> Optional[str]: return d.device_id return None + # Order in which auto-pick prefers label-producing device types when + # the operator doesn't pick one explicitly. Quest first because it's + # the richer ground-truth source -- if both are connected during a + # comparison session, we want the wider label vector by default. + AUTO_LABEL_TYPE_PREFERENCE = ("quest_hand", "lask5") + def _auto_pick_label(self) -> Optional[str]: - """First connected label-producing device (LASK5), if any.""" - for d in self.devices.values(): - if d.device_type == "lask5": - return d.device_id + """First connected label-producing device, by type preference.""" + for preferred_type in self.AUTO_LABEL_TYPE_PREFERENCE: + for d in self.devices.values(): + if d.device_type == preferred_type: + return d.device_id return None + # Default match windows per label-device family. Quest WebXR has higher + # end-to-end latency than LASK5's ESP-NOW path (browser -> WS -> server), + # so a tighter window would reject too many sensor frames as unpaired. + DEFAULT_WINDOW_MS_BY_TYPE = { + "lask5": 100, + "quest_hand": 175, + } + DEFAULT_WINDOW_MS_FALLBACK = 100 + def start_recording(self, sensor_device_id: Optional[str] = None, label_device_id: Optional[str] = None, filename: Optional[str] = None, - window_ms: int = 100, + window_ms: Optional[int] = None, label_count: int = 4) -> ActiveCapture: """Start a paired recording. @@ -599,9 +893,13 @@ def start_recording(self, record sensor frames only (the paired CSV will have no label columns). filename: CSV name. JSONL sidecars derived from it. - window_ms: temporal match window in milliseconds (default 100). + window_ms: temporal match window in milliseconds. If None, picked + per-device-type from DEFAULT_WINDOW_MS_BY_TYPE + (lask5=100, quest_hand=175, else 100). label_count: how many label_* columns to write per row (default 4 - for the standard LASK5 piston count). + for the standard LASK5 piston count). Ignored when + the label device is quest_hand -- in that case the + writer infers from the first packet's values length. """ if self.recording is not None: raise RuntimeError("Already recording -- stop the current capture first") @@ -632,6 +930,22 @@ def start_recording(self, effective_label_count = label_count if label_device_id else 0 + # Quest hand tracking sends a wide joint vector whose width depends on + # the headset / WebXR implementation (Quest 3S = 26 joints * 7 floats = + # 182 per hand). Rather than hardcode it, pass None so CaptureWriter + # derives the column count from the first label packet. + label_dev_for_width = self.devices.get(label_device_id) if label_device_id else None + label_device_type = label_dev_for_width.device_type if label_dev_for_width else None + if label_device_type == "quest_hand": + effective_label_count = None + + # Pick the match window: explicit arg wins; otherwise per-device-type + # default (Quest needs a wider window than LASK5 because WebXR + # latency is higher than ESP-NOW). + if window_ms is None: + window_ms = self.DEFAULT_WINDOW_MS_BY_TYPE.get( + label_device_type or "", self.DEFAULT_WINDOW_MS_FALLBACK) + # Build paths name = filename or f"capture_{int(time.time())}.csv" if not name.endswith(".csv"): @@ -641,6 +955,12 @@ def start_recording(self, stem = csv_path.with_suffix("") # data/raw/merged/foo (no .csv) sensor_sidecar = stem.with_suffix(".sensor.jsonl") label_sidecar = stem.with_suffix(".label.jsonl") + # Labels-schema sidecar is only written for label sources whose + # column meaning isn't obvious from device_type alone -- v1: Quest. + labels_schema_sidecar: Optional[Path] = ( + Path(str(stem) + ".labels.schema.json") + if label_device_type == "quest_hand" else None + ) writer = CaptureWriter( output_path=str(csv_path), @@ -677,6 +997,7 @@ def start_recording(self, matcher=matcher, sensor_jsonl=sensor_stream, label_jsonl=label_stream, + labels_schema_path=labels_schema_sidecar, ) self.log_buffer.info("recording", "started: {} (sensor={}, label={}, window={}ms)".format( @@ -692,6 +1013,7 @@ def start_recording(self, auto = { "sensor_device_id": sensor_device_id, "label_device_id": label_device_id, + "label_source": label_device_type, # "lask5" | "quest_hand" | None "window_ms": window_ms, "sensor_shape": [sensor_dev.rows, sensor_dev.cols], "started_at": self.recording.started_at, @@ -747,6 +1069,17 @@ def stop_recording(self) -> Optional[dict]: "stopped: {} -- {} matched / {} sensor frames ({}%), {}s".format( rec.path.name, rec.matched_count, rec.sensor_frames_seen, round(rec.match_rate * 100, 1), round(rec.duration_s, 1))) + # If any rows had to be padded/truncated to the locked label width, + # call it out -- it means the label source sent variable-length + # frames (e.g. partial hand tracking), and those rows have some + # zero-filled joint columns. + if rec.label_width_mismatch_count: + self.log_buffer.warn("recording", + "{}: {} row(s) padded/truncated to locked label width {} " + "(variable-length label frames -- some joint columns are " + "zero-filled)".format( + rec.path.name, rec.label_width_mismatch_count, + rec.locked_label_count)) result = { "filename": rec.path.name, "rows": rec.row_count, @@ -759,10 +1092,14 @@ def stop_recording(self) -> Optional[dict]: "unpaired_sensor": rec.unpaired_sensor_count, "sensor_frames_seen": rec.sensor_frames_seen, "label_packets_seen": rec.label_packets_seen, + "label_width_mismatch": rec.label_width_mismatch_count, "match_rate": round(rec.match_rate, 3), "sidecars": { "sensor": str(rec.path.with_suffix("")) + ".sensor.jsonl", "label": (str(rec.path.with_suffix("")) + ".label.jsonl") if rec.label_jsonl else None, + "labels_schema": (str(rec.labels_schema_path) + if (rec.labels_schema_path and rec.labels_schema_written) + else None), }, } self.recording = None @@ -819,7 +1156,7 @@ def delete_capture(self, name: str) -> bool: p.unlink() # Also delete sidecars if present stem = p.with_suffix("") - for suffix in (".sensor.jsonl", ".label.jsonl", ".meta.json"): + for suffix in (".sensor.jsonl", ".label.jsonl", ".meta.json", ".labels.schema.json"): sidecar = Path(str(stem) + suffix) if sidecar.exists(): try: diff --git a/pc/src/openmuscle/web/static/app.js b/pc/src/openmuscle/web/static/app.js index 474637c..90be312 100644 --- a/pc/src/openmuscle/web/static/app.js +++ b/pc/src/openmuscle/web/static/app.js @@ -18,10 +18,31 @@ const selStatus = document.getElementById('captures-sel-status'); const checkAll = document.getElementById('captures-check-all'); const modelsBody = document.getElementById('models-body'); const modelsCount = document.getElementById('models-count'); +const openFolderBtn = document.getElementById('captures-open-folder'); + +// Ask the server to open the captures folder in the OS file manager. +// If `name` is given, highlight that capture file inside the folder. +async function revealCaptureFolder(name) { + try { + const r = await fetch('/api/reveal', { + method: 'POST', + headers: {'Content-Type': 'application/json'}, + body: JSON.stringify({name: name || null}), + }); + if (!r.ok) throw new Error(await readError(r)); + } catch (e) { + alert('Could not open folder: ' + e.message); + } +} + +if (openFolderBtn) { + openFolderBtn.onclick = () => revealCaptureFolder(null); +} // Per-user pick preferences that survive a refresh const STORE_SENSOR = 'om.sensor_device_id'; const STORE_LABEL = 'om.label_device_id'; +const STORE_HAND = 'om.hand_target'; // last successfully-applied "host:port" — auto-restored on next launch // Set of capture filenames currently checked in the table const selectedCaptures = new Set(); @@ -41,6 +62,11 @@ function connectWS() { ws.onopen = () => { wsStatus.textContent = 'connected'; wsStatus.className = 'badge online'; + // Re-arm the hand-target auto-restore: every fresh WS connect (which + // includes server restarts) gets a chance to re-apply the saved hand + // target. Otherwise the operator has to remember to click Apply + // after every `openmuscle web` restart. + handTargetRestoreAttempted = false; }; ws.onclose = () => { wsStatus.textContent = 'disconnected'; @@ -72,6 +98,7 @@ function handleTick(msg) { } renderActiveSession(); renderDevices(); + renderDiscovery(msg.discovery || []); renderRecordPickers(); renderRecording(); const dev = selectedDevice(); @@ -84,6 +111,175 @@ function handleTick(msg) { const lask = lastDevices.find(d => d.device_type === 'lask5'); renderLask(lask); renderInference(msg.inference); + // Comparator + top-bar pipeline strip are Studio-shell additions. + // They derive everything from the per-tick snapshot, so they update + // in lockstep with the underlying bars and the WS message. + renderResiduals(lask, msg.inference); + renderPipelinePills(msg, lask); + // quest_hand 3D viewer: when a hand label source is streaming, swap the + // LASK5 piston comparator for a live 3D hand (the pistons are zeros for + // a hand source). No-op when no quest_hand device is present. + renderHandViewer(lastDevices.find(d => d.device_type === 'quest_hand'), + msg.inference); +} + +// ---------- quest_hand 3D viewer ---------- + +// Drives the Three.js hand viewer (window.OMHandViewer, loaded as a module). +// Shows the REAL captured hand from the live quest_hand device's flat joint +// `values`, plus the model's PREDICTED hand from inference.piston_values when +// a quest-trained model (>= 25 joints * 7 floats) is running. Toggles the +// .hand-mode class on .comparator so CSS hides the LASK5 pistons in favor of +// the viewer. +function renderHandViewer(questDev, inference) { + const comparator = document.querySelector('.comparator'); + const viewerReady = window.OMHandViewer && window.OMHandViewer.isReady; + if (!questDev) { + if (comparator) comparator.classList.remove('hand-mode'); + if (viewerReady && window.OMHandViewer.isReady()) window.OMHandViewer.setVisible(false); + return; + } + // Lazy-init the viewer on first quest_hand sighting (the module may still + // be loading right at page open; guard with isReady). + if (window.OMHandViewer && !window.OMHandViewer.isReady()) { + const el = document.getElementById('hand-viewer-canvas'); + if (el) window.OMHandViewer.init(el); + } + if (!(window.OMHandViewer && window.OMHandViewer.isReady())) return; + + if (comparator) comparator.classList.add('hand-mode'); + window.OMHandViewer.setVisible(true); + + const realFlat = Array.isArray(questDev.values) ? questDev.values : null; + // Predicted hand: only when the live model emits a full hand vector. + let predFlat = null; + const pv = inference && inference.piston_values; + if (Array.isArray(pv) && pv.length >= 25 * 7) predFlat = pv; + window.OMHandViewer.update(realFlat, predFlat); + + // Reuse the existing GT meta slot to label the hand source. + const gtMeta = document.getElementById('lask-meta'); + if (gtMeta) { + const hz = (typeof questDev.hz === 'number') ? questDev.hz.toFixed(0) : '0'; + const nJoints = realFlat ? Math.floor(realFlat.length / 7) : 0; + gtMeta.textContent = `Quest hand · ${nJoints} joints · ${hz} Hz`; + } +} + +// ---------- native V4 discovery (Sources rail) ---------- + +let _discoveryProbeWired = false; + +function renderDiscovery(discovery) { + const list = document.getElementById('discovery-list'); + const count = document.getElementById('discovery-count'); + if (!list) return; + if (!_discoveryProbeWired) wireDiscoveryProbe(); + + const subs = discovery.filter(d => d.subscribed).length; + if (count) count.textContent = discovery.length + ? `${subs}/${discovery.length} subscribed` : ''; + + if (!discovery.length) { + list.innerHTML = '
  • No V4 sources discovered yet…
    • '; + return; + } + list.innerHTML = discovery.map(d => { + // State badge: subscribed (green) / error (red) / known (grey). + let stateCls = 'known', stateTxt = 'known'; + if (d.subscribed) { stateCls = 'subscribed'; stateTxt = 'subscribed'; } + else if (d.sub_error) { stateCls = 'err'; stateTxt = 'error'; } + const btnTxt = d.subscribed ? 'Unsubscribe' : 'Subscribe'; + const btnAct = d.subscribed ? 'unsubscribe' : 'subscribe'; + const age = (d.age_s != null) ? `${d.age_s.toFixed(0)}s ago` : ''; + const errLine = d.sub_error + ? `
    ${escapeHtml(d.sub_error)}
    ` + : ''; + return ` +
  • +
    + ${escapeHtml(d.device_id)} + ${stateTxt} +
    +
    + ${escapeHtml(d.device_type)} + ${escapeHtml(d.ip)}:${d.cmd_port} + via ${escapeHtml(d.source)} + ${age} +
    + ${errLine} + +
    • `; + }).join(''); + + list.querySelectorAll('.src-btn').forEach(btn => { + btn.onclick = async (e) => { + e.stopPropagation(); + const id = btn.dataset.id; + const act = btn.dataset.act; + btn.disabled = true; + btn.textContent = act === 'subscribe' ? 'Subscribing…' : 'Unsubscribing…'; + try { + const res = await fetch(`/api/discovery/${act}`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ device_id: id }), + }); + if (!res.ok) { + const err = await res.json().catch(() => ({})); + setProbeMsg(err.detail || `${act} failed`, true); + } + } catch (err) { + setProbeMsg(String(err), true); + } + // Next WS tick re-renders the true state; no manual refresh needed. + }; + }); +} + +function wireDiscoveryProbe() { + const form = document.getElementById('discovery-probe-form'); + const input = document.getElementById('discovery-probe-ip'); + if (!form || !input) return; + _discoveryProbeWired = true; + form.onsubmit = async (e) => { + e.preventDefault(); + const raw = input.value.trim(); + if (!raw) return; + // Accept "ip" or "ip:port". + let ip = raw, cmd_port = null; + if (raw.includes(':')) { + const parts = raw.split(':'); + ip = parts[0]; + const p = parseInt(parts[1], 10); + if (!isNaN(p)) cmd_port = p; + } + setProbeMsg(`probing ${ip}…`, false); + try { + const res = await fetch('/api/discovery/probe', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(cmd_port ? { ip, cmd_port } : { ip }), + }); + if (res.ok) { + const d = await res.json(); + setProbeMsg(`found ${d.device_id} (${d.device_type})`, false); + input.value = ''; + } else { + const err = await res.json().catch(() => ({})); + setProbeMsg(err.detail || `no V4 source at ${ip}`, true); + } + } catch (err) { + setProbeMsg(String(err), true); + } + }; +} + +function setProbeMsg(text, isError) { + const el = document.getElementById('discovery-probe-msg'); + if (!el) return; + el.textContent = text; + el.classList.toggle('err', !!isError); } // ---------- device list ---------- @@ -370,6 +566,7 @@ function renderActiveSession() { ${armBit}${subj} · ${s.capture_count || 0} captures · ${formatUptime(dur)}${gestures}
    +
    @@ -377,6 +574,8 @@ function renderActiveSession() { ${s.notes ? `
    ${escapeHtml(s.notes)}
    ` : ''} `; document.getElementById('session-end-btn').onclick = endSession; + const addBtn = document.getElementById('active-session-add-btn'); + if (addBtn) addBtn.onclick = () => openLinkModal(activeSession); sessionStartBtn.disabled = true; sessionStartBtn.title = 'End the current session before starting a new one'; capturesFilterLabel.textContent = `· filtered to ${s.name || s.id}`; @@ -400,6 +599,137 @@ async function refreshPastSessions() { } catch (e) { /* best-effort */ } } +// Sessions whose capture list is currently expanded in the UI. Persisted +// across re-renders (refreshPastSessions can fire on its own) so a poll +// doesn't collapse what the user just opened. +const expandedSessions = new Set(); + +// ---------- Add-captures-to-session picker modal ---------- +// +// Lets the operator retroactively assign past recordings (made without an +// active session) to a session. The picker shows every capture NOT +// currently linked to the target session, with checkboxes for bulk add. +// +// Wires up: +// - "+ Add captures" button in each past-session card +// - "×" remove button on each capture in the expanded view + +const linkModal = document.getElementById('link-modal'); +const linkSessionName = document.getElementById('link-session-name'); +const linkCaptureList = document.getElementById('link-capture-list'); +const linkAddBtn = document.getElementById('link-add-btn'); +let linkSessionId = null; // current session being edited +const linkSelected = new Set(); // capture names currently checked + +function openLinkModal(session) { + linkSessionId = session.id; + linkSelected.clear(); + linkSessionName.textContent = session.name || session.id; + linkAddBtn.disabled = true; + linkAddBtn.textContent = 'Add 0 captures'; + linkCaptureList.innerHTML = '
    Loading captures…
    '; + linkModal.classList.add('open'); + linkModal.setAttribute('aria-hidden', 'false'); + + // Fetch the full capture list, filter out ones already in this session. + fetch('/api/captures') + .then(r => r.ok ? r.json() : Promise.reject('fetch failed')) + .then(list => { + const alreadyLinked = new Set(session.captures || []); + const candidates = list.filter(c => !alreadyLinked.has(c.name)); + if (!candidates.length) { + linkCaptureList.innerHTML = '
    All captures are already in this session.
    '; + return; + } + // Render rows with checkbox + name + meta summary + (if linked + // to a different session) an annotation so the operator doesn't + // accidentally yank a capture out of another session. + linkCaptureList.innerHTML = candidates.map(c => { + const meta = c.meta || {}; + const otherSession = (meta.tags || []).find(t => t.startsWith('session:')); + const otherNote = otherSession + ? `⚠ ${escapeHtml(otherSession)}` + : ''; + const kb = (c.size_bytes / 1024).toFixed(1); + return ``; + }).join(''); + linkCaptureList.querySelectorAll('input[type=checkbox]').forEach(cb => { + cb.onchange = () => { + if (cb.checked) linkSelected.add(cb.dataset.name); + else linkSelected.delete(cb.dataset.name); + const n = linkSelected.size; + linkAddBtn.disabled = (n === 0); + linkAddBtn.textContent = `Add ${n} capture${n === 1 ? '' : 's'}`; + }; + }); + }) + .catch(err => { + linkCaptureList.innerHTML = '
    Could not load captures.
    '; + console.warn('link picker fetch:', err); + }); +} + +function closeLinkModal() { + linkModal.classList.remove('open'); + linkModal.setAttribute('aria-hidden', 'true'); + linkSessionId = null; + linkSelected.clear(); +} + +linkModal.querySelectorAll('[data-close]').forEach(el => { + el.addEventListener('click', closeLinkModal); +}); +document.addEventListener('keydown', e => { + if (e.key === 'Escape' && linkModal.classList.contains('open')) closeLinkModal(); +}); + +linkAddBtn.onclick = async () => { + if (!linkSessionId || linkSelected.size === 0) return; + linkAddBtn.disabled = true; + linkAddBtn.textContent = 'Adding…'; + try { + const r = await fetch(`/api/sessions/${encodeURIComponent(linkSessionId)}/captures`, { + method: 'POST', + headers: {'Content-Type': 'application/json'}, + body: JSON.stringify({capture_names: [...linkSelected]}), + }); + if (!r.ok) throw new Error(await readError(r)); + const result = await r.json(); + if ((result.skipped || []).length) { + // Surface skips inline -- e.g. "already in another session" + console.warn('some captures skipped:', result.skipped); + } + closeLinkModal(); + await refreshPastSessions(); + await refreshCaptures(); + } catch (e) { + alert('Add failed: ' + (e.message || e)); + linkAddBtn.disabled = false; + const n = linkSelected.size; + linkAddBtn.textContent = `Add ${n} capture${n === 1 ? '' : 's'}`; + } +}; + +async function removeCaptureFromSession(sessionId, captureName) { + if (!confirm(`Remove ${captureName} from this session?\n(The capture file itself stays — just the link is cleared.)`)) return; + try { + const r = await fetch( + `/api/sessions/${encodeURIComponent(sessionId)}/captures/${encodeURIComponent(captureName)}`, + {method: 'DELETE'} + ); + if (!r.ok) throw new Error(await readError(r)); + await refreshPastSessions(); + await refreshCaptures(); + } catch (e) { + alert('Remove failed: ' + (e.message || e)); + } +} + function renderPastSessions() { if (!pastSessions.length) { pastSessionsList.innerHTML = '
    No past sessions yet.
    '; @@ -408,21 +738,59 @@ function renderPastSessions() { pastSessionsList.innerHTML = pastSessions.map(s => { const dur = (s.ended_at && s.started_at) ? Math.floor(s.ended_at - s.started_at) : null; const armBit = s.arm ? escapeHtml(s.arm) + ' arm' : '—'; - const captures = s.capture_count || (s.captures || []).length; - return `
    -
    + const captureList = Array.isArray(s.captures) ? s.captures : []; + const captureCount = s.capture_count != null ? s.capture_count : captureList.length; + const isOpen = expandedSessions.has(s.id); + const caret = captureList.length ? (isOpen ? '▾' : '▸') : '·'; + // The captures sub-list is a sibling div, toggled by .hidden. We + // render it eagerly (with .hidden if closed) so the open/close + // animation isn't required and so screen readers can find it. + const capturesInner = captureList.length + ? captureList.map(name => ` +
  • + ${escapeHtml(name)} + + + + download + + +
    • `).join('') + : '
  • No captures linked to this session.
    • '; + + return `
    +
    + ${caret} ${escapeHtml(s.name || s.id)} - ${armBit} · ${escapeHtml(s.subject || '—')} · ${captures} captures${dur != null ? ' · ' + formatUptime(dur) : ''} + ${armBit} · ${escapeHtml(s.subject || '—')} · ${captureCount} captures${dur != null ? ' · ' + formatUptime(dur) : ''}
    +
    ${s.notes ? `
    ${escapeHtml(s.notes)}
    ` : ''} +
      ${capturesInner}
    `; }).join(''); + + // Stop session-action buttons from triggering the row-toggle handler + pastSessionsList.querySelectorAll('.session-actions button').forEach(btn => { + btn.addEventListener('click', e => e.stopPropagation()); + }); + + // Toggle expand/collapse when the session header row is clicked + pastSessionsList.querySelectorAll('[data-toggle-session]').forEach(head => { + head.onclick = () => { + const sid = head.dataset.toggleSession; + if (expandedSessions.has(sid)) expandedSessions.delete(sid); + else expandedSessions.add(sid); + renderPastSessions(); + }; + }); + pastSessionsList.querySelectorAll('button[data-delete-session]').forEach(btn => { btn.onclick = async () => { const sid = btn.dataset.deleteSession; @@ -435,6 +803,34 @@ function renderPastSessions() { } catch (e) { alert('Delete failed: ' + e.message); } }; }); + + // Per-capture actions inside the expanded list + pastSessionsList.querySelectorAll('button[data-reveal-cap]').forEach(btn => { + btn.onclick = (e) => { + e.stopPropagation(); + revealCaptureFolder(btn.dataset.revealCap); + }; + }); + pastSessionsList.querySelectorAll('button[data-edit-cap]').forEach(btn => { + btn.onclick = (e) => { + e.stopPropagation(); + openMetaModal(btn.dataset.editCap); + }; + }); + pastSessionsList.querySelectorAll('button[data-unlink-cap]').forEach(btn => { + btn.onclick = (e) => { + e.stopPropagation(); + removeCaptureFromSession(btn.dataset.fromSession, btn.dataset.unlinkCap); + }; + }); + pastSessionsList.querySelectorAll('button[data-add-to-session]').forEach(btn => { + btn.onclick = (e) => { + e.stopPropagation(); + const sid = btn.dataset.addToSession; + const session = pastSessions.find(s => s.id === sid); + if (session) openLinkModal(session); + }; + }); } pastSessionsToggle.onclick = () => { @@ -643,6 +1039,7 @@ function renderCaptures(list) { ${escapeHtml(date)} + download @@ -669,6 +1066,9 @@ function renderCaptures(list) { capturesBody.querySelectorAll('button[data-edit]').forEach(btn => { btn.onclick = () => openMetaModal(btn.dataset.edit); }); + capturesBody.querySelectorAll('button[data-reveal]').forEach(btn => { + btn.onclick = () => revealCaptureFolder(btn.dataset.reveal); + }); updateSelectionStatus(); } @@ -1035,7 +1435,43 @@ function renderInference(inf) { }); } +// One-shot: if the server has no hand_target on first snapshot but we have +// one saved in localStorage, auto-apply it so launching `openmuscle web` +// doesn't lose the address every time. UDP-only (the only protocol we +// support); port defaults to 3145. +let handTargetRestoreAttempted = false; +function maybeRestoreHandTarget(inf) { + if (handTargetRestoreAttempted) return; + if (!inf) return; // wait for first inference snapshot + handTargetRestoreAttempted = true; // one-shot regardless of outcome + if (inf.hand_target) return; // server already has one (e.g. --hand on CLI) + const saved = localStorage.getItem(STORE_HAND); + if (!saved) return; + autoApplyHandTarget(saved); +} + +async function autoApplyHandTarget(raw) { + let host = raw, port = 3145; + if (raw.includes(':')) { + const idx = raw.lastIndexOf(':'); + host = raw.slice(0, idx); + const portN = parseInt(raw.slice(idx + 1), 10); + if (Number.isFinite(portN) && portN > 0 && portN < 65536) port = portN; + } + try { + await fetch('/api/inference/hand', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ host, port }), + }); + } catch (e) { + console.warn('hand target auto-restore failed', e); + } +} + function renderInferenceControls(inf) { + maybeRestoreHandTarget(inf); + const hasModel = !!(inf && inf.model); const enabled = !!(inf && inf.enabled); @@ -1111,6 +1547,10 @@ async function applyHandTarget() { body: JSON.stringify({ host, port }), }); if (!r.ok) throw new Error(await readError(r)); + // Persist so next launch auto-restores. Clear on explicit empty + // so the operator can "forget" the target deliberately. + if (host) localStorage.setItem(STORE_HAND, raw); + else localStorage.removeItem(STORE_HAND); // Force the snapshot side to refresh by clearing the cache so the // next tick syncs the (possibly normalized) value back into the input. lastSnapshotHand = undefined; @@ -1124,6 +1564,110 @@ inferHandInput.addEventListener('keydown', (e) => { if (e.key === 'Enter') applyHandTarget(); }); +// ---------- Studio shell: comparator residuals (Δ) ---------- + +// Compute per-piston residual (predicted - ground_truth) and write it into +// the .delta-row elements in the comparator. Color-codes by direction so +// the operator can see at a glance whether the model is over- or under- +// shooting each finger. +// +// CLOSE_THRESHOLD picked at 0.05 (5% of the 0..1 scale) — below that, the +// difference is below the noise floor of the LASK5 measurement itself. +const RESIDUAL_CLOSE_THRESHOLD = 0.05; + +function renderResiduals(laskDev, inf) { + const deltaRows = document.querySelectorAll('#comparator-deltas .delta-row'); + if (!deltaRows.length) return; + const gt = laskDev && Array.isArray(laskDev.values) ? laskDev.values : null; + const pred = inf && Array.isArray(inf.piston_values) ? inf.piston_values : null; + + deltaRows.forEach((row, i) => { + const valEl = row.querySelector('.delta-val'); + row.classList.remove('over', 'under', 'close'); + if (!gt || !pred || i >= gt.length || i >= pred.length) { + if (valEl) valEl.textContent = '--'; + return; + } + const g = pistonFraction(gt[i]); + const p = pistonFraction(pred[i]); + const d = p - g; + valEl.textContent = (d >= 0 ? '+' : '') + d.toFixed(2); + if (Math.abs(d) < RESIDUAL_CLOSE_THRESHOLD) row.classList.add('close'); + else if (d > 0) row.classList.add('over'); + else row.classList.add('under'); + }); +} + +// ---------- Studio shell: top-bar pipeline status strip ---------- + +// Set a pipe-pill's status + value text. State controls colour: +// 'live' -- blue accent (data flowing) +// 'ok' -- green (idle but healthy) +// 'warn' -- orange +// 'bad' -- red +// '' -- neutral grey +function setPipePill(id, state, valText) { + const el = document.getElementById(id); + if (!el) return; + el.classList.remove('ok', 'warn', 'bad', 'live'); + if (state) el.classList.add(state); + const valEl = el.querySelector('.pipe-val'); + if (valEl) valEl.textContent = valText; +} + +function renderPipelinePills(msg, laskDev) { + // SENSOR pill = the active flexgrid (the one driving the heatmap) + const dev = selectedDevice(); + if (dev && dev.device_type === 'flexgrid') { + const stale = dev.last_seen_age > 2.0; + setPipePill('pipe-sensor', stale ? 'warn' : 'live', `${dev.hz.toFixed(0)}Hz`); + } else { + setPipePill('pipe-sensor', '', '--'); + } + + // LABEL pill = LASK5 stream + if (laskDev) { + const stale = laskDev.last_seen_age > 2.0; + setPipePill('pipe-label', stale ? 'warn' : 'live', `${laskDev.hz.toFixed(0)}Hz`); + } else { + setPipePill('pipe-label', '', '--'); + } + + // CAPTURE pill + if (recordingState) { + const matchRate = recordingState.match_rate ?? 0; + const cls = matchRate < 0.5 ? 'bad' : (matchRate < 0.9 ? 'warn' : 'live'); + setPipePill('pipe-capture', cls, `REC ${recordingState.rows ?? 0}r`); + } else if (activeSession) { + setPipePill('pipe-capture', 'ok', `session: ${activeSession.name || activeSession.id}`); + } else { + setPipePill('pipe-capture', '', 'idle'); + } + + // MODEL pill + const inf = msg.inference; + if (inf && inf.model && inf.enabled) setPipePill('pipe-model', 'live', inf.model); + else if (inf && inf.model && !inf.enabled) setPipePill('pipe-model', 'ok', inf.model + ' (paused)'); + else setPipePill('pipe-model', '', 'none'); + + // HAND pill = UDP forwarding target + if (inf && inf.hand_target) setPipePill('pipe-hand', 'live', inf.hand_target); + else setPipePill('pipe-hand', '', 'off'); +} + +// ---------- Studio shell: diagnostics drawer ---------- + +const diagToggle = document.getElementById('diag-toggle'); +const diagBody = document.getElementById('diag-body'); +if (diagToggle && diagBody) { + diagToggle.onclick = () => { + const isHidden = diagBody.classList.toggle('hidden'); + diagToggle.setAttribute('aria-expanded', isHidden ? 'false' : 'true'); + diagToggle.textContent = (isHidden ? '▸' : '▾') + ' Diagnostics & logs'; + // Logs poll runs unconditionally; we just hide the DOM. Cheap. + }; +} + // ---------- utils ---------- function escapeHtml(s) { diff --git a/pc/src/openmuscle/web/static/hand-viewer.js b/pc/src/openmuscle/web/static/hand-viewer.js new file mode 100644 index 0000000..ea5cbb7 --- /dev/null +++ b/pc/src/openmuscle/web/static/hand-viewer.js @@ -0,0 +1,269 @@ +// OpenMuscle desktop 3D hand viewer. +// +// Renders quest_hand joint data as a 3D skeleton in the Studio "Live" stage, +// replacing the LASK5 4-piston comparator (which shows zeros for a hand +// label source). Shows the REAL captured hand and, when a quest-trained +// model is running, the model's PREDICTED hand overlaid -- the desktop +// counterpart to the VR ghost hand. +// +// Loaded as an ES module from index.html; exposes a small imperative API on +// window.OMHandViewer so the plain (non-module) app.js can drive it: +// OMHandViewer.init(containerEl) +// OMHandViewer.update(realFlat, predFlat) // flat [px,py,pz,rx,ry,rz,rw]*N +// OMHandViewer.setVisible(bool) +// +// Both hands are transformed into WRIST-LOCAL space (subtract the wrist +// position, rotate by the inverse wrist orientation) before drawing, so the +// hand always appears in a canonical palm orientation regardless of how it +// was held, and REAL vs PRED is a direct shape comparison. (This is the same +// wrist-relative idea tracked for training in issue #2, used here purely for +// visualization.) + +import * as THREE from 'three'; + +// Canonical WebXR hand joint order (25 joints). Index i in a flat values +// array occupies [i*7 .. i*7+6] = px,py,pz, rx,ry,rz,rw. +const N_JOINTS = 25; +const FLOATS_PER_JOINT = 7; + +// Bone connectivity as [parentIdx, childIdx] pairs. Wrist = 0; then 4 thumb +// joints (1..4), then 5 each for index/middle/ring/pinky. +const BONES = [ + // thumb + [0, 1], [1, 2], [2, 3], [3, 4], + // index + [0, 5], [5, 6], [6, 7], [7, 8], [8, 9], + // middle + [0, 10], [10, 11], [11, 12], [12, 13], [13, 14], + // ring + [0, 15], [15, 16], [16, 17], [17, 18], [18, 19], + // pinky + [0, 20], [20, 21], [21, 22], [22, 23], [23, 24], +]; + +// Fingertip joint indices, for slightly larger tip markers. +const TIPS = new Set([4, 9, 14, 19, 24]); + +const COLOR_REAL = 0x34d399; // emerald +const COLOR_PRED = 0xfbbf24; // amber + +let scene, camera, renderer, container; +let raf = null; +let visible = false; +let autoRotate = true; +let yaw = 0.6, pitch = -0.25; // view angles (radians) +let dragging = false, lastX = 0, lastY = 0; + +// One reusable hand rig = 25 joint spheres + bone line-segments + a label. +function makeHandRig(color, opacity) { + const group = new THREE.Group(); + const jointMat = new THREE.MeshBasicMaterial({ + color, transparent: opacity < 1, opacity, + }); + const tipGeo = new THREE.SphereGeometry(0.007, 10, 8); + const jointGeo = new THREE.SphereGeometry(0.0045, 8, 6); + const joints = []; + for (let i = 0; i < N_JOINTS; i++) { + const m = new THREE.Mesh(TIPS.has(i) ? tipGeo : jointGeo, jointMat); + group.add(m); + joints.push(m); + } + // Bones: one BufferGeometry with 2 vertices per bone, updated each frame. + const positions = new Float32Array(BONES.length * 2 * 3); + const boneGeo = new THREE.BufferGeometry(); + boneGeo.setAttribute('position', new THREE.BufferAttribute(positions, 3)); + const boneMat = new THREE.LineBasicMaterial({ + color, transparent: opacity < 1, opacity: Math.min(1, opacity + 0.1), + }); + const bones = new THREE.LineSegments(boneGeo, boneMat); + group.add(bones); + return { group, joints, bones, positions }; +} + +let realRig, predRig; + +// Scratch objects reused per update (no per-frame allocation). +const _wristPos = new THREE.Vector3(); +const _wristQuatInv = new THREE.Quaternion(); +const _p = new THREE.Vector3(); + +// Transform a flat values array into wrist-local joint positions and write +// them into the rig. Returns false (and hides the rig) if the data is +// missing/degenerate. `out` is an array of N_JOINTS THREE.Vector3 to fill. +function layoutHand(flat, rig, outPositions) { + if (!flat || flat.length < N_JOINTS * FLOATS_PER_JOINT) { + rig.group.visible = false; + return false; + } + _wristPos.set(flat[0], flat[1], flat[2]); + // Wrist quaternion -> inverse, with a degeneracy guard (a bad model can + // emit a near-zero quat; inverting that yields NaN). + const qx = flat[3], qy = flat[4], qz = flat[5], qw = flat[6]; + const qLenSq = qx * qx + qy * qy + qz * qz + qw * qw; + let useRot = false; + if (qLenSq > 1e-6) { + _wristQuatInv.set(qx, qy, qz, qw).normalize().invert(); + useRot = true; + } + for (let i = 0; i < N_JOINTS; i++) { + const b = i * FLOATS_PER_JOINT; + _p.set(flat[b], flat[b + 1], flat[b + 2]).sub(_wristPos); + if (useRot) _p.applyQuaternion(_wristQuatInv); + rig.joints[i].position.copy(_p); + outPositions[i].copy(_p); + } + // Update bone vertices from the laid-out joint positions. + const pos = rig.positions; + for (let k = 0; k < BONES.length; k++) { + const [a, c] = BONES[k]; + const pa = outPositions[a], pc = outPositions[c]; + const o = k * 6; + pos[o] = pa.x; pos[o + 1] = pa.y; pos[o + 2] = pa.z; + pos[o + 3] = pc.x; pos[o + 4] = pc.y; pos[o + 5] = pc.z; + } + rig.bones.geometry.attributes.position.needsUpdate = true; + rig.group.visible = true; + return true; +} + +const _realOut = Array.from({ length: N_JOINTS }, () => new THREE.Vector3()); +const _predOut = Array.from({ length: N_JOINTS }, () => new THREE.Vector3()); + +// Auto-framing target + fit radius, recomputed from the real hand each update +// so any hand size/pose fills the viewport (the hand extends up from the +// wrist, so looking at the wrist alone wastes the lower half of the view). +const _focus = new THREE.Vector3(); +let _fitRadius = 0.1; + +function computeFraming(pts) { + _focus.set(0, 0, 0); + for (let i = 0; i < N_JOINTS; i++) _focus.add(pts[i]); + _focus.multiplyScalar(1 / N_JOINTS); + let maxR = 0; + for (let i = 0; i < N_JOINTS; i++) { + const d = pts[i].distanceTo(_focus); + if (d > maxR) maxR = d; + } + _fitRadius = maxR || 0.1; +} + +let _lastW = 0, _lastH = 0; + +function resize() { + if (!container || !renderer) return; + const w = container.clientWidth || 320; + const h = container.clientHeight || 240; + // Skip no-op resizes: setVisible(true) fires every tick, and setSize + // clears the drawing buffer even when the size hasn't changed. + if (w === _lastW && h === _lastH) return; + _lastW = w; _lastH = h; + renderer.setSize(w, h, false); + camera.aspect = w / h; + camera.updateProjectionMatrix(); +} + +// Place the orbit camera from the current yaw/pitch. Hands are wrist-centered +// at the origin, so the camera always looks at (0,0,0). Shared by the animate +// loop and snapshot() so a snapshot never renders from an unpositioned camera. +function positionCamera() { + // Distance to fit a sphere of _fitRadius given the camera's vertical FOV, + // with margin. Orbit around the hand centroid (_focus), not the wrist. + const half = THREE.MathUtils.degToRad(camera.fov) / 2; + const r = Math.max(0.12, (_fitRadius / Math.sin(half)) * 1.4); + camera.position.set( + _focus.x + r * Math.cos(pitch) * Math.sin(yaw), + _focus.y + r * Math.sin(pitch), + _focus.z + r * Math.cos(pitch) * Math.cos(yaw), + ); + camera.lookAt(_focus); +} + +function animate() { + raf = requestAnimationFrame(animate); + if (!visible) return; + if (autoRotate && !dragging) yaw += 0.005; + positionCamera(); + renderer.render(scene, camera); +} + +const OMHandViewer = { + init(containerEl) { + if (renderer) return; // idempotent + container = containerEl; + scene = new THREE.Scene(); + camera = new THREE.PerspectiveCamera(45, 1, 0.01, 10); + renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true }); + renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2)); + // The canvas must never drive layout: setSize(..., false) sets the + // canvas width/height ATTRIBUTES (drawing buffer), and without CSS + // sizing those become the layout size. The container then grows to + // fit, resize() reads the bigger clientWidth, and the loop runs the + // canvas up to the GPU's max texture size. CSS-pin it to the + // container instead. + renderer.domElement.style.width = '100%'; + renderer.domElement.style.height = '100%'; + container.appendChild(renderer.domElement); + + realRig = makeHandRig(COLOR_REAL, 1.0); + predRig = makeHandRig(COLOR_PRED, 0.55); + predRig.group.visible = false; + scene.add(realRig.group); + scene.add(predRig.group); + + // Drag to rotate (pauses auto-rotate while dragging). + const el = renderer.domElement; + el.style.cursor = 'grab'; + el.addEventListener('pointerdown', (e) => { + dragging = true; lastX = e.clientX; lastY = e.clientY; + el.style.cursor = 'grabbing'; el.setPointerCapture(e.pointerId); + }); + el.addEventListener('pointermove', (e) => { + if (!dragging) return; + yaw -= (e.clientX - lastX) * 0.01; + pitch = Math.max(-1.4, Math.min(1.4, pitch + (e.clientY - lastY) * 0.01)); + lastX = e.clientX; lastY = e.clientY; + }); + const endDrag = () => { dragging = false; el.style.cursor = 'grab'; }; + el.addEventListener('pointerup', endDrag); + el.addEventListener('pointercancel', endDrag); + // Double-click resets to auto-rotate. + el.addEventListener('dblclick', () => { autoRotate = true; }); + + window.addEventListener('resize', resize); + resize(); + animate(); + }, + + // realFlat: live captured hand. predFlat: model prediction (or null/short + // for a non-quest model -> predicted hand hidden). + update(realFlat, predFlat) { + if (!renderer) return; + if (layoutHand(realFlat, realRig, _realOut)) computeFraming(_realOut); + if (predFlat && predFlat.length >= N_JOINTS * FLOATS_PER_JOINT) { + layoutHand(predFlat, predRig, _predOut); + } else { + predRig.group.visible = false; + } + }, + + setVisible(v) { + visible = !!v; + if (container) container.style.display = v ? '' : 'none'; + if (v) resize(); + }, + + // Render one frame synchronously and return it as a PNG data URL. The + // synchronous render + toDataURL captures the drawing buffer before the + // compositor clears it, so it works without preserveDrawingBuffer. Handy + // for frame thumbnails and for headless verification of the render. + snapshot() { + if (!renderer) return null; + positionCamera(); + renderer.render(scene, camera); + return renderer.domElement.toDataURL('image/png'); + }, + + isReady() { return !!renderer; }, +}; + +window.OMHandViewer = OMHandViewer; diff --git a/pc/src/openmuscle/web/static/index.html b/pc/src/openmuscle/web/static/index.html index b795c34..ed1ac14 100644 --- a/pc/src/openmuscle/web/static/index.html +++ b/pc/src/openmuscle/web/static/index.html @@ -3,179 +3,328 @@ - OpenMuscle Live + OpenMuscle Studio + + + -
    -

    OpenMuscle Live

    -
    +
    +
    +

    OpenMuscle Studio

    +
    + + + + +
    disconnected
    -
    -
    -

    Devices

    -
      -
    • Waiting for a device to send a packet…
      • -
    -
    +
    -
    -
    -

    Heatmap

    - -
    - -
    + +
    +
    +

    1 Live

    + what the band feels · what the model thinks +
    -
    -
    -

    LASK5 — Ground Truth

    - no device -
    -
    -
    P1
    --
    -
    P2
    --
    -
    P3
    --
    -
    P4
    --
    -
    -
    - joystick - - --, -- -
    -
    +
    + + -
    -

    Record

    -
    - - - - + +
    +
    + +
    + +
    + + +
    +
    +

    Ground truth vs Predicted

    +
    + GT: no device + MODEL: no model loaded +
    +
    + +
    + +
    +
    P1
    --
    +
    P2
    --
    +
    P3
    --
    +
    P4
    --
    +
    + + +
    +
    Δ--
    +
    Δ--
    +
    Δ--
    +
    Δ--
    +
    + + +
    +
    P1̂
    --
    +
    P2̂
    --
    +
    P3̂
    --
    +
    P4̂
    --
    +
    +
    + + +
    +
    +
    + ● real + ● predicted + drag to rotate +
    +
    + + +
    + + --, -- +
    + + +
    + + predictions to a robot hand: see Stage 4 +
    +
    -
    -
    -
    -

    Sessions

    -
    - + +
    +
    +

    2 Capture

    + session + record · paired sensor/label rows go to disk +
    + +
    + +
    +
    + Session + +
    +
    +
    No active session — recordings won't be grouped. Click "New session" to start one.
    +
    +
    + + +
    +
    + Record +
    +
    + + + + +
    +
    -
    -
    No active session — recordings won't be grouped. Click "New session" to start one.
    -
    + +
    -
    -
    -

    Captures

    -
    - 0 selected - + +
    +
    +

    3 Data & Models

    + captures become models · audit honestly · iterate +
    + +
    + +
    +
    +

    Captures

    +
    + 0 selected + +
    +
    + + + + + + + + + + + + + + +
    NameMetaSizeModified
    No captures saved yet.
    +
    + + +
    + +
    +
    + + +
    +
    +

    Models

    + +
    + + + + + + + + + + + + + + +
    NameCreatedR²MSEFeatures × Labels
    No models trained yet.
    - - - - - - - - - - - - - - -
    NameMetaSizeModified
    No captures saved yet.
    -
    -
    -
    -

    Models

    - + +
    +
    +

    4 Output

    + where the predictions are sent +
    +
    + + + no hand target
    - - - - - - - - - - - - - - -
    NameCreatedR²MSEFeatures × Labels
    No models trained yet.
    -
    -
    -

    Logs

    -
    - - - -
    + +
    + +
    - Open-Muscle · FlexGrid · github + Open-Muscle · FlexGrid Studio · github
    - + - + + + +