Pass DIVE camera registrations to 2-cam/3-cam VIAME pipelines - #1780
Pass DIVE camera registrations to 2-cam/3-cam VIAME pipelines#1780romleiaj wants to merge 10 commits into
Conversation
|
Paired with VIAME/VIAME#278. |
5811f58 to
58e1959
Compare
Pass DIVE camera registrations to 2-cam/3-cam VIAME pipesMulticam pipes never got any alignment info from DIVE — the transform paths were This sends the dataset's registration to the pipe and turns the camera order What's newAssign cameras dialog. Running a 2-cam or 3-cam pipe now opens a dialog with Registration hand-off. DIVE writes one registration file per camera into the Blocked before it fails. DIVE now knows which inputs a pipe warps, so if a TestingNeeds VIAME from
NotesStandalone on The old automatic header-matching path is gone — the dialog always supplies the |
Multicam pipelines previously received no transform information from DIVE; the old h5 workflow relied on paths hardcoded inside the pipes. Desktop and web now hand the dataset's camera registration to the pipeline's warp processes, mirroring the stereo calibration flow: - One standard <camera>_to_<reference>_registration.json per non-reference camera is written into the job work dir; warpN (matching the pipeline camera position) receives its file via -s warpN:transformation_file=..., with the pair and direction pinned through the reader's from_camera/to_camera config since a pair may be stored in either orientation. - Pipeline camera order is the registration reference first (the pipes warp everything onto camera 1's frame), then display order. Which detector a pipe runs on which input is the pipe's documented contract, not something DIVE infers. - Only pairs registering a camera directly onto the reference are supported; pairs between two non-reference cameras are explicitly unsupported for pipelines and never reach the job. Cameras with points-only (unfitted) pairs get no settings. - Web sends the registration as a multicam_registration job param built from dataset meta; the worker writes the files. Requires VIAME's dive transform_2d_io reader and warp_detections process. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The web path built its registration files from the pre-v2 correspondence
shape: a flat list of {id, a, b} per camera pair, emitted as pairs[].points
and stamped version 1. Format v2 replaced that with observations -- one
entry per image pair, each carrying its own points -- so this did not
merely write an outdated file, it raised KeyError: 'a' on the first
correspondence and failed the job in the worker. Every web multicam
pipeline run on a v2-registered dataset would have died there.
Build observations instead, as the inverse of
registration_output._from_registration_pairs: the store's imageA/imageB
become the file's imageLeft/imageRight, frame/enabled/source/stats carry
through, and each point's a/b pair becomes one `leftX leftY rightX rightY`
row. Stamp version 2.
The desktop path was never affected -- it writes through
buildPerCameraRegistrationFiles, which the format-v2 work already moved --
which is exactly why this was easy to miss: desktop produces correct v2
files while the worker crashes.
VIAME's dive transform reader only consumes the matrices, so the
observations travel for provenance and so a file round-trips back into
DIVE without losing which frame contributed what. Tests now assert the
version, which nothing did before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
- cameraRegistration.spec.ts: JsonMeta does not exist (use JsonConfig); type the fs-extra mock the way common.spec.ts does so vue-tsc accepts the default export and statSync parameter types. - crud_rpc.py: drop unused pathlib import. - multicam_pipeline.py: black formatting. - test_multicam_pipeline.py: VIAME's dive reader ignores the file version; it is DIVE's format-v2 loader that rejects other versions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
Which dataset camera feeds which inputN of a multicam pipe is the pipe's contract -- the arctic seal 3-cam pipe runs the thermal detector on input3 and projects optical boxes onto input2 -- but DIVE was inferring it from display order (reference camera first, then the rest), so a dataset imported as EO, IR, UV silently ran the thermal detector on UV. Pipes now state their slots with a `# Camera Order: EO, UV, IR` header, parsed into pipeline metadata on desktop and web. At run time each slot is matched to a dataset camera by name: an exact camera name, or a name segment sharing the slot's role (EO ~ eo/rgb/optical/color/vis, IR ~ ir/thermal/lwir/flir, UV ~ uv/ultraviolet); other tokens match literally. A slot matching no camera or several refuses the run with a message that names the pipe's slots and the dataset's cameras. Camera 1 of the resolved order is the warp target the registration files are built onto. Pipes without the header keep the reference-first fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
… datasets Which dataset camera feeds which inputN of a 2-cam/3-cam pipe was decided silently -- from the pipe's `# Camera Order:` header matched against camera names, and before that from display order. Make it explicit and visible: - Datasets carry a sensor role per camera (`cameraRoles`: eo / ir / uv), inferred once at import from the camera name and, failing that, from tokens in its image file names (KAMERA style _rgb / _ir / _uv). Only a unanimous answer is recorded. Desktop and web import both set it; it is a mutable config key so it round-trips through load/save on both. - Running a 2-cam/3-cam pipe opens a camera-assignment step: one row per pipeline camera (the header's slots, or bare input1..N) with the dataset camera DIVE proposes -- by role when both sides have one, else by name -- which the user confirms or changes. Unfilled or duplicated slots block the run. Confirming role-labelled slots saves the roles back onto the dataset (opt-out checkbox), so a corrected role wins over a misleading name next time and for every other pipe. - The confirmed order travels as pipelineParams.cameraOrder; desktop and web validate it against the dataset's cameras and use it verbatim. Runs without it (CLI) still resolve the header by role then name; pipes with no header keep the reference-first fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
…re time A 2-cam/3-cam pipe whose warpN camera had no registration onto camera 1 launched anyway and died inside kwiver with "Path does not exist: registration_camera2_to_camera1.json". Do the check before the job exists: - Pipeline discovery (desktop + web) records registrationWarps: the input positions of `process warpN :: warp_detections | warp_image` in the pipe body, e.g. [2, 3]. - The camera-assignment dialog shows, for each warped row, whether the chosen camera has a fitted Aligned View registration onto camera 1, and blocks Run with "Register X -> Y in Aligned View first" when it does not. - Desktop (viame.ts) and web (crud_rpc) refuse the run with the same message before creating the job, so CLI and API callers get it too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
…stration-missing message Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
The design settled on the camera-assignment step always supplying the input1..N order, which left the earlier automatic header resolution as a dead path with a lot of code behind it. Remove it and flatten what is left: - Drop resolvePipelineCameraOrder / candidatesForSlot / camerasMatchingSlot / pipelineCameraNames / pipelineOrderedCameraNames on the client and resolve_pipeline_camera_order / candidates_for_slot / cameras_matching_slot / pipeline_camera_order on the server. Without a confirmed cameraOrder (API callers) both backends now use the dataset's stored camera order, exactly as main does. One matcher, camerasForSlot, remains for the dialog prefill. - buildRegistrationPipelineArgs takes the pipe's warp positions and throws the register-first error itself, instead of viame.ts reverse-engineering the fitted set from the -s keys it had just built; its three guards for "this camera has a fitted pair onto camera 1" become one. - Web import infers camera roles from names only (no per-folder item scan); the server's infer_camera_role drops its image-name fallback accordingly. Desktop keeps the image-name fallback since it has the list in hand. No change on the dialog path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
Both places that decide which camera feeds which inputN fell back to Object.keys(multiCam.cameras) when no confirmed order was supplied. That is the JSON object's key order, which #1857 established is not a dependable statement of rig order -- it varies across loads, and multiCam.cameraOrder is what the dataset actually persists. Read orderedMultiCamCameraNames instead, the same helper the viewer tabs, the registration panel and RunPipelineMenu already use, so the EO-first/IR-last heuristic only applies to datasets imported before the order was persisted. The UI path always supplies a confirmed order, so this only changed non-UI callers -- but multiCamOrderFor validates confirmed-vs-dataset as a sorted set, so a wrongly-ordered fallback would have gone unchallenged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011PTsH5DETMmFFi75xpf1Qx
58e1959 to
6769ed7
Compare
Multicam pipelines previously received no transform information from DIVE; the old h5 workflow relied on paths hardcoded inside the pipes. Desktop and web now hand the dataset's camera registration to the pipeline's warp processes, mirroring the stereo calibration flow:
Requires VIAME's dive transform_2d_io reader and warp_detections process.