Quality Gates For Egocentric RGB-D

Robot-ready egocentric RGB-D data is not just a visual artifact. It is a set of claims.

Depth claims something about geometry. Timing metadata claims something about synchronization. Hand tracks claim something about left and right hand state. Object masks claim something about visible and task-relevant objects. Captions claim something about actions. Review state claims something about what a trained person has checked.

StableBrowse's quality system is built around making those claims explicit.

The pipeline has reached 100 hours/day of post-processing throughput and is scaling toward 1,000 hours/day. At that scale, quality cannot depend on a person noticing every issue by watching videos. It has to be encoded into the data contract: confidence masks, validity flags, frame accounting, review states, manifest validation, QA summaries, provenance, and final promotion rules.

This post describes how StableBrowse thinks about quality for depth, timing, hands, masks, captions, and human review.

Quality Is A Claim

Preview videos are useful. They help humans see whether depth feels plausible, whether hand tracks remain stable, whether masks drift, and whether captions describe the action.

But a clean overlay is not a quality proof.

The pipeline needs machine-checkable answers:

• Did this stage cover the declared frame range?
• Are frame indices monotonic and gap-free?
• Do all artifacts agree on the same output window?
• Are invalid depth regions explicit?
• Does a mask have enough valid depth to support spatial use?
• Is a hand track semantically left or right, or only a generic detection?
• Is a 3D hand keypoint valid, triangulated, depth-backed, or unavailable?
• Is a caption a draft, edited, reviewed, or approved?
• Was an object mask generated automatically, corrected by a human, or approved?
• Which calibration and timing evidence support this episode?

That is why StableBrowse treats quality as metadata, manifests, and review state, not only visual inspection.

Calibrated Depth

StableBrowse produces computed stereo-derived depth from egocentric stereo. The important word is "computed."

This is not a blanket claim that every pixel contains perfect measured depth. Useful stereo-derived depth depends on the evidence behind it: camera calibration, lens model, eye ordering, rectification, baseline, frame pairing, timestamp semantics, and confidence estimation.

The depth stage begins before inference. The pipeline has to know which image is left, which image is right, how distortion should be handled, which transform direction applies, and which frames correspond. If the geometry is inconsistent, a depth model can still produce a dense-looking map. It just will not be a trustworthy robotics artifact.

StableBrowse's depth quality gates therefore include geometry-aware checks and confidence reporting:

• per-session calibration reference
• intrinsics and stereo extrinsics parsed into a stable schema
• rectified coordinate conventions
• depth resolution and frame-index contract
• validity or confidence masks
• invalid regions preserved instead of visually filled
• render reports for QA previews
• provenance tying depth to source frames and calibration

That matters for downstream robotics teams. A model training pipeline can choose to ignore low-confidence regions, filter masks by valid-depth fraction, or inspect depth distributions around hands and task objects. Hidden uncertainty removes that choice.

Timing Evidence

Robotics teams care about time. RGB, depth, hands, motion, captions, and actions all become less useful if the alignment story is vague.

StableBrowse treats timing evidence according to what the capture source actually provides. Different sources can expose different levels of timing proof:

• host arrival timestamps
• frame-associated metadata
• camera exposure or hardware trigger timestamps
• high-rate motion logs
• calibrated camera-to-motion timeshifts
• same-clock evidence across sensors
• per-session clock mappings and residuals where needed

Those are not interchangeable. A frame-associated motion row is useful, but it is not the same claim as raw high-rate motion data on the same hardware clock as camera exposure. A calibrated timeshift is valuable, but it has to be applied deliberately. A p99 synchronization claim is not the same as every-frame synchronization.

This precision is part of quality, not a disclaimer after the fact. The pipeline should represent timing provenance clearly so downstream teams know what they can assume for a given dataset.

The same principle applies to motion and pose outputs. A 6-DoF pose branch can be useful while still carrying coverage, divergence, drift, and scale flags. StableBrowse does not need to collapse every motion estimate into a single "passed" label. It needs to preserve the evidence and boundaries that make the output usable.

Mask-Depth Joins

A 2D mask says which pixels belong to an object or hand. A mask joined with calibrated depth becomes more useful:

• valid-depth fraction inside the mask
• median and percentile depth
• approximate 3D centroid estimates
• per-frame spatial summaries
• per-track depth coverage
• flags for frames where depth is unavailable

These statistics are not just dashboards. They are quality gates and filtering tools.

They also become part of the dataset interface. Customers can filter tracks by spatial reliability without rejoining raw masks and depth themselves.

For example, a generated object mask may look clean in RGB but cover a region with low depth confidence. A hand mask may be visually plausible but lack enough valid depth for a particular 3D use case. A segment may extend beyond the available depth window. The correct behavior is to surface that state explicitly, not to fabricate completeness.

The architecture also matters. Mask-depth joins can run after the expensive GPU stages have produced masks and depth. That lets the pipeline reuse saved artifacts and add spatial value without rerunning models unnecessarily.

Hands Are The Interface

In egocentric robotics data, hands carry intent.

Objects tell you what is present. Hands tell you what is being done. A task may involve one hand stabilizing an object while the other manipulates it, both hands folding or stretching a deformable item, one hand pressing a control, or a hand releasing an object after a state change.

For that reason, StableBrowse treats hand state as a first-class output, not background context.

The hand pipeline can produce several different artifacts:

• independent left and right hand state
• 2D keypoints with confidence and visibility
• stereo-derived 3D keypoints where geometry supports them
• validity flags instead of invented points
• visible hand masks
• mask-depth statistics
• QA overlays for human inspection
• provenance tying hand artifacts back to frames and calibration

The important quality bar is semantic stability. A high raw detection rate is not enough if the track identity fragments every time a hand leaves the frame, crosses the other hand, or becomes occluded. For manipulation learning, a stable right-hand track across a procedure can be more valuable than many short-lived detections that cover more pixels.

StableBrowse's review and QA process therefore looks at more than missing-mask rate:

• left/right identity
• temporal continuity
• mask drift
• occlusion behavior
• coordinate consistency
• whether hand artifacts can be joined to depth

Hand Masks

Hands are unusually difficult segmentation targets. They deform. They move quickly. They occlude objects. They get occluded by objects. They touch the scene. They often appear near frame edges. In egocentric footage, they can be close to the camera and partially visible for long periods.

That makes hand segmentation a different problem from generic object segmentation.

A broad prompt can attach to nearby fabric, tools, surfaces, or machine parts. A box-style prompt can increase visual coverage while adding semantic noise. A temporally propagated mask can look good until an occlusion or re-entry changes the scene. A hand detector can produce useful keypoints while still leaving visible-mask boundaries uncertain. A mesh projection can be useful for 3D reasoning while still not being the same thing as visible-pixel segmentation.

StableBrowse handles that by treating hand artifacts as layered signals:

The pipeline keeps those distinctions visible so customers can choose the artifact level that fits their training or evaluation task.

Object Relevance

Object segmentation in egocentric robotics has a relevance problem.

The object that matters is not always the largest, sharpest, or most visually salient object. It may be the object being held, the target of a hand action, the object whose state changes, or the tool that matters for a step. In a cluttered workspace, generic segmentation can be visually impressive while missing the task-relevant object.

StableBrowse uses generated mask candidates and propagation to reduce manual work, but object labels become dependable only when the workflow preserves human judgment. A trained annotator can choose the correct seed, identify the task-relevant object, add correction points, reject drift, and approve the final track.

The system is designed to avoid hidden shortcuts. Display-time limits should not become dataset limits. If later tracks exist, they should not disappear because an early QA view was simplified. If depth is unavailable for a range, the output should say so. If a mask is generated but not reviewed, its state should remain generated.

That is how object segmentation becomes a dataset workflow rather than a one-off demo.

Propagation Vs. Review

Interactive annotation and long video propagation have different latency requirements.

When an annotator creates or corrects a seed, the interface should feel immediate. That is a human-in-the-loop task. When a propagation job extends that seed across a long clip, the correct system behavior is usually asynchronous: submit the job, show state, let the annotator continue working, and return generated candidates when the worker finishes.

Keeping those regimes separate matters for quality. A human seed should remain a human assertion. A propagated track should remain a generated extension until reviewed. If the track drifts, the reviewer should be able to correct the segment without losing the original evidence.

It also matters for infrastructure. Long propagation jobs can use bounded temporal windows, keep memory under control, and write outputs by frame index. Interactive tools can stay responsive because they are not waiting for full-video propagation in the foreground.

Reproducible QA

Visual QA is essential, but it has to be downstream of the data.

StableBrowse generates overlay videos, depth previews, mask dashboards, and comparison views from saved artifacts. Those previews help engineers and annotators inspect results quickly. They should also carry enough provenance to explain what they show: source artifacts, render settings, frame counts, coverage, and any known limitations.

That separation prevents a common quality mistake. A render issue should not automatically invalidate the underlying prediction. A browser compatibility issue should not make a mask artifact disappear. A projection overlay can misrepresent a prediction if coordinate transforms or render settings are wrong. The source artifact and the QA render report let reviewers tell the difference.

The rule is: QA previews accelerate review, but the manifest and saved artifacts remain the source of truth.

Caption Review

Language labels are especially easy to overstate.

A vision-language model can produce a useful first pass for captions and action descriptions. It can reduce blank-page work, suggest scene summaries, and help create consistent draft metadata. But a fluent caption is not automatically a correct robotics label.

StableBrowse's captioning workflow uses a vision-language model for the first pass, followed by review and correction by a team of trained annotators.

That human review is not ornamental. Annotators improve specificity, remove hallucinations, normalize vocabulary, align text to visible evidence, and make sure task-relevant details are represented. They can distinguish:

• "moves hand near object" from "grasps object"
• "holds" from "places"
• "opens" from "removes"
• "object visible" from "object being manipulated"

The review unit is structured. Depending on the dataset, annotators may check fields such as:

• action verb
• manipulated object
• left/right hand involvement
• contact state
• object state before and after the action
• task phase
• visible objects
• caption text
• approval status

The output should tell downstream users whether a caption is model-drafted, human-edited, reviewed, or approved. That state is part of the label.

Scaling Human Review

At 100 hours/day, human review cannot be an informal spreadsheet and a folder of videos. At 1,000 hours/day, even minor workflow inefficiencies become limiting.

StableBrowse's annotation workflow is designed around fixed work units, draft/final separation, async generation, and explicit approval state.

The review interface can show model drafts, generated mask candidates, QA previews, and saved annotations, but storage state has to remain unambiguous. Adding an object to temporary storage is not the same as promoting it to the review UI. A new draft should not overwrite a newer human edit. A final label should not be inferred from a file merely existing somewhere in storage.

The annotation system treats review state as a versioned record, not as whatever file happens to exist in storage. Drafts, generated candidates, human edits, approvals, and final promoted artifacts have separate meanings, and conflict-aware saves prevent a newer human edit from being silently replaced.

Versioned drafts and conflict-aware saves are part of the quality system. They let annotators work quickly while preserving auditability. The final dataset should know what was model-generated, what was edited, what was reviewed, what was approved, and what was promoted.

The same principle applies to captions and action metadata. A model draft, a human edit, and an approved label should be distinguishable states. Completion should require the relevant segments to be approved, not merely present.

Review State

Many dataset problems come from collapsed state.

If "generated," "reviewed," and "approved" all mean the same thing in storage, quality drift becomes likely. If a retry overwrites a human edit, trust erodes. If a UI shows a preview that has not been promoted into the final manifest, downstream consumers can get confused. If one annotator's draft can replace another's newer work without an explicit conflict state, review becomes hard to audit.

StableBrowse avoids that by treating review state as a first-class part of the artifact contract.

The system separates:

That structure lets trained annotators work efficiently without making the final dataset ambiguous.

Scope Boundaries

Good robotics data should make its limits visible.

StableBrowse does not need to pretend that computed stereo depth is perfect at every pixel. It needs to expose confidence. It does not need to pretend that generated masks are the same as approved labels. It needs to carry review state. It does not need to pretend that all hardware sources provide identical timing proof. It needs to preserve timing provenance.

Clear boundaries help customers use the data correctly:

• Use confidence masks to filter depth.
• Use validity flags for 3D hand keypoints.
• Use review status to select approved captions and masks.
• Use mask-depth statistics to choose spatially reliable segments.
• Use timing provenance to decide which tasks require which capture sources.
• Use pose quality flags when a trajectory is useful but not a ground-truth motion claim.

This is more useful than a single glossy output with hidden assumptions.

Quality gates emit metrics continuously, not only at final review. Depth confidence, rectification checks, hand-track continuity, mask-depth coverage, mask drift, caption review state, and manifest completeness are all stage outputs that can be monitored as the pipeline scales.

A Quality-Gated Episode

A processed StableBrowse episode can include:

• RGB video or frame windows tied to exact frame indices
• computed stereo-derived depth with explicit invalid regions
• calibration and timing provenance
• confidence and validity fields
• left/right hand state
• 2D and 3D hand signals where supported
• visible hand masks
• object mask candidates and approved tracks
• mask-depth statistics
• captions and action metadata
• trained annotator review and correction status
• QA previews generated from saved artifacts
• manifests for final promotion and downstream consumption

The point is not that every customer needs every artifact. The point is that when an artifact is present, its status and provenance are visible.

Why It Matters

Robotics models learn from details that ordinary video datasets often blur together.

Was the object only visible, or was it manipulated? Which hand touched it? Did contact happen before or after the state change? Is depth reliable inside the object mask? Did the caption describe what was visible or what the annotator inferred? Does the frame window align with the motion metadata? Is this mask generated or approved?

Those details are the difference between data that looks good and data that trains well.

StableBrowse's post-processing pipeline is built to preserve those details at scale. Throughput matters because robotics needs volume. Quality gates matter because volume without trust becomes expensive noise.

The goal is clear and hard to build: scale egocentric RGB-D data production while keeping depth, timing, hands, masks, captions, and review state honest.