Building Egocentric RGB-D At Scale

Robotics teams do not need another pile of videos. They need data products that survive training, evaluation, analysis, and model iteration.

That is the difference between collecting egocentric stereo footage and producing egocentric RGB-D data at scale. A raw recording is only the beginning. A usable robotics dataset needs calibrated depth, frame-aligned sensor metadata, hand and object signals, captions and action metadata, masks, review state, manifests, and enough provenance to explain exactly how a sample was produced.

StableBrowse's post-processing pipeline is built around that standard. The pipeline has reached 100 hours/day of post-processing throughput and is scaling toward 1,000 hours/day. The hard engineering problem is not simply running more vision models. It is keeping the full data factory fast while preserving the quality contracts that downstream robotics teams depend on.

This post describes the system design behind that pipeline.

The Dataset Contract

For robotics, a processed episode is useful only when its assumptions are explicit.

A final output should answer questions like:

• Which recording did this come from?
• Which frame window does this artifact represent?
• Which calibration and timing evidence were used?
• Which stage produced this artifact, using which configuration?
• Which masks have valid depth inside them?
• Which hand track is left or right?
• Which captions, masks, and tracks are generated, reviewed, approved, or still candidates?
• Which QA previews came from saved artifacts rather than from a one-off render?
• Which outputs are customer-facing deliverables and which are diagnostics?

Those questions sound operational, but they become model-quality questions quickly. If a chunk boundary duplicates a frame, an action label can cover the wrong moment. If decoder pre-roll leaks into exported data, RGB, depth, motion, masks, and captions can disagree about time. If a generated mask is treated as reviewed, training labels become less trustworthy. If depth confidence is hidden, downstream teams cannot filter geometry reliably.

StableBrowse treats post-processing as a data factory, not a model demo.

Raw capture is immutable. Work areas are execution-scoped. Chunks are deterministic. Artifacts carry provenance. Final delivery is a promotion step after validation. A worker writing a file is not the same as a dataset being complete.

That distinction is the spine of the pipeline.

Ingest

Egocentric capture can arrive in several shapes: stereo video, calibration files, motion logs, timing metadata, capture app manifests, or capture-source-specific packages. Some recordings come from field collections. Some come from controlled environments. Some are side-by-side stereo streams. Others need to be normalized from sensor containers before the shared post-processing path begins.

The ingestion layer turns those inputs into a session.

The pipeline does not start because one object appears in storage. It starts when a session completion marker exists and the required input set can be validated. This prevents a common pipeline risk: treating a partial upload as processable data.

Once a session is admitted, the orchestrator validates the expected files, resolves capture metadata, builds the run manifest, and creates the stage graph. From that point forward, every stage operates against explicit session identity and concrete input references.

The identity travels with the work:

• collection date or session grouping
• worker, camera, and clip identity where available
• frame range and chunk identity
• calibration reference
• timing reference
• stage name and attempt number
• input and output artifact references
• quality status
• review status where applicable

In lower-volume work, this can feel like bookkeeping. At 100 hours/day, it is what keeps the system inspectable. At 1,000 hours/day, it is what keeps the system operable.

Normalize Once

The first expensive system boundary is not the first neural network. It is decode and normalization.

Raw video is not always directly seekable, frame-indexed, or ready for parallel processing. Sensor logs can contain video, motion, calibration, and timing in different structures. Downstream stages should not each rediscover how to split a recording, parse timestamps, align stereo frames, or interpret calibration. That creates duplicate work and, worse, duplicate interpretations.

StableBrowse uses a format-normalizing preprocessing stage that runs once per session:

• validates the session and its required inputs
• creates canonical frame and timing indexes
• parses calibration into a stable schema
• checks timing monotonicity
• catches corrupt or truncated inputs early
• produces deterministic chunk specifications

After that boundary, the rest of the system consumes a shared contract instead of source-specific raw details.

The control plane ships references, not pixels. Stage workers receive chunk references and artifact manifests. They read from object storage, write results back to object storage, and report status to the ledger. Frames are not moved through per-request RPC calls. Pipeline data does not live inside GPU worker volumes. Persistent volumes are used for weights, compiled engines, and other reusable runtime assets, not as the source of truth.

This boundary looks straightforward on paper, but it is one of the main reasons the pipeline can scale horizontally. Object storage owns durable artifacts. The orchestrator owns state. Workers own computation for a bounded stage and attempt.

Canonical Chunks

StableBrowse scales by processing chunks, but a chunk is more than a convenient time slice.

A canonical chunk is the unit of retry, validation, frame accounting, and final assembly. It has deterministic identity and an expected output window. It may need additional context internally, but the exported data window remains exact.

That distinction matters because compressed video and temporal estimators often need context before the first published frame. A decoder may need pre-roll from a prior keyframe. A tracker or propagation stage may benefit from nearby frames. The pipeline allows that context internally, but exported RGB, depth, hand state, masks, captions, motion metadata, and QA summaries must represent only the declared output frames.

Every stage has to respect two concepts:

• decode frames are context
• output frames are the dataset

That protects downstream joins. A preview video can look fine while frame accounting is wrong. A manifest cannot. The universal join key is the frame index in the canonical output window.

Chunking also makes recovery practical. If a worker is interrupted, the orchestrator can rerun or resume that chunk without invalidating the whole recording. Completed chunks remain completed. Raw inputs remain untouched. Final assembly waits until the required chunk set has passed validation.

This is also how preemptible capacity becomes usable. An interrupted worker should be an efficiency event, not a data-integrity event. Checkpoints and deterministic output paths let the pipeline reuse completed stages and rebuild only what is missing.

The Stage Graph

StableBrowse's post-processing graph is built around a few core branches:

• preprocessing and canonical chunk generation
• stereo-derived depth
• hand state and hand masks
• object masks and tracks
• captions and action metadata
• motion and timing-derived artifacts
• mask-depth joins and statistics
• QA renders and dashboards
• final manifest, validation, and promotion

Some stages are strict dependencies. Mask-depth statistics cannot run until depth and masks exist. Final promotion cannot happen until required branches have reported status and validation has completed.

Other stages can run independently. Captions do not need to block depth inference. Motion quality checks do not need to block object-mask propagation. CPU-heavy joins, validation, and QA renderers should not stay inside an inference worker merely because they are adjacent to model output. QA preview rendering should be reproducible from saved artifacts rather than treated as the artifact itself.

The orchestrator keeps those boundaries clear. Each branch reports operational completion, quality status, and review state separately. A generated caption can be a draft. A mask can be a candidate. A depth chunk can be computed while carrying a confidence profile. A motion branch can complete operationally while still exposing quality flags.

This is how the system avoids converting "work happened" into "data is ready."

Independent Components

Each branch is separately scalable.

Depth, hand tracking, object segmentation, captions/action metadata, motion, joins, QA rendering, and final assembly do not share one monolithic worker. Each component declares its own input contract, output schema, quality gates, runtime image, concurrency limits, and retry behavior. The orchestrator coordinates them, but the components can be scaled, replaced, paused, or rerun independently.

The shared boundary is the canonical chunk artifact. Decode, frame indexing, and rectification happen once, then depth, hands, segmentation, motion, and QA consume the same frame geometry instead of each stage re-deriving it. That gives every component the same frame index, calibration reference, and coordinate convention.

That matters because the stages have different economics and operational profiles:

The production shape uses Modal-hosted GPU workers for heavy self-hosted inference branches. Workers keep model weights, compiled engines, and model state warm where possible, read and write pipeline artifacts directly through object storage, and report per-stage status back to the ledger. Autoscaling is stage-aware: GPU worker floors can be raised for a processing window and dropped back to zero when the relevant ledger states are terminal.

Depth does not need to wait for object-segmentation capacity. Caption review backlog does not keep depth GPUs alive. A QA-render queue can lag behind without changing the deliverable mask JSONL. If a component is paused for inspection or review, that hold must be explicit and visible in run state.

The same modularity makes model upgrades less disruptive. A hand-pose or mesh component can change under the hood without changing what downstream customers consume: the canonical hand artifact still carries frame numbers, track IDs, handedness, validity, provenance, and QA state. A segmentation model can be swapped only if it satisfies the same mask, track, depth-stat, and review contracts. Model choice is a component decision. The data contract remains stable.

Full-Loop Inference

The phrase "inference optimization" often collapses into one number: model frames per second. That number matters, but it is not the system bottleneck by itself.

At 30 fps, 100 hours/day represents 10.8 million frames per day. Scaling to 1,000 hours/day represents 108 million frames per day. The pipeline has to move those frames through decode, rectification, resize, tensor preparation, inference, post-processing, compression, artifact writes, manifest updates, QA rendering, review workflows, and final validation.

The actual throughput target is useful output per day.

StableBrowse therefore optimizes the whole loop around inference, not only the model call.

The point is not that every stage uses every optimization. The point is that each stage has an explicit execution contract, with the selected engine and configuration recorded in provenance:

Those optimizations are useful because they are connected to the data contract. Throughput that silently changes resolution, drops awkward frames, caps tracks, truncates windows, or hides missing depth is not production throughput.

The Moving Bottleneck

Once a model path is optimized, the bottleneck often moves into the ring around the model.

For a video propagation branch, the propagation kernel may not be the limiting step. The expensive parts can be video decode, model/session initialization, video-state construction, prompt normalization, mask post-processing, output materialization, and rendering review videos from saved results. Long object-segmentation clips also create a memory-shape problem: an unbounded propagation design can end up holding decoded frames, embeddings, temporal state, and materialized masks for too much of the video at once.

StableBrowse handles this with bounded temporal windows. The worker keeps only an active minute-scale window of video state in memory, writes outputs by frame index, and stitches windows through the manifest rather than relying on one huge in-memory propagation state. The windowing policy is a systems decision, not a quality shortcut: it exists to keep memory bounded while preserving the declared frame contract.

For stereo depth, inference can be only one part of the cost. The pipeline still has to write tensor bundles, transfer them, read them back for mask-depth joins, validate shapes, and render browser-friendly QA views. Thousands of tiny depth files can become a storage-request and listing problem even when the GPU stage is fast. For hand and caption branches, CPU parsing, coordinate transforms, compression, JSONL emission, and artifact upload can become visible once the model call is no longer the slowest line item.

The practical issue is rarely that someone intentionally runs a manifest-validation job on a GPU. It is that CPU-heavy work stays packaged inside a GPU worker because it is adjacent to inference code. A worker that initializes a model quickly but spends most of its wall time decoding, rendering, transferring, compressing, or writing small artifacts is not optimized in any meaningful system sense.

StableBrowse profiles those boundaries stage by stage. Some work should stay close to the GPU because moving it would add transfer overhead or break batching. Some work should move to CPU capacity near storage because it is dominated by file reads, joins, validation, rendering, or manifests. The pipeline is designed to make that separation possible without changing the dataset contract.

Artifact Layout

At scale, storage layout becomes part of the compute path.

Per-frame depth tensors are convenient during an experiment. They are painful at production volume. A stage that writes one object per frame can create a request, listing, and transfer problem that slows every later stage. The solution is not just "use faster storage." The output schema has to reflect how downstream stages consume data.

StableBrowse favors per-chunk packed artifacts and stage-level JSONL or columnar summaries. Depth can be stored as chunked tensor bundles with masks and frame indices. Segmentation masks can be emitted as per-chunk encoded rows. Mask-depth statistics can be written as compact summaries. QA videos can be rendered from saved artifacts without rerunning inference.

That layout has three benefits:

• downstream joins read coherent units instead of thousands of tiny files
• retries are still chunk-scoped and idempotent
• final delivery can promote accepted artifacts without dragging diagnostics into customer-facing output

Artifact layout is also how the system preserves quality signals. Missing depth should be explicit. Invalid regions should remain invalid. A mask row should carry the frame and track it belongs to. A QA preview should point back to the artifacts that generated it.

The same rule applies to review interfaces. A UI may sample, paginate, or cap what it displays for usability, but the artifact writer should still emit the complete accepted track set. Display constraints are UI behavior, not dataset semantics.

Callbacks And Heartbeats

Some post-processing branches take long enough that the orchestrator should not block compute while waiting. For those stages, StableBrowse uses callback-style execution: the control plane submits work, records a run reference, receives heartbeats and status updates, and resumes when the worker reports completion.

This shape is important for GPU jobs with queue wait. A wall-clock timeout sized around expected compute time can misinterpret capacity delay as a worker problem. Queue-aware status is more robust. A stage can be alive but waiting for capacity. A worker can be alive but inside a long video propagation section. A review stage can be waiting on humans without indicating an infrastructure problem.

Queue wait is not treated as a compute error. Long stages heartbeat from the moment work is accepted, including time spent waiting behind other chunks. The orchestrator holds a task reference, not a running machine, and hard timeouts exist only as runaway protection. Fan-out stays inside the compute plane; the control plane tracks stage state, artifact pointers, and progress without polling a custom global queue.

Interactive seed creation and full-video propagation are intentionally separated. A seed mask needs low-latency feedback for the annotator. Propagation needs asynchronous execution, bounded memory, checkpointed writes, and visible job state.

Backpressure causes queuing, not ambiguity. CPU queues, GPU queues, API concurrency limits, and human review queues are separate pressure points. The stage ledger should show which one is blocking progress.

The system keeps each kind of state separate:

This separation lets the system scale horizontally without turning every transient event into a manual incident.

Manifests Beat Markers

Completion markers are useful. They are not sufficient.

A robust pipeline validates the manifest behind the marker:

• Do referenced artifacts exist?
• Do frame counts match the declared output window?
• Are timestamps monotonic?
• Did the stage write the expected schema?
• Are masks in the expected coordinate space?
• Did a QA render come from saved artifacts?
• Did the output cover the declared frame range?
• Is a semantic label generated, edited, reviewed, or approved?

Those checks prevent a subtle class of quality issue: a marker says complete, but the data contract is incomplete.

A chunk is complete only when its checkpoint validates the declared frame range, input reference, output list, artifact sizes, stage configuration, runtime version, attempt number, wall time, compute time, and QA scalars. The run ledger is an index over this truth, not a replacement for validated artifacts.

StableBrowse's final promotion step reads validated stage outputs, builds the final manifest, records provenance, and promotes deliverable artifacts out of the execution-scoped work area. Work areas can contain retries, diagnostics, and intermediate artifacts. The final dataset should contain only the accepted contract.

Final success therefore means:

• required stage outputs exist
• manifests validate
• frame windows align
• QA summaries are present
• review state is represented
• deliverable artifacts have been promoted
• provenance points back to source inputs and processing stages

Work-bucket completion alone is not the finish line.

Observability

At 100 hours/day, logs are not enough. The system has to answer operational questions from the run ledger and stage metrics:

• Where is this recording in the graph?
• Which stage is blocking it?
• Is the blocker queue wait, compute, storage, review, or validation?
• What did this chunk cost?
• Which runtime and configuration produced it?
• Are QA metrics drifting relative to the expected range?

StableBrowse treats those metrics as part of the production surface. Queue depth, chunk latency, GPU utilization, CPU utilization, output bytes, cost per processed hour, retry count, and QA drift are tracked by stage. Those signals decide whether the next bottleneck is compute, decode, storage layout, review, or a quality regression.

QA Artifacts

Overlays, dashboards, and browser-friendly preview videos are essential. They let engineers and annotators inspect depth, masks, hand tracks, captions, timing alignment, and edge cases quickly.

But they are not the source of truth.

A preview render should be reproducible from saved artifacts. If a browser codec needs a compatibility render, the masks should not disappear. If an overlay has a projection issue, the underlying predictions should not be invalidated without checking the artifact report. If a dashboard summary is regenerated, it should point back to the same manifest.

This separation makes quality review faster without making the data product dependent on a visual side effect. It also lets visual QA run on CPU capacity after inference completes, instead of keeping GPU workers alive to render videos.

The Final Output

A StableBrowse processed episode can include:

• synchronized RGB frames or video windows
• computed stereo-derived depth with confidence and invalid regions preserved
• calibration and timing provenance
• hand state, including left/right identity where available
• 2D and 3D hand signals with validity flags
• hand masks and mask-depth statistics
• object mask candidates, tracks, and review status
• captions and action metadata
• trained annotator review and correction status
• QA previews and dashboards generated from saved artifacts
• manifests tying artifacts to source inputs and processing stages

The exact output package depends on the customer use case, but the underlying principle is the same: artifacts should not appear as anonymous files. They should be traceable robotics data.

Toward 1,000 Hours/Day

The path from 100 hours/day to 1,000 hours/day is not only "more machines."

More machines help only when the pipeline can preserve correctness under parallelism. The scaling work is in the contracts:

• stable session and chunk identities
• references instead of frame payloads in the control plane
• idempotent retries
• bounded temporal windows
• per-stage autoscaling windows
• stage-specific concurrency
• CPU/GPU placement
• packed artifact layout
• manifest validation
• stage-first observability
• storage lifecycle
• review-state separation
• quality metrics that surface the right cause

Scaling also changes which costs matter. A minor storage inefficiency becomes noticeable. A noisy retry policy becomes expensive. A manual QA step without assignment and approval state becomes a bottleneck. A stage that writes artifacts in a hard-to-join format slows every later stage. A GPU branch that spends too much time decoding, initializing, rendering, or writing is not ready for the next order of magnitude.

The architecture is built so throughput and quality scale together. The target is not just 1,000 hours/day of compute. It is 1,000 hours/day of data that remains auditable, reviewable, and useful for robotics.

The Engineering Lesson

The hard part of egocentric RGB-D post-processing is the boundary between computer vision and production systems.

Vision models produce signals. Robotics datasets need evidence. The pipeline has to connect them: calibration, timing, frame accounting, hand identity, depth confidence, mask provenance, human review, retries, manifests, and final promotion.

That is why StableBrowse treats post-processing as a core product surface. The output is not "a model ran on video." The output is a data product with enough structure for robotics teams to train on it, evaluate it, and trust what each artifact means.

That is the standard the pipeline is designed around as it moves from 100 hours/day toward 1,000 hours/day.