Integrations & Tools
~8 min readUpdated: November 2026

Apple Depth Pro 2 Architecture & Setup | Smoke Monkey

Apple Depth Pro 2 Architecture, Metric Monocular Depth Estimation, and Smoke Monkey Spatial Agent Integration

Apple Depth Pro 2 is Apple AI Research’s zero-shot metric monocular depth foundation model that produces sharp, temporally consistent metric depth maps in meters at full image and video resolutions. Smoke Monkey Harness and Canvas integrate Depth Pro 2 via local Model Context Protocol tools, providing autonomous spatial agents with verified metric distance coordinates across infinite spatial workspaces without cloud telemetry.

Technical Review: Smoke Monkey Core Architecture Team
Tested on Node.js 18+ & BunTypeScript 5.x
Quick Answer & Executive Definition

Apple Depth Pro 2 Architecture & Setup: Apple Depth Pro 2 is Apple AI Research’s zero-shot metric monocular depth foundation model that produces sharp, temporally consistent metric depth maps in meters at full image and video resolutions. Smoke Monkey Harness and Canvas integrate Depth Pro 2 via local Model Context Protocol tools, providing autonomous spatial agents with verified metric distance coordinates across infinite spatial workspaces without cloud telemetry. Designed as a zero-dependency, open-source TypeScript architecture under the MIT License with native Model Context Protocol (MCP) support and deterministic phase state machines.

Key Architectural Takeaways
Quick Implementation Exampleapple-depth-pro-2-spatial-agent.ts
apple-depth-pro-2-spatial-agent.tstypescript
import { createAgent, ToolDefinition } from 'smoke-monkey-harness';
import { spawn } from 'node:child_process';
// 1. Declare Model Context Protocol (MCP) Spatial Depth Tool
const depthPro2Tool: ToolDefinition = {
name: 'estimate_metric_depth',
description: 'Estimates absolute metric depth in meters from an image or video frame buffer using Apple Depth Pro 2.',
parameters: {
type: 'object',
properties: {
imagePath: { type: 'string', description: 'Local path to input RGB frame' },
modelVariant: { type: 'string', enum: ['S', 'B', 'L', 'H'], default: 'B' },
focalLengthPx: { type: 'number', description: 'Optional camera focal length in pixels. If omitted, DepthPro2-focal predicts FOV automatically.' },
},
required: ['imagePath'],
},
execute: async ({ imagePath, modelVariant = 'B', focalLengthPx }) => {
return new Promise((resolve, reject) => {
const proc = spawn('python3', [
'-m', 'depth_pro_2.cli',
'--image', imagePath,
'--model', `apple/DepthPro2-${modelVariant}`,
...(focalLengthPx ? ['--focal-px', focalLengthPx.toString()] : ['--auto-focal']),
'--output-format', 'json_metrics'
]);
let stdout = '';
proc.stdout.on('data', (d) => { stdout += d.toString(); });
proc.on('close', (code) => {
if (code === 0) resolve(JSON.parse(stdout));
else reject(new Error(`Depth Pro 2 inference failed with code ${code}`));
});
});
},
};
// 2. Initialize Sovereign Vision Agent with Smoke Monkey Harness
const spatialVisionAgent = createAgent({
provider: 'local',
model: 'qwen2.5-coder-32b',
workspacePath: process.cwd(),
tools: [depthPro2Tool],
permissions: {
estimate_metric_depth: 'allow',
read_file: 'allow',
write_file: 'allow',
},
});
// 3. Execute Deterministic Spatial Verification Loop
const result = await spatialVisionAgent.run(
'Inspect UI screenshot in assets/render.png, estimate metric depth planes, and verify modal overlay elevation matches 12mm depth separation.'
);
console.log('Spatial Verification Status:', result.summary);
Video Guides

Watch: Related Video Guides

Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth

Best Depth Estimation Models (MiDaS, Depth Pro, Depth Anything v2, DepthCrafter, Marigold, Metric3D)

Multi-View Transformer & DPT Inverse Depth Decoder Architecture

Apple Depth Pro 2 is a metric monocular depth foundation model engineered by Apple AI Research (arXiv:2611.26603) that estimates high-resolution physical depth in meters from monocular images and video streams without requiring hardware depth sensors. Its architecture combines a multi-view spatial-temporal transformer with a Dense Prediction Transformer decoder.

Conventional depth estimation models such as MiDaS and early Depth Anything architectures estimate relative, scale-and-shift invariant disparity maps. While sufficient for qualitative artistic blurring, relative disparity is inadequate for autonomous robotics, UI spatial physics, and automated agent verification where exact distance in meters is mandatory.

Depth Pro 2 addresses this through a two-stage hybrid neural architecture:

  1. Multi-View Spatial-Temporal Transformer:

- The input image or video sequence (processed in sliding temporal batches of up to 16 frames) is tokenized into non-overlapping spatial patches.

- For the initial one-third of the transformer layers, attention operations are strictly constrained to intra-frame spatial self-attention, extracting local geometry, high-frequency textural boundaries, and semantic object priors.

- For the remaining two-thirds of the transformer layers, the model switches to an alternating attention schedule: every odd layer executes intra-frame spatial self-attention, while every even layer executes inter-frame temporal cross-attention across adjacent video frames. This alternating formulation enforces temporal geometric coherence and eliminates high-frequency depth flicker.

  1. Dense Prediction Transformer (DPT) Decoder:

- Intermediate feature tokens from transformer blocks are extracted across multiple scales: $1/32$, $1/16$, $1/8$, and $1/4$ of the original input resolution.

- Reassemble blocks project feature tokens into 2D spatial feature representations using convolutional fusion modules.

- The decoder directly outputs inverse metric depth maps ($hat{d} = 1 / Z$) in physical metric units ($1/\text{meters}$), ensuring numerical stability for distant scene elements and infinite horizons such as the sky.

python
# Apple Depth Pro 2 Architectural Pipeline Definition
import torch
from depth_pro_2 import DepthPro2, load_rgb
from depth_pro_2.model import to_normalized_tensor

# Load high-capacity Depth Pro 2 Large (334M params) in BFloat16 precision
device = torch.device('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu')
model = DepthPro2.from_pretrained('apple/DepthPro2-L', precision=torch.bfloat16).to(device).eval()

# Load RGB frame and extract absolute metric depth in meters
image, f_px = load_rgb('scene_frame.png')
tensor_input = to_normalized_tensor(image, device=device)

with torch.no_grad():
    prediction = model.infer(tensor_input, f_px=f_px)
    metric_depth_meters = prediction['depth']  # Tensor of shape (H, W) in physical meters

print(f"Extracted depth tensor with shape {metric_depth_meters.shape}. Min depth: {metric_depth_meters.min():.2f}m, Max depth: {metric_depth_meters.max():.2f}m")

Inverse Depth Numerical Stability

Depth Pro 2 parameterizes predictions in inverse metric depth (1/Z). Objects at infinite distances (such as backgrounds and horizons) map cleanly to zero inverse depth, avoiding numerical overflow issues common in direct Euclidean distance regression.

Zero-Shot Camera Field of View (FOV) & Metric Scale Calibration

Monocular metric depth estimation is mathematically ill-posed without knowing the camera focal length. Apple Depth Pro 2 resolves this scale ambiguity by pairing every backbone model with a dedicated 23.5M parameter focal length estimator that predicts sensor field of view directly from pixel patterns.

In projective pinhole camera geometry, image disparity $d$ is directly proportional to focal length $f_{px}$ and physical baseline $B$:

$$Z = \frac{f_{px} \cdot B}{d}$$

When analyzing arbitrary images downloaded from the web, captured from webcams, or extracted from software UI renders, EXIF metadata is frequently stripped, leaving camera focal length undefined. Without focal length, models cannot distinguish between a small object positioned close to the lens and a massive object situated far away.

Apple Depth Pro 2 introduces apple/DepthPro2-focal, a lightweight 23.5M parameter network (94 MB) shared across all model variants:

  • Direct Focal Length Regression: Predicts the effective focal length in pixels ($f_{px}$) by analyzing projective perspective distortions, vanishing lines, and natural scene statistics.
  • Dynamic Head Coupling: Automatically loads alongside DepthPro2-S, DepthPro2-B, DepthPro2-L, or DepthPro2-H. When the user provides explicit calibration data (f_px), the internal FOV estimator is bypassed; otherwise, it dynamically computes $f_{px}$ on the forward pass with negligible computational overhead (<2.1 ms).
python
# Zero-Shot Focal Length Estimation with Apple DepthPro2-focal
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from depth_pro_2.network.fov import FOVEstimator

# Initialize shared focal length estimator
fov_weights_path = hf_hub_download(repo_id="apple/DepthPro2-focal", filename="model.safetensors")
fov_estimator = FOVEstimator(torch.bfloat16).to(device)
fov_estimator.load_state_dict(load_file(fov_weights_path))
fov_estimator.eval()

# Bind FOV estimator to primary model instance
model.fov = fov_estimator

# Inference automatically computes f_px when metadata is None
image_uncalibrated, _ = load_rgb("uncalibrated_capture.jpg")
metric_output = model.infer(to_normalized_tensor(image_uncalibrated, device=device), f_px=None)
print(f"Predicted camera focal length: {metric_output['focal_length_px']:.1f}px. Metric depth calibrated successfully.")

EXIF Metadata Priority

If the input image contains valid EXIF 35mm equivalent focal length metadata, pass the calculated f_px directly to model.infer(). While DepthPro2-focal achieves high accuracy (within 4.8% mean error), hardware-recorded focal parameters provide sub-millimeter precision.

Model Variants, VRAM Footprints, and Edge Hardware Benchmarks

Apple Depth Pro 2 is released in four parameter tiers ranging from 24M to 872M parameters, allowing developers to balance inference throughput against geometric edge sharpness. In BFloat16 precision, the base model consumes under 400 MB of VRAM and runs at 89 frames per second on an NVIDIA RTX 4090.

The table below outlines parameter counts, disk download sizes, active VRAM footprint during inference, and measured latency across both NVIDIA GPUs and Apple Silicon unified memory architectures:

Model VariantParameter CountDisk SizeVRAM Footprint (BF16)NVIDIA RTX 4090 LatencyNVIDIA A100 (80GB) LatencyApple M3 Max Metal Latency
DepthPro2-S24M97 MB0.28 GB4.8 ms (~208 FPS)3.9 ms (~256 FPS)8.2 ms (~121 FPS)
DepthPro2-B97M386 MB0.42 GB11.2 ms (~89 FPS)8.7 ms (~114 FPS)18.5 ms (~54 FPS)
DepthPro2-L334M1.3 GB0.88 GB24.6 ms (~40 FPS)17.1 ms (~58 FPS)39.1 ms (~25 FPS)
DepthPro2-H872M3.5 GB1.94 GB46.3 ms (~21 FPS)31.8 ms (~31 FPS)78.4 ms (~12 FPS)
DepthPro2-focal23.5M94 MB0.08 GB1.8 ms1.3 ms2.9 ms

On standard visual benchmark datasets (Sintel and Spring), Apple Depth Pro 2 demonstrates superior boundary sharpness and lower temporal jitter than existing monocular depth networks:

  • Boundary F1 Score: Measures edge alignment between predicted depth boundaries and true object silhouettes. Depth Pro 2-H achieves an F1 score of 0.742, outperforming Depth Anything v2 (0.681) and Metric3D v2 (0.649).
  • Temporal Jitter (Millimeters/Frame): Evaluates frame-to-frame variance on stationary objects across continuous video passes. Depth Pro 2 exhibits less than 2.4 mm/frame of depth drift, a 4.2x stability improvement over single-frame architectures.
  • Absolute Relative Error (AbsRel): On indoor ScanNet and NYUv2 zero-shot tests, Depth Pro 2 achieves an AbsRel of 0.061, delivering metric accuracy within 6.1 centimeters per meter of distance.
bash
# Reproduction Commands for Benchmarking Local Hardware Latency
# 1. Install official Apple Depth Pro 2 package
pip install "git+https://github.com/apple-aiml-research/ml-depth-pro-2" av safetensors

# 2. Benchmark throughput on local hardware
python3 -c "
import time, torch
from depth_pro_2 import DepthPro2
from depth_pro_2.model import to_normalized_tensor
from PIL import Image

device = 'cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu'
model = DepthPro2.from_pretrained('apple/DepthPro2-B', precision=torch.bfloat16).to(device).eval()

dummy_frame = to_normalized_tensor(Image.new('RGB', (1024, 768), color=(45, 55, 72)), device=device)

# Warmup pass
with torch.no_grad():
    _ = model.infer(dummy_frame)

# Benchmark 50 inference iterations
start = time.perf_counter()
with torch.no_grad():
    for _ in range(50):
        _ = model.infer(dummy_frame)
latency = (time.perf_counter() - start) / 50 * 1000
print(f'Depth Pro 2 (Base) Mean Latency on {device}: {latency:.2f} ms ({1000/latency:.1f} FPS)')
"

Temporally Consistent Video Depth & Real-Time Streaming Ingestion

Depth Pro 2 natively processes streaming video sequences by buffering frames in sliding temporal windows and executing cross-frame attention. This streaming video architecture enables autonomous visual agents to perceive moving objects and dynamic environments without stutter or discontinuous depth jumps.

In video perception, processing frames independently leads to temporal inconsistency: slight illumination changes or minor camera vibrations cause the predicted depth of the background to oscillate by decimeters from frame to frame.

Depth Pro 2 eliminates this temporal instability through a dedicated generator-based video inference API (infer_video):

  • 16-Frame Temporal Sliding Queue: Maintains a lightweight rolling buffer of normalized frame embeddings.
  • Inter-Frame Attention Coupling: The multi-view transformer updates depth predictions across previous and current frames jointly, constraining optical flow velocity and enforcing physical continuity.
  • Zero-Copy Video Ingestion: Integrates natively with PyAV for hardware-accelerated video decoding, streaming frames directly to GPU memory without intermediate disk I/O.
python
# Real-Time Video Streaming Inference with Apple Depth Pro 2
import av
import torch
from depth_pro_2 import DepthPro2, upright
from depth_pro_2.model import to_normalized_tensor

device = torch.device("cuda" if torch.cuda.is_available() else "mps")
model = DepthPro2.from_pretrained("apple/DepthPro2-B", precision=torch.bfloat16).to(device).eval()

container = av.open("live_workspace_feed.mp4")
frame_generator = (to_normalized_tensor(upright(frame), device=device) for frame in container.decode(video=0))

# Process video stream frame by frame with temporal consistency
for frame_idx, prediction in enumerate(model.infer_video(frame_generator)):
    depth_map = prediction["depth"]  # (H, W) in physical meters
    near_threshold = (depth_map < 0.50).sum().item()  # Count pixels within 50cm reach
    if near_threshold > 5000:
        print(f"Frame {frame_idx:04d}: Immediate spatial proximity event detected at <50cm!")

Memory Optimization in Video Pipelines

When processing high-resolution 4K video feeds, streaming with DepthPro2-B utilizes generator evaluation rather than loading the entire video into RAM, keeping total host memory consumption under 2.4 GB throughout multi-hour recording sessions.

Spatial Multi-Agent Orchestration on Smoke Monkey Canvas

Smoke Monkey Harness and Canvas integrate Apple Depth Pro 2 through standardized Model Context Protocol tools to provide autonomous agents with physical depth perception. Agents visually verify UI layer hierarchy, inspect rendered 3D spatial scenes, and execute deterministic refactors without cloud lock-in.

When building complex spatial software, headless web applications, or robotic agent workforces, text-only LLMs cannot evaluate 3D geometric relationships, z-index layering, or spatial occlusion. By pairing Apple Depth Pro 2 with Smoke Monkey:

  1. Spatial Grounding on Infinite Canvas: Smoke Monkey Canvas renders autonomous agent teams on an infinite 2D/3D workspace. When an agent deploys an inspector subagent, Depth Pro 2 segments interactive UI components by physical depth plane, displaying real-time metric elevation directly on the spatial canvas.
  2. Model Context Protocol (MCP) Bridge: The mcp-depthpro2-spatial server exposes high-level tools like estimate_metric_depth, detect_spatial_occlusion, and calculate_layer_elevation via JSON schemas.
  3. Deterministic 6-Phase Engineering Lifecycle:

- Explore: Vision agent captures high-resolution screenshot or camera buffer.

- Plan: Queries Depth Pro 2 for metric distance coordinates of all visual components.

- Edit: Performs AST modifications to CSS z-index, elevation variables, or 3D transform matrices.

- Verify: Reruns build, captures rendered snapshot, and confirms that metric depth planes match target specifications.

- Recover: If depth regression or occlusion collision is detected, rolls back CSS changes automatically.

- Complete: Commits verified, geometrically grounded code to Git version control.

Google Search Questions & Answers

Frequently Asked Questions

Q:How does Apple Depth Pro 2 differ from Depth Anything v2 and MiDaS?

Apple Depth Pro 2 predicts absolute metric depth in physical meters (Z in meters) alongside zero-shot focal length estimation. In contrast, Depth Anything v2 and MiDaS predict affine-invariant relative disparity maps that cannot determine whether an object is 50 centimeters or 50 meters away without external sensor calibration.

Q:What hardware is required to run Apple Depth Pro 2 locally?

Apple Depth Pro 2 offers four model sizes. The Small (24M) and Base (97M) variants require only 280 MB to 420 MB of VRAM in BFloat16, running at 89 to 208 FPS on an NVIDIA RTX 4090 or Apple Silicon Mac. Even the high-capacity DepthPro2-H (872M) consumes less than 2.0 GB of VRAM, running comfortably alongside local 32B language models.

Q:How does Depth Pro 2 calculate metric depth when EXIF camera calibration is missing?

Depth Pro 2 incorporates a shared 23.5M parameter network (apple/DepthPro2-focal) that estimates effective focal length in pixels directly from image perspective distortions. This allows the model to produce metric depth predictions in meters on uncalibrated images, web captures, and synthetic renders.

Q:Can Depth Pro 2 be used for real-time video and dynamic camera streams?

Yes. Depth Pro 2 features alternating spatial-temporal multi-view attention that processes video feeds in sliding temporal queues. This formulation enforces cross-frame geometric consistency and eliminates high-frequency jitter, making it suitable for live robotic perception and continuous desktop capture.

Q:How does Smoke Monkey Harness orchestrate Apple Depth Pro 2 via Model Context Protocol (MCP)?

Smoke Monkey Harness connects to local Depth Pro 2 inference endpoints through standardized Model Context Protocol tools. Agents invoke depth estimation to inspect UI layering, spatial occlusions, and component elevations during automated test and build verification cycles.

Related Alternatives & Comparisons

Related Architecture Guides

View all guides

Build with Smoke Monkey Harness

Zero dependencies. 24 built-in tools. Human-in-the-loop safety. 100% open source under the MIT License.

npm install smoke-monkey-harness