🤯 Sapiens2: AI Sees the World 👁️

SAPIENS2: A New Foundation Model for Human-Centric Vision
Sapiens2 represents a significant advancement in human-centric computer vision, addressing the inherent challenges of modeling complex human forms with high accuracy and robustness. This new foundation model offers a substantial leap over its predecessor, Sapiens, across numerous benchmarks, demonstrating improved performance in diverse visual tasks.

THE CHALLENGES OF HUMAN-CENTRIC COMPUTER VISION
Human-centric computer vision faces unique difficulties compared to traditional object recognition. Unlike static objects, humans possess articulated structures, intricate surface details, and exhibit extreme variation in pose, clothing, lighting, and ethnicity. Accurately understanding these complexities simultaneously across arbitrary real-world images is a computationally intensive and conceptually challenging task. Existing methods often struggle to capture the nuances of human appearance, leading to inaccuracies in segmentation, depth estimation, and other related tasks.

A COMBINED APPROACH: MAE AND CL
The design of Sapiens2 incorporates a sophisticated combination of two key techniques: Masked Autoencoder (MAE) and contrastive learning (CL). MAE, a form of masked image modeling (MIM), forces the model to learn spatial details and textures by reconstructing masked image patches. However, MAE primarily learns through compression and struggles to capture high-level semantic understanding. Contrastive learning (CL), like DINO and SimCLR, organizes representations semantically by treating different views of the same image as similar. While CL excels at capturing semantic relationships, its aggressive augmentation strategies can strip away crucial appearance cues, leading to representation drift.

ADDRESSING REPRESENTATION DRIFT WITH JOINT OBJECTIVES
Sapiens2 directly tackles this problem by combining both MAE and CL objectives. A masked image reconstruction loss (LMAE) preserves low-level fidelity, while a global contrastive loss (LCL) using a student-teacher framework based on DINOv3 maintains semantic coherence. Critically, color augmentations are restricted to the MAE objective, preserving the appearance cues essential for photorealistic tasks. This joint objective, represented as L = LMAE + λLCL, ensures a balanced approach that leverages the strengths of both techniques.

A CURATED AND DIVERSE TRAINING DATASET
Achieving a dataset of 1 billion human images required a meticulous multi-stage filtering pipeline. Starting from a web-scale pool of approximately 4 billion images, Meta AI employed bounding box detection, head-pose estimation, aesthetic and realism scoring, CLIP-based feature filtering, and text-overlay detection to identify and isolate relevant images. The resulting corpus consistently contains at least one prominent person with a minimum short-side resolution of 384 pixels.

DATA DIVERSITY AND QUALITY
To ensure broad representation, the research team utilized perceptual hashing and deep-feature nearest-neighbor pruning for deduplication. Visual embeddings were clustered, and selective sampling was applied to balance the dataset across poses, viewpoints, occlusion levels, clothing types, and lighting conditions. Notably, no task labels or human-specific priors were injected during pretraining, relying solely on the raw image data.

MODEL ARCHITECTURE AND SCALE
Sapiens2 offers four model sizes: 0.4B, 0.8B, 1B, and 5B parameters, each operating at native 1K resolution. The 5B model represents the highest-FLOPs vision transformer reported to date, achieving 15.722 TFLOPs. For 4K resolution, a hierarchical windowed attention design is implemented, employing windowed self-attention locally within spatial windows, followed by a [CLS]-guided pooling step. This architecture maintains compatibility with MAE-style pretraining, preventing information leakage during masked token handling.

OPTIMIZED ARCHITECTURAL COMPONENTS
Several architectural improvements enhance Sapiens2's stability and performance. RMSNorm replaces LayerNorm, Grouped-Query Attention (GQA) is utilized in mid-depth blocks for increased throughput, QK-Norm ensures robust high-resolution training, and SwiGLU feed-forward layers are incorporated. The decoder employs pixel-shuffle upsampling for sub-pixel reasoning, and the decoder output resolution is increased to 1K for base backbones and 2K for 4K backbones.

IMPROVED TASK-SPECIFIC SUPERVISION
A critical advancement is the scale and quality of task-specific supervision. Compared to the original Sapiens, Sapiens2 utilizes 10x more labels, reaching approximately 1 million labels per task. Fine-tuning is performed on five downstream tasks using lightweight task-specific heads, leaving the backbone unchanged.

EVALUATION AND PERFORMANCE RESULTS
The 11K-image in-the-wild pose test set demonstrates a 4 mAP improvement with Sapiens2-5B compared to Sapiens-2B. On body-part segmentation, the smallest model, Sapiens2-0.4B, achieves 79.5 mIoU, a 21.3% gain over Sapiens-2B. Sapiens2-5B reaches 82.5 mIoU, a 24.3% improvement. For surface normal estimation, Sapiens2-0.4B achieves a mean angular error of 8.63°, outperforming DAViD-L at 10.73°. The 5B model reduces this further to 6.73°, with a median angular error of just 3.08°. Albedo estimation shows consistent improvement across all model sizes, with Sapiens2-5B achieving an MAE of 0.012 and a PSNR of 32.61 dB. In pointmap estimation, all Sapiens2 model sizes outperform MoGe. Dense probing evaluations confirm Sapiens2-5B’s superior performance, surpassing previous state-of-the-art models.