GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives

1]XPeng Inc. 2]Peking University 3]The University of Hong Kong 4]University of California, Berkeley 5]Princeton University 6]National University of Singapore 7]Tsinghua University 8]HKUST (GZ) \contribution[*]Equal contribution \contribution[‡]Project lead \contribution[†]Corresponding authors \projecthttps://groundingpi.github.io/ \codehttps://github.com/groundingpi/GroundingPI \modelhttps://huggingface.co/GroundingPI/GroundingPI \titleteaserfigures/teaser.pdf

Abstract

Precise grounding matters. It specifies which object is the target and where that object is, even in clutter and for tiny objects, and it has to be fast enough for closed-loop control. Yet vision-language-action (VLA) and world-action models (WAMs) take perception from general-purpose vision-language and video-generation backbones, which still fail in these settings. We introduce GroundingPI, a 4B grounding foundation model that generates points and boxes as quantized coordinates in a shared vocabulary. Training combines multimodal and spatial pretraining, supervised fine-tuning, and reinforcement learning with GRPO, using supervision from public datasets and dedicated data engines. Against 44 baselines across 34 grounding benchmarks spanning 11 perceptual capabilities, GroundingPI establishes a new state of the art, averaging 73.68%, above the larger GPT-6 Astra (71.54%). As a downstream visual backbone, GroundingPI improves performance on robotic manipulation and autonomous driving. On RoboTwin 2.0, it outperforms every mainstream backbone we evaluate in all four out-of-distribution settings, by up to 24.8% relative to the strongest backbone. On RoboCasa-GR1, GroundingPI trained with 50% of the demonstrations outperforms those baselines trained with 75%. On nuScenes, used as the visual backbone, GroundingPI attains an average open-loop L2 error of 0.296 m. We systematically analyze GroundingPI’s pretraining in scale and data composition. Downstream autonomous driving and robotic manipulation improve as the pretraining is scaled. Analyzing the data recipe across these 11 perceptual capabilities shows dense grounding’s substantial benefits for both, and OCR’s potential as a catalyst for perceptual learning. These results support grounding as a perceptual foundation, and dedicated perceptual pretraining as a promising direction for foundation models of physical intelligence.

Abstract

\abstractlist

Contents

Appendix Contents

1 Introduction

Visual grounding connects language to objects, locations, and interaction-relevant structures, making it a core perceptual capability of vision-language models (VLMs). Despite recent progress in grounding UI elements (Lin et al., 2024; Liu et al., 2025), regions (Lai et al., 2024; Ren et al., 2024), and task-relevant entities (Zhang et al., 2023; Jiang et al., 2026; Yu et al., 2025), achieving broad and precise grounding for physical intelligence (Black et al., 2024; Wang et al., 2026b; Kim et al., 2024) remains challenging.

When tidying a cluttered desk, one finds the mug and its handle before deciding how to grasp and move it, establishing where to act before determining how to act. Vision-language-action (VLA) models (Kim et al., 2024; Black et al., 2024; NVIDIA et al., 2025) and world-action models (WAMs) (Kim et al., 2026; Ye et al., 2026; Wang et al., 2026b) increasingly build on general-purpose vision-language and video-generation backbones. However, these backbones are not primarily optimized for the precise grounding required by physical interaction and leave perception as a bottleneck for downstream action learning.

Our evaluations reveal substantial weaknesses in general-purpose VLMs and considerable room for improvement in dedicated models such as Rex-Omni and LocateAnything (Jiang et al., 2026; Wang et al., 2026a). These limitations are particularly evident in demanding settings such as dense scenes and tiny objects, where Qwen3.5 models struggle and even GPT-6 Astra leaves room for improvement. Such perceptual gaps can leave scarce action data responsible for both perceptual and action learning. Compute and latency constraints further limit reliance on larger backbones for action execution (NVIDIA et al., 2025; Chen et al., 2026b). These gaps motivate building a more capable grounding foundation model.

Recent policy recipes already incorporate perceptual learning: the progression from \pi_{0} (Black et al., 2024) to \pi_{0.5} (Intelligence et al., 2025) adds bounding-box prediction within a broader knowledge-insulation recipe (Black et al., 2024; Intelligence et al., 2025; Driess et al., 2025), while other approaches introduce auxiliary modules and learning objectives to enhance perception (Yu et al., 2026; Song et al., 2026; Tu et al., 2026; Chen et al., 2026c). We argue that a more natural and scalable approach is to develop strong grounding as a native capability of the foundation model. We pursue a grounding foundation model built on visual primitives, aiming to advance broad, precise perception and investigate its value for physical intelligence. We ask:

To address these questions, we introduce GroundingPI, a 4B-parameter grounding foundation model for a broad range of perception tasks, built on visual primitives including points and bounding boxes. A shared vocabulary with quantized coordinates unifies diverse perception tasks as language-conditioned structured generation, covering grounding, referring, pointing, OCR, GUI and layout grounding, and visual prompting (). Training combines multimodal and spatial pretraining, supervised fine-tuning, and reinforcement learning with GRPO, using supervision from public datasets and dedicated data engines.

Evaluated against 44 baselines across 34 grounding benchmarks spanning 11 perceptual capabilities, GroundingPI achieves 73.68% on average, establishing a new state of the art, averaging 73.68%, above the larger GPT-6 Astra (71.54%). We compare with GPT-6 Astra specifically to explore the potential and limits of improving basic perceptual performance. We further evaluate transfer to autonomous driving on nuScenes and robotic manipulation on RoboTwin 2.0 and RoboCasa-GR1. Within each manipulation benchmark, we fix the Action DiT, action data, and training budget. GroundingPI demonstrates strong transfer to physical intelligence tasks, consistently outperforming all evaluated mainstream backbones, in all four OOD settings, with relative improvements of up to 24.8% over the strongest baseline. Action-data efficiency also improves in robotic manipulation: on RoboCasa-GR1, GroundingPI trained with 50% of demonstrations outperforms all compared baselines trained with 75%. These results support the value of a stronger perceptual foundation for downstream action learning.

Our analyses offer insights into perceptual pretraining and the design of embodied foundation models. Scaling and data ablations show that both pretraining scale and data composition matter for grounding and downstream transfer. Basic grounding provides an important foundation, dense grounding substantially benefits autonomous driving and robotic manipulation, and OCR shows potential as a catalyst for perceptual learning. Our design insights focus on perceptual foundations for System-1-style execution (NVIDIA et al., 2025; Chen et al., 2026b). We discuss possible directions for designing such embodied foundation models to complement the high-level reasoning and planning of frontier models such as Astra. Visual primitives may further enable physical prompting, specifying targets, locations, and structures alongside language.

In summary, our major contributions are as follows:

• A strong grounding foundation model. We introduce GroundingPI, a 4B model built on visual primitives, with a staged training recipe and state-of-the-art grounding performance.
• Transfer toward physical intelligence. Autonomous driving and robotic manipulation evaluations demonstrate the value of this perceptual foundation, including strong ID and OOD performance and improved action-data efficiency.
• Insights into perceptual pretraining and future embodied paradigms. We analyze how pretraining scale and data composition shape grounding and transfer, and discuss implications for System-1 foundation-model design and its complementary role in future embodied systems.

2 Related Work

2.1 Visual Grounding Models

Visual grounding has evolved from closed-set object detection (Redmon et al., 2016; Carion et al., 2020) to language-conditioned localization beyond fixed vocabularies through grounded image-text pretraining (Li et al., 2022; Liu et al., 2024). Generative grounding models adopt autoregressive next-token prediction to produce labels and quantized coordinates as structured sequences, providing a shared interface across diverse perception tasks (Chen et al., 2022; Xiao et al., 2024; Jiang et al., 2026), while LocateAnything explores precise grounding with parallel box decoding (Jiang et al., 2026; Wang et al., 2026a). Spatial affordance prediction and spatial referring extend these capabilities toward robotic interaction (Yuan et al., 2024; Zhou et al., 2026; Wu et al., 2026). These advances motivate studying broad and precise grounding as a transferable perceptual foundation for physical intelligence.

2.2 Foundation Models for Physical Intelligence

Physical-intelligence models increasingly inherit complementary priors from large-scale pretraining. VLA models build on VLMs to transfer broad visual-semantic knowledge and language grounding (Kim et al., 2024; Black et al., 2024), while WAMs adapt video and world models to leverage spatiotemporal and visual-dynamics priors (Kim et al., 2026; Ye et al., 2026; Wang et al., 2026b). Embodied foundation models further combine semantic, dynamic, and embodied experience toward more general physical intelligence (Intelligence et al., 2025; NVIDIA et al., 2025; Dang et al., 2026; NVIDIA, 2026). However, these broad capabilities do not necessarily provide the precise, task-conditioned perception required for physical interaction, leaving a capability mismatch between pretraining and action learning. Together, this capability mismatch motivates closer study of how precise perception can support action learning.

2.3 Perceptual Learning for Action Transfer

Recent policy recipes strengthen perception through bounding-box prediction, task-relevant region reconstruction, affordance representations, and spatial auxiliary supervision (Intelligence et al., 2025; Song et al., 2026; Yu et al., 2026; Tu et al., 2026; You et al., 2026; Shen et al., 2026). These approaches demonstrate the value of perception, but usually introduce it as a task-specific proxy, auxiliary objective, or intermediate representation within action learning. Such mechanisms cover selected perceptual signals, often depend on additional annotations or policy-specific components, and are difficult to scale across the diverse scenes and perceptual demands of physical intelligence. Together, these limitations motivate a perception-native foundation model: one that learns broad and precise perception before action adaptation, so downstream policies can build on perception rather than recover it from task-specific proxies.

3 GroundingPI: A Grounding Foundation Model

3.1 Model Architecture and Grounding Formulation

GroundingPI is an autoregressive grounding foundation model for language-conditioned perception. As shown in Figure 1, it couples a MoonViT-V2 (Kimi K3) visual encoder (Team et al., 2026) with a Qwen3-4B language decoder (Yang et al., 2025a). A learnable projector aggregates adjacent 2\times 2 visual features and maps them to the language embedding space through a two-layer MLP with GELU and output normalization.

Given an image I and a query P, the encoder E_{\psi} and projector C_{\phi} produce visual embeddings V=C_{\phi}(E_{\psi}(I)). The decoder generates a variable-length response Y autoregressively, with p_{\theta}(Y\mid I,P)=\prod_{i=1}^{|Y|}p_{\theta}(y_{i}\mid V,P,y_{<i}). Semantic labels, protocol markers, and 1,000 quantized coordinate tokens (<0>–<999>) share the output vocabulary; responses are variable-length; an absent queried category yields None. Further input/output conventions are detailed in Section 7.1.

3.2 GroundingPI Data

Training data combines public datasets with annotations produced by our data engine.

Our data engine fuses multi-teacher annotations at the field level and applies task-specific validation (Figure 2). Accepted labels train a unified grounding expert for iterative annotation, while complementary teachers and local observations resolve uncertain cases to expand supervision and refine existing labels. Field validation, task dependencies, and label revision are detailed in Section 7.3.

3.3 Training Design

3.3.1 Base VLM Training

Base training first aligns the projector using image-caption supervision while freezing both backbones. It then updates all modules in two stages: joint multimodal pretraining on text and image–text supervision, followed by general visual/video understanding through question answering and captioning. All stages use causal next-token prediction. Loss normalization and optimization details are provided in Section 7.4.

3.3.2 Supervised Fine-Tuning

SFT aligns coordinates with visual locations and learns the shared output protocol. Under teacher forcing, we minimize \mathcal{L}_{\mathrm{SFT}}=-|\mathcal{S}|^{-1}\sum_{t\in\mathcal{S}}\log p_{\theta}(y_{t}\mid I,P,y_{<t}), where \mathcal{S} contains assistant-response labels, protocol markers, and coordinates, excluding prompt and visual positions. Supervised adaptation first updates all modules, then freezes the vision encoder and projector for language-side refinement. Module schedules and hyperparameters are given in Section 7.4.

3.3.3 Reinforcement Post-Training

To directly optimize localization, target coverage, and text–geometry consistency, we apply group relative policy optimization (GRPO) (Shao et al., 2024; Ping et al., 2026) with G=8 autoregressive responses per image–query pair (Figure 3). For grounding, R_{\mathrm{set}} measures coverage through reference-wise maximum-IoU matching with class validation. The complementary R_{\mathrm{strict}} combines multi-threshold F1, localization, format, count, and ordering scores, penalizing duplicate and oversized boxes. The independently standardized rewards yield \widetilde{A}_{i}=0.7Z(R_{\mathrm{set},i})+0.3Z(R_{\mathrm{strict},i}). For OCR, Hungarian one-to-one matching evaluates text and geometry across complete word/line views or complementary references. Format gating and output penalties produce one composite reward, giving \widetilde{A}_{i}=Z(R_{\mathrm{OCR},i}). Here Z standardizes each active reward within a prompt; the combined advantages are then standardized across the response batch. We optimize the clipped GRPO objective with a frozen SFT reference, updating only language parameters. Reward definitions and optimization details are provided in Sections 7.6.1, 7.6.2 and 7.6.

4 GroundingPI Model towards Physical Intelligence

GroundingPI provides a strong perceptual interface for action learning. We study whether its precise, language-conditioned grounding representations translate into stronger action-learning capabilities in two downstream physical-intelligence paradigms: trajectory prediction for autonomous driving and continuous control for robot manipulation. Each domain retains its native action learner, while GroundingPI serves as the shared perceptual foundation.

4.1 Autonomous Driving

We formulate autonomous driving on nuScenes (Caesar et al., 2020) as language-conditioned trajectory prediction (). Given the current front-camera image and historical ego states, the token-based interface represents the future trajectory as six ordered waypoints at 0.5-second intervals over a three-second horizon. Each waypoint specifies a cumulative position in the current ego frame, with forward and left axes measured in meters. Fixed metric ranges map these coordinates to GroundingPI’s existing <0>–<999> vocabulary. The language head generates the trajectory autoregressively, and response-only cross-entropy provides the adaptation objective. Decoding recovers metric waypoints while preserving their temporal order. Coordinate conventions and the open-loop L2 metric are detailed in Section 8.1.

4.2 Robot Manipulation

\pi-style robotics manipulation learning.

For robot manipulation, we follow the \pi-style flow-matching policy (Black et al., 2024) and adopt the layer-wise connection used in StarVLA (Community, 2026) (). The pretrained foundation encodes the visual observation and language instruction once; its intermediate features are projected and resampled, then injected into the corresponding cross-attention layers of an Action DiT. Conditioned on these features and the robot state, the action expert denoises a noisy action chunk into continuous controls.

Controlled comparison across foundation models.

To compare perceptual foundations under the same action-learning setup, we follow the two adaptation pathways implemented in StarVLA: Wan-\pi for video-generation backbones and the StarVLA \pi-style pathway for vision-language foundations. This yields four controlled model families: general-purpose vision-language models, video-generation backbones, grounding models, and embodied foundation models. Within each benchmark, we keep the Action DiT, robot data, action representation, optimization budget, and inference procedure fixed, varying only the pretrained foundation. We compare their action acquisition and generalization, with full implementation details in Section 8.2.

5 Experiments

In this section, we ask:

5.1 Grounding Benchmark Results

Implementation Details and Evaluation Setup.

We evaluate GroundingPI against 44 baselines on 34 benchmarks spanning 11 perceptual capabilities, covering specialized detectors, general-purpose VLMs, grounding specialists, and embodied foundations. All numerical results and conclusions involving our models are based on the mean of ten runs, using five random seeds with two runs per seed. Owing to the high computational cost, other baselines that we evaluate locally for the main leaderboard are averaged over three runs, using three random seeds with one run per seed. GPT-6 Astra is evaluated with thinking effort set to High. We use F1mIoU for box grounding and OCR, F1@Point for object pointing, and task-specific accuracy for spatial and robo and GUI grounding.

Benchmark suite.

Our evaluation covers common and long-tailed detection on COCO (Lin et al., 2014) and LVIS (Gupta et al., 2019), and dense and tiny-object detection on Dense200 (Jiang et al., 2026) and VisDrone (Zhu et al., 2018). Referring grounding uses RefCOCO and RefCOCO+ (Yu et al., 2016) and RefCOCOg (Mao et al., 2016; Nagaraja et al., 2016), together with HumanRef (Jiang et al., 2025). Spatial and GUI grounding are evaluated on RoboSpatial (Song et al., 2025), RefSpatial (Zhou et al., 2026), ScreenSpot-V2 (Wu et al., 2025), ScreenSpot-Pro (Li et al., 2025), and OSWorld-G (Xie et al., 2026). OCR uses HierText (Long et al., 2022), ICDAR2015 (Karatzas et al., 2015), TotalText (Ch’Ng & Chan, 2017), and SROIE (Huang et al., 2019); document layout grounding uses DocLayNet (Pfitzmann et al., 2022) and M6Doc (Cheng et al., 2023). We additionally evaluate exemplar-based visual prompting on FSC147 (Ranjan et al., 2021) and detection benchmarks described above.

Main Results.

GroundingPI achieves 73.68% Avg, outperforming similarly sized models and remaining competitive with GPT-6 Astra (71.54%). Figure 4 summarizes this breadth. The selected comparisons below distinguish gains in coverage and localization from the remaining task-specific limitations.

Reporting conventions.

We report percentage scores for selected baselines; bold marks column bests, including ties (lower is better only for parse error). Model-name stars denote externally reported rows; entry-level stars denote source/support exceptions. --, N/A, and UNK indicate unreported values, unsupported evaluations, and unspecified zero-shot status, respectively. Daggers flag prompt/parser uncertainty, including DeepSeek GUI; affected scores are descriptive rather than definitive capability estimates. Under our evaluation protocols, GroundingDINO lacks GUI/OCR/layout interfaces, and Kimi-K3 lacks compatible pointing/OCR/layout outputs. SenseNova-Vision’s GUI evaluation is incompatible. Detailed exceptions appear in Section 12.

Detection and referring grounding.

In Table 1, GroundingPI reaches 74.53 on Dense200, versus Astra’s 65.04 and Qwen3-VL-4B’s 14.02, directly addressing the dense-scene perceptual gap, although VisDrone remains challenging. RefCOCO avg is the unweighted mean of RefCOCO, RefCOCOg, and RefCOCOplus.

Robot, spatial, and GUI grounding.

Table 2 shows strong spatial transfer. RefSpatial averages its Location and Placement splits. GroundingPI also improves over the selected compact baselines on all benchmarks, while Astra retains a substantial advantage, exposing a remaining limit.

OCR and document layout.

Table 3 extends the same structured interface to text and document regions: GroundingPI reaches 72.47 on SROIE and 74.82 on M6Doc, compared with Astra’s 53.57 and 60.59. The gains are not uniform: Astra remain stronger on TotalText, and SenseNova-Vision slightly leads on DocLayNet.

Object pointing.

Following Rex-Omni (Jiang et al., 2026), SAM-derived object masks (Kirillov et al., 2023) determine point correctness, with F1@Point balancing missed objects and false positives. GroundingPI leads six of the seven columns in Table 4; Astra leads Dense200 pointing, despite GroundingPI’s stronger box result, showing that point selection and boundary precision remain distinct challenges.

Additional visual-prompt capability.

Exemplar-based visual prompting on FSC147, Dense200, COCO, and LVIS is evaluated as an additional capability; the complete comparisons are provided in Appendix 12.9.

5.2 Physical Intelligence Performance

We evaluate autonomous driving on nuScenes (Caesar et al., 2020) and manipulation on RoboTwin 2.0 (Chen et al., 2025) and RoboCasa-GR1 (Nasiriany et al., 2024; NVIDIA et al., 2025). Figure 5 shows lower driving error at every reported horizon and the highest success rate in five of six manipulation settings, including all four OOD settings. These results highlight the value of grounding perception for physical intelligence across distinct action learners and embodiments.

5.2.1 Autonomous Driving Performance

GroundingPI attains an average open-loop L2 error of 0.296 m, improving on Qwen3-VL-4B (0.301 m) and RynnBrain (0.308 m); the trajectory interface and metric are defined in Section 8.1. This modest, consistent gain suggests that embodied specialization alone need not supply the perception most useful for driving. Reading road signs also motivates accurate and timely OCR in vision-based driving, although this evaluation does not isolate sign reading or measure its latency. For latency-sensitive execution, these results motivate compact perceptual foundations; they do not establish closed-loop performance or a deployment limit for larger models.

5.2.2 Robot Manipulation Performance

Controlled Backbone Comparison.

We evaluate the vision-language and video-generation foundations summarized in Figure 5, spanning the backbone families used in VLA and WAM systems. To compare their contribution to action learning under controlled downstream conditions, we couple every backbone to the same layer-wise Action DiT. Following the conditioning design of \pi_{0} (Black et al., 2024), intermediate features from a single backbone forward pass are projected and resampled to a common interface, then injected into the corresponding cross-attention blocks of the action expert. Within each benchmark, all runs keep the Action DiT architecture, action-training data, optimization budget, action representation, and inference procedure fixed, and vary only the pretrained foundation. This controlled comparison allows us to assess how effectively each foundation supports downstream action learning and how well the resulting policies generalize out of distribution. Section 8.2 provides the implementation.

Evaluation Protocol.

We report task success rate (SR) across two representative embodiments: fixed-base bimanual tabletop manipulation on RoboTwin2.0 and humanoid dexterous-hand manipulation on RoboCasa-GR1. In-distribution (ID) evaluation follows the full training and evaluation protocol of each benchmark. For out-of-distribution (OOD) evaluation, policies are trained on ID data and tested under held-out conditions.

• Fixed-base bimanual — RoboTwin2.0. This widely adopted benchmark spans 50 tabletop manipulation tasks (Chen et al., 2025). RoboTwin2.0-Full trains and evaluates on both Clean and Randomized data, whereas RoboTwin2.0-Clean2Random trains only on Clean demonstrations and evaluates on Randomized scenes. The released Randomized setting retains the same manipulation tasks but varies scene clutter, lighting, table and background textures, tabletop height, and language instructions. It therefore primarily evaluates scene robustness under environmental variation.
• Humanoid dexterous-hand — RoboCasa-GR1. This benchmark evaluates fine-grained manipulation from head-camera observations without wrist cameras (Nasiriany et al., 2024; NVIDIA et al., 2025). Its full protocol measures ID performance, while OOD evaluation uses three held-out suites (Chen et al., 2026a): Unseen Appearance applies novel textures to familiar object–container pairs; Unseen Combinations places seen objects in novel container pairings; and Unseen Object Types introduces novel object categories. These suites evaluate entity generalization across object appearances, object categories, and object–container combinations.

Together, the two protocols evaluate scene robustness and entity generalization, two complementary requirements of an embodied foundation model. Both require stable, task-conditioned perception before action learning, making them direct tests of whether a pretrained foundation provides reusable representations for manipulation.

Overall Performance.

Under the matched downstream architecture and training protocol, Figure 5 compares backbones with distinct pretraining objectives. We organize the compared backbones into four groups: general-purpose VLMs (Qwen3-VL-4B (Bai et al., 2025a) and PaliGemma-3B (Beyer et al., 2024)), video-generation backbones (Wan2.2-TI2V-5B (Wan et al., 2025) and Cosmos-Predict2.5-2B (Ali et al., 2025)), grounding models (LocateAnything-3B (Wang et al., 2026a) and Rex-Omni-3B (Jiang et al., 2026)), and embodied foundation models (RynnBrain (Dang et al., 2026) and RynnBrain1.1 (Li et al., 2026)). Across these heterogeneous pretraining sources, GroundingPI ranks first in five of the six evaluations, including RoboTwin2.0 Full and every OOD setting. It achieves 66.20% on RoboTwin2.0 Full and remains competitive on RoboCasa-GR1 Full at 37.75%, compared with RynnBrain’s 39.00%. These results support precise spatial perception as an effective foundation for action learning across two robot embodiments, with consistent performance under distribution shift.

Generalization under Complementary Shifts.

RoboTwin2.0-Clean2Random evaluates scene robustness, while RoboCasa-GR1 evaluates entity generalization across appearances, object types, and object–container relations. GroundingPI ranks first in all four OOD settings. On RoboTwin2.0, it reaches 17.60%, with Rex-Omni second at 14.10%. On RoboCasa-GR1, RynnBrain ranks second across Type, Appearance, and Container OOD. Rex-Omni’s strength in scene robustness is consistent with its emphasis on detection, referring, and coordinate prediction (Jiang et al., 2026). RynnBrain’s strength in entity generalization is consistent with its broader language-conditioned embodied and semantic understanding, including spatiotemporal localization and physically grounded reasoning (Dang et al., 2026). These observations highlight the complementary strengths of precise perceptual grounding and broader embodied semantic understanding.

Perception Before Action.

Figure 6 illustrates a conceptual progression from action-only adaptation of general VLMs, through perceptual supervision in the \pi family, to grounding as a native foundation capability (Black et al., 2024; Intelligence et al., 2025; Driess et al., 2025). GroundingPI retains broad VQA-style, language-conditioned semantic knowledge while emphasizing precise spatial perception through grounding and pointing supervision. Visual primitives provide an interface for physical prompting, specifying targets, locations, and structures alongside language. Together with precise spatial perception, this interface supports scene robustness and entity generalization. Our transfer results evaluate the pretrained foundation for physical intelligence rather than a separate prompting intervention. The data-composition ablation in Section 5.3.2 further examines which forms of grounding supervision transfer to action.

5.2.3 Data Efficiency

Figure 7 evaluates data efficiency under the RoboCasa-GR1 Full/ID protocol using 25%, 50%, 75%, and 100% of the robot demonstrations. We vary only the amount of action data, keeping the architecture, action-learning approach, and other training choices unchanged. GroundingPI leads all three reduced-data settings. At 50% data, its 28.75% SR exceeds every baseline at 75%, where RynnBrain achieves the highest score of 27.75%.

These results support prioritizing precise spatial perception before action learning. The same physical-prompting interface allows scarce action demonstrations to focus on control rather than relearning basic perception. Together with the preceding results, this suggests that precise grounding supports effective and generalizable action learning as well as more data-efficient adaptation.

5.3 Ablations and Analysis

5.3.1 Effects of Grounding Training

Figure 8 tracks the aggregate grounding score together with manipulation success on the RoboCasa-GR1 ID and OOD splits as pretraining grows from 88.4B to 221B tokens. This scaling sweep examines how perception and downstream action learning evolve together.

Grounding and Action Learning Improve with Scale.

As pretraining scale increases, the grounding score rises steadily and robotic manipulation performance improves on both splits. Scaling grounding pretraining therefore strengthens precise perception and transfers to more effective and generalizable action learning.

Grounding Gains Track Action-Learning Gains.

The curves move together across the scaling trajectory, with generalization tracking grounding quality most closely. This correspondence links the two observations: better grounding provides a stronger perceptual foundation for downstream action learning.

5.3.2 What Transfers from Grounding to Action?

The scaling study in Section 5.3.1 establishes that more grounding pretraining improves downstream action learning. We now ask what that data should contain. Figure 9 decomposes the training mixture into basic and dense grounding, referring, pointing, and auxiliary OCR, layout, and GUI supervision.

Dense Grounding Provides the Spatial Core.

In the available leave-one-out comparisons, removing dense grounding degrades both manipulation splits and nuScenes performance more than removing referring or robo pointing. In driving and manipulation, relevant targets may be densely packed or small relative to the image, coupling dense and tiny-object perception through a shared need for fine-grained spatial discrimination. Dense multi-object supervision therefore provides the core precise spatial perception that transfers across downstream domains.

Auxiliary Perception Amplifies Grounding: OCR as a Potential Catalyst.

Removing the OCR-containing group causes the largest manipulation losses on both splits (5.33/3.63 pp), yet this group alone transfers poorly. We hypothesize that OCR catalyzes perceptual learning: sharp character boundaries demand local precision, while transcription binds visual regions to semantics, providing a localized captioning proxy task for vision–language alignment. Such supervision could strengthen fine-grained ViT features and amplify the benefits of dense spatial supervision for grounding and downstream action learning. Additional OCR mixing experiments favor combining cold-start and batch-level supervision for manipulation transfer; the OCR-specific mechanism remains a hypothesis (Section 9.4).

Complementary Supervision Builds the Strongest Interface.

No reduced mixture matches the full mixture simultaneously across manipulation performance and nuScenes localization. Dense grounding supplies the spatial core, auxiliary perception sharpens the visual representation, and referring and pointing connect that representation to language and action-relevant targets. Together with the scaling study, these results show that scale determines how much grounding capability is learned, while composition determines whether it forms an effective and generalizable interface for action learning.

5.3.3 Discussion

Effects of Base Model.

The backbone comparisons suggest possible sensitivities beyond supervision. RynnBrain1.1 trails RynnBrain in all six manipulation settings despite improving driving. Its Qwen3.5 base combines Gated DeltaNet with gated attention, whereas RynnBrain uses Qwen3-VL (Li et al., 2026; Dang et al., 2026; Qwen Team, 2026a). Recurrent compression and output gating may affect access to spatial features under a new action objective; Section 10.2 derives conditional memory and gradient effects (Yang et al., 2025b; Qiu et al., 2026).

DeepStack enriches visual evidence through intermediate feature injection, but may also increase the demands of aligning several feature levels with an action readout (Bai et al., 2025a; Meng et al., 2024). GroundingPI instead couples MoonViT-V2 to a full-attention Qwen3-4B decoder through a single visual interface. Since RynnBrain and Qwen3-VL both use DeepStack, their driving difference cannot isolate this mechanism (Section 10.3). The older Qwen2.5-VL foundations of Rex-Omni and LocateAnything also leave base-model quality as a possible factor. These comparisons motivate controlled studies of feature accessibility and alignment; they do not establish architectural causes of the observed rankings.

Spatial Supervision: A First-Principles Hypothesis.

From first principles, a useful starting point is what spatial information the visual input can actually determine. An image records projected structure, while its absolute metric scale may remain ambiguous. Human perception illustrates this limitation: we may readily identify and localize a distant object yet struggle to judge whether it is 25 m or 30 m away. A raw L_{1} depth loss nevertheless assigns a 5 m error, which need not reflect the perceptual difficulty; a fixed metric tolerance also has different implications for close-range manipulation and distant scenes. Under perspective projection, small image-space errors can further translate into larger metric depth and 3D localization errors at longer ranges. These considerations motivate a hypothesis: grounding may help elicit and develop spatial intelligence in VLMs by forcing spatial understanding to become explicit through precise localization. Predicting boxes and points ties language to specific visual regions, providing low-level supervision that may compel the model to preserve and use fine-grained spatial evidence. Relative depth, such as scene-normalized values in [0,1], could offer complementary supervision of depth ordering and scene structure without requiring absolute scale. We therefore conjecture that prioritizing these visually grounded targets may better cultivate transferable perception, with metric calibration learned for downstream action. This is a hypothesis about perceptual pretraining, and the proposed advantage over absolute metric supervision remains untested in our study.

Design of Embodied Foundation Models.

A useful distinction may be between deliberative planning (System 2) and fast perception–action execution (System 1), as explored in dual-system robotics (NVIDIA et al., 2025; Bu et al., 2025). Planning can benefit from extensive knowledge, coding, and long-horizon reasoning; execution requires timely feedback, precise interaction perception, and reliable control. The analogy to coding agents is functional: sophisticated reasoning has limited practical value when generated programs repeatedly fail to run, just as strong planning can be constrained by unreliable physical execution.

General VLMs, including Qwen and the PaliGemma base of \pi_{0} (Bai et al., 2025a; Black et al., 2024), offer valuable starting points, but may not best serve every execution role. GroundingPI explores a perception-native alternative: learning where to interact before limited action data teach how to act. Astra provides a frontier reference for grounding quality; comparison with it probes the perceptual frontier rather than its suitability for direct VLA adaptation. Such frontier models may instead serve higher-level planning roles. Our results motivate stronger perceptual foundations for System 1; a deployed dual-system controller and its latency remain to be evaluated (Section 10.4).

5.3.4 Ablation Study

Figure 10 supports the chosen grounding architecture and training recipe. Quantized coordinates improve Avg from 71.21 to 73.68, while textual coordinates run at 0.25\times the relative speed. MoonViT-V2 reaches 73.68, versus 73.17 with MoonViT and 71.65 with Qwen3-ViT; potential implications for visual alignment are discussed in Section 10.3. Removing Stage III reduces Avg to 72.86. These comparisons support the complete design choices without isolating every architectural difference.

Table 5 shows compact serialization: GroundingPI uses 7.6/5.1 tokens per box on COCO/Dense200, versus 148.8/74.5 for SEED1.5-VL. Figure 11 separately relates GroundingPI’s generation time and output length to predicted object count. Shared protocol overhead is amortized in dense outputs, while autoregressive generation cost remains (Section 9.5).

6 Conclusion

We introduced GroundingPI, a 4B-parameter grounding foundation model for broad and precise perception. Across 34 grounding benchmarks, it outperforms similarly sized models and remains competitive with GPT-6 Astra, while transferring effectively to driving and manipulation.

Reliable physical interaction requires connecting instructions to precise spatial targets, a capability that general semantic understanding alone does not guarantee and scarce action supervision may not adequately develop. Dedicated grounding pretraining directly addresses this perceptual gap, supporting better manipulation generalization and action-data efficiency. Our analyses highlight basic grounding as an important foundation, dense grounding as valuable for physical interaction, and OCR as a potential catalyst for perceptual learning. These findings motivate distinct embodied foundations: broad reasoning for System 2 planning, and precise perception for reliable System 1 execution, the direction explored by GroundingPI.

AI Use Statement

In this work, we used generative AI tools to assist with code development, to polish the writing, and to produce some of the figures. We also used generative AI models as part of the data annotation pipeline to generate pseudo-labels for model training. We did not use generative AI tools to develop the research ideas or methodology, to design or interpret the experiments, or to outline the paper. We have reviewed all AI-assisted work. We take responsibility for the final content of this work, including text, claims, data annotations, or artifacts produced with the aid of generative AI.

Ethics Statement

This work does not involve human-subject studies. All grounding datasets and evaluation benchmarks used in this work were obtained from publicly available or appropriately licensed sources in accordance with their respective terms and licenses. Driving was evaluated offline on nuScenes, and robot manipulation in simulation; no real-world vehicles or robots were deployed in these experiments. GroundingPI is developed as a general-purpose visual grounding foundation model for research, with downstream experiments intended to study how precise spatial perception transfers to physical-intelligence tasks rather than to demonstrate real-world autonomous deployment. We encourage appropriate safety evaluation and human oversight before applying such models to real-world embodied systems. The authors declare no conflicts of interest.

Reproducibility Statement

We will publicly release the code, model weights, and evaluation data for GroundingPI to support systematic and reproducible evaluation. The release will include training configurations and evaluation scripts with standardized task prompts, output parsers, and metric implementations. The input/output protocol, coordinate conventions, architecture, tokenizer, and visual processing are detailed in Sections 7.1 and 7.2; data construction and validation in Section 7.3; and base-model training, spatial supervised fine-tuning, and optimization hyperparameters in Section 7.4. Downstream interfaces, the shared action architecture, matched training budgets, and driving and manipulation evaluation protocols are specified in Sections 8.1 and 8.2. Data-efficiency, scaling, training-mixture, and OCR-mixing studies are documented in Sections 9.1, 9.2, 9.3 and 9.4. The benchmark suite, evaluation metrics, reporting conventions, and complete grounding results are provided in Section 12, including the visual-prompting evaluations in Section 12.9.

Research Scope, Data Use, and Institutional Disclaimer

This work originated from exploratory academic research undertaken by the project leader Qize Yu during their internship at Xpeng Inc. The project was conducted exclusively for scientific investigation and academic publication and does not involve commercial applications, product development, or commercial deployment. All data used in this project were used solely for academic research and maintained under strict segregation from the company’s commercial model development and deployment activities. No project data were used to train, fine-tune, evaluate, or otherwise support commercial models, products, or services. Internal legal review of the dataset materials was completed on September 21, 2026. This work neither uses nor discloses business data containing users’ private or personally identifiable information. The research-only scope described here does not modify or supersede the applicable terms and licenses of the source datasets.

The views, methods, findings, and conclusions presented in this paper are those of the authors and do not represent the official positions, technical direction, product roadmap, or commercial commitments of Xpeng Inc. The company’s support for this research should not be construed as endorsement of any commercial application. Neither the research findings nor their publication constitute a claim of readiness, safety, or suitability for commercial deployment.

Acknowledgments

We thank Xpeng Inc. for providing computational and data resources in support of this academic research, and the data team for their assistance with data preparation and research support. We are particularly grateful to Professor Ping Luo for his guidance on the research ideas and manuscript writing. We also thank Xinghang Li, Qing Li, Baiqiao Yin, Xinyu Wei, Jiadi You, Linhao Zhou, Qiman Wu, Ziteng Cui, Haojun Zhang, Min Chen, Hao Li, Hanzhen Zhang and Zhuo Li for their valuable suggestions and constructive feedback.

References

1. Ali et al. (2025) [1] Ali, A., Bai, J., Bala, M., Balaji, Y., Blakeman, A., Cai, T., Cao, J., Cao, T., Cha, E., Chao, Y.-W., et al. World simulation with video foundation models for physical ai. arXiv preprint arXiv:2511.00062, 2025.
2. Bai et al. (2025a) [2] Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, J., Tu, J., Wan, J., Wang, P., Wang, P., Wang, Q., Wang, Y., Xie, T., Xu, Y., Xu, H., Xu, J., Yang, Z., Yang, M., Yang, J., Yang, A., Yu, B., Zhang, F., Zhang, H., Zhang, X., Zheng, B., Zhong, H., Zhou, J., Zhou, F., Zhou, J., Zhu, Y., and Zhu, K. Qwen3-vl technical report, 2025a. URL https://arxiv.org/abs/2511.21631.
3. Bai et al. (2025b) [3] Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., and Lin, J. Qwen2.5-vl technical report, 2025b. URL https://arxiv.org/abs/2502.13923.
4. Beyer et al. (2024) [4] Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D., Neumann, M., Alabdulmohsin, I., Tschannen, M., Bugliarello, E., et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024.
5. Black et al. (2024) [5] Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. \pi_{0}: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024.
6. Bu et al. (2025) [6] Bu, Q., Li, H., Chen, L., Cai, J., Zeng, J., Cui, H., Yao, M., and Qiao, Y. Towards synergistic, generalized, and efficient dual-system for robotic manipulation, 2025. URL https://arxiv.org/abs/2410.08001.
7. Caesar et al. (2020) [7] Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: A multimodal dataset for autonomous driving. In 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 11618–11628. IEEE, 2020.
8. Carion et al. (2020) [8] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In European conference on computer vision, pp. 213–229. Springer, 2020.
9. Chen et al. (2026a) [9] Chen, B., Chen, Y., Qiu, L., Bai, J., Ge, Y., and Ge, Y. UniT: Toward a unified physical language for human-to-humanoid policy learning and world modeling. arXiv preprint arXiv:2604.19734, 2026a. 10.48550/arXiv.2604.19734. URL https://arxiv.org/abs/2604.19734.
10. Chen et al. (2026b) [10] Chen, H., Liu, J., Gu, C., Liu, Z., Zhang, R., Li, X., He, X., Guo, Y., Fu, C.-W., Zhang, S., et al. Fast-in-slow: a dual-system vla model unifying fast manipulation within slow reasoning. Advances in Neural Information Processing Systems, 38:98049–98083, 2026b.
11. Chen et al. (2022) [11] Chen, T., Saxena, S., Li, L., Fleet, D. J., and Hinton, G. Pix2seq: A language modeling framework for object detection, 2022. URL https://arxiv.org/abs/2109.10852.
12. Chen et al. (2025) [12] Chen, T., Chen, Z., Chen, B., Cai, Z., Liu, Y., Li, Z., Liang, Q., Lin, X., Ge, Y., Gu, Z., et al. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088, 2025.
13. Chen et al. (2026c) [13] Chen, Y., Jiang, M., Zheng, K., Liang, J., Tie, C., Lu, H., Wu, R., and Dong, H. Pa3ff:learning part-aware dense 3d feature field for generalizable articulated object manipulation. In International Conference on Learning Representations, volume 2026, 2026c.
14. Cheng et al. (2023) [14] Cheng, H., Zhang, P., Wu, S., Zhang, J., Zhu, Q., Xie, Z., Li, J., Ding, K., and Jin, L. M6doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi-annotation category dataset for modern document layout analysis, 2023. URL https://arxiv.org/abs/2305.08719.
15. Ch’Ng & Chan (2017) [15] Ch’Ng, C. K. and Chan, C. S. Total-text: A comprehensive dataset for scene text detection and recognition. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pp. 935–942. IEEE, 2017.
16. Comanici et al. (2025) [16] Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
17. Community (2026) [17] Community, S. Starvla: A lego-like codebase for vision-language-action model developing. arXiv preprint arXiv:2604.05014, 2026.
18. Covert et al. (2025) [18] Covert, I., Sun, T., Zou, J. Y., and Hashimoto, T. Locality alignment improves vision-language models. In International Conference on Learning Representations, volume 2025, pp. 83127–83165, 2025.
19. Cui et al. (2025) [19] Cui, C., Sun, T., Lin, M., Gao, T., Zhang, Y., Liu, J., Wang, X., Zhang, Z., Zhou, C., Liu, H., Zhang, Y., Lv, W., Huang, K., Zhang, Y., Zhang, J., Zhang, J., Liu, Y., Yu, D., and Ma, Y. Paddleocr 3.0 technical report, 2025. URL https://arxiv.org/abs/2507.05595.
20. Dai et al. (2021) [20] Dai, X., Chen, Y., Xiao, B., Chen, D., Liu, M., Yuan, L., and Zhang, L. Dynamic head: Unifying object detection heads with attentions. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7369–7378. ieee, 2021.
21. Dang et al. (2026) [21] Dang, R., Guo, J., Hou, B., Leng, S., Li, K., Li, X., Liu, J., Mao, Y., Wang, Z., Yuan, Y., Zhu, M., Lin, X., Bai, Y., Jiang, Q., Zhao, Y., Zeng, M., Gao, J., Jiang, Y., Cen, J., Huang, S., Wang, L., Zhang, W., Liu, C., Yang, J., Lu, S., and Zhao, D. RynnBrain: Open embodied foundation models. arXiv preprint arXiv:2602.14979, 2026. 10.48550/arXiv.2602.14979. URL https://arxiv.org/abs/2602.14979.
22. Deitke et al. (2025) [22] Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 91–104. IEEE, 2025.
23. Deng et al. (2025) [23] Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., Shi, G., and Fan, H. Emerging properties in unified multimodal pretraining, 2025. URL https://arxiv.org/abs/2505.14683.
24. Driess et al. (2025) [24] Driess, D., Springenberg, J. T., Ichter, B., Yu, L., Li-Bell, A., Pertsch, K., Ren, A. Z., Walke, H., Vuong, Q., Shi, L. X., and Levine, S. Knowledge insulating vision-language-action models: Train fast, run fast, generalize better, 2025. URL https://arxiv.org/abs/2505.23705.
25. Guo et al. (2025) [25] Guo, D., Wu, F., Zhu, F., Leng, F., Shi, G., Chen, H., Fan, H., Wang, J., Jiang, J., Wang, J., et al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062, 2025.
26. Gupta et al. (2019) [26] Gupta, A., Dollar, P., and Girshick, R. Lvis: A dataset for large vocabulary instance segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5351–5359. IEEE, 2019.
27. Han et al. (2026) [27] Han, X., Li, J., Deng, K., Chen, Z., Shi, X., Wang, S., Li, B., Wang, L., Xie, S., You, X., Quan, J., Cai, Z., Diao, H., Liu, Z., Yang, L., Lin, D., and Wang, Q. Vision as unified multimodal generation, 2026. URL https://arxiv.org/abs/2607.06560.
28. Huang et al. (2019) [28] Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., and Jawahar, C. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1516–1520. IEEE, 2019.
29. Intelligence et al. (2025) [29] Intelligence, P., Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Galliker, M. Y., Ghosh, D., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Ren, A. Z., Shi, L. X., Smith, L., Springenberg, J. T., Stachowicz, K., Tanner, J., Vuong, Q., Walke, H., Walling, A., Wang, H., Yu, L., and Zhilinsky, U. \pi_{0.5}: a vision-language-action model with open-world generalization, 2025. URL https://arxiv.org/abs/2504.16054.
30. Jiang et al. (2025) [30] Jiang, Q., Wu, L., Zeng, Z., Ren, T., Xiong, Y., Chen, Y., Qin, L., and Zhang, L. Referring to any person. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 21667–21678. IEEE, 2025.
31. Jiang et al. (2026) [31] Jiang, Q., Huo, J., Chen, X., Xiong, Y., Zeng, Z., Chen, Y., Ren, T., Yu, J., and Zhang, L. Detect anything via next point prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25472–25483, 2026.
32. Karatzas et al. (2015) [32] Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V. R., Lu, S., Shafait, F., Uchida, S., and Valveny, E. Icdar 2015 competition on robust reading. In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 1156–1160, 2015. 10.1109/ICDAR.2015.7333942.
33. Kim et al. (2024) [33] Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024.
34. Kim et al. (2026) [34] Kim, M. J., Gao, Y., Lin, T.-Y., Lin, Y.-C., Ge, Y., Lam, G., Liang, P., Song, S., Liu, M.-Y., Finn, C., et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning. arXiv preprint arXiv:2601.16163, 2026.
35. Kirillov et al. (2023) [35] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In 2023 IEEE/CVF international conference on computer vision (ICCV), pp. 3992–4003. IEEE, 2023.
36. Lai et al. (2024) [36] Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., and Jia, J. Lisa: Reasoning segmentation via large language model, 2024. URL https://arxiv.org/abs/2308.00692.
37. Li et al. (2025) [37] Li, K., Meng, Z., Lin, H., Luo, Z., Tian, Y., Ma, J., Huang, Z., and Chua, T.-S. Screenspot-pro: Gui grounding for professional high-resolution computer use. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 8778–8786, 2025.
38. Li et al. (2026) [38] Li, K., Hou, B., Zhu, M., Zhang, T., Cheng, Z., Wang, Z., Leng, S., Li, X., Lin, X., Yao, B., Zeng, M., Liu, J., Dang, R., Guo, J., Huang, S., Zhao, H., Ping, H., Zhao, Y., Zhao, T., Wang, K., Lu, T., Xue, S., Tang, J., Wang, Y., Wang, Z., Gao, J., Lu, S., Liu, C., Yang, J., Chen, M., and Zhao, D. Rynnbrain 1.1: Towards more capable and generalizable embodied foundation model, 2026. URL https://arxiv.org/abs/2607.17977.
39. Li et al. (2022) [39] Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. Grounded language-image pre-training. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10955–10965. IEEE, 2022.
40. Liang et al. (2023) [40] Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control. In 2023 IEEE International conference on robotics and automation (ICRA), pp. 9493–9500. IEEE, 2023.
41. Lin et al. (2024) [41] Lin, K. Q., Li, L., Gao, D., Yang, Z., Wu, S., Bai, Z., Lei, W., Wang, L., and Shou, M. Z. Showui: One vision-language-action model for gui visual agent, 2024. URL https://arxiv.org/abs/2411.17465.
42. Lin et al. (2014) [42] Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740–755. Springer, 2014.
43. Lipman et al. (2023) [43] Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling, 2023. URL https://arxiv.org/abs/2210.02747.
44. Liu et al. (2024) [44] Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38–55. Springer, 2024.
45. Liu et al. (2025) [45] Liu, Z., Xie, J., Ding, Z., Li, Z., Yang, B., Wu, Z., Wang, X., Sun, Q., Liu, S., Wang, W., Ye, S., Li, Q., Dong, X., Yu, Y., Lu, C., Mo, Y., Yan, Y., Tian, Z., Zhang, X., Huang, Y., Liu, Y., Su, W., Luo, G., Yue, X., Qi, B., Chen, K., Zhou, B., Qiao, Y., Chen, Q., and Wang, W. Scalecua: Scaling open-source computer use agents with cross-platform data, 2025. URL https://arxiv.org/abs/2509.15221.
46. Long et al. (2022) [46] Long, S., Qin, S., Panteleev, D., Bissacco, A., Fujii, Y., and Raptis, M. Towards end-to-end unified scene text detection and layout analysis. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1039–1049. IEEE, 2022.
47. Loshchilov & Hutter (2019) [47] Loshchilov, I. and Hutter, F. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101.
48. Lu et al. (2026) [48] Lu, Z., Chai, Y., Guo, Y., Yin, X., Liu, L., Wang, H., Xiao, H., Ren, S., Zhao, P., Liu, G., et al. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 17608–17616, 2026.
49. Mao et al. (2016) [49] Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A. L., and Murphy, K. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11–20, 2016.
50. Meng et al. (2024) [50] Meng, L., Yang, J., Tian, R., Dai, X., Wu, Z., Gao, J., and Jiang, Y.-G. Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms. Advances in Neural Information Processing Systems, 37:23464–23487, 2024.
51. Nagaraja et al. (2016) [51] Nagaraja, V. K., Morariu, V. I., and Davis, L. S. Modeling context between objects for referring expression understanding. In European conference on computer vision, pp. 792–807. Springer, 2016.
52. Nasiriany et al. (2024) [52] Nasiriany, S., Maddukuri, A., Zhang, L., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., and Zhu, Y. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems, 2024.
53. NVIDIA (2026) [53] NVIDIA. Cosmos 3: Omnimodal world models for physical ai. arXiv preprint arXiv:2606.02800, 2026.
54. NVIDIA et al. (2025) [54] NVIDIA, Bjorck, J., Castañeda, F., Cherniadev, N., et al. GR00T N1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025.
55. Peebles & Xie (2023) [55] Peebles, W. and Xie, S. Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4172–4182. IEEE, 2023.
56. Pfitzmann et al. (2022) [56] Pfitzmann, B., Auer, C., Dolfi, M., Nassar, A. S., and Staar, P. Doclaynet: A large human-annotated dataset for document-layout segmentation. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp. 3743–3751, 2022.
57. Ping et al. (2026) [57] Ping, B., Chen, Z., Hui, T., Yu, Q., Li, C., Yan, J., and Chang, B. LongAct: Harnessing intrinsic activation patterns for long-context reinforcement learning, 2026. URL https://arxiv.org/abs/2604.14922.
58. Qin et al. (2025) [58] Qin, Y., Ye, Y., Fang, J., Wang, H., Liang, S., Tian, S., Zhang, J., Li, J., Li, Y., Huang, S., Zhong, W., Li, K., Yang, J., Miao, Y., Lin, W., Liu, L., Jiang, X., Ma, Q., Li, J., Xiao, X., Cai, K., Li, C., Zheng, Y., Jin, C., Li, C., Zhou, X., Wang, M., Chen, H., Li, Z., Yang, H., Liu, H., Lin, F., Peng, T., Liu, X., and Shi, G. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL https://arxiv.org/abs/2501.12326.
59. Qiu et al. (2026) [59] Qiu, Z., Wang, Z., Zheng, B., Huang, Z., Wen, K., Yang, S., Men, R., Yu, L., Huang, F., Huang, S., et al. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. Advances in Neural Information Processing Systems, 38:100092–100118, 2026.
60. Qwen Team (2026a) [60] Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026a. URL https://qwen.ai/blog?id=qwen3.5.
61. Qwen Team (2026b) [61] Qwen Team. Qwen3.6-27B: Flagship-level coding in a 27B dense model, April 2026b. URL https://qwen.ai/blog?id=qwen3.6-27b.
62. Qwen Team (2026c) [62] Qwen Team. Qwen3.7: The agent frontier, May 2026c. URL https://qwen.ai/blog?id=qwen3.7.
63. Qwen Team (2026d) [63] Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, August 2026d. URL https://qwen.ai/blog?id=qwen3.8.
64. Rajbhandari et al. (2020) [64] Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: international conference for high performance computing, networking, storage and analysis, pp. 1–16. IEEE, 2020.
65. Ranjan et al. (2021) [65] Ranjan, V., Sharma, U., Nguyen, T., and Hoai, M. Learning to count everything. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3393–3402. IEEE, 2021.
66. Redmon et al. (2016) [66] Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788, 2016.
67. Ren et al. (2024) [67] Ren, Z., Huang, Z., Wei, Y., Zhao, Y., Fu, D., Feng, J., and Jin, X. Pixellm: Pixel reasoning with large multimodal model, 2024. URL https://arxiv.org/abs/2312.02228.
68. Shao et al. (2024) [68] Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.
69. Shen et al. (2026) [69] Shen, Z., Liang, J., Lu, J., Jiang, F., Wang, Y., Wei, C., Liu, J., Yang, J., Yu, Q., You, J., Hao, C., He, G., Xie, C., and Wu, R. LD4WAM: Learning latent dynamics from human videos for world action models, 2026. URL https://arxiv.org/abs/2608.22403.
70. Song et al. (2025) [70] Song, C. H., Blukis, V., Tremblay, J., Tyree, S., Su, Y., and Birchfield, S. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15768–15780. IEEE, 2025.
71. Song et al. (2026) [71] Song, W., Zhou, Z., Zhao, H., Chen, J., Ding, P., Yan, H., Huang, Y., Tang, F., Wang, D., and Li, H. Reconvla: Reconstructive vision-language-action model as effective robot perceiver. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 18549–18557, 2026.
72. Team et al. (2026) [72] Team, K., Bai, T., Bai, Y., Bao, Y., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., Charles, Y., et al. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026.
73. Tu et al. (2026) [73] Tu, R., Shukla, A., Yoo, S., Li, X., Li, J., Xie, J., Su, H., and Tu, Z. Sg-vla: Learning spatially-grounded vision-language-action models for mobile manipulation. arXiv preprint arXiv:2603.22760, 2026.
74. Wan et al. (2025) [74] Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., Zeng, J., Wang, J., Zhang, J., Zhou, J., Wang, J., Chen, J., Zhu, K., Zhao, K., Yan, K., Huang, L., Feng, M., Zhang, N., Li, P., Wu, P., Chu, R., Feng, R., Zhang, S., Sun, S., Fang, T., Wang, T., Gui, T., Weng, T., Shen, T., Lin, W., Wang, W., Wang, W., Zhou, W., Wang, W., Shen, W., Yu, W., Shi, X., Huang, X., Xu, X., Kou, Y., Lv, Y., Li, Y., Liu, Y., Wang, Y., Zhang, Y., Huang, Y., Li, Y., Wu, Y., Liu, Y., Pan, Y., Zheng, Y., Hong, Y., Shi, Y., Feng, Y., Jiang, Z., Han, Z., Wu, Z.-F., and Liu, Z. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.
75. Wang et al. (2026a) [75] Wang, S., Liu, S., Kuang, Y., Wei, X., Liu, Y., Li, Z., Man, Y., Chen, G., Tao, A., Liu, G., et al. Locateanything: Fast and high-quality vision-language grounding with parallel box decoding. In European Conference on Computer Vision, pp. 336–357. Springer, 2026a.
76. Wang et al. (2026b) [76] Wang, Y., Huang, S., Li, M., Zhang, C., Liang, J., Jin, W., Chen, Y., Chi, X., Zhou, D., Yu, Q., et al. Openwam: An open, modular exploration towards systematic world-action model pretraining. arXiv preprint arXiv:2609.07398, 2026b.
77. Wu et al. (2026) [77] Wu, T., Kong, X., Chen, Y., Yu, Q., Ye, H., Li, J., Wang, Y., and Dong, H. SUGAR: A scalable human-video-driven generalizable humanoid loco-manipulation learning framework, 2026. URL https://arxiv.org/abs/2605.20373.
78. Wu et al. (2024) [78] Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024.
79. Wu et al. (2025) [79] Wu, Z., Wu, Z., Xu, F., Wang, Y., Sun, Q., Jia, C., Cheng, K., Ding, Z., Chen, L., Liang, P. P., et al. Os-atlas: Foundation action model for generalist gui agents. In International Conference on Learning Representations, volume 2025, pp. 5090–5108, 2025.
80. Xiao et al. (2024) [80] Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M., Liu, C., and Yuan, L. Florence-2: Advancing a unified representation for a variety of vision tasks. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4818–4829. IEEE, 2024.
81. Xie et al. (2026) [81] Xie, T., Deng, J., Li, X., Yang, J., Wu, H., Chen, J., Hu, W., Wang, X., Xu, Y., Wang, Z., et al. Scaling computer-use grounding via user interface decomposition and synthesis. Advances in Neural Information Processing Systems, 38, 2026.
82. Yang et al. (2025a) [82] Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
83. Yang et al. (2025b) [83] Yang, S., Kautz, J., and Hatamizadeh, A. Gated delta networks: Improving mamba2 with delta rule. In International Conference on Learning Representations, volume 2025, pp. 29687–29707, 2025b.
84. Ye et al. (2025) [84] Ye, J., Zhang, X., Xu, H., Liu, H., Wang, J., Zhu, Z., Zheng, Z., Gao, F., Cao, J., Lu, Z., Liao, J., Zheng, Q., Huang, F., Zhou, J., and Yan, M. Mobile-agent-v3: Fundamental agents for gui automation, 2025. URL https://arxiv.org/abs/2508.15144.
85. Ye et al. (2026) [85] Ye, S., Ge, Y., Zheng, K., Gao, S., Yu, S., Kurian, G., Indupuru, S., Tan, Y. L., Zhu, C., Xiang, J., et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026.
86. You et al. (2026) [86] You, J., Yu, Q., Chen, Y., Cai, M., Zhong, Z., Wang, Y., Ping, B., Liang, J., Shen, Z., Yan, H., Li, Y., Wu, R., Qi, X., and Chen, Y. AffordanceWAM: Affordance-aware joint world-action modeling for robot manipulation, 2026. URL https://arxiv.org/abs/2609.22332.
87. Yu et al. (2025) [87] Yu, E., Lin, K., Zhao, L., Yin, J., Wei, Y., Peng, Y., Wei, H., Sun, J., Han, C., Ge, Z., Zhang, X., Jiang, D., Wang, J., and Tao, W. Perception-r1: Pioneering perception policy with reinforcement learning, 2025. URL https://arxiv.org/abs/2504.07954.
88. Yu et al. (2016) [88] Yu, L., Poirson, P., Yang, S., Berg, A. C., and Berg, T. L. Modeling context in referring expressions. In European conference on computer vision, pp. 69–85. Springer, 2016.
89. Yu et al. (2026) [89] Yu, Q., You, J., Wang, Y., Liang, J., Ping, B., Tian, Y., Chen, Y., Cai, M., Gong, Z., Wu, R., et al. AffordanceVLA: A vision-language-action model empowering action generation through affordance-aware understanding. arXiv preprint arXiv:2606.06155, 2026.
90. Yuan et al. (2024) [90] Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R., Murali, A., Mousavian, A., and Fox, D. Robopoint: A vision-language model for spatial affordance prediction for robotics, 2024. URL https://arxiv.org/abs/2406.10721.
91. Yue et al. (2025) [91] Yue, Z., Lin, Z., Song, Y., Wang, W., Ren, S., Gu, S., Li, S., Li, P., Zhao, L., Li, L., et al. Mimo-vl technical report. arXiv preprint arXiv:2506.03569, 2025.
92. Zhang et al. (2022) [92] Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., and Shum, H.-Y. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022. URL https://arxiv.org/abs/2203.03605.
93. Zhang et al. (2023) [93] Zhang, H., Li, H., Li, F., Ren, T., Zou, X., Liu, S., Huang, S., Gao, J., Zhang, L., Li, C., and Yang, J. Llava-grounding: Grounded visual chat with large multimodal models, 2023. URL https://arxiv.org/abs/2312.02949.
94. Zhao et al. (2024) [94] Zhao, Z., Kang, H., Wang, B., and He, C. Doclayout-yolo: Enhancing document layout analysis through diverse synthetic data and global-to-local adaptive perception, 2024. URL https://arxiv.org/abs/2410.12628.
95. Zhou et al. (2026) [95] Zhou, E., An, J., Chi, C., Han, Y., Rong, S., Zhang, C., Wang, P., Wang, Z., Huang, T., Sheng, L., et al. Roborefer: Towards spatial referring with reasoning in vision-language models for robotics. Advances in Neural Information Processing Systems, 38:28404–28481, 2026.
96. Zhu et al. (2018) [96] Zhu, P., Wen, L., Bian, X., Ling, H., and Hu, Q. Vision meets drones: A challenge, 2018. URL https://arxiv.org/abs/1804.07437.

\beginappendix

\WF@box

7 Grounding Model Details

\WF@box

7.1 Input/Output Protocol

GroundingPI uses the same language-conditioned interface for box and point prediction. A query specifies the task and its semantic targets; the response associates each label, referring expression, or OCR transcription with a geometric payload. The examples below illustrate the protocol rather than training records. Displayed line breaks are for readability and are omitted in the serialized response.

Vocabulary and entry structure.

Table 6 summarizes the token roles. Coordinates are atomic vocabulary entries; labels and punctuation use ordinary language tokens. Every entry follows the template

<|object_ref_start|>label<|object_ref_end|> <|box_start|>payload<|box_end|>

A box uses four consecutive coordinate tokens in xyxy order; a point uses two in (x,y) order. Multiple instances with the same label share one wrapper, with tuples separated by commas without spaces. Distinct entries are separated by a comma and a space. Explicitly queried but absent categories retain their label and use the ordinary text payload None.

Task prompts.

Table 7 lists the canonical user prompts. Category names are joined by </c> without additional spaces or commas; requested category strings are preserved in grounding and layout responses. Referring prompts provide the target description, and OCR responses use recognized text as their labels. Dense and ordinary grounding share a prompt. Visual prompts encode example boxes with the same coordinate vocabulary as the output. The image and user prompt are supplied through the native multimodal chat template.

Illustrative responses.

The following example contains two cups and an absent car. The comma after the first entry is followed by a space in the serialized response.

<|object_ref_start|>cup<|object_ref_end|> <|box_start|><10><20><30><40>,<50><60><70><80><|box_end|>, <|object_ref_start|>car<|object_ref_end|> <|box_start|>None<|box_end|><|im_end|>

Pointing uses the same wrapper with two coordinates per instance:

<|object_ref_start|>cup<|object_ref_end|> <|box_start|><20><30>,<60><70><|box_end|><|im_end|>

OCR binds the transcription directly to its text region:

<|object_ref_start|>OPEN<|object_ref_end|> <|box_start|><100><200><400><300><|box_end|><|im_end|>

Coordinate conventions.

Image coordinates are normalized relative to image width and height. An integer coordinate v on the intermediate [0,1000] grid is mapped to q(v)=\lfloor(999v+500)/1000\rfloor and encoded by the corresponding atomic coordinate token. Image-space coordinates are recovered by multiplying q/999 by the corresponding image dimension. Source xywh boxes are converted to xyxy before normalization, and polygon annotations are converted to their axis-aligned enclosing boxes. Point targets follow their task definition; box centers are used only where the annotation convention specifies them. Non-finite, out-of-range, inverted, or degenerate geometry is rejected rather than silently repaired.

Ordering and response boundaries.

Within a label, box instances are stably sorted by x_{1} and unordered points by x; ordered trajectories retain temporal order, including repeated locations. Trajectories use a separate metric-coordinate convention described in Section 8.1. The marker <|box_end|> closes one payload, whereas <|im_end|> ends the response. Structural and coordinate tokens identify the entries and their geometry.

\WF@box

7.2 Architecture, Tokenizer, and Visual Processing

GroundingPI combines a MoonViT-V2 (Kimi K3) visual encoder, a learnable multimodal projector, and a Qwen3-4B language backbone. Visual embeddings are inserted into the language sequence, which uses one-dimensional rotary position indices. All 36 language layers use full attention. Table 8 summarizes the architecture.

Vocabulary and parameterization.

The tokenizer extends the 151,669-entry base vocabulary with 1,000 coordinate tokens and </c>, giving 152,670 entries. Coordinate IDs are 151669–152668, and the category separator has ID 152669. EOS and padding are distinct (151645 and 151643). The input embedding and output head are untied; all semantic, structural, and coordinate tokens are predicted by the same vocabulary head. The complete multimodal model contains approximately 4.844B parameters, including the visual encoder and vocabulary matrices (Table 9); Qwen3-4B denotes the language-backbone family.

\WF@box

7.3 Data Engine

Candidate generation and field fusion.

The engine combines complementary predictions for object localization, segmentation, text recognition, GUI elements, and document regions. For category-level grounding, category discovery precedes category-conditioned localization. Candidate instances are mapped to the original image and aligned within a common query scope and annotation granularity. Fusion operates separately on semantic and geometric fields: verified information is retained, while disagreements trigger additional evidence for the affected fields. Object–part containment and word–line relations are preserved rather than merged as duplicates.

Task-dependent validation.

Each field is marked as accepted, rejected, or unresolved. Validation checks geometry, syntax, semantic correspondence, and consistency with available evidence. Required fields are determined by the query, independently of which fields a teacher produces; missing required fields remain unresolved. A supervision item is accepted only when all required fields are accepted. Queries requiring all instances additionally require a coverage check over the relevant region. For an absent-target answer, absence is itself a required fact to verify; an empty prediction does not establish it. Consequently, an unresolved instance can block an all-target or counting query while leaving independently verified local supervision usable. Multi-teacher agreement supports validation but does not guarantee annotation correctness.

Targeted observations and expert iteration.

Accepted annotations train a unified grounding expert, whose subsequent predictions pass through the same fusion and validation procedure. Complementary teachers and local observations are invoked when semantic identity, geometry, or coverage remains unresolved. Crop predictions are mapped back to the original coordinates; supervision requiring detail unavailable in the original training input retains the necessary local view. Additional evidence can both add annotations and revise existing labels. Revisions propagate to dependent tasks, and invalidated labels are withdrawn until their dependencies are resolved.

Deriving supervision.

A verified instance record can support several tasks: category–box pairs provide grounding, attributes and relations support referring, valid instance regions support pointing, exemplar correspondence supports visual prompting, and text regions paired with transcriptions support OCR. Each derived query retains its own required fields and coverage conditions. This reuse preserves task-specific semantics while sharing the underlying visual evidence.

\WF@box

7.4 Base VLM Training (Pretrain 1)

Training organization.

GroundingPI training comprises three successive phases: (i) Base VLM training (Pretrain 1), (ii) coordinate alignment (Pretrain 2), also termed supervised fine-tuning (SFT), and (iii) reinforcement learning (RL) with GRPO. Pretrain 1 contains Stages 1–3 below, while Pretrain 2 corresponds to Stage 4. Thus, Stage 1–4 numbering describes the supervised training recipe within the first two phases. Each stage starts from the preceding checkpoint, and RL follows the Stage 4 SFT checkpoint. The aggregate training exposure is 500.41\mathrm{B}+221.17\mathrm{B} tokens.

Stage 1: Vision–language projector alignment.

We freeze the visual encoder and language model and update only the projector using image-caption supervision. Causal cross-entropy on the assistant response aligns visual features with the language input space, without an additional feature-distance or contrastive objective. The context and packing lengths are both 8192 tokens.

Stage 2: Joint multimodal pretraining.

We unfreeze the visual encoder, projector, and language model and jointly train on text-only and general image–text data. Text examples provide full-token causal supervision, while multimodal examples supervise assistant responses. The language model uses a peak learning rate of 10^{-5}, and the visual encoder and projector each use 10^{-6}. The context and packing lengths remain 8192 tokens.

Stage 3: General visual and video understanding.

We continue updating all modules on general visual question answering and instruction-following data, image captions, and videos. The context and packing lengths increase to 32768 tokens to accommodate longer multimodal sequences. This stage does not separately mix in the spatial-specialization data or an independent text-only quota; image-caption supervision remains part of the general visual training mixture.

\WF@box

7.5 Coordinate Alignment (Pretrain 2 / SFT)

Stage 4: Spatial perception specialization.

Starting from the Stage 3 checkpoint, coordinate alignment updates all modules using eight spatial task groups: detection, GUI grounding, referring-expression grounding, referring-expression pointing, OCR, document layout, dense pointing and counting, and visual prompting. This phase is the SFT phase described in the main text. It uses an 8192-token context and packing length and applies autoregressive supervision to semantic labels, box and point coordinates, and protocol markers. Quantized coordinate tokens share the same cross-entropy objective as other valid response tokens; no additional IoU regression loss is introduced.

Supervision and loss normalization.

Stages 1–4 use causal next-token prediction. Let \mathcal{S}_{d} contain valid sample–position pairs (b,t) in domain d\in\{\mathrm{text},\mathrm{vlm}\}. The domain loss is \mathcal{L}_{d}=-\bigl(\max(|\mathcal{S}_{d}|,1)\bigr)^{-1}\sum_{(b,t)\in\mathcal{S}_{d}}\log p_{\theta}(z_{b,t}\allowbreak\mid V_{b},z_{b,<t}), with V_{b}=\varnothing for text-only examples. Text supervision covers valid causal targets; multimodal supervision covers assistant responses, excluding prompt, visual, padding, and empty thinking-prefix positions. Stage 2 minimizes \mathcal{L}_{\mathrm{text}}+\mathcal{L}_{\mathrm{vlm}}, with each domain normalized over its valid targets across data-parallel workers. A missing domain contributes zero. Stages 1, 3, and 4 use \mathcal{L}_{\mathrm{vlm}} without a separate text-only domain. In Stage 4, this assistant-only loss includes the semantic, coordinate, and protocol tokens of structured spatial responses. Packing preserves each sample’s causal boundaries and supervision mask.

Optimization.

Table 10 summarizes the configuration for Pretrain 1 and Pretrain 2. The supervised training recipe uses 128 GPUs, BF16 precision, ZeRO-1, FlashAttention, and activation recomputation. Only the projector is trainable in Stage 1; Stages 2–4 update all modules. Language parameters include the input embedding and untied output head. All four stages use one gradient-accumulation step, AdamW with betas (0.9,0.95) and \epsilon=10^{-8}, and gradient clipping at 1.0. Weight decay is zero in Stage 1 and 0.1 thereafter. Cosine schedules use 3% warmup and decay to 2% of the peak learning rate in Stage 1 and 10% in Stages 2–4. The configured budget is one epoch per stage. Global batches count packed sequences; context length is the complete multimodal sequence budget, distinct from the visual budget of 1024 projected tokens. Seed and data seed are both 42.

\WF@box

7.6 Reinforcement Learning (RL)

RL is the third overall training phase. We initialize the policy from the coordinate-aligned Pretrain 2 / SFT checkpoint and optimize complete generated outputs with task-specific GRPO rewards.

Policy update.

We sample image–query pairs q=(I,P) from the training distribution \mathcal{D} and draw G=8 responses per pair from \pi_{\mathrm{old}}(\cdot\allowbreak\mid q). Let \mathcal{B}=\{(q_{i},Y_{i})\}_{i=1}^{N} denote the complete response batch, with each sampled prompt repeated for its G responses. Grounding uses two active reward components with weights 0.7 and 0.3; OCR uses one composite reward with weight 1. Inactive task components are excluded. For active component h, Z_{h}(R_{h,i})=(R_{h,i}-\mu_{h,q_{i}})/(s_{h,q_{i}}+\delta), where \mu_{h,q_{i}} and s_{h,q_{i}} are the mean and sample standard deviation over responses to the same prompt, and \delta=10^{-8}. Thus \widetilde{A}_{i}=0.7Z_{\mathrm{set}}(R_{\mathrm{set},i})+0.3Z_{\mathrm{strict}}(R_{\mathrm{strict},i}) for grounding and \widetilde{A}_{i}=Z_{\mathrm{OCR}}(R_{\mathrm{OCR},i}) for OCR. These values are standardized across all N responses: A_{i}=(\widetilde{A}_{i}-\mu_{\widetilde{A}})/(s_{\widetilde{A}}+\delta). A constant component within a prompt contributes zero before this final normalization.

The same response-level advantage is assigned to each output token. Define the token context c_{i,t}=(q_{i},y_{i,<t}) and policy ratio r_{i,t}=\pi_{\theta}(y_{i,t}\allowbreak\mid c_{i,t})/\pi_{\mathrm{old}}(y_{i,t}\allowbreak\mid c_{i,t}). We maximize the response-averaged objective

\mathcal{J}(\theta)=\mathbb{E}_{\mathcal{B}}\!\left[\frac{1}{N}\sum_{i=1}^{N}\frac{1}{|Y_{i}|}\sum_{t=1}^{|Y_{i}|}\left\{\min\!\left(r_{i,t}A_{i},\operatorname{clip}(r_{i,t},1-\epsilon,1+\epsilon)A_{i}\right)-\beta d_{i,t}\right\}\right].

Following GRPO (Shao et al., 2024), the sampled KL penalty is d_{i,t}=\rho_{i,t}-\log\rho_{i,t}-1\geq 0, where \rho_{i,t}=\pi_{\mathrm{ref}}(y_{i,t}\allowbreak\mid c_{i,t})/\pi_{\theta}(y_{i,t}\allowbreak\mid c_{i,t}). Its expectation equals D_{\mathrm{KL}}(\pi_{\theta}\|\pi_{\mathrm{ref}}) when the token is sampled from \pi_{\theta}; evaluated on old-policy rollouts, it is a sampled regularizer rather than an unbiased current-policy KL estimate. We use a frozen copy of the Pretrain 2 / SFT checkpoint as the reference, \epsilon=0.2, and \beta=0.02. Advantages and the old/reference policies are held fixed during the policy update. Only nonpadding response positions contribute to the objective. The vision encoder and projector remain frozen throughout reinforcement post-training.

\WF@box

7.6.1 Grounding Rewards

Set completeness.

For a nonempty reference set \{(g_{j},c_{j})\}_{j=1}^{n}, with n\geq 1, let \{(b_{k},\hat{c}_{k})\}_{k=1}^{m} denote the predictions. If m=0, set R_{\mathrm{set}}=0 without performing a match. Otherwise, select k_{j}^{\star}=\arg\max_{1\leq k\leq m}\operatorname{IoU}(g_{j},b_{k}) for each reference, then validate its class: s_{j}=\operatorname{IoU}(g_{j},b_{k_{j}^{\star}})\mathbb{I}[c_{j}=\hat{c}_{k_{j}^{\star}}]. With S=\sum_{j=1}^{n}s_{j}, soft recall and precision are R_{s}=S/n and P_{s}=S/m, and R_{\mathrm{set}}=2P_{s}R_{s}/(P_{s}+R_{s}+10^{-8}). Matching is independent for each reference and can reuse a prediction; this F1-style coverage surrogate can therefore exceed one. Empty reference sets lie outside this definition.

Strict localization and output quality.

The complementary reward is a weighted sum of the components in Table 12, clipped to [0,1]. Its detection term is \overline{F}=(F_{0.50}+F_{0.75}+F_{0.95})/3, where F_{\tau} is detection F1 at IoU threshold \tau; the matched-box IoU term provides continuous localization feedback. Format, count, and ordering scores assess the structured response, while nonnegative duplicate and oversized-box penalties discourage redundant or imprecise predictions. These auxiliary scores are distinct from the OCR count and format terms below. Both grounding components are standardized separately before their weighted combination, as described in Section 7.6.

\WF@box

7.6.2 OCR Rewards

Joint text–geometry matching.

Let P=\{(b_{i},s_{i})\}_{i=1}^{n} and T=\{(\hat{b}_{j},\hat{s}_{j})\}_{j=1}^{m} contain predicted and reference text instances, with n\geq 0, m\geq 1, and valid geometry and transcriptions. Define I_{ij}=\operatorname{IoU}(b_{i},\hat{b}_{j}). Text normalization \mathcal{N} applies Unicode NFKC, case folding, and alphanumeric filtering; symbol-only strings retain distinct Unicode-based keys. For nonempty normalized transcriptions, edit similarity is E_{ij}=1-\operatorname{Lev}(\mathcal{N}(s_{i}),\mathcal{N}(\hat{s}_{j}))/\max(|\mathcal{N}(s_{i})|,|\mathcal{N}(\hat{s}_{j})|). Blank transcriptions and empty reference sets lie outside these definitions. All affinities use Hungarian maximum-weight one-to-one assignment. If M(A) is the assigned affinity sum, define F(A)=2M(A)/(n+m); with no predictions, M(A)=F(A)=0.

The hard affinity is A^{\mathrm{hard},\tau}_{ij}=\mathbb{I}[\mathcal{N}(s_{i})=\mathcal{N}(\hat{s}_{j})]\mathbb{I}[I_{ij}\geq\tau], giving H_{\tau}=F(A^{\mathrm{hard},\tau}) and

\overline{H}=\tfrac{1}{10}\sum_{\tau\in\{0.50,0.55,\ldots,0.95\}}H_{\tau}.

The soft affinity is A^{\mathrm{soft}}_{ij}=\sqrt{I_{ij}}\,E_{ij}\mathbb{I}[I_{ij}\geq 0.10]\mathbb{I}[E_{ij}\geq 0.20], with S_{\mathrm{soft}}=F(A^{\mathrm{soft}}). These terms provide strict text–region agreement and continuous feedback for partial matches. Count agreement is C=\min(n,m)/\max(n,m), with C=0 when n=0. The reference-view score is V(P,T)=\operatorname{clip}_{[0,1]}(0.45\overline{H}+0.35S_{\mathrm{soft}}+0.10H_{0.50}+0.10C).

Word/line granularity.

Complete, nonempty word and line references T_{w} and T_{l} receive scores V_{w}=V(P,T_{w}) and V_{l}=V(P,T_{l}). Let V_{\mathrm{hi}}=\max(V_{w},V_{l}) and V_{\mathrm{lo}}=\min(V_{w},V_{l}). For a local granularity group g, predictions P_{g} are scored against each complete local representation: G_{g}=\max\{V(P_{g},T_{w}^{g}),V(P_{g},T_{l}^{g})\}. This selects between whole word and line views rather than combining isolated matches from incompatible views.

For singleton reference j, let B_{j} and S_{j} contain its valid box and text alternatives. Define

I_{ij}^{\star}=\max_{b\in B_{j}}\operatorname{IoU}(b_{i},b),\qquad E_{ij}^{\star}=\max_{s\in S_{j}}E(s_{i},s),

with the maxima taken independently. The corresponding hard and soft affinities use these starred quantities, with exact text agreement given by E_{ij}^{\star}=1, and retain Hungarian one-to-one assignment. The singleton consensus score is G_{\mathrm{cons}}=0.60\overline{H}^{\star}+0.40S^{\star}. Let \Gamma denote the annotation-dependent aggregate of the granularity and singleton scores; conflicting singleton blocks receive reduced weight. The global-view score D and content score Q_{g} use the coefficients in Table 13.

Set m_{g}=\max(|T_{w}|,|T_{l}|,1) and C_{g}=\min(n,m_{g})/m_{g}. Format score f_{g} is 1 for complete valid output, 0.6 for incomplete structure with reliably parseable instances, 0.3 for partially valid instances, and 0 for unparseable output. The final reward is R_{g}=\operatorname{clip}_{[0,1]}((Q_{g}+0.10f_{g}C_{g})f_{g}^{2}-\Pi), where \Pi=0.05P_{\mathrm{dup}}+0.10P_{\mathrm{over}}+0.05P_{\mathrm{invalid}} penalizes duplicate, excessive, and invalid predictions.

Complementary references for complex text.

For curved, rotated, or complex text arrangements, consider three nonempty reference views: two primary views T_{a},T_{b} and a supplementary view T_{c}. Each receives W=0.85V+0.15U, where U is text–geometry F1 at IoU 0.50 using exact NFKC/case-folded surface strings with punctuation retained. Primary scores are fused as W_{p}=0.70\max(W_{a},W_{b})+0.30\min(W_{a},W_{b}), then W_{t}=0.82W_{p}+0.18W_{c}. When a nonempty auxiliary geometry view is available, Q_{c}=0.95W_{t}+0.05\Gamma_{\mathrm{aux}}, with \Gamma_{\mathrm{aux}}=0.60\overline{H}_{\mathrm{geo}}+0.30S_{\mathrm{geo}}+0.10C_{\mathrm{geo}}; otherwise Q_{c}=W_{t}. The auxiliary term evaluates geometry only.

Set m_{c}=\operatorname{median}(|T_{a}|,|T_{b}|,|T_{c}|)\geq 1 and C_{c}=\min(n,m_{c})/m_{c}. Format scores f_{c} are 1, 0.45, 0.35, 0.15, and 0 for strictly valid output, complete output with extraneous content, incomplete but parseable structure, recoverable instances without a valid overall structure, and unparseable output, respectively. The reward is R_{c}=\operatorname{clip}_{[0,1]}((0.90Q_{c}+0.10f_{c}C_{c})f_{c}^{2}-\Pi-0.08P_{\mathrm{large}}). The last term penalizes boxes covering over 80% of the image without supporting reference geometry.

Each OCR sample selects the branch appropriate to its annotations. All content, format, coverage, and penalty terms are combined into one scalar before within-prompt standardization. OCR subterms are not independently standardized; the subsequent response-batch normalization follows Section 7.6. Training rewards provide optimization feedback and are distinct from the benchmark metrics.

\WF@box

8 Training Details and Evaluation Setup

This appendix describes the downstream interfaces and task-specific protocols for the physical-intelligence study in Section 5.2. Grounding-model optimization is provided separately in Sections 7.4, 7.5 and 7.6.

\WF@box

8.1 Autonomous Driving

Observation and trajectory interface.

The driving task uses the current front-camera image and up to seven ego-state records at 0.5-second intervals, including the current state and covering at most three seconds of history. Historical positions are expressed relative to the current ego pose, together with available velocity, acceleration, and steering information. Short histories retain their observed length; missing states are not replaced by fabricated zeros. The output is six cumulative future positions over three seconds, Y=((x_{1},y_{1}),\ldots,(x_{6},y_{6})), with x forward and y left, measured in meters. Current visual observations and historical states condition the prediction; future waypoints are the supervised targets.

Metric coordinates and serialization.

The trajectory interface uses fixed ranges x\in[-60,60] and y\in[-20,20] meters. For axis bounds [a,b] with a<b and an in-range coordinate z, define the integer code q_{z}=\operatorname{round}_{\mathrm{even}}(999(z-a)/(b-a))\in\{0,\ldots,999\} and reconstruct \hat{z}=a+(b-a)q_{z}/999. Rounding uses ties to even. These are metric-coordinate bins, distinct from image normalization in Section 7.1. GroundingPI’s existing atomic coordinate vocabulary represents the trajectory through one trajectory entry with six ordered coordinate pairs. Temporal order and repeated points are retained; the output describes cumulative locations, so no additional cumulative sum is applied. The adaptation objective is causal cross-entropy on the trajectory response.

Open-loop metric.

Predictions are decoded to meters and compared with unquantized reference waypoints. Let \bar{d}_{k} be the mean Euclidean error at future step k. The cumulative-horizon metric is \mathrm{L2}@h=(2h)^{-1}\sum_{k=1}^{2h}\bar{d}_{k} for h\in\{1,2,3\} seconds, and the reported average is \mathrm{L2}_{\mathrm{Avg}}=\tfrac{1}{3}\sum_{h=1}^{3}\mathrm{L2}@h. This averages errors up to each horizon, rather than only the endpoint errors. Open-loop trajectory error measures agreement with recorded trajectories; it does not establish closed-loop control performance.

\WF@box

8.2 Robot Manipulation

The manipulation study compares the foundations used by vision-language-action (VLA) and world-action model (WAM) systems without allowing their native action heads, conditioning paths, or optimization budgets to become additional variables. We replace the model-specific control modules with the same backbone-to-action interface and the same fixed-capacity action expert. The evaluated variable is therefore the pretrained foundation that supplies the representation for action learning.

The compared foundations are Qwen3-VL-4B (Bai et al., 2025a) and PaliGemma-3B (Beyer et al., 2024); Wan2.2-TI2V-5B (Wan et al., 2025) and Cosmos-Predict2.5-2B (Ali et al., 2025); LocateAnything-3B (Wang et al., 2026a) and Rex-Omni-3B (Jiang et al., 2026); and RynnBrain-2B (Dang et al., 2026) and RynnBrain1.1-2B (Li et al., 2026), alongside GroundingPI. These groups characterize the pretrained foundations rather than different downstream action heads.

\WF@box

8.2.1 Unified Backbone-to-Action Interface

Layer-wise backbone features.

For each observation and language instruction, the foundation backbone is executed once. Given a backbone with N_{B} transformer blocks, we sample eight hidden states at normalized depths \ell_{j}=\operatorname{round}(j(N_{B}-1)/7) for j\in\{0,\ldots,7\}. This rule preserves comparable relative depths across foundations with different numbers of layers and avoids selecting backbone-specific layers. Each sampled state H_{\ell_{j}} is normalized, mapped from the backbone width D_{B} to the common action width by a backbone-specific linear projector, augmented with a learned depth embedding, and compressed by a shared learned-query resampler: Z_{j}=\operatorname{LN}(H_{\ell_{j}})W_{B}+e_{j} and C_{j}=\operatorname{Resampler}_{64}(Z_{j}). The projector W_{B} is shared across the eight depths of a backbone, and the same resampler is shared across both depths and models. Every cross-attention block consequently receives the same number and width of condition tokens, independent of the backbone’s native width or token count.

Controlling the VLA–WAM comparison.

For VLA foundations, the eight states are taken from the language-conditioned vision–language stack. For WAM foundations, they are taken from the video-generation backbone under a fixed feature-extraction state, including the observed-frame construction, diffusion timestep, condition mask, noise realization, input resolution, and number of frames. In both cases, a single backbone forward pass produces all eight conditions, which are cached and reused throughout action denoising. We remove any model-specific raw-text or proprioceptive cross-attention bypass: language, perception, and dynamics information can reach the action expert only through the evaluated foundation representations. Thus, the two paradigms differ in the pretrained foundation that constructs the physical representation, while sharing the same downstream control architecture and compute schedule.

\WF@box

8.2.2 Fixed Layer-wise Action DiT

The common action expert is a fixed-depth, fixed-width \pi-style (Black et al., 2024) Action DiT (Peebles & Xie, 2023) trained with flow matching (Lipman et al., 2023). It contains 16 atomic transformer blocks, alternating between layer-wise cross-attention and action self-attention. Blocks 0,2,\ldots,14 attend to C_{0},C_{1},\ldots,C_{7}, respectively, and blocks 1,3,\ldots,15 perform self-attention over the action-side sequence. The architecture therefore contains eight cross-attention blocks, eight self-attention blocks, and 16 feed-forward networks; it is not a stack of 16 paired self- and cross-attention blocks.

The action-side input concatenates the encoded proprioceptive state, learned planning tokens, and a noisy action chunk with its flow timestep. Timestep-conditioned adaptive layer normalization is applied before attention, whereas the feed-forward path uses standard layer normalization. Only the action-token positions are decoded. Action and state input/output projections are embodiment-specific, but the transformer capacity and conditioning interface are identical across all foundations. The complete shared configuration is summarized in Table 14.

Given a normalized action chunk a, Gaussian noise \epsilon, and flow timestep t, training constructs a_{t}=(1-t)\epsilon+ta with target velocity v^{\star}=a-\epsilon and minimizes \lVert\hat{v}_{\theta}(a_{t},t)-v^{\star}\rVert_{2}^{2} on the action positions. Every model uses the same noise distribution, action normalization, and solver settings listed in Table 14.

\WF@box

8.2.3 Training Budget and Evaluation Protocol

Within each benchmark, every foundation is trained on the same action demonstrations with the same sampling and augmentation pipeline. RoboTwin 2.0 Full uses the Clean and Randomized protocols, whereas Clean2Random uses only Clean demonstrations for training and holds Randomized scenes out for evaluation. RoboCasa-GR1 uses the same task suite and demonstration pool for every foundation. The reduced-data experiments change only the available fraction of the action dataset. All other optimization settings are matched across foundations, as summarized in Table 15.

This protocol fixes the quantities most likely to confound a backbone comparison: action-head depth and width, condition-token budget, state and action paths, action data, batch size, number of updates, optimizer and schedule, flow objective, and inference computation. Backbone-specific width projectors are the only interface parameters whose size varies with the native backbone width; they are shared across depth and remain small relative to the common action expert. The comparison therefore measures how effectively the representations inherited from VLA- and WAM-style foundations support effective and generalizable action learning under matched downstream capacity and supervision.

\WF@box

9 Additional Transfer and Ablation Analyses

\WF@box

9.1 Data Efficiency and the Burden on Action Demonstrations

Figure 7 varies the fraction of RoboCasa-GR1 demonstrations while retaining the action architecture and remaining training choices. GroundingPI leads every reduced-data setting. Its 28.75% SR with half the demonstrations exceeds the strongest baseline trained on three quarters, RynnBrain at 27.75%. At full data, RynnBrain reaches 39.00%, compared with GroundingPI’s 37.75%. Thus, the main advantage in this experiment is effective adaptation with limited demonstrations, rather than the highest full-data ID score.

The distinction between learning where to interact and how to act interprets this result in terms of the paper’s motivation. A reusable perceptual foundation may let downstream supervision focus more on control, but the experiment does not separately measure perceptual and motor-learning sample complexity. It also does not vary physical prompts. The evidence concerns downstream demonstration efficiency in manipulation; it does not establish lower total pretraining cost, a universal data-scaling law, or driving-data efficiency.

\WF@box

9.2 Grounding and Action Scaling

Table 16 records the performance trajectory in Figure 8, ordered by increasing grounding-training exposure. The initial backbone, downstream action architecture, action demonstrations, and action-training recipe are held fixed.

Table 16: Performance at successive grounding-training exposures. The last row is the full configuration. Exposure order Grounding Avg RoboCasa-GR1 ID SR RoboCasa-GR1 OOD Avg SR 1 61.69 33.92 26.59 2 69.71 35.92 30.00 3 72.22 38.75 32.06 4 (full) 73.68 37.75 33.38 The endpoint gains are 11.99 pp in grounding, 3.83 pp in ID SR, and 6.78 pp in OOD Avg SR, using the recorded values before rounding. Grounding and OOD performance improve at every step, whereas ID SR falls by 1.00 pp at the final step. The evidence supports joint improvement in perception and transfer, especially generalization, but not a monotonic law connecting grounding score to every action metric. Four settings without repeated-seed uncertainty estimates are insufficient to establish a scaling law or determine whether the final ID decrease is systematic.

RoboCasa-GR1 OOD Avg weights Container, Appearance, and Type by their evaluation-suite sizes:

\mathrm{SR}_{\mathrm{OOD}}=\frac{14\,\mathrm{SR}_{\mathrm{Container}}+18\,\mathrm{SR}_{\mathrm{Appearance}}+32\,\mathrm{SR}_{\mathrm{Type}}}{64}.

These are evaluation weights, not training-mixture proportions. The aggregate is computed before rounding the displayed suite scores.

\WF@box

9.3 Which Perceptual Capabilities Transfer?

The six groups retain the naming in Figure 9: A Basic Grounding; B Dense Grounding; C Referring; D Basic Pointing; E Robo Pointing; and F Else (OCR, Layout, GUI). Group membership specifies task inclusion only. Table 17 reports the downstream results; the action-learning setup is unchanged across configurations.

Basic grounding provides an important foundation.

Adding A and D to F raises ID SR from 24.42% to 30.92%, OOD Avg SR from 7.28% to 29.28%, and reduces driving error from 0.391 to 0.310 m. This supports the importance of basic object localization together with pointing, particularly for generalization. Since A and D change jointly and no A-only leave-one-out row is available, their individual contributions cannot be identified. The table does not establish that either group is independently necessary or sufficient.

Dense grounding has broad marginal value.

Removing B reduces ID/OOD SR by 4.67/3.47 pp and increases L2 error by 0.015 m, larger changes than removing C or E. Dense and tiny-object perception share the need to preserve small spatial distinctions and separate nearby instances. Such demands plausibly recur in cluttered manipulation and distant road objects, making this supervision relevant across domains. The intervention removes the dense group as a whole; it does not disentangle object size, crowding, or annotation coverage.

Why tiny targets demand precise localization.

For equal axis-aligned boxes of width w>0 and height h>0 displaced only horizontally by \delta, with |\delta|<w, the intersection and union areas are (w-|\delta|)h and (w+|\delta|)h. Thus, for 0<\tau<1,

\operatorname{IoU}(\delta)=\frac{w-|\delta|}{w+|\delta|},\qquad\operatorname{IoU}(\delta)\geq\tau\;\Longleftrightarrow\;|\delta|\leq w\frac{1-\tau}{1+\tau}.

The tolerated absolute displacement shrinks linearly with target width. This illustrates the precision demanded by tiny targets and complements the need to separate nearby instances in dense scenes.

Embodied labels are not the only route to action.

Removing E reduces ID/OOD SR by 1.92/1.53 pp and leaves driving error unchanged at the displayed precision. This is a smaller marginal effect than removing B, not evidence that embodied supervision is useless: its information may overlap with other groups, and the tasks may not emphasize every affordance it teaches. Removing C causes 2.08/1.09 pp losses and a 0.002 m error increase. Existing language understanding may reduce its marginal benefit, but that explanation would require a controlled change to the language foundation.

Complementarity, not independent effect sizes.

Removing F produces the largest manipulation decreases, 5.33/3.63 pp, yet F alone is the weakest configuration. These observations establish complementarity at the group level. The full mixture is best on both manipulation metrics and ties the best displayed driving error; it is not uniquely best on every metric. Leave-one-out changes depend on the retained groups and should not be added as independent contributions. No interaction effect or OCR-only causal effect is identified by this incomplete factorial design.

\WF@box

9.4 OCR as a Perceptual Catalyst: A Hypothesis

Table 18: OCR mixing schedules. We vary when OCR supervision is incorporated to probe its catalytic role in downstream transfer. Cold-start + batch mixing corresponds to the full six-group configuration. RoboCasa-GR1 nuScenes OCR schedule ID SR (%\uparrow) OOD Avg. SR (%\uparrow) L2 Avg. (m\downarrow) Cold-start mixing 35.17 32.38 0.296 Batch mixing 34.92 32.53 0.296 Cold-start + batch mixing 37.75 33.38 0.296 OCR mixing schedules. To probe OCR’s catalytic role, we compare cold-start mixing, batch mixing, and their combination (Table 18). The combined schedule achieves the highest manipulation success on both ID and OOD splits, while nuScenes L2 remains unchanged at the reported precision. This pattern is consistent with complementary benefits from early OCR exposure and continued joint training for manipulation transfer, motivating the following analysis of transcription beyond geometric supervision. OCR may complement grounding by coupling fine visual discrimination with region–text alignment, providing a localized captioning proxy for perceptual learning (Section 5.3.2). This motivation is consistent with the benefits of local visual semantics (Covert et al., 2025). The hypothesis concerns the contribution of transcription beyond shared geometric supervision.

Transcription beyond geometry.

Partition supervised OCR response positions into box, transcription, and protocol roles, \mathcal{T}_{b}, \mathcal{T}_{t}, and \mathcal{T}_{p}, with nonempty union \mathcal{T}. The SFT loss decomposes as

\mathcal{L}_{\mathrm{OCR}}=\mathcal{L}_{b}+\mathcal{L}_{t}+\mathcal{L}_{p},\qquad\mathcal{L}_{r}=-\frac{1}{|\mathcal{T}|}\sum_{i\in\mathcal{T}_{r}}\log p_{\theta}(y_{i}\allowbreak\mid I,Q,y_{<i}),\quad r\in\{b,t,p\}.

Because transcription precedes coordinates, teacher-forced box prediction already conditions on the reference text. A box-only control therefore removes \mathcal{L}_{t} while retaining protocol supervision, all transcription tokens, and the original denominator |\mathcal{T}|. Comparing it with the full loss isolates the additional transcription gradient under identical conditioning and matched training budgets. Deleting transcription tokens changes the box-prediction task; renormalizing over unmasked positions changes the weight of the retained losses.

Local transfer condition.

For trainable visual parameters \theta_{v}, let g_{t}=\nabla_{\theta_{v}}\mathcal{L}_{t} and g_{g}=\nabla_{\theta_{v}}\mathcal{L}_{g}, where \mathcal{L}_{g} is a non-text grounding loss. With other parameters fixed and Hessian norm bounded by H along the update,

\mathcal{L}_{g}(\theta_{v}-\eta g_{t})-\mathcal{L}_{g}(\theta_{v})\leq-\eta\langle g_{g},g_{t}\rangle+\frac{H\eta^{2}}{2}\lVert g_{t}\rVert_{2}^{2}.

Positive alignment permits a local decrease for sufficiently small \eta>0; total OCR-gradient alignment could instead arise from box supervision alone. For AdamW, the corresponding first-order diagnostic is \langle g_{g},\Delta\theta_{v}\rangle using the actual update. The catalyst hypothesis predicts that adding transcription loss under the fixed-conditioning control improves non-text dense/tiny grounding and its subsequent transfer to action.

\WF@box

9.5 Output Representation and Generation Cost

The coordinate ablation favors quantization by 2.47 pp in grounding Avg. The reported textual speed is 0.25\times the quantized configuration’s speed; the figure does not specify a closed-loop action-latency metric. The visual-encoder gaps are 0.51 pp for MoonViT and 2.03 pp for Qwen3-ViT relative to MoonViT-V2, while reinforcement learning contributes 0.82 pp under the reported comparison. These are configuration-level results; the encoder comparison also changes pretrained visual representations and does not isolate cross-layer fusion.

In Table 5, SEED1.5-VL statistics are external values from (Jiang et al., 2026), rather than a new reproduction. Dividing the reported tokens per box gives approximately 19.6\times and 14.6\times fewer tokens for GroundingPI on COCO and Dense200. These ratios quantify serialization, not measured speedups: model computation, sampling, and the distribution of predictions also matter. Four atomic coordinates specify a box, with additional tokens for labels, separators, and wrappers. Sharing one label wrapper across multiple instances amortizes overhead, explaining why tokens per box can fall in dense images even as total output length rises. Figure 11 is consistent with increasing generation cost as more boxes are emitted; it neither measures action execution nor establishes real-time performance.

\WF@box

10 Embodied-Foundation Design: Mechanisms and System Roles

We analyze how pretrained visual features reach the manipulation action interface under a new objective. These local, conditional analyses characterize feature access and adaptation; the cross-model comparisons do not isolate architectural causes.

\WF@box

10.1 Architectural Facts and Scope of Comparison

RynnBrain-2B uses Qwen3-VL with full-attention language layers (Dang et al., 2026; Bai et al., 2025a). RynnBrain1.1-2B uses Qwen3.5 with 18 linear-attention and six full-attention layers, plus attention-output gating (Li et al., 2026; Qwen Team, 2026a). Both employ DeepStack; here this denotes intermediate ViT-feature injection into early language layers (Li et al., 2026; Bai et al., 2025a). Qwen3-VL injects projected features into its first three language layers. GroundingPI combines MoonViT-V2, a single visual interface, and a full-attention Qwen3-4B decoder.

RynnBrain1.1 trails RynnBrain in manipulation but improves driving, motivating analysis of task-dependent feature access. The comparison fixes downstream capacity and optimization while varying complete pretrained foundations. RynnBrain and Qwen3-VL share DeepStack, whereas GroundingPI also differs in encoder, connector, pretraining, positional treatment, and language-model scale. The Qwen2.5-VL foundations of Rex-Omni and LocateAnything leave general base-model quality as another potential influence alongside grounding specialization (Jiang et al., 2026; Wang et al., 2026a).

\WF@box

10.2 Gated Memory and Attention: Conditional Transfer Sensitivities

For the eight-depth manipulation interface in Section 8.2, write

C_{s}=\mathcal{R}\!\left(\operatorname{LN}(H_{\ell_{s}})W_{B}+e_{s}\right),\quad s=0,\ldots,7,\qquad\hat{y}=\mathcal{A}_{\phi}(\xi;C_{0},\ldots,C_{7}).

Here \mathcal{R} is the resampler, W_{B} the shared projector, e_{s} the depth embedding, and \mathcal{A}_{\phi} the Action DiT predicting flow velocity. Local derivatives hold parameters and action-side inputs \xi fixed.

Recurrent transmission.

For one GDN head (Yang et al., 2025b), let k_{t},q_{t}\in\mathbb{R}^{d_{k}}, v_{t}\in\mathbb{R}^{d_{v}}, and S_{t}\in\mathbb{R}^{d_{v}\times d_{k}}, with

S_{t}=S_{t-1}A_{t}+\beta_{t}v_{t}k_{t}^{\top},\qquad A_{t}=\alpha_{t}(I-\beta_{t}k_{t}k_{t}^{\top}),\qquad o_{t}=S_{t}q_{t}.

Assume 0\leq\alpha_{t},\beta_{t}\leq 1, \lVert k_{t}\rVert_{2}\leq 1, and uniformly \lVert q_{t}\rVert_{2}\leq Q, consistent with normalized keys and queries. Fix all keys, queries, gates, other values, and the preceding state, and perturb only v_{j}. Expanding the recurrence gives

\displaystyle\Delta o_{t} \\ \displaystyle=\kappa_{jt}\Delta v_{j},\qquad\kappa_{jt}=\beta_{j}k_{j}^{\top}A_{j+1}\cdots A_{t}q_{t},\quad t\geq j, \\ \displaystyle|\kappa_{jt}| \\ \displaystyle\leq\beta_{j}Q\prod_{r=j+1}^{t}\alpha_{r}.

The ordered product is I for t=j; indices denote sequence positions, not physical distance. The bound follows from \lVert A_{t}\rVert_{2}\leq\alpha_{t} and decays as \bar{\alpha}^{t-j} if \alpha_{r}\leq\bar{\alpha}\in(0,1) uniformly along the path.

Access through action conditions.

Treat o_{1},\ldots,o_{T} as independent inputs to the downstream network. Let R_{t}=\partial\hat{y}/\partial o_{t} include subsequent backbone layers, all eight feature readouts, projection, resampling, and the Action DiT. The conditional value-path Jacobian is

J_{j}^{\mathrm{rec}}=\sum_{t=j}^{T}\kappa_{jt}R_{t},\qquad\nabla_{v_{j}}^{\mathrm{rec}}\mathcal{L}_{\mathrm{act}}=(J_{j}^{\mathrm{rec}})^{\top}\nabla_{\hat{y}}\mathcal{L}_{\mathrm{act}}.

The t=j term retains access through the token’s own hidden state. Action sensitivity thus depends jointly on \kappa_{jt} and R_{t}, including downstream amplification or cancellation; image perturbations can additionally follow residual and full-attention paths.

Output gating.

For gated softmax attention (Qiu et al., 2026), write b=W_{o}(g\odot u), where u is the attention output and g=\sigma(z) is an elementwise sigmoid gate. With \lambda=\nabla_{b}\mathcal{L}_{\mathrm{act}},

\nabla_{u}\mathcal{L}_{\mathrm{act}}=g\odot W_{o}^{\top}\lambda,\qquad\nabla_{z}\mathcal{L}_{\mathrm{act}}=g\odot(1-g)\odot u\odot W_{o}^{\top}\lambda.

Small gates attenuate branch-feature gradients, while saturation attenuates gate-logit gradients; residual paths and optimizer rescaling remain. Spatial readout from the eight resampled conditions is therefore the relevant diagnostic of action-feature accessibility, beyond retention or output-gate values alone.

\WF@box

10.3 Multi-level Visual Injection and Action Readout

DeepStack enriches visual evidence through intermediate feature injection (Meng et al., 2024; Bai et al., 2025a). Under limited action adaptation, its utility may depend on compatibility with the shared projector and resampler in Equation 6. GroundingPI uses a single ViT-to-language interface, but both designs provide eight feature levels to the Action DiT. The hypothesis in Section 5.3.3 therefore concerns adaptation of a shared cross-depth readout.

Readout sensitivity.

For vectorized states, write h_{\ell+1}=F_{\ell}(h_{\ell})+U_{\ell}f_{\ell}, where U_{\ell} inserts projected visual features f_{\ell} at visual-token positions. Let \mathcal{I} index injection layers, \mathcal{J}_{\ell}=\partial F_{\ell}/\partial h_{\ell}, h_{m_{s}}=\operatorname{vec}(H_{\ell_{s}}), and c_{s}=\operatorname{vec}(C_{s}). With initial state, parameters, and action-side inputs fixed,

\displaystyle B_{\ell} \\ \displaystyle=\sum_{s:m_{s}>\ell}\frac{\partial\hat{y}}{\partial c_{s}}\frac{\partial c_{s}}{\partial h_{m_{s}}}\mathcal{J}_{m_{s}-1}\cdots\mathcal{J}_{\ell+1}U_{\ell}, \\ \displaystyle\delta\hat{y} \\ \displaystyle=\sum_{\ell\in\mathcal{I}}B_{\ell}\,\delta f_{\ell}+O(\lVert\delta f\rVert_{2}^{2}).

Empty products are identities; \delta f concatenates feature perturbations, and the remainder assumes locally bounded second derivatives. Each B_{\ell} includes all affected action readouts. This quantifies sensitivity to feature perturbations, which are not themselves prediction errors.

Task alignment under finite adaptation.

For two injection settings with other parameters and inputs unchanged, let e_{0}=\hat{y}_{0}-y^{\star} denote error relative to the flow-matching target and d=\hat{y}_{1}-\hat{y}_{0} the prediction change. Then

\mathbb{E}\lVert e_{0}+d\rVert_{2}^{2}-\mathbb{E}\lVert e_{0}\rVert_{2}^{2}=2\mathbb{E}[e_{0}^{\top}d]+\mathbb{E}\lVert d\rVert_{2}^{2}.

Additional evidence helps when correction of existing error outweighs the quadratic term; for small changes, d\approx\sum_{\ell}B_{\ell}\delta f_{\ell}. If extra branches can be zeroed with all other components unchanged, the multi-injection model retains the single-interface model as a special case, so path count alone cannot raise optimal task loss. Any practical difficulty instead concerns learning a compatible readout within the available data and optimization budget. Shared projection and resampling couple adaptation across feature levels, while depth embeddings and separate action cross-attention blocks can compensate. A matched-capacity comparison of shared and depth-specific projectors across action-data budgets would directly test this compatibility hypothesis.

\WF@box

10.4 Designing Perceptual Foundations for System 1

Different roles may favor different foundations.

A useful design hypothesis separates knowledge-intensive deliberation from rapid perception–action execution. A planning system may benefit from extensive world knowledge, coding, and long-horizon reasoning. An execution system must translate the current goal and observations into reliable interaction while incorporating timely feedback. Existing dual-system architectures provide concrete precedents: RoboDual couples a generalist with a specialist policy, and GR00T N1 couples vision–language interpretation with a diffusion action module (Bu et al., 2025; NVIDIA et al., 2025). These systems motivate the distinction; they do not establish that one universal division of modules is optimal.

Here, System 1 denotes the functional perception–action execution loop. Its module boundaries need not match the naming in another architecture: GR00T N1 calls its VLM System 2 and its diffusion action module System 1. Our proposal concerns the perceptual foundation supplying an action learner, rather than identifying an unadapted grounding VLM with a complete controller. The benefit sought is precise, task-conditioned information available when an action must be selected or corrected.

Execution quality can limit the value of planning.

The coding-agent analogy highlights the need for a dependable execution layer: increasingly sophisticated generated programs have limited practical value if they repeatedly fail to run or if their feedback cannot guide correction. In robotics, stronger plans similarly depend on correctly identifying interaction targets and executing feasible actions. Code as Policies offers a concrete connection between these roles by generating programs that combine perception outputs with control APIs (Liang et al., 2023). Our analogy motivates evaluating the complete feedback loop; it is not an empirical comparison between coding and robot-learning systems.

What perception-native pretraining may contribute.

Dense and tiny-object grounding requires distinctions that can matter directly for interaction: which instance is intended, where it is, and which local region is relevant. Basic grounding supplies reusable localization, while OCR may complement it through local visual precision and semantic alignment. These observations suggest pretraining priorities for a foundation supporting execution. Broad semantic knowledge remains useful, but may be insufficient when the limiting factor is spatial discrimination. The controlled backbone comparison supports this concern for the evaluated Qwen- and PaliGemma-based foundations; it does not establish inferiority of every adaptation recipe or every member of the \pi family. Grounding also leaves dynamics, contact, and feedback control to the downstream policy.

Interpreting the Astra comparison.

Astra is a frontier reference for grounding quality in this study. Comparing a compact grounding model with it assesses how closely specialized perception can approach a strong general-purpose reference across the evaluated capabilities. It does not define a theoretical performance ceiling or evaluate Astra as a VLA backbone. Assigning a frontier model to higher-level planning and a perception-native foundation to execution is a prospective design, compatible with the observed capability differences but not tested as an integrated system here.

Tests of the proposed division.

A direct evaluation would hold the planner and action expert fixed while changing the perceptual foundation, then measure task success, robustness under distribution shift, and observation-to-action latency. Perception failures should be distinguished from planning and control failures, especially for small, crowded, or ambiguous targets. Conversely, varying planner capability under a fixed execution layer would test when better reasoning helps and when interaction remains the limiting factor. The present results motivate these experiments through grounding quality, controlled action transfer, and manipulation-data efficiency; they do not yet demonstrate a real-time dual-system controller.

\WF@box

11 Limitations and Future Work

GroundingPI’s aggregate strength is not uniform dominance: tiny-box localization, high-IoU boundaries, TotalText OCR, FSC147 visual prompting, and frontier-level GUI grounding retain clear gaps. Prompt and parser compatibility also constrain several baseline measurements; unsupported evaluations are not evidence of absent intrinsic capability. The manipulation comparison controls the downstream recipe, but pretrained foundations differ in scale, architecture, and supervision. Driving is evaluated open loop, and action-data efficiency is tested only in manipulation. OCR-specific causality and the proposed architectural mechanisms remain hypotheses requiring matched interventions. Finally, physical prompting and complementary System-1 execution alongside frontier reasoning are directions suggested by the interface, rather than capabilities independently established by the current experiments.

\WF@box

12 Comprehensive Grounding Benchmark Results

This appendix reports the complete recorded baseline comparisons underlying the selected main-text tables, including local evaluation results and externally reported baselines. Published reference scores replace local entries only for the explicitly marked substitutions. Missing metrics are not reconstructed from other scores.

Reporting conventions.

The notation defined in Section 5.1 applies throughout. R and P denote recall and precision. For box grounding, the nine columns report R/P/F1 at IoU 0.50 and 0.95, followed by their mIoU aggregates. Object pointing uses point-in-mask R/P/F1. OCR uses the recorded loose-match F1 metrics; parse error is the percentage of outputs that cannot be parsed. Named dataset-family averages are unweighted arithmetic means and are reported only when every component is available. Threshold-specific and mean-overlap metrics are retained as recorded; missing recall or precision is not inferred from F1. The headline Avg is the mean of 11 independently computed capability scores. Benchmark-level results below are reported separately and do not define this average through a direct mean of their table entries. Size groups use the language backbone configuration (the 7B understanding branch for MoT models and total language-model parameters for MoE models). Qwen3.7-Max, Kimi-K2.6, Kimi-K3, and GPT-6 Astra are grouped separately in the >1T category.

Benchmark suite and metrics.

Common and long-tailed detection use COCO (Lin et al., 2014) and LVIS (Gupta et al., 2019); dense and tiny-object detection use Dense200 (Jiang et al., 2026) and VisDrone (Zhu et al., 2018). Referring grounding covers HumanRef (Jiang et al., 2025), RefCOCO and RefCOCO+ (Yu et al., 2016), and RefCOCOg (Mao et al., 2016; Nagaraja et al., 2016). Spatial pointing uses RefSpatial (Zhou et al., 2026) and RoboSpatial (Song et al., 2025). GUI grounding uses ScreenSpot-Pro (Li et al., 2025), ScreenSpot-V2 (Wu et al., 2025), and OSWorld-G (Xie et al., 2026). OCR covers HierText (Long et al., 2022), ICDAR2015 (Karatzas et al., 2015), TotalText (Ch’Ng & Chan, 2017), and SROIE (Huang et al., 2019); layout grounding uses DocLayNet (Pfitzmann et al., 2022) and M6Doc (Cheng et al., 2023). Visual prompting uses FSC147 (Ranjan et al., 2021) and exemplar-conditioned detection. RefCOCO avg is the arithmetic mean of the three family-level F1mIoU scores, distinct from the separately reported RefCOCOg validation/test columns. RefSpatial avg averages Location and Placement. ScreenSpot reports action accuracy, and OSWorld-G reports exact accuracy. Scores marked as external retain their source protocol and are not presented as locally reproduced measurements.

\WF@box

12.1 Common and Long-tailed Object Detection

COCO.

GroundingPI achieves 62.98 F1mIoU, exceeding Rex-Omni (56.28), LocateAnything Hybrid (59.12), and Astra (62.75). Its 81.16 F1 at IoU 0.50 reflects strong object coverage, but its 22.84 at IoU 0.95 trails several baselines. The aggregate gain therefore should not be read as uniformly superior boundary precision: extremely strict localization remains a limitation even on common objects.

LVIS.

GroundingPI reaches 56.02 F1mIoU and 75.22 F1 at IoU 0.50, improving over Astra’s 54.97 and 72.58. SenseNova-Vision remains slightly ahead on F1mIoU (56.12) and more clearly ahead at IoU 0.95 (28.88 versus 22.42). Together with COCO, these results support broad category coverage while identifying high-IoU localization as a separate challenge.