If you're evaluating general-purpose robot policies, VLA models, or robotic arm agents, don't just ask "what's the success rate." Before deployment, you need answers to at least 7 questions:
- Does the test set share visual overlap with the training environment?
- Has the task set already been saturated by the model?
- Does the success rate come with a confidence interval?
- When it fails, can you pinpoint wrong grasps, drops, collisions, or step errors?
- How much does performance drop when language instructions become vague, overly detailed, or rephrased?
- How much does performance drop with messier scenes, more distractors, or longer task sequences?
- Is the trajectory short, smooth, and acceptable—not "bumping around but eventually succeeding"?
This article is based on the RoboLab evaluation methodology published on the NVIDIA Developer Blog on 2026-07-11 (source: NVIDIA Technical Blog, verified 2026-07-14). NVIDIA positions RoboLab as a simulation-based evaluation platform for real-world robot policy deployment, designed to address visual/task overlap, benchmark saturation, diagnostic gaps, and statistical trustworthiness. NVIDIA also states that RoboLab research will power the open-source NVIDIA Isaac Lab-Arena, with key features planned for productization in August 2026.
This article covers the evaluation workflow only and does not assume RoboLab or Isaac Lab-Arena is ready for all production robotic systems.
Who Needs This Evaluation
This is for three types of teams:
| What You're Doing | Why You Need It |
|---|---|
| Training or fine-tuning a robot foundation model / VLA policy | Static benchmark scores can mask generalization failures |
| Deploying a robotic arm, mobile robot, or humanoid policy in the real world | Real-world testing is expensive, slow, and hard to reproduce—use high-quality simulation to filter risk first |
| Procuring or evaluating a third-party robot policy | A single success rate from a vendor isn't enough; you need diagnosable metrics |
If you're working on general text, image, or code agents, this isn't your priority read. But the evaluation principles transfer: don't just look at aggregate scores—look at coverage, failure types, confidence intervals, and real task complexity.
Why Success Rate Isn't Enough
NVIDIA's article identifies at least four common problems in robot policy evaluation.
| Problem | Risk | How to Address It |
|---|---|---|
| Visual overlap between training and evaluation | Scores may just reflect memorized simulation scenes, not real-world generalization | Use higher-fidelity or different simulation sources; avoid sharing the same visual distribution |
| Fixed benchmark saturation | All models approach perfect scores, losing discriminative power | Continuously generate new tasks, retire saturated ones |
| Binary success/failure is too coarse | You can't tell if failure is due to language understanding, perception, grasping, collision, or step ordering | Track graded scores, trajectory quality, and failure events |
| Too few rollouts | A single success rate has no statistical trustworthiness | Report confidence intervals, not just the mean |
NVIDIA provides a statistical example: with an observed 90% success rate and only 70 rollouts, a 95% Clopper-Pearson confidence interval spans 15.4 percentage points. Tightening it to roughly ±2 points requires about 1,030 rollouts. This specific example doesn't apply to every task directly, but it illustrates one thing: small-sample success rates are not suitable for go/no-go decisions.
Define Task Coverage Before Running Models
A key RoboLab practice is capability-tagged tasks. NVIDIA breaks general manipulation capabilities into at least three categories:
| Capability | What It Tests | Example Task |
|---|---|---|
| Visual | Color, size, semantic category recognition | Put the small red cup in the bin |
| Procedural | Stacking, reorienting, tool affordances, action sequencing | Turn the mugs right-side up and stack the red ones on the shelf |
| Relational | Spatial reasoning, counting, conjunctions and disjunctions | Pick the orange or the lime and put it in the bowl |
Your first evaluation table can be built around these dimensions:
| Dimension | Minimum Coverage |
|---|---|
| Object categories | Training objects + real-world critical objects + distractors |
| Visual attributes | Color, size, shape, material, similar objects |
| Instruction types | Clear, vague, overly detailed, paraphrased |
| Operation types | Pick, place, sort, stack, open/close, tool use |
| Spatial relations | Left/right, inside/outside, near/far, on/under |
| Task horizon | Single-step, 2-4 steps, long chain |
| Scene complexity | Clean table, few distractors, crowded scene, visual noise |
Don't lump all tasks into a single aggregate success rate. Break results down by capability tag, or you won't know whether the model is weak at visual recognition, language understanding, or motion control.
Metrics at Four Levels
Pre-deployment evaluation can be organized into four layers.
1. Binary Success Rate
Still track success/failure—it's the easiest to tie to business goals. But every success rate should include:
- Number of rollouts
- Number of random seeds or scene variations
- Confidence interval
- Task tags
- Robot embodiment / policy version
Don't compare two models that differ by only 2-3 percentage points when their confidence intervals overlap significantly.
2. Graded Task Scores
NVIDIA's RoboLab uses graded task scores, giving partial credit for multi-step tasks. These are more diagnostic than binary success.
Example:
| Task | Binary Result | Graded Breakdown |
|---|---|---|
| Grasped correct object but placed it wrong | Fail | Perception and grasping passed; placement or spatial reasoning failed |
| Completed goal but picked up an extra distractor | Success or partial success | Task objective met, but there was an erroneous object interaction |
| First two steps correct, third step out of order | Fail | Long-horizon step sequencing insufficient |
The value of graded scores is telling the team what to fix next: data, planner, language grounding, or low-level control.
3. Trajectory Quality
NVIDIA's article mentions that RoboLab measures path length and SPARC (Spectral Arc-Length) as motion smoothness metrics, and also tracks end effector velocity. The reason is practical: a robot "completing the task" doesn't mean it's ready for deployment.
Before deployment, at least check:
- Does the path明显绕远
- Are motions jerky, with abrupt stops or back-and-forth试探
- Does it接近collision boundaries
- Does it move at unacceptable speeds
- Does it frequently re-grasp or readjust
In warehouses, factories, hospitals, and homes, trajectory quality is directly tied to safety, wear, energy consumption, and user trust.
4. Failure Event Logging
NVIDIA emphasizes failure event logging: automatically tracking wrong-object grasps, dropped objects, and gripper collisions, pinpointed to specific frames in the episode. You should build a similar event table.
| Event | What to Log |
|---|---|
| Wrong object grasp | Instruction, target object, grasped object, visual similarity |
| Drop | Grasp pose, velocity, object material, drop timing |
| Collision | Collision object, robot part, scene layout |
| Step omission | Expected subtask, actual action sequence |
| Over-operation | Extra picks, extra placements, repeated actions |
| Language misunderstanding | Original instruction, alternative phrasing, failure differences |
This table is far more useful than "it failed 12 times." It directly guides data augmentation and policy repair.
Three Types of Complexity Must Be Tested Separately
Real-world deployment won't give you a clean tabletop and standard sentences. NVIDIA's article breaks complexity into language, scene, and task horizon—a structure worth adopting directly.
Language Complexity
Write at least three instruction variants for each task:
| Type | Example |
|---|---|
| Clear | Take all 3 bananas out of the bin |
| Vague | Clean out the fruit from the bin |
| Overly detailed | First look inside the bin, find all the yellow elongated fruit, then take them out one by one and place them on the left side of the table |
NVIDIA's initial benchmark provides vague, default, and specific instruction variants, and observes that current models remain brittle to vague phrasing—and sometimes too much detail also degrades performance. Your evaluation should also record: is the model failing to understand the goal, or being misled by excessive detail?
Scene Complexity
Run each task across at least three scene types:
- No distractors
- A few similar distractors
- Crowded, occluded, or visually noisy scenes
If a model only succeeds on a clean tabletop, it's not ready for the real world.
Task Horizon
Track short and long tasks separately:
| Task Horizon | What to Look For |
|---|---|
| Single-step | Basic perception, grasping, and placement |
| 2-4 steps | Subtask sequencing, state maintenance, error recovery |
| Long chain | Whether early small errors cascade into total failure |
NVIDIA's article notes that most policies still struggle with long-horizon tasks, and observes that no policy could successfully complete more than four complex subtasks. This finding comes from the article's specific context and shouldn't be extrapolated to all systems—but it's enough to remind teams: long tasks must be evaluated separately.
Run Sensitivity Analysis, Not Gut Feeling
Common pre-deployment debates: "Is it the camera angle?" "The lighting?" "Is the table too cluttered?" Testing each variable in isolation leads to combinatorial explosion.
NVIDIA's approach is to run evaluations across many scene variations simultaneously and use sensitivity analysis to identify which environmental variables most influence success or failure. The article mentions using Neural Posterior Estimation (NPE) to estimate the relationship between variables and outcomes.
Real-world teams can start with a simplified version:
| Variable | Example Values |
|---|---|
| Camera angle | Front, side, high angle |
| Lighting | Bright, shadowed, reflective |
| Number of distractors | 0, 3, 8 |
| Target object position | Center, edge, occluded |
| Instruction style | Clear, vague, detailed |
| Task horizon | 1 step, 3 steps, 5 steps |
Log these variables for every rollout. Even without NPE, you can identify weak spots through grouped statistics.
A Two-Week Evaluation Plan
Round one doesn't need to be comprehensive. The goal is to prove the evaluation system can surface problems.
| Phase | Action | Deliverable |
|---|---|---|
| Days 1-2 | Select 20-40 representative tasks, tag them as visual/procedural/relational | Task table and tag distribution |
| Days 3-4 | Write 3 language variants per task, prepare 2-3 scene complexity levels | Instruction set and scene set |
| Days 5-7 | Run initial rollouts, track success rate, graded scores, failure events, trajectory metrics | Baseline report |
| Days 8-9 | Increase rollout count, add confidence intervals for key tasks | Confidence report |
| Days 10-11 | Run variable grouping or sensitivity analysis on high-failure tasks | Failure hypothesis |
| Days 12-14 | Decide whether to fix data, update policy, adjust task boundaries, or block certain deployment scenarios | Deployment gate checklist |
A deployment gate can be quite specific:
- Key tasks meet internal success thresholds with reported confidence intervals.
- High-risk failure events are below threshold (e.g., collisions, drops, grasping dangerous objects).
- Vague instruction failures trigger clarification rather than blind execution.
- Long-task failures result in safe stops.
- Performance degradation under increased scene complexity stays within acceptable bounds.
- Humans can review failure frames, trajectories, and variable logs.
Common Mistakes
Focusing only on leaderboard scores
Leaderboard scores are useful for model selection, not deployment decisions. Deployment needs your tasks, your robot, your scenes, and your risk thresholds.
Treating simulation success as real-world success
Simulation is a necessary proxy, not the final answer. NVIDIA's article also positions simulation evaluation as a large-scale diagnostic path before real-world deployment, not a replacement for real-world acceptance testing.
Never updating the task set
Once a model maxes out on a fixed task set, the benchmark loses discriminative power. Add new tasks with every model upgrade, hardware change, or deployment scenario shift.
Not logging failure context
Without failure events and frame-level logs, teams can only guess. At minimum, log wrong grasps, drops, collisions, step omissions, and language misunderstandings.
Comparing models with too few rollouts
With small sample sizes, a few percentage points of difference are meaningless. Look at confidence intervals first, then judge whether the difference is large enough.
FAQ
What is RoboLab?
According to NVIDIA's 2026-07-11 technical blog, RoboLab is a simulation-based robot policy evaluation platform developed by NVIDIA Research. It supports robot-agnostic tasks, rapid task generation, graded diagnostics, failure event logging, and complexity analysis. NVIDIA states that RoboLab research will power the open-source NVIDIA Isaac Lab-Arena, with key features planned for productization in August 2026.
Can I use RoboLab as a production evaluation tool today?
Don't assume that. NVIDIA's article provides a website, paper, and GitHub code, and states that key features are planned for Isaac Lab-Arena. Treat it as a research and methodology reference for now. Whether it enters your production pipeline depends on code maturity, your robot stack, and internal validation.
Why use the Clopper-Pearson confidence interval?
Because robot rollout success/failure can be modeled as a binomial outcome. Clopper-Pearson gives the success rate a confidence interval, preventing small-sample偶然 results from being mistaken for true capability. NVIDIA's article uses the 90% success rate example to show that 70 rollouts still carry significant uncertainty.
Should evaluation tasks be written manually or generated by an agent?
Both are valid. NVIDIA's article notes that RoboLab supports coding agents using agent skills to generate new tasks in the workflow. But critical tasks and high-risk scenarios should still be reviewed by engineers to prevent generated tasks from偏离 real deployment conditions.
How does this relate to general AI agent evaluation?
The principles are shared: don't just look at aggregate success rates—examine task coverage, step-by-step performance, failure causes, confidence intervals, and real complexity. The difference is that robotics adds trajectory quality, collisions, drops, velocity, and real physical environment variables.