Lecture 19 · Act V · Applications

Robotics and embodied AI.

A sentence enters; a trajectory leaves. Between them: symbols, geometry, feedback, and a monitor that can say no.

9 sections 5 layers 3 policy families 1 interactive lab
00 · Learning objectivesLecture 19 · Robotics and embodied AI

By the end of this lecture

Four things you can do.

01

Separate the planning layers

Task planning, motion planning, control — and the feasibility query that links them.

02

Explain what a VLA guarantees

SayCan, RT-2, OpenVLA: a policy from images and language to actions, with no formal guarantee.

03

Diagnose grounding failures

A plan can be valid for the wrong symbol; find the first divergence before the robot moves.

04

Place a shield in the loop

State its guarantee relative to the specification and model, and why refusal is an outcome.

S/V · Lecture 19
00 · The hookLecture 19 · Robotics and embodied AI

A tabletop, one sentence

“Put the red cube in the bowl.”

red cube red sphere bowl knife
One sentence. Six models. Six places to be wrong.
Which object? Grounding: words to referents.
What first? Task planning: preconditions and effects.
Can it reach? Motion planning: geometry and collisions.
How? Control: feedback at 1–10 ms.
Did it work? Monitoring: prediction versus observation.
Was it allowed? Shielding: a specification over the trace.
S/V · Lecture 19
01 · The robot stackLecture 19 · Robotics and embodied AI

The reference architecture

Every layer is a model.

instruction + scene “put the red cube…” camera, robot state raw grounding objects, attributes relations, affordances 30–300 ms task planner preconditions, effects search over facts 0.1–10 s motion planner configuration space collisions, reachability 0.1–10 s controller feedback, dynamics torques, impedance 1–10 ms specification φ safety over traces monitor / shield allow · refuse · stop every step
The arrows are representation changes. The monitor sits outside and can refuse.

The seam is where a type error becomes a robot error. Pixels → symbols → actions → geometry → torques.

S/V · Lecture 19
02 · Task, motion, controlLecture 19 · Robotics and embodied AI
02 / 08

Three layers, three promises.

A discrete proof, a continuous path, a feedback law. None of them is a promise about the world.

S/V · Lecture 19
02 · Task, motion, controlLecture 19 · Robotics and embodied AI

Discrete · STRIPS / PDDL

A plan is a proof about a model.

si+1 = (si ∖ del(ai)) ∪ add(ai)
(2.1)

Apply a only if s ⊨ pre(a). A plan is a sequence whose last state satisfies the goal. Soundness and completeness are relative to the domain.

The planner cannot check the geometry. It does not know if the arm can reach.

The lab's plan

1. pick(blue_cube) 2. place(blue_cube, table) 3. pick(red_cube) 4. place(red_cube, bowl) pre: on(red_cube, table), hand_empty

The first action is not the object in the sentence. The bowl is occupied.

S/V · Lecture 19
02 · Task, motion, controlLecture 19 · Robotics and embodied AI

Continuous · two rates

A path is not a plan.

Motion planning

Search Cfree for a collision-free path from qstart to qgoal. PRM and RRT are probabilistically complete; RRT* is asymptotically optimal.

Promises: a path exists and stays in free space. Does not promise a good grasp or a trackable trajectory.

Control

At 1–10 ms, map the measured state and a reference trajectory to torques, velocities, or impedance. PID, computed torque, MPC.

Promises: local stability and tracking bounds under assumptions. Does not promise task success or safety.

The guarantee changes with the layer. Completeness for the search, stability for the loop. Neither is a statement about the task.
S/V · Lecture 19
02 · Task, motion, controlLecture 19 · Robotics and embodied AI

The seam

An action is applicable only if a motion exists.

∃ τ : τ ∈ Cfree,   τ(0) = qi−1,   τ(1) ∈ Grasp(o),   si−1 ⊨ pre(ai)
(2.3)

Plan first, then check

Fast. Incomplete: an infeasible plan forces backtracking over plans.

Motion first, then abstract

Feasible motions. Loses symbolic completeness: the motion may not realize the action.

Interleave

TAMP: ask for a refinement of each action; geometric failures prune the symbolic search. Complete and expensive.

Garrett et al., Integrated Task and Motion Planning, 2021.

S/V · Lecture 19
03 · Language policiesLecture 19 · Robotics and embodied AI
03 / 08

Language proposes; the body disposes.

SayCan, RT-2, OpenVLA — three answers to “can a language model produce an action?”

S/V · Lecture 19
03 · Language policiesLecture 19 · Robotics and embodied AI

SayCan · 2022

Plausible × feasible.

at = argmaxa ∈ A   pLM(a | instruction, history) · Va(st)
(3.1)

LLM

Which skill makes sense next for the instruction?

Value function

How likely is this skill to succeed from this state?

Product

Grounds the proposal in the body and the scene. Greedy; no completeness.

The robot is the language model's hands and eyes. A plausible sentence that the robot cannot execute gets a low value.

S/V · Lecture 19
03 · Language policiesLecture 19 · Robotics and embodied AI

RT-2 · 2023

Actions as tokens.

# one sequence, one objective <img> put the red cube in the bowl → [ACTION 128] [ACTION 47] [ACTION 9] …

Co-fine-tune a vision-language model on robot trajectories and web vision-language data.

Represent each action as a text token, so language answers and actions share one decoder.

What changes: semantic knowledge from web data transfers to control; novel objects and commands improve.

What does not: no explicit state, no plan, no safety guarantee. Still a policy.

Brohan et al., arXiv:2307.15818.

S/V · Lecture 19
03 · Language policiesLecture 19 · Robotics and embodied AI

OpenVLA · 2024

Open weights change the economics.

7B

parameters. Llama 2 + fused DINOv2 / SigLIP features.

970k

real-world demonstrations; discretized actions; fine-tunable.

16.5%

absolute task-success gain over RT-2-X (55B) across 29 tasks, per the paper's abstract.

Still a policy. It does not know its own preconditions, cannot explain a plan, and cannot certify an action as safe.

S/V · Lecture 19
03 · Language policiesLecture 19 · Robotics and embodied AI

What a VLA does not guarantee

A policy, not a planner.

No task-success guarantee. Generalization is empirical; new objects, layouts, and compositions can break it.
No explicit state. History is compressed into context or weights; the facts it believes cannot be inspected.
No planner. It emits an action or a short chunk; long horizons need decomposition.
No safety guarantee. Safety is a property of the trace, not of the action head.
No calibration by default. Confidently wrong is a normal failure mode.
It inherits its data. Background, position, viewpoint, and the experimenter's hand become part of the policy.

Put it in a stack: a task planner for horizons, a shield for safety, a monitor for recovery.

S/V · Lecture 19
04 · Grounding and affordancesLecture 19 · Robotics and embodied AI

Interactive · task plan / grounding

Same instruction, two referents.

red cube red sphere bowl
The dashed ring is what the system grounded. It is not always the referent.
grounded red_cube
plan pick(red_cube) → place(red_cube, bowl)
monitor goal holds

Correct grounding: the plan puts the cube in the bowl and the goal holds. Switch to the detector swap and run the same instruction.

S/V · Lecture 19
04 · Grounding and affordancesLecture 19 · Robotics and embodied AI

Reference, not lookup

Words → candidates → a binding.

“the red cube” colour = red shape = cube constraints detected candidates red_cube · red, cube · 0.94 red_sphere · red, sphere · 0.69 blue_cube · blue, cube · — scores are colour × shape confidence top-2 margin matters bind red_cube the planner's symbol the monitor's referent if margin small ask, do not guess ambiguity is an outcome

Grounding is an argmax with a refusal option. If the top two candidates are close, the honest output is a question.

S/V · Lecture 19
04 · Grounding and affordancesLecture 19 · Robotics and embodied AI

Two different questions

Naming is not acting.

Grounding

Which object does the phrase refer to? What is its colour, shape, position, relation?

Failure: the wrong referent. Test: top-k candidates and margin.

Affordance

What can this body do with it? Grasp width, reach, friction, stability, sharpness.

Failure: a grasp that cannot succeed. Test: grasp prediction and reachability.

A VLM can know the word “bowl” and not know the bowl is too deep for the gripper. Reference and action are separate predictions, and they fail in different places.
S/V · Lecture 19
04 · Grounding and affordancesLecture 19 · Robotics and embodied AI

The failure that hides

Valid in the wrong symbols.

Planner output

pick(blue_cube) place(blue_cube, table) pick(red_sphere) # was meant to be red_cube place(red_sphere, bowl) pre: on(red_sphere, table), hand_empty eff: on(red_sphere, bowl), hand_empty
Every precondition holds. Every effect is applied. The plan reaches its goal.
The task goal is over the cube. The bowl holds the sphere. The plan is a success for the wrong symbol.
The planner cannot see it. The error happened upstream, in grounding.

Catch it with a goal over the intended referent, or ask when the grounding margin is small.

S/V · Lecture 19
05 · Shielding and safetyLecture 19 · Robotics and embodied AI
05 / 08

A filter that can say no.

A specification over traces, a shield over actions — and a guarantee that is only as good as its model.

S/V · Lecture 19
05 · Shielding and safetyLecture 19 · Robotics and embodied AI

Preemptive or post-hoc

The policy proposes; the shield disposes.

policy / planner proposes a shield a ∈ Safe(s)? spec φ over traces synthesized from a game allowed action or refusal or backup robot executes every step
πshield(s) = argmaxa ∈ Safe(s) Q(s, a)
(5.1)

Alshiekh et al., arXiv:1708.08611. The guarantee is relative to φ, the abstraction, and the transition model.

S/V · Lecture 19
05 · Shielding and safetyLecture 19 · Robotics and embodied AI

Four places, four blind spots

Where the verifier enters.

LevelMechanismGuaranteesBlind spot
TaskPlan validation; action constraints; criticNo forbidden action in the planWrong symbols or goal
MotionCollision, joint, speed, force limitsTrajectory stays in limitsTask safety; dynamic obstacles
RuntimeMonitor the trace; stop or switchViolations detectedSpec gaps; detection delay
PolicyAction filter around a learned policyNo unsafe command sentWrong abstraction; conservatism

They are complementary. A task-level check cannot see a collision; a motion check cannot see a wrong goal.

S/V · Lecture 19
05 · Shielding and safetyLecture 19 · Robotics and embodied AI

Two axes, four outcomes

Safety is not success.

Safe success

Goal holds, specification holds. The only outcome you want to ship.

Unsafe success

Goal holds, specification violated. The knife in the bowl. The metric says yes; the trace says no.

Safe failure

Goal fails, specification holds. A wrong grounding with the shield on: safe, and still wrong.

Unsafe failure

Both fail. The case that should never reach a robot.

A shield cannot fix a wrong goal. It can only refuse to make the wrong goal happen unsafely.

S/V · Lecture 19
06 · EvaluationLecture 19 · Robotics and embodied AI

Diagnosis, not a leaderboard

Success is not enough.

Goal interpretation. Did the agent read the instruction correctly?
Subgoal decomposition. Did it break the goal into the right steps?
Action sequencing. Did it order the actions correctly?
Transition modelling. Did it predict what each action changes?

Embodied Agent Interface evaluates these four modules with fine-grained errors: hallucinated preconditions, affordance errors, planning errors.

Li et al., arXiv:2410.07166. ALFWorld tests text-to-embodied transfer; LIBERO tests lifelong transfer across 130 tasks.

A score says whether. A module metric says why.

Report safety separately. Report the initial-state distribution, the paraphrases, and the number of trials.

S/V · Lecture 19
07 · Failure modesLecture 19 · Robotics and embodied AI

A diagnostic table

Attribute the first divergence.

SymptomLayerLikely cause
Picks the wrong objectGroundingAmbiguity, attribute swap, wrong frame
Plan valid, impossibleTask–motion seamMissing geometric precondition
Arm collidesMotion / controlStale map, tracking error
Object slipsControl / affordanceForce, friction, bad grasp pose
Unsafe action executesSpec / monitorNo shield, wrong spec, blind spot
Goal holds, safety failsSpecificationSuccess metric ignores safety

Fix the first divergent layer, not the last one that produced a symptom.

S/V · Lecture 19
07 · Failure modesLecture 19 · Robotics and embodied AI

Instrument the seams

Log the parse, the plan, and the monitor.

PARSE put the red cube in the bowl GROUND red_cube 0.94 | red_sphere 0.69 → bind red_cube PLAN pick(blue_cube) … 4 actions; BFS expanded 12 states CHECK reach(red_cube) ok | bowl empty ok EXEC place(red_sphere, bowl) MON on(red_cube, bowl) = false | bowl holds red_sphere FIRST DIVERGENCE: grounding

The trace is not a log of failures; it is a log of assumptions. Each line is a model's claim, and the first claim that is false is the bug.

S/V · Lecture 19
07 · Failure modesLecture 19 · Robotics and embodied AI

The sharp version

The plan was valid. The symbols were wrong.

A planner is sound relative to its model. A shield is sound relative to its specification. Neither is sound relative to the world.

Grounding is where the world enters the symbols. Monitoring is where it comes back.

S/V · Lecture 19
08 · RecapLecture 19 · Robotics and embodied AI

What you can do now

Four moves, one tabletop.

Separate

Task, motion, control; the feasibility query links them.

Explain

SayCan, RT-2, OpenVLA; a VLA is a policy, not a planner.

Diagnose

A plan valid for the wrong symbol; the first divergence is grounding.

Shield

Safety and goal correctness are orthogonal; refusal is an outcome.

S/V · Lecture 19
Speaker notes