learning from observation

Learning from observation tackles the most natural and hardest form of imitation: you see what the expert did but not how they did it. The demonstrations are sequences of states — frames of a video of someone tying a knot, say — with no recorded actions. This is exactly the situation a child is in when copying an adult, and it is what lets agents learn from the vast amount of action-free demonstration available online.

The missing actions make plain behavioral cloning impossible, since there are no labels to predict. Methods fill the gap in two broad ways: infer the likely action between consecutive states with a learned inverse-dynamics model and then clone, or skip actions entirely and match the distribution of state transitions the expert produced, often with an adversarial discriminator over states or state pairs.

An extra wrinkle is embodiment mismatch — a human hand and a robot gripper reach the same state differently — so the cleanest formulations reason about states and goals rather than low-level motions.