From Recognizing Actions to Understanding People

One long-standing idea in cognitive science, known as the Machiavellian intelligence hypothesis, is that intelligence evolved in large part to meet the demands of social life rather than those of the physical environment. Whether or not this is true, the capability it describes remains difficult for current video models. Models have become pretty good at recognizing actions, but they still struggle to infer what people are trying to accomplish, why they chose a particular action, and what they are likely to do next. This is hard because people are not passive parts of the environment. They have goals, act on incomplete information, and change their behavior in response to what they expect others to do. Understanding behavior is therefore inherently strategic. It requires inferring goals that are never stated, tracking what each person does and does not know, and anticipating how people will interpret and respond to one another’s actions.

The challenge is finding a setting where this kind of strategic reasoning can be evaluated rigorously. In everyday video, behavior can be complex, but intentions and causal explanations are rarely observable. Thus, it is often impossible to determine whether a model’s explanation is correct. This is a large part of why our group studies sports. Before moving into research, I played basketball in Lithuania and later in college in the United States. I had long wanted to turn that experience into a research problem. Team sports turn out to be an excellent testbed for studying strategic reasoning. They are full of strategic interaction. Players coordinate with teammates, compete with opponents, hide intentions, exploit expectations, and continuously adjust to one another. These interactions also produce definitive outcomes, including scores, wins, turnovers, and fouls. Combined with explicit rules, statistics, and expert analysis, those outcomes make it possible to evaluate strategic and causal explanations against objective evidence. This is rarely possible in everyday video.

Sports also let us study capabilities beyond interpreting behavior, including forecasting outcomes and deciding what to do next. AI has made major leaps on these problems in closed-world environments such as Go and StarCraft, where the state and available actions are explicitly defined. Extending those capabilities to continuous, real-world multi-agent video remains an open problem.

AI has made major advances in closed-world environments with explicit states and actions.  Extending these capabilities to real-world multi-agent video remains an open problem.

We call this set of capabilities Strategic Video Intelligence (SVI). To measure it, we built SVI-Bench, the first large-scale benchmark that spans the progression from perception through causal reasoning and simulation to agency in real-world multi-agent video. It contains 35,000 hours of basketball, soccer, and hockey broadcasts, aligned with action annotations, expert commentary, game reports, and structured statistics. Its nine tasks move from parsing who is where, doing what to explaining why actions lead to outcomes and anticipating likely consequences. They then progress to simulating counterfactual futures and goal-directed strategies, and finally to agency, where a system must decide what evidence to gather and how to combine it across videos and documents.

Strategic Video Intelligence (SVI) progresses from perceiving what happened to explaining why, simulating what could happen under different conditions, and deciding what to do next.

Performance falls sharply as the tasks move beyond perception. On fine-grained perceptual questions, the strongest video model reaches roughly 74% and nearly matches the performance of human experts. Scores are much lower on tasks that require explaining events, forecasting outcomes, or gathering evidence across multiple sources. On the hardest task, where the system must search through videos and documents to determine what information it needs to answer a complex analytical question, the best model achieves only about 5% accuracy.

Across SVI-Bench, the top score falls from roughly 74% on the fine-grained perceptual question-answering task to 5% on the hardest agentic task.

In one strategic reasoning example, the inbounder deliberately threw the ball off a defender’s back, recovered it, and scored. Frontier models, including Gemini and GPT, instead described a back screen or an alley-oop. This illustrates a recurring failure mode. When faced with an unusual play, models fall back on familiar basketball patterns that are not present in the video.

The inbounder intentionally threw the ball off the defender’s back, recovered it, and scored. Frontier models instead described generic basketball actions that did not occur.

The broader significance of SVI-Bench is that the problem it exposes extends well beyond sports to any setting where multiple people or agents interact and make decisions in a dynamic environment. The same capabilities matter in surgery, emergency response, autonomous driving, and military operations. In each setting, an AI system must understand why events unfold, anticipate how different actions could change the situation, and decide how to respond based on what it observes. SVI-Bench makes these capabilities measurable in real-world video, where outcomes can be verified.