From Recognizing Actions to Understanding People

One long-standing idea in cognitive science, known as the Machiavellian intelligence hypothesis, is that intelligence evolved in large part to meet the demands of social life rather than those of the physical environment. Whether or not this is true, the capability it describes remains difficult for current video models. Models have become pretty good at recognizing actions, but they still struggle to infer what people are trying to accomplish, why they chose a particular action, and what they are likely to do next. This is hard because people are not passive parts of the environment. They have goals, act on incomplete information, and change their behavior in response to what they expect others to do. Understanding behavior is therefore inherently strategic. It requires inferring goals that are never stated, tracking what each person does and does not know, and anticipating how people will interpret and respond to one another’s actions.

The challenge is finding a setting where this kind of strategic reasoning can be evaluated rigorously. In everyday video, behavior can be complex, but intentions and causal explanations are rarely observable. Thus, it is often impossible to determine whether a model’s explanation is correct. This is a large part of why our group studies sports. Before moving into research, I played basketball in Lithuania and later in college in the United States. I had long wanted to turn that experience into a research problem. Team sports turn out to be an excellent testbed for studying strategic reasoning. They are full of strategic interaction. Players coordinate with teammates, compete with opponents, hide intentions, exploit expectations, and continuously adjust to one another. These interactions also produce definitive outcomes, including scores, wins, turnovers, and fouls. Combined with explicit rules, statistics, and expert analysis, those outcomes make it possible to evaluate strategic and causal explanations against objective evidence. This is rarely possible in everyday video.

Sports also let us study capabilities beyond interpreting behavior, including forecasting outcomes and deciding what to do next. AI has made major leaps on these problems in closed-world environments such as Go and StarCraft, where the state and available actions are explicitly defined. Extending those capabilities to continuous, real-world multi-agent video remains an open problem.

AI has made major advances in closed-world environments with explicit states and actions.  Extending these capabilities to real-world multi-agent video remains an open problem.

We call this set of capabilities Strategic Video Intelligence (SVI). To measure it, we built SVI-Bench, the first large-scale benchmark that spans the progression from perception through causal reasoning and simulation to agency in real-world multi-agent video. It contains 35,000 hours of basketball, soccer, and hockey broadcasts, aligned with action annotations, expert commentary, game reports, and structured statistics. Its nine tasks move from parsing who is where, doing what to explaining why actions lead to outcomes and anticipating likely consequences. They then progress to simulating counterfactual futures and goal-directed strategies, and finally to agency, where a system must decide what evidence to gather and how to combine it across videos and documents.

Strategic Video Intelligence (SVI) progresses from perceiving what happened to explaining why, simulating what could happen under different conditions, and deciding what to do next.

Performance falls sharply as the tasks move beyond perception. On fine-grained perceptual questions, the strongest video model reaches roughly 74% and nearly matches the performance of human experts. Scores are much lower on tasks that require explaining events, forecasting outcomes, or gathering evidence across multiple sources. On the hardest task, where the system must search through videos and documents to determine what information it needs to answer a complex analytical question, the best model achieves only about 5% accuracy.

Across SVI-Bench, the top score falls from roughly 74% on the fine-grained perceptual question-answering task to 5% on the hardest agentic task.

In one strategic reasoning example, the inbounder deliberately threw the ball off a defender’s back, recovered it, and scored. Frontier models, including Gemini and GPT, instead described a back screen or an alley-oop. This illustrates a recurring failure mode. When faced with an unusual play, models fall back on familiar basketball patterns that are not present in the video.

The inbounder intentionally threw the ball off the defender’s back, recovered it, and scored. Frontier models instead described generic basketball actions that did not occur.

The broader significance of SVI-Bench is that the problem it exposes extends well beyond sports to any setting where multiple people or agents interact and make decisions in a dynamic environment. The same capabilities matter in surgery, emergency response, autonomous driving, and military operations. In each setting, an AI system must understand why events unfold, anticipate how different actions could change the situation, and decide how to respond based on what it observes. SVI-Bench makes these capabilities measurable in real-world video, where outcomes can be verified.

Long Video Understanding as a Language Modeling Problem

For the past several years, our group at UNC has focused on long video understanding. Quite surprisingly, we found that the most effective solution to this problem is often very simple. The key idea is to decouple short-term perception from long-term reasoning. We first use a visual captioner to generate textual descriptions of short clips densely sampled from a long video, along with transcribed speech. Then, we feed these temporally ordered captions and transcribed speech into a Large Language Model (LLM) for long-term reasoning.

Empirically, these simple language-driven systems (LLoVi, VideoTree, SiLVR) frequently outperform sophisticated video-native solutions across numerous benchmarks (see SiLVR for full comparisons) while offering significant advantages.

An illustration of our Simple Language-based Video Reasoning framework (SiLVR). Top: the video is segmented into short clips and paired with speech. A clip captioner processes each segment to generate visual descriptions. The speech is transcribed using ASR. Bottom: an LLM takes the question, transcribed speech, and dense visual descriptions and produces the final answer.

First, by decoupling vision and reasoning, we can leverage the most powerful LLMs (even 1T+ parameters). In contrast, video-native approaches have to fit both the visual features and the language model into one GPU memory budget, which in practice caps them at a few hundred frames and a much smaller LLM (e.g., ~7B parameters). This makes long-form video analysis very difficult. Furthermore, decoupling perception and long-term reasoning makes the framework easy to upgrade since every new captioner or LLM (released frequently these days) improves the system with minimal effort, and without any additional training.

Our language-driven video understanding models (LLoVi, SiLVR) use orders of magnitude more parameters for language processing than for vision.

The effectiveness of these simple language-driven frameworks raises some interesting questions: Is vision meant to be a peripheral module that feeds a language brain? Should we keep using orders of magnitude more parameters on language than on vision, as we do now? And how do we reconcile the design of these systems with what we know about our own brains, which devote most of their capacity to vision and only a small part to language?

Some would argue that such simple language-driven approaches only work when the problem has a strong language interface. Beyond long videos, there are many video problems where the inputs and outputs are difficult to express in language. For example, fine-grained temporal understanding (TimeBlind), skill analysis that requires capturing subtle motion cues (ExAct, BASKET), or even problems like generating background music for a silent video, where the result has to match the content and the rhythm of the video (VMAs, V2M-Zero).

While it's true that the current language-driven approaches cannot solve these problems (yet), I'd point out that LLMs have turned out to be surprisingly good at reasoning over representations that aren't natural language, like bounding boxes, pose coordinates, or the raw grids of ARC-AGI. Their massive pretraining seems to give them the ability to generalize quite well over data structures beyond plain text. Therefore, you can imagine solutions to some of these problems, like fine-grained understanding or skill analysis, where an LLM reasons over intermediate visual representations such as object tracks and pose sequences. Of course, finding the right intermediate representations that can be effectively processed by LLMs is still hard for many visual problems (e.g., video-to-music generation or dexterous robot manipulation). Additionally, we still need powerful visual models that can extract such intermediate representations directly from raw pixels. However, in my view, the likely outcome is that the dominant approaches for long video understanding (and possibly many other video understanding problems) will be built on top of such intermediate representations using frontier LLMs. If so, then vision may indeed become a peripheral module feeding a language brain, the reverse of how our own intelligence is organized, but perhaps that was never a requirement.

TimeSformer, five years later.

ICML 2026 is coming up, which means it's been five years since I presented TimeSformer at ICML 2021. It turned out to be the most cited paper I've published, and it's still the only paper I've had at ICML. With the 5-year anniversary approaching I wanted to write down how TimeSformer actually came together.

The idea itself wasn't groundbreaking. ViT had just shown that a pure transformer could work well on images, and it was obvious that video was next. At its core, TimeSformer was a simple idea. It treated a video as a sequence of small image patches, much like a language transformer treats a sentence as a sequence of words, and used self-attention to relate those patches to one another. However, the obvious way to do that, i.e., comparing every patch with every other patch across space and time, was too expensive to run on video clips longer than a few frames. To address this issue, we proposed to split attention into two steps. First, each patch attends only to the patches at the same location in the other frames, which handles time. Then it attends only to the patches within its own frame, which handles space. That cut the cost from roughly T×N comparisons per patch to T+N, and made it possible to train on longer, high-resolution clips. In comparison, the joint attention variant couldn't even fit those same clips in the GPU memory. Surprisingly, the divided attention variant wasn't just cheaper, it was also more accurate.

Two ways of applying self-attention to video. In joint space-time attention (left), a query patch (blue) attends to every patch in every frame at once (red). In divided space-time attention (right), the same query attends only along two axes: over time, to the patch at its own location in the neighboring frames (green), and over space, to the patches within its own frame (red). The three rows are the previous, current, and next frames (t−δ, t, t+δ). Splitting attention this way cuts the cost per patch from roughly T×N comparisons to T+N compared to the joint attention variant.

At the time, not everyone thought this was the right direction. A number of people at Facebook AI Research (FAIR), including in my own group, believed a mix of convolution and attention was better suited to video than a pure transformer. Meanwhile several other groups, a few at FAIR and at least one at Google that I knew of, were working on nearly the same idea. Most of them were vision researchers aiming for ICCV, so we deliberately targeted ICML to get our work out first. One funny detail was related to the naming of the method. We went back and forth on what to call the paper, and the two candidates were ViViT and TimeSformer. In the end, we went with TimeSformer. About seven weeks later, a paper came out of Google on the same topic, named ViViT.

Divided space-time attention scales better than joint attention. Top: FLOPs grow far more slowly as the spatial crop (left) and the number of input frames (right) increase, and joint attention runs out of memory in the shaded regions. Bottom: our divided space-time attention is also more accurate than joint attention on both Kinetics-400 (K400) and Something-Something V2 (SSv2).

ICML reviewers criticized the paper for a lack of novelty and gave it fairly low scores. Looking back, the field was moving so fast that within a few months several papers came out with better results. If our paper hadn't been accepted at ICML, it probably wouldn't have been accepted anywhere. Fortunately, it got in. One thing I did differently with this paper was to release the code before publication, which I hadn't done before. I think that ended up being one of the biggest reasons it got adopted so quickly. People could build on it right away instead of waiting until it was published.

For all it could do, TimeSformer had significant limitations. It needed large-scale pretraining to work well, and it struggled with fine-grained temporal understanding. Five years later, I'd say the latter problem is still largely unsolved, as shown by our recent TimeBlind benchmark. Fittingly, TimeBlind was rejected from ICML 2026, so it won't be my second ICML paper after all:)