Long Video Understanding as a Language Modeling Problem

For the past several years, our group at UNC has focused on long video understanding. Quite surprisingly, we found that the most effective solution to this problem is often very simple. The key idea is to decouple short-term perception from long-term reasoning. We first use a visual captioner to generate textual descriptions of short clips densely sampled from a long video, along with transcribed speech. Then, we feed these temporally ordered captions and transcribed speech into a Large Language Model (LLM) for long-term reasoning.

Empirically, these simple language-driven systems (LLoVi, VideoTree, SiLVR) frequently outperform sophisticated video-native solutions across numerous benchmarks (see SiLVR for full comparisons) while offering significant advantages.

An illustration of our Simple Language-based Video Reasoning framework (SiLVR). Top: the video is segmented into short clips and paired with speech. A clip captioner processes each segment to generate visual descriptions. The speech is transcribed using ASR. Bottom: an LLM takes the question, transcribed speech, and dense visual descriptions and produces the final answer.

First, by decoupling vision and reasoning, we can leverage the most powerful LLMs (even 1T+ parameters). In contrast, video-native approaches have to fit both the visual features and the language model into one GPU memory budget, which in practice caps them at a few hundred frames and a much smaller LLM (e.g., ~7B parameters). This makes long-form video analysis very difficult. Furthermore, decoupling perception and long-term reasoning makes the framework easy to upgrade since every new captioner or LLM (released frequently these days) improves the system with minimal effort, and without any additional training.

Our language-driven video understanding models (LLoVi, SiLVR) use orders of magnitude more parameters for language processing than for vision.

The effectiveness of these simple language-driven frameworks raises some interesting questions: Is vision meant to be a peripheral module that feeds a language brain? Should we keep using orders of magnitude more parameters on language than on vision, as we do now? And how do we reconcile the design of these systems with what we know about our own brains, which devote most of their capacity to vision and only a small part to language?

Some would argue that such simple language-driven approaches only work when the problem has a strong language interface. Beyond long videos, there are many video problems where the inputs and outputs are difficult to express in language. For example, fine-grained temporal understanding (TimeBlind), skill analysis that requires capturing subtle motion cues (ExAct, BASKET), or even problems like generating background music for a silent video, where the result has to match the content and the rhythm of the video (VMAs, V2M-Zero).

While it's true that the current language-driven approaches cannot solve these problems (yet), I'd point out that LLMs have turned out to be surprisingly good at reasoning over representations that aren't natural language, like bounding boxes, pose coordinates, or the raw grids of ARC-AGI. Their massive pretraining seems to give them the ability to generalize quite well over data structures beyond plain text. Therefore, you can imagine solutions to some of these problems, like fine-grained understanding or skill analysis, where an LLM reasons over intermediate visual representations such as object tracks and pose sequences. Of course, finding the right intermediate representations that can be effectively processed by LLMs is still hard for many visual problems (e.g., video-to-music generation or dexterous robot manipulation). Additionally, we still need powerful visual models that can extract such intermediate representations directly from raw pixels. However, in my view, the likely outcome is that the dominant approaches for long video understanding (and possibly many other video understanding problems) will be built on top of such intermediate representations using frontier LLMs. If so, then vision may indeed become a peripheral module feeding a language brain, the reverse of how our own intelligence is organized, but perhaps that was never a requirement.