TimeSformer, five years later.

ICML 2026 is coming up, which means it's been five years since I presented TimeSformer at ICML 2021. It turned out to be the most cited paper I've published, and it's still the only paper I've had at ICML. With the 5-year anniversary approaching I wanted to write down how TimeSformer actually came together.

The idea itself wasn't groundbreaking. ViT had just shown that a pure transformer could work well on images, and it was obvious that video was next. At its core, TimeSformer was a simple idea. It treated a video as a sequence of small image patches, much like a language transformer treats a sentence as a sequence of words, and used self-attention to relate those patches to one another. However, the obvious way to do that, i.e., comparing every patch with every other patch across space and time, was too expensive to run on video clips longer than a few frames. To address this issue, we proposed to split attention into two steps. First, each patch attends only to the patches at the same location in the other frames, which handles time. Then it attends only to the patches within its own frame, which handles space. That cut the cost from roughly T×N comparisons per patch to T+N, and made it possible to train on longer, high-resolution clips. In comparison, the joint attention variant couldn't even fit those same clips in the GPU memory. Surprisingly, the divided attention variant wasn't just cheaper, it was also more accurate.

At the time, not everyone thought this was the right direction. A number of people at Facebook AI Research (FAIR), including in my own group, believed a mix of convolution and attention was better suited to video than a pure transformer. Meanwhile several other groups, a few at FAIR and at least one at Google that I knew of, were working on nearly the same idea. Most of them were vision researchers aiming for ICCV, so we deliberately targeted ICML to get our work out first. One funny detail was related to the naming of the method. We went back and forth on what to call the paper, and the two candidates were ViViT and TimeSformer. In the end, we went with TimeSformer. About seven weeks later, a paper came out of Google on the same topic, named ViViT.

ICML reviewers criticized the paper for a lack of novelty and gave it fairly low scores. Looking back, the field was moving so fast that within a few months several papers came out with better results. If our paper hadn't been accepted at ICML, it probably wouldn't have been accepted anywhere. Fortunately, it got in. One thing I did differently with this paper was to release the code before publication, which I hadn't done before. I think that ended up being one of the biggest reasons it got adopted so quickly. People could build on it right away instead of waiting until it was published.

For all it could do, TimeSformer had significant limitations. It needed large-scale pretraining to work well, and it struggled with fine-grained temporal understanding. Five years later, I'd say the latter problem is still largely unsolved, as shown by our recent TimeBlind benchmark. Fittingly, TimeBlind was rejected from ICML 2026, so it won't be my second ICML paper after all:)