Schedule

COMP 790: Visual Recognition with Transformers (Spring 2022)

About

Date

Presenters

Jan 10

Gedas

Topic

Course Overview;

Project Details

Paper Critique Details

Papers

[1] Attention Is All You Need

Length

75min

Slides

pdf

Overview of Self-Attention and Transformers

Jan 12

Gedas

Vision Transformers

[2] An image is worth 16x16 words: Transformers for image recognition at scale

45min

pdf

Detailed Paper Discussion #1

30min

pdf

Jan 17

No Class (MLK Day)

Jan 19

Gedas

Data-Efficient Vision Transformers

[3] Training data-efficient image transformers & distillation through attention

[4] Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

25min+QA

pdf

Jan 24

Yi-Lin

Somnath

Mixing Convolutions with Self-Attention

[5] Early Convolutions Help Transformers See Better

[6] ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

25min+QA

pdf

25min+QA

pdf

Jan 26

Sherry,

Shiv &

Yining

Understanding Vision Transformers

[7] Do Vision Transformers See Like Convolutional Neural Networks?

45min

Detailed Paper Discussion #2

30min

pdf

Jan 31

Mohaiminul

Qin

Efficient Self-Attention Schemes

[8] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

[9] XCiT: Cross-Covariance Image Transformers

25min+QA

pdf

25min+QA

pdf

Feb 2

David

Yunkui

Object Detection

[10] DE⫶TR: End-to-End Object Detection with Transformers

[11] Pix2Seq: A Language Modeling Framework for Object Ddetection

25min+QA

pdf

25min+QA

pdf

Feb 7

Yining

Sherry

Dense Pixel-Level Prediction

[12] Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

25min+QA

pdf

[13] Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

25min+QA

pdf

Feb 9

Gedas

Sam

Object Tracking

[14] TrackFormer: Multi-Object Tracking with Transformers

[15] End-to-End Video Instance Segmentation with Transformers

25min+QA

pdf

25min+QA

pdf

Feb 14

Project Proposal

(Part 1)

Report due Feb 20, 11:59 PM

Feb 16

Project Proposal

(Part 2)

Report due Feb 20, 11:59 PM

Feb 21

Mike

Gedas

3D Recognition

[16] Point Transformer

[17] An End-to-End Transformer Model for 3D Object Detection

25min+QA

pdf

Feb 23

Gedas

Video Classification

[18] Is Space-Time Attention All You Need for Video Understanding?

[19] Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

25min+QA

pdf

25min+QA

pdf

Feb 28

Text-Guided Image Generation

[20] Zero-Shot Text-to-Image Generation

Detailed Paper Discussion #3

45min

30min

pdf

Mar 2

Learning Visual Models from Text

[21] Learning Transferable Visual Models From Natural Language

Detailed Paper Discussion #4

45min

30min

pdf

David &

Somnath

Sam &

Yi-Lin

pdf

Mar 9

Mohaiminul,

Yan-Bo &

Akshay

Multimodal Few-Shot Learning

[23] Multimodal Few-Shot Learning with Frozen Language Models

Detailed Paper Discussion #6

45min

30min

pdf

Mar 7

General Multimodal Perception

[22] Hierarchical Perceiver

Detailed Paper Discussion #5

45min

30min

pdf

Angelos &

Ryan

pdf

Mar 14

No Class (Spring Break)

Mar 16

No Class (Spring Break)

Mar 21

Project Milestone

(Part 1)

Report due Mar 27, 11:59 PM

Mar 23

Project Milestone

(Part 2)

Report due Mar 27, 11:59 PM

Mar 28

Akshay

Angelos

Learning from Video, Audio and Text

[24] VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

[25] MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

25min+QA

pdf

25min+QA

pdf

Mar 30

Avinash

Shiv

Other Multimodal Applications

[26] Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

25min+QA

pdf

[27] Attention Bottlenecks for Multimodal Fusion

25min+QA

pdf

Apr 4

Mike & Eli

Transformer Generalization

[28] Pretrained transformers as universal computation engines

Detailed Paper Discussion #7

45min

30min

pdf

Apr 6

Eli

Maitrey

Self-Supervised Learning

[29] Generative Pretraining from Pixels

[30] Emerging properties in self-supervised vision transformers

25min+QA

pdf

25min+QA

pdf

Apr 11

Yilin,

Nurislam &

Avinash

Self-Supervised Learning

[31] Masked Autoencoders Are Scalable Vision Learners

Detailed Paper Discussion #8

45min

30min

pdf

Yilin

Ryan

Beyond Self-Attention

[32] MLP-Mixer: An all-MLP Architecture for Vision

25min+QA

pdf

25min+QA

pdf

Apr 13

Apr 18

Nurislam

Yan-Bo

Beyond Self-Attention

[34] Patches Are All You Need?

[35] MetaFormer is Actually What You Need for Vision

25min+QA

pdf

[33] Pay Attention to MLPs

Back to Convolutions

Detailed Paper Discussion #9

45min

30min

pdf

[36] A ConvNet for the 2020s

Apr 20

Yunkui,

Qin &

Maitrey

pdf

Apr 27

Apr 25

Final Presentation

(Part 1)

Final Presentation

(Part 2)

Report due May 1, 11:59 PM