Google Genie 2 Introduction

Genie 2 operates as an autoregressive latent diffusion model trained on a large video dataset. The process begins with an image prompt (which can be generated by Imagen 3 or be a real photo) that defines the desired environment. The system first passes the input through an autoencoder, then processes the latent frames using a large transformer model with a causal mask similar to language models. During inference, Genie 2 generates the environment frame-by-frame in an autoregressive manner, taking into account past frames and user actions while using classifier-free guidance to improve action controllability. The model demonstrates remarkable capabilities including long-term memory (remembering off-screen elements), physics simulation, lighting effects, and complex character animations.