AnimateDiff is a method that allows you to create videos using pre-existing stable diffusion text-to-image models.

It achieves this by inserting motion module layers into the frozen text-to-image model and training on video clips to extract motion priors. These motion modules are applied after the ResNet and attention modules in the stable diffusion UNet. The goal is to introduce coherent motion between image frames. To support these modules, we introduce the concepts of MotionAdapter and UNetMotionModel. They conveniently integrate these motion modules with existing stable diffusion models.

Note: The SDXL checkpoint of AnimateDiff is a test version.