Diffusion models have achieved great success in image generation. However, when leveraging this idea for video generation, we face significant challenges in maintaining the consistency and continuity across video frames. This is mainly caused by the lack of an effective framework to align frames of videos with desired temporal features while preserving consistent semantic and stochastic features. In this work, we propose a novel Sector-Shaped Diffusion Model (S2DM) whose sector-shaped diffusion region is formed by a set of ray-shaped reverse diffusion processes starting at the same noise point. S2DM can generate a group of intrinsically related data sharing the same semantic and stochastic features while varying on temporal features with appropriate guided conditions. We apply S2DM to video generation tasks, and explore the use of optical flow and posture information as temporal conditions. Experimental results show that S2DM outperforms many existing methods for video generation.
For text-to-video generation tasks where temporal conditions are not explicitly given, we propose a two-stage generation strategy which can decouple the generation of temporal features from semantic-content features. We show that, without additional training, our model integrated with another temporal conditions generative model can still achieve comparable performance with existing works.
Reference
Ours
Disco(CVPR'2024)
Magic Animate(CVPR'2024)
Ours
(Flow Conditioned)
Ours
(Text Conditioned)
LFDM(CVPR'2023)
A lady with glasses, wearing a black T-shirt and black jeans is right arm swipe to the right
A lady with glasses, wearing a floral sweater and black jeans is knock on door
A lady with glasses, wearing a red sweater and brown pants is draw circle clockwise
A lady with glasses, wearing a red sweater and brown pants is knock on door
A lady with glasses, wearing a red sweater and brown pants is stand to sit
A man wearing a gray T-shirt and green pants is draw circle counter clockwise
A man wearing a gray T-shirt and green pants is right arm swipe to the left
A man with glasses, wearing a black coat and blue jeans is squat
A man with glasses, wearing a blue shirt and black jeans is draw triangle
Ours
(Flow Conditioned)
Ours
(Text Conditioned)
LFDM(CVPR'2023)
Person 010 is making surprise expression
Person 020 is making fear expression
Person 020 is making happiness expression
Person 039 is making fear expression
Person 076 is making happiness expression
Person 065 is making fear expression
Ours
Ours
(Training Shared Noise)
(Sampling Non-shared Noise)
Ours
(Training Non-shared Noise)
(Sampling Non-shared Noise)
A man with glasses, wearing a black coat and blue jeans is tennis forehand swing
A man with glasses, wearing a blue shirt and black jeans is walking
Person 020 is making happiness expression
Person 076 is making happiness expression
TBA