Diffusion models have achieved great success in image generation. However, when leveraging this idea for video generation, we face significant challenges in maintaining the consistency and continuity across video frames. This is mainly caused by the lack of an effective framework to align frames of videos with desired temporal features while preserving consistent semantic and stochastic features.
In this work, we propose a novel Sector-Shaped Diffusion Model (S2DM) whose sector-shaped diffusion region is formed by a set of ray-shaped reverse diffusion processes starting at the same noise point. S2DM can generate a group of intrinsically related data sharing the same semantic and stochastic features while varying on temporal features with appropriate guided conditions. We apply S2DM to video generation tasks, and explore the use of optical flow as temporal conditions. Our experimental results show that S2DM outperforms many existing methods in the task of video generation without any temporal-feature modelling modules. For text-to-video generation tasks where temporal conditions are not explicitly given, we propose a two-stage generation strategy which can decouple the generation of temporal features from semantic-content features.
We show that, without additional training, our model integrated with another temporal conditions generative model can still achieve comparable performance with existing works.
Ours
(Flow Conditioned)
Ours
(Text Conditioned)
LFDM
A lady with glasses, wearing a black T-shirt and black jeans is right arm swipe to the right
A lady with glasses, wearing a floral sweater and black jeans is knock on door
A lady with glasses, wearing a red sweater and brown pants is draw circle clockwise
A lady with glasses, wearing a red sweater and brown pants is knock on door
A lady with glasses, wearing a red sweater and brown pants is stand to sit
A man wearing a gray T-shirt and green pants is draw circle counter clockwise
A man wearing a gray T-shirt and green pants is right arm swipe to the left
A man with glasses, wearing a black coat and blue jeans is squat
A man with glasses, wearing a blue shirt and black jeans is draw triangle
Ours
(Flow Conditioned)
Ours
(Text Conditioned)
LFDM
Person 010 is making surprise expression
Person 020 is making fear expression
Person 020 is making happiness expression
Person 039 is making fear expression
Person 076 is making happiness expression
Person 065 is making fear expression
Ours
Ours
(Training Shared Noise)
(Sampling Non-shared Noise)
Ours
(Training Non-shared Noise)
(Sampling Non-shared Noise)
A man with glasses, wearing a black coat and blue jeans is tennis forehand swing
A man with glasses, wearing a blue shirt and black jeans is walking
Person 020 is making happiness expression
Person 076 is making happiness expression
TBA