Adobe has announced that its Firefly Video Model will be available in beta later this year, bringing a series of generative AI video tools to the company’s video editing software, including Premiere Pro and After Effects. The company released a series of previews for the tools, showcasing the capabilities in Adobe’s Creative Cloud applications, including text-to-video and image-to-video features.
Firefly was launched in March 2023, enabling users to fill parts of a scene with AI-generated objects or automatically remove distractions from a video. Now the company says it has worked with the video editing community to advance the Firefly Video Model, allowing users to generate video to fill gaps in their timeline, and add new elements to existing footage.
Adobe noted that the model is commercially safe, and only trained on content the business has permission to use, as opposed to Adobe users’ content.
“The ever-increasing demand for fresh, short-form video content means editors, filmmakers and content creators are being asked to do more and in less time,” Ashley Still, Senior Vice President & General Manager of Creative Product Group at Adobe, wrote in a blog post. “At Adobe, we’re leveraging the power of AI to help editors expand their creative toolset so they can work in these other disciplines, delivering high-quality results on the timelines their clients require.”
Fresh angles
The blog post mentions common editorial tasks eased by the new Firefly model, such as navigating gaps in footage, removing unwanted objects from a scene, and smoothing jump cut transitions.
Perhaps the most attention-grabbing function is the Text-to-Video tool, whereby users can input text prompts, camera controls, and reference images to generate B-Roll that fills gaps in the timeline. Adobe gave a series of examples, all of which it said were generated in under two minutes.
The example below was generated by the prompt: “Cinematic closeup and detailed portrait of a reindeer in a snowy forest at sunset. The lighting is cinematic and gorgeous and soft and sun-kissed, with golden backlight and dreamy bokeh and lens flares. The color grade is cinematic and magical.”
Adobe also highlighted the tool’s ability to generate videos of the natural world, allowing editors to create establishing shots that may have been missed during production. The model additionally enables a variety of camera controls, including angle, motion and zoom.
The following example was generated using the prompt: “Footage of a camera on a drone flying over a desert with wind blowing over the dunes creating waves in the sand below.”
The model also includes Image-to-Video capabilities, where users can upload a reference frame and generate a clip based on the image. The next example takes the image of the galaxy and uses a simple text promt to produce the video below.
“We are delivering AI tools to help facilitate an even better creative process that helps take the tedium out of post-production,” said Ashley Still’s blog post. “Our AI tools give editors more time to explore new creative ideas, the part of the job they love, while also setting them up for successful collaboration with the larger team.”
Asset assists
The latest genAI tools from Adobe look set to have a massive impact on video, significantly reducing production costs and empowering smaller creative enterprises to create brand-safe video assets. However The Verge notes that videos produced by the text-to-video and image-to-video tools have a maximum length of five seconds.
The update also positions Adobe as a rival to Sora, OpenAI’s text-to-video model. In April, Adobe said it was looking to introduce Sora into its software, and the company is presumably still looking to integrate third-party genAI tools. But the software firm also emphasised the “commercially safe” elements of Firefly, potentially helping assauge the concerns around copyright infringement that plague OpenAI’s models.
And advertising agencies are already partnering with tech companies to gain access to these AI tools. Last year, holding group Omnicom announced a partnership with Adobe, granting its agencies access to Firefly’s creative genAI models. And in May, WPP partnered with Anthropic, integrating the startup’s suite of genAI tools into the agency group’s operating system.
