Generative AI for Innovative Photo & Video Editing & Creation - APPs and Enabling Datasets

7 min readDec 18, 2023

In the ever-evolving realm of digital artistry, the emergence of Generative Artificial Intelligence (AI) has unlocked a vast array of opportunities. This revolutionary technology is reshaping the limits of photo and video editing and revolutionizing the way we create, perceive, and engage with visual content.

In addition, it was not long before the concept of AI editing was combined with pre-existing images or photographs. While not as complicated as video editing, manual image editing is still a time-consuming process, especially when it comes to correcting imperfections, enhancing facial features, and removing unwanted elements.

Generative AI for innovative photos and videos, for example, can turn a few words or a simple drawing into stunning works of art, as exemplified by AI tools such as DALL-E 3, Canva, Runway’s Gen 2, and Adobe’s Firefly Image 2 and Project Stardust, etc.

Therefore, with the gradual penetration of AIGC technology into fields such as entertainment, media, and games, efficiency and creativity can be significantly improved.

Let’s take a look at some of the well-known AI-powered image and video editing and creation tools to find out their strengths and challenges, and what we can do to improve efficiency and unleash creativity.

1. AI-powered Image & Video Editing & Creation APPs

1.1 Firefly Image 2

At the Adobe Max event, Adobe launched a text-to-image AI tool — Firefly Image 2, not only to generate better image quality, the default generation resolution increased to 2048 × 2048.

Also added other features, including the “image-to-image” to imitate similar styles, as well as generating settings to adjust, such as depth of field and motion blur and other effects.

In addition, like DALL-E 3, Adobe Firefly can also directly give prompt suggestions.

Some users have also mentioned that when you ask Firefly Image Model 2 to generate a photo-like image, both portrait and body part, the detail and realism is quite good:

Image source: https://twitter.com/icreatelife/status/1734469772572574160

Image source: https://twitter.com/creacas_ai/status/1711838852644556847

In addition, the comparison of Firefly and 3 other AI tools showed that Firefly Image 2 had a stunning result in terms of portrait realism.

Image source: https://twitter.com/ryuya__52/status/1732986128494710864

To render more beautiful pixels and detail for users, Alexandru Costin, Adobe’s VP of generative AI and sensei, said, “Firefly is an ensemble of multiple models and I think we’ve increased their sizes by a factor of three.”

The company also increased the dataset by almost a factor of two, which in turn should give the model a better understanding of what users are asking for.

1.2 Project Stardust

Adobe has also unveiled a cutting-edge artificial intelligence (AI) photo editing tool. Called Project Stardust, the tool features an innovative “object-aware editing engine” that automatically detects and selects individual objects in ordinary photos. This allows users to effortlessly manipulate and reposition these objects as if they were on separate layers, with the software automatically filling in the background to blend seamlessly with the surroundings.

https://youtu.be/DtKeu9tSZYA

In addition, Project Stardust offers a contextual feature that intelligently detects the next steps in the design process, facilitating quick edits. A demonstration shows the software’s ability to identify and remove distracting elements, such as a crowd of blurry people in the background, by presenting a “Remove Distractors” button on the taskbar. When clicked, the button immediately removes the unwanted elements.

The demo video also shows the versatility of Project Stardust, as Adobe Digital Imaging Product Manager Aya Philémon effortlessly changes a model’s clothing. By selecting the desired item and providing a description, the software generates AI-generated alternatives that are seamlessly integrated into the image.

https://www.youtube.com/watch?v=PxlFsdj0Tew

1.3 Runway Gen 2

On November 3, Runway released an update to its Gen-2 model that increased the clarity of the generated video to 4K precision, improving the fidelity and consistency of the results. On November 20, the Motion Brush feature went live, making it possible to move anything stationary to any position in the image with a single brush.

Runway is an innovative AI-powered platform that can generate video from text, images, or video clips. Users can create stunning visuals using only written words and enhance any image with simple text prompts.

In a recent update, Runway launched the Runway Gen2 version upgrade, introducing a more advanced AI model and officially launching the Motion Brush feature. Simply “paint” any object on the screen with the brush to make the specified object move.

Not only can you make a static character move, but even the slightest movement of her skirt and head is natural.

You can control the angle of the camera’s shooting position (i.e., camera movement) and even rotate the subject in the image.

Still, the Motion Brush feature also has some shortcomings.

For example, the brush may not recognize selected objects, the movement options may be quite limited, it may not be possible to specify more than one direction of movement at a time, and so on.

1.4 Canva

Canva is an AI-powered platform offering seamless photo transformations and a variety of design tools. Its features include an AI photo editor, an extensive template library, and graphic elements. Available on Android and iOS, it combines photo and video editing in one app.

Canva is an AI-powered platform that offers seamless photo transformations and a variety of design tools. Its features include an AI photo editor, an extensive template library, and graphic elements. Available on Android and iOS, it combines photo and video editing in one app.

Canva’s Magic Expand helps expand images in any direction to correct framing, save, and adjust orientation. Magic Grab allows you to select and manipulate themes similar to a Canva template. Magic Edit transforms images based on a written prompt, while Magic Eraser emphasizes the subject by erasing unnecessary details. Background Remover makes it easy to remove backgrounds from images and videos, and Magic Animate automatically adds motion to designs through animations and transitions.

Image source: https://www.canva.com/magic/

2. Enhance AI-powered image and video editing and creation capabilities with high-quality datasets

With the rapid development of AI, Maadaa.ai believes that AI-powered image and video editing tools will continue to surprise us.

It turns out that most of the performance improvements we can achieve come from a data-centric approach.

In fact, Andrew Ng, who proposed the concept of data-centric AI, once showed the experimental test results in the figure below in a speech to prove the importance of high-quality data.

Image source: https://towardsdatascience.com/from-model-centric-to-data-centric-artificial-intelligence-77e423f3f593

Therefore, high-quality data is an absolute necessity for AI to generate images and videos with accuracy, clarity, and efficiency. Without it, AI models can result in inconsistencies, inaccuracies, and blurred content.

In addition, high-quality data allows AI models to offer advanced features, improve training, and generalize better.

According to Alexandru Costin, Adobe’s VP of generative AI and Sensei, the Firefly model’s dataset has nearly doubled. This increase should help the model better understand the needs and requests of users.

The importance of high-quality data in AI-driven image and video processing and creation cannot be overstated.

Based on years of experience in AI dataset services, maadaa.ai has released AI Photo-Video Editing Standard Dataset Lite (Version 2.0), which includes 16 typical scenarios and 25 fine-annotated sub-datasets.

2.1 Specific Fine-Segmentation Datasets for Precise Object Detection and Editing

It includes data ranging from food contours to object and hair segmentation, helping the AI identify fine details in images. This improves accuracy in tasks such as background removal, object isolation, and targeted enhancement.

2.2 Human-Body Segmentation for Advanced Body-related Editing

Master body-related editing with our human body segmentation datasets. Gain a detailed understanding of human body shapes and postures, enabling advanced editing such as reshaping and posture adjustment to enhance your photo and video editing capabilities.

2.3 Facial Segmentation for Realistic and Personalized Facial Editing

The datasets provide a detailed understanding of human facial features, enabling the algorithms to identify and manipulate these features with high precision, contributing to advanced editing features such as beauty filters and realistic facial animation.

Read on for more details:

AI Photo-VIdeo Editing Standard Dataset Lite (Version 2.0)

Take your digital content creation to the next level with AI-powered photo and video editing apps, the industry's…

maadaa.ai

3. Conclusion

Generative AI has revolutionized the editing and creation of photos and videos, resulting in innovative tools like Runway Gen2, Firefly Image 2, Project Stardust, and Canva. These amazing AI tools can transform your ideas into stunning visuals with just a few words or a click on the screen, making the creative process more accessible and efficient.

As we explore the potential of Generative AI for photo and video editing, it becomes clear that a data-centric approach is crucial. High-quality datasets are essential for improving the accuracy, clarity, and efficiency of AI models. As these technologies continue to evolve, we can expect a future where image and video editing and creation are not only easier and faster but also more creative and engaging.