Testing new Image2Video workflows 🖼 👉 📽

The next level of generative video is officially here, and we didn't have to wait for OpenAI's SORA to be publicly released to get it either!

Runway Gen-3, KLING - AI Videos, and LUMA LABS Dream Machine all now offer advanced easy-to-access image-to-video workflow features which we will be testing and comparing throughout this newsletter.

All three of these platforms are included in our new training and workshops.

You can now book these sessions directly through our website: https://seedily.ai/

Platforms like ComfyUI, Krea, and Kaiber offer similar workflows using image references.

However, they are either less accessible or lack the specific capabilities of the three platforms we focus on in this newsletter.

Nevertheless, they are all worth exploring depending on your generative video needs.

To provide a fair comparison of each platform's image-to-video capabilities, we used the same Midjourney 6.1 images and identical prompts for each. This approach ensured consistency, keeping the prompts short enough to fit the input requirements of all three platforms.

Additionally, we ensured fairness and consistency by not cherry-picking the best results from each platform.

Instead, we used the first result each platform provided, making it a true test of performance!

Midjourney v6.1 Image reference 1: MEERKAT

Runway output (converted to GIF format from mp4): Best prompt adherence

KLING - AI Videos output (converted to GIF format from mp4): Most natural movement

LUMA LABS Dream Machine output (converted to GIF format): Lack of character movement

Midjourney v6.1 image reference: LYNX 🐆

Runway output (converted to GIF format from mp4): Best prompt adherence

Kling AI output (converted to GIF): most natural movement, least warping

Midjourney v6.1 image reference: GOAT 🐐

Runway output (converted to GIF format from mp4): Best prompt adherence

Kling AI output (converted to GIF format): Head turns the wrong way, most natural again.

Dream machine output (converted to GIF format from mp4): No head turn, only zoom

We would have loved to delve deeper into these experiments in this newsletter, as we do in our workshops. However, LinkedIn is already struggling with the number of GIFs we've included in this newsletter so far. So, we'll limit our discussion of the image and video (converted to GIF) outputs for now.

WE COVER A LOT MORE IN OUR NEW WORKSHOPS!

Book a session directly through our new website: https://seedily.ai/

Conclusions about comparissons

Even with just these three animals, some clear conclusions could be drawn.

Firstly, Dream Machine struggled with our prompts. It generated its own motions, which, although smooth, did not align with our prompt instructions really at all. Notably, we used the Loop function on each generation, even though Dream Machine uniquely supports it. We also avoided using the end keyframe feature to ensure fairness as no other platform in has this feature, so we were not using Dream Machine to its full capabilities.

Secondly, Kling AI produced by far the most natural movement for each animal. However, it occasionally failed to follow the short prompts accurately. For instance, while it moved the animal's head each time, it only directed it toward the camera once (in the case of the lynx) and failed to add LED glowing lights to the meerkat as instructed. Nonetheless, if you value natural movement and have patience, Kling AI is the clear winner in this department.

Thirdly, Runway AI excelled at understanding prompts. Although some movements appeared unnatural compared to Kling AI's output, Runway nearly always executed the tasks as prompted. Despite its limitations with image input types and the 16:9 aspect ratio restriction, Runway was the overall winner. The other platforms, accepting nearly any aspect ratio, have an advantage in broader use cases as of this writing.

Which image-to-video tools have you been using?

Leave a comment below to let us know what we should test and report on next! 👇✌️