Animating, storyboarding or movie making in general is a craft of it’s own, but with the rising capabilities of txt2img ai, the creative routine will change drastically. With the introduction of Stable Diffusion 1.5++ professional tools become available to be executed on one GPU standalone setup at you home studio. Here is one current approach with DEFORUM( link ) – but watch out for more actual solutions!
img2img + prompt experiments
We can use images as starting points, or inital prompts, so to say. Considering this fact, one question becomes obvious: is the system capable to conclude the existing image in a consistent and relevant manner, when moving the camera beyond the scope of the image?
The first approach is simply starting up with an image and applying diverse prompts to it over time
{
"0": "futuristic androgyn human neutral face, translucent shimmer organic bio cyberpunk devices, wobbly bubbly translucent skin, pipes and cables, complementary colors contrast realistic 4k 8k octane, blurry synthetic inner organs smooth background",
"40": "futuristic androgyn human happy face transforms into translucent bubbly bio organic pipes and cables flowing vortex underwater, bright translucent complementary colors contrast realistic 4k 8k octane, blurry inner body organs smooth lit up background",
"60": "energy rays and god ray morph futuristic androgyn human extatic face, translucent bio organic tiny particles and macro bubbles underwater, electronic wires and circuits, wetware , bright translucent colorful iriscendent complementary colors contrast realistic lowkey rim lights 4k 8k octane, blurry illuminated smooth background",
"100": "futuristic androgyn human neutral face, translucent shimmer organic bio cyberpunk device on androgyn human body wobbly bubbly translucent skin, pipes and cables, complementary colors contrast realistic 4k 8k octane, blurry synthetic inner organs smooth background"
}
The second approach adds camera motion to the prompt+image setup. From slight wiggling and zooming to a 360 degree rotation of the inital image.
3D rendering as input prompt
{
"0": "exoskeleton organic glove arm device curvy bubbly forms, silicone translucent reflective wet skin color, mint eyes, hand moving, realistic 3D render octane 4k 8k",
"40": "exoskeleton organic inflated blown up cloth oversize floating tentacles arm glove hand device curvy bubbly forms, silicone reflective wet translucent skin color, mint eyes, realistic 3D render octane 4k 8k",
"70": "organic animal floating no gravity curvy bubbly bio biomimikry snake tentacles, wet reflective silky silicone translucent skin color, mint eyes, realistic 3D render octane 4k 8k"
}
videos as input prompt
First quick experiments and sketches with keyframe promps
Using the ai not only for visualizing prompts but to also interpolate and animate between multiple prompt clouds is a first, very exciting approach to dive into the subject. Check out the results and original prompt setup.
{
"0": "translucent shimmer organic bio cyberpunk device skin device wobbly bubbly skin, pipes and cables, complementary colors contrast realistic 4k 8k octane",
"60": "translucent bubbly bio organic pipes and cables flowing vortex underwater, bright translucent complementary colors contrast realistic 4k 8k octane",
"80": "translucent bio organic tiny particles and macro bubbles underwater, electronic wires and circuits, wetware , bright translucent colorful iriscendent complementary colors contrast realistic lowkey rimlights 4k 8k octane",
"100": "translucent shimmer organic bio cyberpunk device skin device wobbly bubbly skin, pipes and cables, complementary colors contrast realistic 4k 8k octane"
}
slight variation in rotation and ymotion of the seeds with a very low framerate of 5fps
Even with the prompt reference to an androgyn human body, we obviously get a focus on a very normative female body – well :/
{
"0": "translucent shimmer organic bio cyberpunk device on androgyn human body parts wobbly bubbly translucent skin, pipes and cables, complementary colors contrast realistic 4k 8k octane, blurry synthetic inner organs smooth background --no female",
"40": "(body parts) transforms into translucent bubbly bio organic pipes and cables flowing vortex underwater, bright translucent complementary colors contrast realistic 4k 8k octane, blurry inner body organs smooth lit up background --no female",
"60": "energy rays and god ray morph into (body parts) and translucent bio organic tiny particles and macro bubbles underwater, electronic wires and circuits, wetware , bright translucent colorful iriscendent complementary colors contrast realistic lowkey rimlights 4k 8k octane, blurry illuminated smooth background --no female",
"100": "translucent shimmer organic bio cyberpunk device on androgyn human body wobbly bubbly translucent skin, pipes and cables, complementary colors contrast realistic 4k 8k octane, blurry synthetic inner organs smooth background --no female"
}
Explicitly avoiding the female body direction through –no female brings a bit of improvement, but just a little