late night creative workflows with Stable Diffusion/A1111

Working with A1111 in combination with any pixel design tool can open up a great workflow for unexpected results. The following experiments show different approaches:

prompt to video workflow / StableDiffusion & Runway Gen2

First off, we create a relevant amount of keyframes of any storyline we like to tell. This can be archieve with simple basic sketching and equivalent prompting. The interplay of this, using Stable Diffusion/A1111 by developing a base prompt and a consistent color palette in the drawings, can create stunning consistent pieces of frames, we can use to generate moving pictures from.

using the sketch2img pipeline of Stable Diffusion in combination with variation of main prompt

(realistic close up photo of futuristic androgyn middle aged male gardener in white insect bug shell suit fungi mushroom inspired made from recyled and reused material overgrown with moss and plants :1.6), white glossy translucent materials with curvy voronoi structure, backpack, dirty wet, (background foggy surreal forest wilderness overgrown moss rotten wood:1.4), background rimlight, 8k uhd, dslr, dynamic expressive lighting, high quality, film grain, Fujifilm XT3 ,realistic, detailed, sharpened sharp, professional photography, shiny lights , hyperrealistic photo realistic, sharpen detailed , fine detail, realistic proportions, professional photography DOF, ( renaissance and baroque style : 1.5)

helmet, (deformed iris deformed face deformed pupils:1.7), (semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime:1.3), asian, sexy, text, close up, cropped, out of frame, worst quality, low quality, jpeg artifacts, ( ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck:1.1)(blur blurry:2.0)

collaging to composition

step I : wildly collaging ugly webcam portraits
step II : img2img with loose denoise
step III : img2img with controlNet Depth of step II img as consistency base
set IV – playing with denoise values
step V : overpaint again with some motif change and light rays
step VI : lower resolution nice result
final upscaled result