This experimental approach explores the co-creational potential of working with generative ai on plain color /drawing based interaction. In addition to the visual creation, we use LLM's and proper systemprompts in combination with audio generator models to use the generative graphics as a musical score.
The first approach uses simple sharpen or blur nodes as „preamplification“ of the image before it runs through the diffusion process. With overcooked diffusion parameters and pretty „general“ prompts, we can use the diffusion models to intentionally halucinate and show a proper creative sparing partner.
DOWNLOAD WORKFLOW: basic_img2img_sharpening_preamp_with_upscaler
DOWNLOAD WORKFLOW:basic_img2img_sharpening_preamp_no_upscaler
This experiment comes with a web-based drawing interface, that is based on a GPU accelerated method for smooth and versatile drawing experience even on small mobile devices. This prototype uses the PIXIJS framework ( https://pixijs.com/ ) for proper RenderTexture management and uses a pretty standard import/export image pipeline.
The code can be found at GUTHUB: https://github.com/tristanschulze/superpainter_zero/tree/main
The demo is at: https://turboflip.de/superpaint/
the image is the score
Modern LLM’S are capable to „read“ visual information – respective images. With simple systems prompts, we can translate the image description to a proper sound prompt. As sound generation is also mainly based on FFT spectrum generation – means its visual based – both procedures come with a proper similarity and can be combined fluently.
SYSTEMPROMPT: you get a picture. you analyze the picture according to colors, contours, patterns, motifs, composition and mood. Categorize the image culturally – instead of a description, you give me back a comma-separated prompt by converting your analysis into musical equivalents: musical genres, tonalities, instruments and a poetic, emotional description of the musical auditory mood. Create a complex sonic picture with your description. Avoid stereotypical categorizations. make the image prompt a comma separated list of words. maximum 100 words.
PROMPT GENERATED FROM IMAGE + SYSPROMPT IN Mistral-Small-3.1-24B-Instruct-2503: symphonic prog-rock, midnight mystic flamenco, psychedelic trance fusion,on the tonalities of minor-key orchestral coloratura, theremin, Sheng, gamelan, temple bells, electric sitars, gritty microtonal synths, beads of pearlescent tempos, the creative fusion of sprechstimme vocals speaking ethereal incantations, claustrophobic and claustrophilia-infused, filled with twisting melodies, dripping and spitting with throttled and enforced disjointed harmonies, from salty ocean foam to cosmic ooze turning into flickering neon fire, breakthroughs that erupt profoundly into the sky, orgasmic constellations writhing past ancient chronicles, questioning and claiming echoing crescendos hovering over ambiguous landmarks, a hedonistic journey, riding through the serendipitous maelstrom, a turbulent embrace, dancing at the bottom of a dream, where stars blossom into abyssal rivers and whispers of the cosmos tickle the ocean floor, instrumental
PROMPT GENERATED FROM IMAGE + SYSPROMPT IN Mistral-Small-3.1-24B-Instruct-2503: Post-rock,synthesizer textures,melodic arpeggios,ambient,majestic,dreamy pads soundscapes,atmospheric,ecxperimental,pure vibrancy,yellows,rainbow spectrum:Dynamic,Dream Theater’s version of „Bohemian Rhapsody,“Dramatic,narrative sweep across a vast, evolving landscape,chromatic layers,hypnotic,repeating patterns,Folk psycherlandais,Spiegel im Spiegel,frantic electronic crescendos, expressive
PROMPT GENERATED FROM IMAGE + SYSPROMPT IN Mistral-Small-3.1-24B-Instruct-2503: glitch-hop, vaporwave, chiptune, bitpop, electro, mod, evanescent, chilling, hypnotic, artificial, colorful, ethereal,anguished, pastel, surreal, mystical, poignant, nebulae, bioluminescent, cold,static, circuit-bent, haunting , An electronic symphony blares, a colorful, angular maze of circuits, विधुत् ujjal, pulsing with synthetic life, umbilical chords connected to a motherboard, glowing circuits, eclipsing shadowy giants arranged in the image of God, pulses of the synth-drums, reverberating into infinity, glitching, atemlos, modular synths ascending, resonating chambers of reverberation, pulsating, indifferent, firmly thanatos, irregular, bass drops, haunting digitally layered harmonics, antique samplers, glowing, holographic, squelching, narrow angles of paradise, pulsating in the darkness echoing with sounds, deformed, concealed by age, constantly evolving, a teraflop of information processed each second, exposed, the glowing mazes of circuits and wires that bring the concept of life dancing vapes of digitized smoke, a bright cosmic mist on the canvas of the night, caparisoned, with colorful blankets and suspended ornamentation Rippling, inducing measured, immersive, atmospheric cerebration a celebration of the digital dawn,
