File:X-Y plot of algorithmically-generated AI art demonstrating Hypernetworks.png

An X/Y plot of algorithmically-generated AI artworks depicting a woman in various different settings, created using a custom-trained anime-focused Stable Diffusion-based model known as "Anything V3.0" (with hash 1a7df6b8) created by Furqanil Taqwa. This plot serves to demonstrate the usage of Hypernetworks, a technique created by Kurumuz in 2021 which allows Stable Diffusion-based image generation models to imitate the art style of specific artists, even if the artist is not recognised by the original diffusion model, by applying a small neural network at various points within the larger network.

Hypernetworks are small pre-trained neural networks that steer results towards a particular direction, for example applying visual styles and motifs, when used in conjunction with a larger neural network. The Hypernetwork processes the image by finding key areas of importance such as hair and eyes, and patches them in secondary latent space. They are significantly smaller in filesize compared to DreamBooth models, another method for fine-training AI diffusion models, making Hypernetworks a viable alternative to DreamBooth models in some, but not all, use-cases. Hypernetwork training also requires only 6GB of VRAM, compared to the ~20GB VRAM required for DreamBooth training (although this VRAM requirement can be lowered using DeepSpeed). A downside to Hypernetworks is that they are comparatively less flexible and accurate, and can sometimes lead to unpredictable results. For this reason, Hypernetworks are suited towards applying visual style or cleaning up blemishes in human anatomy, while DreamBooth models are more adept at depicting specific user-defined subjects.

Procedure/Methodology

These images were generated using an NVIDIA RTX 4090; since Ada Lovelace chipsets (using compute capability 8.9, which requires CUDA 11.8) are not fully supported by the pyTorch dependency libraries currently used by Stable Diffusion, I've used a custom build of xformers, along with pyTorch cu116 and cuDNN v8.6, as a temporary workaround. Front-end used for the entire generation process is Stable Diffusion web UI created by AUTOMATIC1111.

Hypernetworks trained on the artstyles of the following artists were used:

• As109, a Chinese artist. Trained on 440 samples using 75,000 steps on 0.0000005 LR.
• Asanagi (朝凪), a Japanese artist and the sole member of the Fatalpulse dōjin circle. Trained using 118,500 steps.
• homunculus (ホムンクルス), a Japanese artist and mangaka. Trained using 90,000 steps on 5e-7 LR with no normalisation, and layer structure 1.0, 2.0, 1.0.
• j.k., a Canadian artist.
• Ohisashiburi (お久しぶり), a Japanese artist. Trained with 1e-5 LR up to 7,000 steps, and 5e-6 LR up to 180,000 steps, with layer structure 1.0, 1.5, 1.5, 1.0, mish activation function, normal weight initialisation, layer norm set to false, dropout usage set to true.
• Takayaki (たかやKi), a Japanese artist and member of the Jenoa Cake (じぇのばけーき) dōjin circle. Trained on 90 samples using 100,000 steps on 0.0000005 LR.

A batch of 768x1024 images were generated with txt2img using the following prompts:

young woman, fully clothed, volumetric lighting, mountain forest background

Negative prompt: nude, lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, artist name

Settings: Steps: 100, Sampler: DPM2, CFG scale: 7, Size: 768x1024, Highres. fix, Denoising strength: 0.7, Clip skip: 2

During the generation of this batch, an X/Y plot was generated using the "X/Y plot" txt2img script, along with the following settings:

• X-axis: Hypernetwork: None, 151abd09, a00f10d3, e0a6b144, 97fa462a, a177a153, 9936f48b
• Y-axis: Prompt S/R: mountain forest background, sitting in front of a computer. corporate office background, romantic date. french cafe background