Masked Image Generation

Learn how to use masked image generation techniques using StableDiffusionXLControlNet and StableDiffusionXLInpaint.

Generate illusions

Edge variation and inpainting

We'll cover two workflows that use masked image generation:

Generate illusions: StableDiffusionXLControlNet with the illusion method generates illusions incorporating the mask.
Edge variation and inpainting: StableDiffusionXLControlNet with the edge method generates variations of an image using edge detection, and StableDiffusionXLInpaint generates inside the mask.

First, initialize Substrate:

Python

TypeScript


from substrate import (
    Substrate,
    RemoveBackground,
    StableDiffusionXLControlNet,
    StableDiffusionXLInpaint,
)
s = Substrate(api_key=YOUR_API_KEY)

Here's the original image:

1. Generate illusions

This workflow uses RemoveBackground to generate a mask, followed by StableDiffusionXLControlNet with the illusion method to generate a two images incorporating the mask into a view of the ocean from above:

Python

TypeScript


mask = RemoveBackground(image_uri="https://media.substrate.run/logo-sq.png", return_mask=True)
controlnet = StableDiffusionXLControlNet(
    image_uri=mask.future.image_uri,
    control_method="illusion",
    conditioning_scale=1,
    prompt="sunlit bright birds-eye view of the ocean, turbulent choppy waves",
    num_images=2,
)
res = s.run(controlnet)

Experimenting with different prompts can produce striking results:

2. Edge variation and inpainting

This workflow uses StableDiffusionXLControlNet to generate a variation of the original, and StableDiffusionXLInpaint to generate a variation inpainting inside a mask.

Python

TypeScript


original = "https://media.substrate.run/logo-nopad-bg-white.png"
controlnet = StableDiffusionXLControlNet(
    image_uri=original,
    control_method="depth",
    prompt="silver disco ball, 4k",
    conditioning_scale=0.8,
    num_images=1,
)
inpaint = StableDiffusionXLInpaint(
    image_uri=controlnet.future.outputs[0].image_uri,
    mask_image_uri=mask.future.image_uri,
    prompt="towers in the futuristic ancient solarpunk city of atlantis",
    num_images=1,
)
res = s.run(inpaint)

res.get(controlnet)

res.get(inpaint)

Structured image generation Transcribe speech