Masked Image Generation

Use a mask to generate images

In this example, we'll generate a mask from a logo image with RemoveBackground, and use that mask to generate variations of the image using two workflows:

Using StableDiffusionXLControlNet with the illusion method to generate illusions incorporating the mask.
Using StableDiffusionXLControlNet with the edge method followed by StableDiffusionXLInpaint to fill a mask with content and then generate inside the mask.

First, initialize Substrate:

Python

TypeScript


from substrate import (
    Substrate,
    RemoveBackground,
    StableDiffusionXLControlNet,
    StableDiffusionXLInpaint,
)
s = Substrate(api_key=YOUR_API_KEY)

Here's the original image:

1. Generate illusions

This workflow uses RemoveBackground to generate a mask, followed by StableDiffusionXLControlNet with the illusion method to generate a two images incorporating the mask into a view of the ocean from above:

Python

TypeScript


mask = RemoveBackground(image_uri="https://media.substrate.run/logo-sq.png", return_mask=True)
controlnet = StableDiffusionXLControlNet(
    image_uri=mask.future.image_uri,
    control_method="illusion",
    conditioning_scale=1,
    prompt="sunlit bright birds-eye view of the ocean, turbulent choppy waves",
    num_images=2,
)
res = s.run(controlnet)

Experimenting with different prompts can produce striking results:

2. Generate a variation and inpaint

This workflow uses RemoveBackground to generate a mask, StableDiffusionXLControlNet with the edge method to generate a variation of the original, and StableDiffusionXLInpaint to generate a variation inpainting inside the mask.

Python

TypeScript


original = "https://media.substrate.run/logo-sq.png"
mask = RemoveBackground(image_uri=original, return_mask=True)
controlnet = StableDiffusionXLControlNet(
    image_uri=original,
    control_method="edge",
    prompt="bright silver disco high contrast",
    conditioning_scale=0.8,
    num_images=1,
)
inpaint = StableDiffusionXLInpaint(
    image_uri=controlnet.future.outputs[0].image_uri,
    mask_image_uri=mask.future.image_uri,
    prompt="towers in the futuristic ancient solarpunk city of atlantis",
    num_images=1,
)
res = s.run(inpaint)

res.get(controlnet)

res.get(inpaint)

Separate foreground & background RAG (coming soon)