Guides
Masked image generation
֍
Use a mask to generate images

In this example, we'll generate a mask from a logo image with RemoveBackground, and use that mask to generate variations of the image using two workflows:

  1. Using StableDiffusionXLControlNet with the illusion method to generate illusions incorporating the mask.
  2. Using StableDiffusionXLControlNet with the edge method followed by StableDiffusionXLInpaint to fill a mask with content and then generate inside the mask.

First, initialize Substrate:

Python
TypeScript

from substrate import (
Substrate,
RemoveBackground,
StableDiffusionXLControlNet,
StableDiffusionXLInpaint,
)
s = Substrate(api_key=YOUR_API_KEY)

Here's the original image:

original

1. Generate illusions

This workflow uses RemoveBackground to generate a mask, followed by StableDiffusionXLControlNet with the illusion method to generate a two images incorporating the mask into a view of the ocean from above:

Python
TypeScript

mask = RemoveBackground(image_uri="https://media.substrate.run/logo-sq.png", return_mask=True)
controlnet = StableDiffusionXLControlNet(
image_uri=mask.future.image_uri,
control_method="illusion",
conditioning_scale=1,
prompt="sunlit bright birds-eye view of the ocean, turbulent choppy waves",
num_images=2,
)
res = s.run(controlnet)

image
no background

Experimenting with different prompts can produce striking results:

image
no background

2. Generate a variation and inpaint

This workflow uses StableDiffusionXLControlNet to generate a variation of the original, and StableDiffusionXLInpaint to generate a variation inpainting inside a mask.

Python
TypeScript

original = "https://media.substrate.run/logo-nopad-bg-white.png"
controlnet = StableDiffusionXLControlNet(
image_uri=original,
control_method="depth",
prompt="silver disco ball, 4k",
conditioning_scale=0.8,
num_images=1,
)
inpaint = StableDiffusionXLInpaint(
image_uri=controlnet.future.outputs[0].image_uri,
mask_image_uri=mask.future.image_uri,
prompt="towers in the futuristic ancient solarpunk city of atlantis",
num_images=1,
)
res = s.run(inpaint)

image
res.get(controlnet)
no background
res.get(inpaint)