What is ControlNet?

ControlNet is a neural network structure to control diffusion models by adding extra conditions. It essentially clones the weights of a large diffusion model and locks the original weights. The clone weights are trainable to learn specific condition inputs. This allows users to guide image generation based on various input conditions like edge maps, segmentation maps, pose estimations, or depth maps, providing finer-grained control over the generated image's composition and style.