Interior Design Image Generation
Stable Diffusion fine-tuned to generate interior designs, with DreamBooth, LoRA and QLoRA, on two T4 GPUs with the model split across both.
My roleComputer Vision and Generative AI Intern at DataInsight Lab
The idea
The problem
Stable Diffusion can draw almost anything, so it has no particular eye for interior design. Giving it one means fine-tuning, and a T4 is a small GPU for a model of this size.
The answer
Use methods that train a small part of the model instead of all of it (DreamBooth, LoRA, QLoRA), and split the model across two T4 GPUs so that it fits.
How the model learns a style
The same base model, nudged toward interiors on hardware that could not hold it alone.
- 1EXAMPLES
- 2SPLIT
- 3TUNE
- 4GENERATE
Interior design images are what the model learns from.
The model is placed across two T4 GPUs, part on each (model parallelism).
DreamBooth, LoRA or QLoRA adjusts the model toward the examples.
The tuned model draws new interiors.
Three ways to fine-tune
What each method is, in one line. These are the general ideas, not results from this project.
DreamBooth
Teaches the model one subject or style from a small set of images.
LoRA
Freezes the model and trains small added matrices, so far fewer weights change.
QLoRA
LoRA on a base model stored at lower precision, which cuts the memory needed further.
The hard parts
A large model, two small GPUs.
A T4 has 16 GB of memory. Model parallelism puts different parts of the model on different GPUs, so together they hold what one cannot.
LoRA and QLoRA train a small set of added weights instead of all of them, which is what makes fine-tuning practical on this hardware.
DreamBooth, LoRA and QLoRA differ in how much memory they need and how much of the model they change.
Built with
| Base model | Stable Diffusion |
|---|---|
| Fine-tuning | DreamBooth, LoRA, QLoRA |
| Hardware | 2×T4 GPUs, with model parallelism |
| Where | DataInsight Lab, FAST-NUCES, June to August 2024 |