Abdul HaqueHire me
Back to my work
INT-08 · Jun – Aug 2024 · internship

Interior Design Image Generation

Stable Diffusion fine-tuned to generate interior designs, with DreamBooth, LoRA and QLoRA, on two T4 GPUs with the model split across both.

Stable DiffusionLoRADreamBooth

My roleComputer Vision and Generative AI Intern at DataInsight Lab

GPUs2×T4one model across both
Methods3DreamBooth, LoRA, QLoRA
SubjectInteriorsinterior design generation

The idea

The problem

Stable Diffusion can draw almost anything, so it has no particular eye for interior design. Giving it one means fine-tuning, and a T4 is a small GPU for a model of this size.

The answer

Use methods that train a small part of the model instead of all of it (DreamBooth, LoRA, QLoRA), and split the model across two T4 GPUs so that it fits.

How the model learns a style

The same base model, nudged toward interiors on hardware that could not hold it alone.

  1. 1EXAMPLES
  2. 2SPLIT
  3. 3TUNE
  4. 4GENERATE
1. EXAMPLES

Interior design images are what the model learns from.

2. SPLIT

The model is placed across two T4 GPUs, part on each (model parallelism).

3. TUNE

DreamBooth, LoRA or QLoRA adjusts the model toward the examples.

4. GENERATE

The tuned model draws new interiors.

Three ways to fine-tune

What each method is, in one line. These are the general ideas, not results from this project.

A SUBJECT

DreamBooth

Teaches the model one subject or style from a small set of images.

FEWER WEIGHTS

LoRA

Freezes the model and trains small added matrices, so far fewer weights change.

LESS MEMORY

QLoRA

LoRA on a base model stored at lower precision, which cuts the memory needed further.

The hard parts

A large model, two small GPUs.

Two small GPUs

A T4 has 16 GB of memory. Model parallelism puts different parts of the model on different GPUs, so together they hold what one cannot.

Training less of the model

LoRA and QLoRA train a small set of added weights instead of all of them, which is what makes fine-tuning practical on this hardware.

Three methods, three trade-offs

DreamBooth, LoRA and QLoRA differ in how much memory they need and how much of the model they change.

Built with

Base modelStable Diffusion
Fine-tuningDreamBooth, LoRA, QLoRA
Hardware2×T4 GPUs, with model parallelism
WhereDataInsight Lab, FAST-NUCES, June to August 2024

Next voyage