← Design + Engineering

AI set design station

FILMPROVISING

Plastic blocks in, a finished set out

RoleAI Engineer · Full-stack
Year2024, updated since
ModelsStable Diffusion · Stability · Runway
StackNext.js · Flask · OpenCV · Firebase

A prototyping station for film set design. A camera watches a miniature set, the designer locks in a scene and a shot, and an image model renders the layout as a finished room. The AI covers the range; the designer makes the calls.

The interface

Try it
The station: a slate across the top with scene, shot, take, lens and location; the live camera feed of the miniature set on the left; the rendered hospital ward on the right; direction, Control and Capture along the bottom.123456

Designer

  1. 1Picks a saved scene and shot on the slate.
  2. 2Gives a line of direction.
  3. 3Sets Control: how far the render may stray from the blocks.
  4. 4Captures the frame.

Model

  1. 5Renders the camera frame as a finished set, keeping the layout of the blocks.
  2. 6Make into video sends the chosen render to Runway.

The interface, redesigned in 2026 and run from its source with sample presets. Camera: a photo of the set. Render: a render the station saved in 2024.

System

Architecture
FILMPROVISING · SYSTEMSET DESIGN STATION · 202401 / THE STATIONBlocks on a sweepWEBCAM · FLASK + OPENCVTHE FRAME IS GRABBED ON CAPTURE02 / THE DESIGNER’S CALLSLocked in, then reusedSCENESaved onceatmosphere,aesthetic, paletteSHOTSaved oncelens, angle,lightingDIRECTIONEach captureone lineof directionCONTROLEach capturehow far it maystray from blocksSCENES, SHOTS AND CHARACTERS SAVED IN FIREBASE03 / THE RENDERSame layout, a finished roomTHE SWAPPABLE PARTSketch controlLOCALSD 1.5, then SDXLA1111, ComfyUI,StreamDiffusionAPIStability sketchCONCEPT FRAMERUNWAY GEN-3Motiononly the framesworth movingFILMPROVISING202401 / THE STATIONBlocks on a sweepWEBCAM · FLASK + OPENCVTHE FRAME IS GRABBED ON CAPTURE02 / THE DESIGNER’S CALLSLocked in, then reusedSCENESaved onceatmosphere,aesthetic, paletteSHOTSaved oncelens, angle,lightingDIRECTIONEach captureone lineof directionCONTROLEach capturehow far it maystray from blocksSCENES AND SHOTS SAVED IN FIREBASE03 / THE RENDERSame layout, a finished roomTHE SWAPPABLE PARTSketch controlLOCALSD 1.5, then SDXLA1111, ComfyUI,StreamDiffusionAPIStability sketchCONCEPT FRAMERUNWAY GEN-3Motiononly the framesworth moving

Why both

Tradeoffs

Exploring runs locally: each render costs nothing, the loop stays fast, and unreleased designs never leave the room. The API comes in when quality matters more than count, and Runway only for the few frames worth turning into motion.

The models changed under it every few months, so the model is the swappable part: the camera, the presets and the interface stay put while it moves from one release to the next.

Beta

20 testers

I tested the beta with 10 set designers and 10 filmmakers, asking how they would use it in their own work. What they liked: rapid iteration, and scene elements they could build once and keep across many generation rounds.

Blocks to a room

Real captures
The station: a miniature set on a paper sweep, camera in front
A render the station saved in 2024

Prototype Demo

0:42

Lock a scene and a shot, set Control, capture. Recorded from the same run as the screens above.