Turn a video clip into a sprite animation

Video models (Veo, Sora, Kling, Runway, Seedance and others) give the best character motion you can get from AI. Polymation writes the prompt that gets a usable clip, then turns the clip into clean frames: the best loop found, the background removed, every frame cropped alike. You get a sprite sheet for your game engine and a Rive flipbook.

You bring
A picture of your character, and a video model to make the clip. Or an existing clip or sprite sheet.
You get
PNG frames, sheet.png + sheet.json (the format Phaser, PixiJS and Aseprite read), and a .riv flipbook.
Needs
The npm install (the one-click extension can't read video), and Chrome or Edge to read the clip.

Ask your assistant

  • Write me a video prompt to make my knight (knight.png) walk in place, with a sword slash as an extra move.
  • Turn walk.mp4 and slash.mp4 into a sprite sheet for Phaser and a Rive file.
  • Clean up this sprite sheet: 4 rows, 6 columns, on a white background, and make it a looping animation.
  • I want my character to face eight directions in my top-down game.

How it works

  1. The brief. sprite_art_brief in video mode writes the prompt for your video model, and a start picture: your character on one flat background colour it doesn't use, standing on a slightly darker floor strip. The floor keeps video models from moving the camera and cutting the feet off. The prompt asks for a still camera and a clip that ends where it starts. Extra moves (a slash, a jump kick) get their own prompts, each starting and ending on the same stance.
  2. You make the clips in your video model with the start picture attached.
  3. Clip to frames. video_to_sprite finds the stretch that loops best and says how seamless it is, keys out the background and floor, crops every frame alike, and writes the frames, the sheet and the .riv. Each move is cropped with the loop, so the character keeps its place, and plays once on a trigger in a state machine called "Moves", then returns to the loop.
  4. Check it with preview_riv_gif and review_motion (see below).

Frames from an image model instead

sprite_art_brief in frames mode draws a pose reference for each frame of a walk, run, jump, idle, attack or wave, and asks your image model for one frame per request. Then remove_sprite_background, normalize_sprite_frames and assemble_sprite_animation make the flipbook.

A sprite sheet you already have

sprite_sheet_to_riv slices the grid, removes the background (art on white included), centres every frame so it stops jittering, and writes the same frames, sheet and .riv. Empty cells at the end can be dropped.

Eight facings

sprite_art_brief in directions mode asks for each facing in its own request: five are drawn, the other three mirrored (or all eight drawn, for a character whose sides differ). assemble_sprite_animation then builds an animation per facing and a state machine with a direction input your game sets, 0 to 7. Rigged characters and 3D models turn too: see Eight-way characters.

Reading the result

  • The loop line gives the jump back to the start as a multiple of an ordinary frame step: up to 1.2 is seamless, above 2.5 shows. Ask your model for a clip that ends where it starts, or pick the span by hand.
  • review_motion reads the rendered frames: where the feet meet the ground, frames that jitter because they aren't lined up, and the loop point.
  • Background left in the frames: give the clip's background colour, or loosen the tolerance.
  • A one-off action rather than a loop: say so, and the whole clip is kept, start to end.

Good to know

  • For a character that flies, swims or floats, ask for no floor: it is placed in the middle instead.
  • Pixel-art characters get prompts for pixel art. Snap the frames onto one grid afterwards: Fix AI pixel art.
  • MP4, WebM and AV1 clips work. HEVC clips need Chrome or Edge installed.
  • A flipbook shows the frames as drawn. To keep one set of art and move it with bones instead, see Rig a character from a picture and Animate your rig from a video.

Related