Rendering video and games in one Bevy engine

Bevy is a game engine written in Rust around an ECS: the world is entities, their data lives in columns, and systems run over those columns every frame. raea uses it as the engine for everything it makes, video included.

One frame, start to finish

A composition is TypeScript and React, written with @raea/motion. It runs on V8 inside Bevy: raea-motion run executes a script inside the engine, and @raea/motion/ecs is Bevy's own World. Interface parts are laid out and drawn by Blitz, a Rust HTML and CSS renderer, into an overlay. Video clips are decoded and composited on the GPU by a K kernel, under that overlay and over the 3D scene. Finished frames are converted to NV12 on the GPU and encoded by VideoToolbox.

Games on the same path

A game is a composition whose state is a pure fold: useGameState(initial, step) applies step to the keys held at every frame. Played live, the keys come from the keyboard; rendered, they come from a log of a play. raea's replay gate checks that both give the same pixels, at SSIM of at least 0.999 on every frame, and its game gate holds each game to 120 fps with no missed frames, input on screen within two frames, and memory under 400 MB.

Only what changed

Because a timeline is data in the engine, raea knows which frames each component renders on. After an edit it steps only those frames again, so changing a caption re-renders the caption's frames and leaves the rest of the reel as it was.