raea/camera

Cameras and the views they make: Camera3d, the Canvas3D a 2D scene draws in, HDR and matchmoved cameras.

View as Markdown

<Camera3d> is Bevy's 3D camera: a transform, a projection and what it renders to. <Canvas3D> is the stage a Bevy-style 2D scene draws in, with Bevy's coordinates (the origin at the centre, +Y up, a unit a pixel). <Matchmove> drives a camera from the solve raea track makes, so 3D stands in filmed footage.

TSX
import { Camera3d } from "raea/camera";
import { Transform } from "raea/transform";

const Shot = () => <Camera3d transform={Transform.fromXyz(-2, 2.5, 5).lookingAt([0, 0, 0])} projection={{ Perspective: { fov: Math.PI / 4 } }} />;
TS
import { Camera3d, Canvas3D, checkMatchmove, Hdr } from "raea/camera"

raea/camera exports 5 components, 1 hook, 3 functions, 9 types. Bevy's counterpart: bevy_camera. Source: sdk/raea/src/camera.ts.

Components

Camera3d

Bevy's Camera3d with its Transform and Projection: it looks along its -Z, +Y up, as Bevy's cameras do.

Bevy: bevy_camera::components::Camera3d, bevy_camera::camera::Camera, bevy_camera::projection::Projection.

PropTypeWhat it does
transformTransformValuewhere the camera stands and faces (Transform.fromXYZ(...).lookingAt(...))
projectionProjectionperspective (the default) or orthographic, as Bevy's Projection
layersnumber[]Bevy's RenderLayers on the camera: it draws only the parts on these layers (a Mesh3d's layers; [0] by default)
ordernumberBevy's Camera order: a camera with a higher one draws after (over) those with lower; 0 by default
clear_colorClearColorConfigBevy's ClearColorConfig: "Default" (the clear colour), "None" (drawn over what the cameras before it drew, as a second pass), or { Custom: colour }
viewportCameraViewportBevy's Viewport: the camera draws into this region of the window only, physical_position from its top left and physical_size big (whole px, at a scale factor of 1)
childrenReactNodecomponents on the camera: post effects, tone mapping, a skybox

Used in: ui/render_ui_to_texture, ui/widgets/viewport_node, 3d/fog_volumes, 3d/fog, 3d/vertex_colors, 3d/rotate_environment_map, 3d/transparency_3d, 3d/spherical_area_lights, and 60 more.

Canvas3D

Only for layered: the 3D is drawn in the page's order, at the box of the nearest positioned element around it (the whole frame when there's none; B3 H7), at that box's size and aspect:

...
Without `layered` it is its children, as written straight in the page.

Takes Canvas3DProps's props.

Hdr

Bevy's Hdr: put it inside the to render it to a float target, tone-mapped after, as Bevy's Hdr camera is (a Skybox's or an emissive's light above 1 then rolls off in the tone mapping instead of clipping).

Bevy: bevy_camera::components::Hdr.

Used in: 3d/rotate_environment_map.

Matchmove

Footage with real 3D standing in it: the clip behind, its camera solved and followed, the subject in front. Fills the frame (its camera's view is the frame's); its children are the 3D in the shot's world, lights included.

Takes MatchmoveProps's props.

TextureView

the GPU texture's number

Bevy: bevy_render::TextureView.

PropTypeWhat it does
texture (required)numberthe id a system lists the Image under in the App's PageTextures

Also takes the props of ImgHTMLAttributes (height, alt, crossOrigin, decoding, fetchPriority, loading, referrerPolicy, sizes, srcSet, useMap, width); HTMLAttributes (slot, style, title, hidden, color, content, part, className, id, defaultChecked, defaultValue, suppressContentEditableWarning, suppressHydrationWarning, accessKey, autoCapitalize, autoFocus, contentEditable, contextMenu, dir, draggable, enterKeyHint, lang, nonce, spellCheck, tabIndex, translate, radioGroup, role, about, datatype, inlist, prefix, property, rel, resource, rev, typeof, vocab, autoCorrect, autoSave, itemProp, itemScope, itemType, itemID, itemRef, results, security, unselectable, popover, popoverTargetAction, popoverTarget, inert, inputMode, is, exportparts); DOMAttributes (children, dangerouslySetInnerHTML, onCopy, onCopyCapture, onCut, onCutCapture, onPaste, onPasteCapture, onCompositionEnd, onCompositionEndCapture, onCompositionStart, onCompositionStartCapture, onCompositionUpdate, onCompositionUpdateCapture, onFocus, onFocusCapture, onBlur, onBlurCapture, onChange, onChangeCapture, onBeforeInput, onBeforeInputCapture, onInput, onInputCapture, onReset, onResetCapture, onSubmit, onSubmitCapture, onInvalid, onInvalidCapture, onLoad, onLoadCapture, onError, onErrorCapture, onKeyDown, onKeyDownCapture, onKeyPress, onKeyPressCapture, onKeyUp, onKeyUpCapture, onAbort, onAbortCapture, onCanPlay, onCanPlayCapture, onCanPlayThrough, onCanPlayThroughCapture, onDurationChange, onDurationChangeCapture, onEmptied, onEmptiedCapture, onEncrypted, onEncryptedCapture, onEnded, onEndedCapture, onLoadedData, onLoadedDataCapture, onLoadedMetadata, onLoadedMetadataCapture, onLoadStart, onLoadStartCapture, onPause, onPauseCapture, onPlay, onPlayCapture, onPlaying, onPlayingCapture, onProgress, onProgressCapture, onRateChange, onRateChangeCapture, onSeeked, onSeekedCapture, onSeeking, onSeekingCapture, onStalled, onStalledCapture, onSuspend, onSuspendCapture, onTimeUpdate, onTimeUpdateCapture, onVolumeChange, onVolumeChangeCapture, onWaiting, onWaitingCapture, onAuxClick, onAuxClickCapture, onClick, onClickCapture, onContextMenu, onContextMenuCapture, onDoubleClick, onDoubleClickCapture, onDrag, onDragCapture, onDragEnd, onDragEndCapture, onDragEnter, onDragEnterCapture, onDragExit, onDragExitCapture, onDragLeave, onDragLeaveCapture, onDragOver, onDragOverCapture, onDragStart, onDragStartCapture, onDrop, onDropCapture, onMouseDown, onMouseDownCapture, onMouseEnter, onMouseLeave, onMouseMove, onMouseMoveCapture, onMouseOut, onMouseOutCapture, onMouseOver, onMouseOverCapture, onMouseUp, onMouseUpCapture, onSelect, onSelectCapture, onTouchCancel, onTouchCancelCapture, onTouchEnd, onTouchEndCapture, onTouchMove, onTouchMoveCapture, onTouchStart, onTouchStartCapture, onPointerDown, onPointerDownCapture, onPointerMove, onPointerMoveCapture, onPointerUp, onPointerUpCapture, onPointerCancel, onPointerCancelCapture, onPointerEnter, onPointerLeave, onPointerOver, onPointerOverCapture, onPointerOut, onPointerOutCapture, onGotPointerCapture, onGotPointerCaptureCapture, onLostPointerCapture, onLostPointerCaptureCapture, onScroll, onScrollCapture, onScrollEnd, onScrollEndCapture, onWheel, onWheelCapture, onAnimationStart, onAnimationStartCapture, onAnimationEnd, onAnimationEndCapture, onAnimationIteration, onAnimationIterationCapture, onToggle, onBeforeToggle, onTransitionCancel, onTransitionCancelCapture, onTransitionEnd, onTransitionEndCapture, onTransitionRun, onTransitionRunCapture, onTransitionStart, onTransitionStartCapture); AriaAttributes (aria-activedescendant, aria-atomic, aria-autocomplete, aria-braillelabel, aria-brailleroledescription, aria-busy, aria-checked, aria-colcount, aria-colindex, aria-colindextext, aria-colspan, aria-controls, aria-current, aria-describedby, aria-description, aria-details, aria-disabled, aria-dropeffect, aria-errormessage, aria-expanded, aria-flowto, aria-grabbed, aria-haspopup, aria-hidden, aria-invalid, aria-keyshortcuts, aria-label, aria-labelledby, aria-level, aria-live, aria-modal, aria-multiline, aria-multiselectable, aria-orientation, aria-owns, aria-placeholder, aria-posinset, aria-pressed, aria-readonly, aria-relevant, aria-required, aria-roledescription, aria-rowcount, aria-rowindex, aria-rowindextext, aria-rowspan, aria-selected, aria-setsize, aria-sort, aria-valuemax, aria-valuemin, aria-valuenow, aria-valuetext).

Hooks

useMatchmove

The solve and this frame's camera, inside a .

TS
function useMatchmove(): MatchmoveState

Functions

checkMatchmove

Checks a Matchmove's props; throws a TypeError or RangeError naming the prop.

TS
function checkMatchmove(p: MatchmoveProps): void

matchmoveIndex

The solve's frame at a time in the clip: the latest at or before it, clamped to the solve.

TS
function matchmoveIndex(solve: MatchmoveSolve, time: number): number

matchmoveView

The footage's view in a frame of another shape: the whole view's size (the footage scaled to cover the frame, times zoom) and the frame's window in it, in frame pixels.

TS
function matchmoveView(footage: { width: number; height: number; }, frame: { width: number; height: number; }, crop?: number | [number, number], zoom?: number): { width: number; height: number; x: number; y: number; }

Types

Camera3dProps

The props of Camera3d, tabled there.

CameraViewport

Bevy's Viewport on a Camera, its depth range Bevy's default 0..1.

FieldTypeWhat it is
physical_position (required)[number, number]the viewport's top-left corner, in px
physical_size (required)[number, number]the viewport's width and height, in px

Canvas3DProps

FieldTypeWhat it is
layeredbooleanthe 3D takes its place in the page's tree order instead of lying under the whole page: page elements before the Canvas3D (and the backgrounds of the elements holding it) are under the 3D, showing wherever it has nothing, and hidden by its parts; positioned elements after it are over it. The 3D's background is transparent. Put the Canvas3D straight in a full-frame AbsoluteFill.
clearColorstringonly with layered: the 3D clears to this CSS colour instead of transparent, so its translucent parts blend over it in linear light, as Bevy's do over its ClearColor (a Bevy 2D scene's rgb(43, 44, 47)), not over the page in sRGB. The 3D then lies under the whole page, as Bevy's camera output lies under all its UI (a GlobalZIndex(-1) too).
childrenReactNodethe 3D scene: cameras, lights and parts

ClearColorConfig

Bevy's ClearColorConfig.

Bevy: bevy_camera::clear_color::ClearColorConfig.

TS
type ClearColorConfig = "Default" | "None" | { Custom: Srgba | LinearRgba }

MatchmoveCamera

FieldTypeWhat it is
t (required)numberseconds into the clip
position (required)Vec3where the camera is, in the solved world (metres)
quaternion (required)[number, number, number, number]camera to world, [x, y, z, w]; the camera looks down its -Z with +Y up
solved (required)booleanfalse: COLMAP didn't register this frame and its pose was interpolated

MatchmoveProps

FieldTypeWhat it is
src (required)stringthe footage: staticFile("clip.mp4")
fromnumberwhere in the clip the shot starts, and how long it runs (seconds; 0 = to the end)
secondsnumberhow long the shot runs, seconds (0: to the clip's end)
subject[number, number, number, number]the subject to keep in front of the 3D and out of the solve: a box around it on the first frame, [x, y, w, h] in 0..1 of the frame (EdgeTAM tracks it from there)
heightnumberthe camera's height over the ground in metres (1.6 by default: handheld)
fovnumberthe vertical field of view in degrees, for a shot without parallax (a tripod, a pan); a solve with structure finds its own
smoothnumberframes of smoothing on the solved path (1 by default; 0 for none)
solveMatchmoveSolvea solve to draw from instead of solving (raea track CLIP -o solve.json, imported)
cropnumber | [number, number]when the frame's shape isn't the footage's (9:16 from 16:9), where the frame sits across it: 0 its left edge, 1 its right, 0.5 the middle (the default); [x, y] for both axes. It may change from frame to frame
zoomnumber1 or more: a tighter window on the footage (a punch-in), still locked
cutoutbooleandraw the subject's cut-out over the 3D (true by default when there's a subject)
toneMapping"none" | "tony" | "aces" | "agx" | "neutral" | "reinhard"the scene's tone mapping: "none" by default, which leaves the footage's colours as they are
childrenReactNodethe 3D that stands in the shot, in the solved world

MatchmoveSolve

What raea track writes and draws from.

FieldTypeWhat it is
mode (required)"sfm" | "rotation" | "static""sfm": a full solve with structure; "rotation": the camera only turns (too little parallax, or too few frames registered); "static": it doesn't move
width (required)numberthe footage's size in pixels, and its frame rate
height (required)numberthe footage's height in pixels
fps (required)numberthe footage's frame rate
from (required)numberwhere in the clip the solve starts, seconds
seconds (required)numberhow long it runs, seconds
frames (required)numberhow many frames it solved
fov (required)numbervertical field of view, degrees; focal length in pixels; the lens's radial distortion
focal (required)numberthe focal length in pixels, at the footage's full size
k1 (required)numberthe lens's radial distortion
registered (required)numberhow many frames COLMAP registered; the rest were interpolated
error (required)number | nullmean reprojection error in pixels (null without structure)
camera_height (required)numberthe camera's height over the ground, metres, that set the world's scale
cameras (required)MatchmoveCamera[]the camera a frame
points (required)Vec3[]the sparse points, in the solved world
ground (required){ y: number; normal: Vec3 }the ground plane: its height (0) and normal (+Y)
cut (required)string | nullthe subject's cut-outs, a path with %04d for the frame (1-based), or null
notes (required)string[]what the solver noticed: fallbacks, an assumed field of view, interpolated frames

MatchmoveState

Each frame's state under a , for the 3D inside it.

FieldTypeWhat it is
solve (required)MatchmoveSolvethe solve it draws from
index (required)numberthe solve's frame shown now (0-based), and the clip's time
time (required)numberthe clip's time at this frame, seconds
camera (required)MatchmoveCamerathe solve's camera for this frame
project (required)(p: Vec3) => [number, number, number]where a point of the solved world lands in the composition, in pixels ([x, y], and its distance in front of the camera; behind it when that's 0 or less)

Projection

Bevy's Projection enum, as it reflects: fov in radians (Bevy's default PI/4).

TS
type Projection = | { Perspective: { fov?: number; near?: number; far?: number } } | { Orthographic: { near?: number; far?: number; scaling_mode?: { FixedVertical: { viewport_height: number } } } }