raea/camera
Cameras and the views they make: Camera3d, the Canvas3D a 2D scene draws in, HDR and matchmoved cameras.
<Camera3d> is Bevy's 3D camera: a transform, a projection and what it renders to. <Canvas3D> is the stage a
Bevy-style 2D scene draws in, with Bevy's coordinates (the origin at the centre, +Y up, a unit a pixel).
<Matchmove> drives a camera from the solve raea track makes, so 3D stands in filmed footage.
import { Camera3d } from "raea/camera";
import { Transform } from "raea/transform";
const Shot = () => <Camera3d transform={Transform.fromXyz(-2, 2.5, 5).lookingAt([0, 0, 0])} projection={{ Perspective: { fov: Math.PI / 4 } }} />;import { Camera3d, Canvas3D, checkMatchmove, Hdr } from "raea/camera"raea/camera exports 5 components, 1 hook, 3 functions, 9 types. Bevy's counterpart: bevy_camera. Source: sdk/raea/src/camera.ts.
Components
Camera3d
Bevy's Camera3d with its Transform and Projection: it looks along its -Z, +Y up, as Bevy's cameras do.
Bevy: bevy_camera::components::Camera3d, bevy_camera::camera::Camera, bevy_camera::projection::Projection.
| Prop | Type | What it does |
|---|---|---|
transform | TransformValue | where the camera stands and faces (Transform.fromXYZ(...).lookingAt(...)) |
projection | Projection | perspective (the default) or orthographic, as Bevy's Projection |
layers | number[] | Bevy's RenderLayers on the camera: it draws only the parts on these layers (a Mesh3d's layers; [0] by default) |
order | number | Bevy's Camera order: a camera with a higher one draws after (over) those with lower; 0 by default |
clear_color | ClearColorConfig | Bevy's ClearColorConfig: "Default" (the clear colour), "None" (drawn over what the cameras before it drew, as a second pass), or { Custom: colour } |
viewport | CameraViewport | Bevy's Viewport: the camera draws into this region of the window only, physical_position from its top left and physical_size big (whole px, at a scale factor of 1) |
children | ReactNode | components on the camera: post effects, tone mapping, a skybox |
Used in: ui/render_ui_to_texture, ui/widgets/viewport_node, 3d/fog_volumes, 3d/fog, 3d/vertex_colors, 3d/rotate_environment_map, 3d/transparency_3d, 3d/spherical_area_lights, and 60 more.
Canvas3D
Only for layered: the 3D is drawn in the page's order, at the box of the nearest positioned element around it
(the whole frame when there's none; B3 H7), at that box's size and aspect:
Takes Canvas3DProps's props.
Hdr
Bevy's Hdr: put it inside the
Bevy: bevy_camera::components::Hdr.
Used in: 3d/rotate_environment_map.
Matchmove
Footage with real 3D standing in it: the clip behind, its camera solved and followed, the subject in front. Fills the frame (its camera's view is the frame's); its children are the 3D in the shot's world, lights included.
Takes MatchmoveProps's props.
TextureView
the GPU texture's number
Bevy: bevy_render::TextureView.
| Prop | Type | What it does |
|---|---|---|
texture (required) | number | the id a system lists the Image under in the App's PageTextures |
Also takes the props of ImgHTMLAttributes (height, alt, crossOrigin, decoding, fetchPriority, loading, referrerPolicy, sizes, srcSet, useMap, width); HTMLAttributes (slot, style, title, hidden, color, content, part, className, id, defaultChecked, defaultValue, suppressContentEditableWarning, suppressHydrationWarning, accessKey, autoCapitalize, autoFocus, contentEditable, contextMenu, dir, draggable, enterKeyHint, lang, nonce, spellCheck, tabIndex, translate, radioGroup, role, about, datatype, inlist, prefix, property, rel, resource, rev, typeof, vocab, autoCorrect, autoSave, itemProp, itemScope, itemType, itemID, itemRef, results, security, unselectable, popover, popoverTargetAction, popoverTarget, inert, inputMode, is, exportparts); DOMAttributes (children, dangerouslySetInnerHTML, onCopy, onCopyCapture, onCut, onCutCapture, onPaste, onPasteCapture, onCompositionEnd, onCompositionEndCapture, onCompositionStart, onCompositionStartCapture, onCompositionUpdate, onCompositionUpdateCapture, onFocus, onFocusCapture, onBlur, onBlurCapture, onChange, onChangeCapture, onBeforeInput, onBeforeInputCapture, onInput, onInputCapture, onReset, onResetCapture, onSubmit, onSubmitCapture, onInvalid, onInvalidCapture, onLoad, onLoadCapture, onError, onErrorCapture, onKeyDown, onKeyDownCapture, onKeyPress, onKeyPressCapture, onKeyUp, onKeyUpCapture, onAbort, onAbortCapture, onCanPlay, onCanPlayCapture, onCanPlayThrough, onCanPlayThroughCapture, onDurationChange, onDurationChangeCapture, onEmptied, onEmptiedCapture, onEncrypted, onEncryptedCapture, onEnded, onEndedCapture, onLoadedData, onLoadedDataCapture, onLoadedMetadata, onLoadedMetadataCapture, onLoadStart, onLoadStartCapture, onPause, onPauseCapture, onPlay, onPlayCapture, onPlaying, onPlayingCapture, onProgress, onProgressCapture, onRateChange, onRateChangeCapture, onSeeked, onSeekedCapture, onSeeking, onSeekingCapture, onStalled, onStalledCapture, onSuspend, onSuspendCapture, onTimeUpdate, onTimeUpdateCapture, onVolumeChange, onVolumeChangeCapture, onWaiting, onWaitingCapture, onAuxClick, onAuxClickCapture, onClick, onClickCapture, onContextMenu, onContextMenuCapture, onDoubleClick, onDoubleClickCapture, onDrag, onDragCapture, onDragEnd, onDragEndCapture, onDragEnter, onDragEnterCapture, onDragExit, onDragExitCapture, onDragLeave, onDragLeaveCapture, onDragOver, onDragOverCapture, onDragStart, onDragStartCapture, onDrop, onDropCapture, onMouseDown, onMouseDownCapture, onMouseEnter, onMouseLeave, onMouseMove, onMouseMoveCapture, onMouseOut, onMouseOutCapture, onMouseOver, onMouseOverCapture, onMouseUp, onMouseUpCapture, onSelect, onSelectCapture, onTouchCancel, onTouchCancelCapture, onTouchEnd, onTouchEndCapture, onTouchMove, onTouchMoveCapture, onTouchStart, onTouchStartCapture, onPointerDown, onPointerDownCapture, onPointerMove, onPointerMoveCapture, onPointerUp, onPointerUpCapture, onPointerCancel, onPointerCancelCapture, onPointerEnter, onPointerLeave, onPointerOver, onPointerOverCapture, onPointerOut, onPointerOutCapture, onGotPointerCapture, onGotPointerCaptureCapture, onLostPointerCapture, onLostPointerCaptureCapture, onScroll, onScrollCapture, onScrollEnd, onScrollEndCapture, onWheel, onWheelCapture, onAnimationStart, onAnimationStartCapture, onAnimationEnd, onAnimationEndCapture, onAnimationIteration, onAnimationIterationCapture, onToggle, onBeforeToggle, onTransitionCancel, onTransitionCancelCapture, onTransitionEnd, onTransitionEndCapture, onTransitionRun, onTransitionRunCapture, onTransitionStart, onTransitionStartCapture); AriaAttributes (aria-activedescendant, aria-atomic, aria-autocomplete, aria-braillelabel, aria-brailleroledescription, aria-busy, aria-checked, aria-colcount, aria-colindex, aria-colindextext, aria-colspan, aria-controls, aria-current, aria-describedby, aria-description, aria-details, aria-disabled, aria-dropeffect, aria-errormessage, aria-expanded, aria-flowto, aria-grabbed, aria-haspopup, aria-hidden, aria-invalid, aria-keyshortcuts, aria-label, aria-labelledby, aria-level, aria-live, aria-modal, aria-multiline, aria-multiselectable, aria-orientation, aria-owns, aria-placeholder, aria-posinset, aria-pressed, aria-readonly, aria-relevant, aria-required, aria-roledescription, aria-rowcount, aria-rowindex, aria-rowindextext, aria-rowspan, aria-selected, aria-setsize, aria-sort, aria-valuemax, aria-valuemin, aria-valuenow, aria-valuetext).
Hooks
useMatchmove
The solve and this frame's camera, inside a
function useMatchmove(): MatchmoveStateFunctions
checkMatchmove
Checks a Matchmove's props; throws a TypeError or RangeError naming the prop.
function checkMatchmove(p: MatchmoveProps): voidmatchmoveIndex
The solve's frame at a time in the clip: the latest at or before it, clamped to the solve.
function matchmoveIndex(solve: MatchmoveSolve, time: number): numbermatchmoveView
The footage's view in a frame of another shape: the whole view's size (the footage scaled to cover the frame, times zoom) and the frame's window in it, in frame pixels.
function matchmoveView(footage: { width: number; height: number; }, frame: { width: number; height: number; }, crop?: number | [number, number], zoom?: number): { width: number; height: number; x: number; y: number; }Types
Camera3dProps
The props of Camera3d, tabled there.
CameraViewport
Bevy's Viewport on a Camera, its depth range Bevy's default 0..1.
| Field | Type | What it is |
|---|---|---|
physical_position (required) | [number, number] | the viewport's top-left corner, in px |
physical_size (required) | [number, number] | the viewport's width and height, in px |
Canvas3DProps
| Field | Type | What it is |
|---|---|---|
layered | boolean | the 3D takes its place in the page's tree order instead of lying under the whole page: page elements before the Canvas3D (and the backgrounds of the elements holding it) are under the 3D, showing wherever it has nothing, and hidden by its parts; positioned elements after it are over it. The 3D's background is transparent. Put the Canvas3D straight in a full-frame AbsoluteFill. |
clearColor | string | only with layered: the 3D clears to this CSS colour instead of transparent, so its translucent parts blend over it in linear light, as Bevy's do over its ClearColor (a Bevy 2D scene's rgb(43, 44, 47)), not over the page in sRGB. The 3D then lies under the whole page, as Bevy's camera output lies under all its UI (a GlobalZIndex(-1) too). |
children | ReactNode | the 3D scene: cameras, lights and parts |
ClearColorConfig
Bevy's ClearColorConfig.
Bevy: bevy_camera::clear_color::ClearColorConfig.
type ClearColorConfig = "Default" | "None" | { Custom: Srgba | LinearRgba }MatchmoveCamera
| Field | Type | What it is |
|---|---|---|
t (required) | number | seconds into the clip |
position (required) | Vec3 | where the camera is, in the solved world (metres) |
quaternion (required) | [number, number, number, number] | camera to world, [x, y, z, w]; the camera looks down its -Z with +Y up |
solved (required) | boolean | false: COLMAP didn't register this frame and its pose was interpolated |
MatchmoveProps
| Field | Type | What it is |
|---|---|---|
src (required) | string | the footage: staticFile("clip.mp4") |
from | number | where in the clip the shot starts, and how long it runs (seconds; 0 = to the end) |
seconds | number | how long the shot runs, seconds (0: to the clip's end) |
subject | [number, number, number, number] | the subject to keep in front of the 3D and out of the solve: a box around it on the first frame, [x, y, w, h] in 0..1 of the frame (EdgeTAM tracks it from there) |
height | number | the camera's height over the ground in metres (1.6 by default: handheld) |
fov | number | the vertical field of view in degrees, for a shot without parallax (a tripod, a pan); a solve with structure finds its own |
smooth | number | frames of smoothing on the solved path (1 by default; 0 for none) |
solve | MatchmoveSolve | a solve to draw from instead of solving (raea track CLIP -o solve.json, imported) |
crop | number | [number, number] | when the frame's shape isn't the footage's (9:16 from 16:9), where the frame sits across it: 0 its left edge, 1 its right, 0.5 the middle (the default); [x, y] for both axes. It may change from frame to frame |
zoom | number | 1 or more: a tighter window on the footage (a punch-in), still locked |
cutout | boolean | draw the subject's cut-out over the 3D (true by default when there's a subject) |
toneMapping | "none" | "tony" | "aces" | "agx" | "neutral" | "reinhard" | the scene's tone mapping: "none" by default, which leaves the footage's colours as they are |
children | ReactNode | the 3D that stands in the shot, in the solved world |
MatchmoveSolve
What raea track writes and
| Field | Type | What it is |
|---|---|---|
mode (required) | "sfm" | "rotation" | "static" | "sfm": a full solve with structure; "rotation": the camera only turns (too little parallax, or too few frames registered); "static": it doesn't move |
width (required) | number | the footage's size in pixels, and its frame rate |
height (required) | number | the footage's height in pixels |
fps (required) | number | the footage's frame rate |
from (required) | number | where in the clip the solve starts, seconds |
seconds (required) | number | how long it runs, seconds |
frames (required) | number | how many frames it solved |
fov (required) | number | vertical field of view, degrees; focal length in pixels; the lens's radial distortion |
focal (required) | number | the focal length in pixels, at the footage's full size |
k1 (required) | number | the lens's radial distortion |
registered (required) | number | how many frames COLMAP registered; the rest were interpolated |
error (required) | number | null | mean reprojection error in pixels (null without structure) |
camera_height (required) | number | the camera's height over the ground, metres, that set the world's scale |
cameras (required) | MatchmoveCamera[] | the camera a frame |
points (required) | Vec3[] | the sparse points, in the solved world |
ground (required) | { y: number; normal: Vec3 } | the ground plane: its height (0) and normal (+Y) |
cut (required) | string | null | the subject's cut-outs, a path with %04d for the frame (1-based), or null |
notes (required) | string[] | what the solver noticed: fallbacks, an assumed field of view, interpolated frames |
MatchmoveState
Each frame's state under a
| Field | Type | What it is |
|---|---|---|
solve (required) | MatchmoveSolve | the solve it draws from |
index (required) | number | the solve's frame shown now (0-based), and the clip's time |
time (required) | number | the clip's time at this frame, seconds |
camera (required) | MatchmoveCamera | the solve's camera for this frame |
project (required) | (p: Vec3) => [number, number, number] | where a point of the solved world lands in the composition, in pixels ([x, y], and its distance in front of the camera; behind it when that's 0 or less) |
Projection
Bevy's Projection enum, as it reflects: fov in radians (Bevy's default PI/4).
type Projection = | { Perspective: { fov?: number; near?: number; far?: number } } | { Orthographic: { near?: number; far?: number; scaling_mode?: { FixedVertical: { viewport_height: number } } } }