raea 0.1.5

The Index release: every picture and video you have, searchable by what's in it, a new app with the chat first, /goal, and three.js and React Three Fiber running on raea's GPU.

View as Markdown

raea 0.1.5 is the Index release. Add your folders, your drives and Apple Photos, and every picture and video in them becomes searchable by what's in it, on your Mac, while you watch the count climb. On Pro and Team your other Macs search it too. It also brings a new app that puts the chat first, with your projects in a rail and less on every screen. You also get /goal for an agent that keeps going until the job is done, a menu on every picture and video, and replies that reach the agent working in your terminal.

Get the app at raea.io (a Mac with Apple silicon, Windows 10 or 11, or Linux on x86_64).

The engine alone:

Terminal
curl -fsSL https://raea.io/install | sh

On Windows, in PowerShell: irm https://raea.io/install.ps1 | iex.

What changed since 0.1.4

Index: everything you've shot, searchable by words. Search, under Everything in the rail, lists the places your pictures live, and Connect brings in another:

  • the usual folders in your home (Pictures, Desktop and the rest), iCloud Drive, any drive you plug in, or any folder you pick;
  • Apple Photos, which means your iCloud Photos and your iPhone's camera roll too. macOS asks you once.

Each picture goes in from the small copy the file or Photos already keeps, embedded by SigLIP 2 on your Mac's Neural Engine. On an M5 Pro, 10,000 pictures were searchable less than a minute after the Index started, about 180 a second, and the first ones in under half a second. The page shows the live count, the rate and the pictures as they go in. Search works while a folder is still going in: type "a red car at night" and the matches come back in under 20 milliseconds. A picture can be the search too: press the corner of any picture for more like it.

A video goes in as a few of its frames: one for every 4 seconds, at most 16, taken from the keyframes it already has, so search finds the moment and not just the file. 47 half-minute clips (23 minutes of video) went in as 371 frames in 2.2 seconds.

On Free nothing leaves your Mac, and nothing is downloaded. Photos is read from the small copies it keeps here, so a library whose originals live in iCloud goes in as it is. A file in iCloud Drive that's only in the cloud goes in from the thumbnail macOS already makes for it (Quick Look), after the folder's local files: about two a second, and the file stays in the cloud.

On Pro and Team, the Index also keeps each picture's small thumbnail and its vector in raea's cloud, never the file, so a search on any of your Macs finds what's on the others too. Each match says which Mac it's on, and a picture that's on both, like one from iCloud Photos, shows once. The thumbnails count toward your storage, about 20 KB each. Take a folder out of the Index and its pictures leave the cloud too.

Google Drive works without its app too. Under Connect, pick files in Google's own picker and they go in from the thumbnails Drive already makes for them, each one searchable about a second after you press Select. Nothing is downloaded, no picture passes through raea's servers, and a match opens the file in Drive. It works on the Free plan.

Google Drive, Dropbox, OneDrive and Box are places of their own under Connect when their app is on your Mac. The files that app keeps here go in at full speed; the ones kept online only are counted on the row and left out for now.

Tell your agents what each library is for. Each place on the Search page takes one sentence in your words, like "Client footage. Read only: never edit or move it." Pick where agents save finished work, too. Every chat starts with those words, and What agents read shows you exactly what they get.

From the terminal, and for your agents: raea index add FOLDER (or photos:), raea index search WORDS, raea index status and raea index remove. The skill teaches agents to reach for raea index search when you say "that photo of…" or "the clip where…".

A new app: the chat first.

  • Your projects are in a rail down the left. Each one wears a 3D emoji, picked from all 4,007 of Noto's emoji and tinted the project's colour or any colour you pick, and it turns toward your pointer. Drag one project onto another to put it inside.
  • A project opens to its chat, with its feed beside it. Source, Tasks and Controllers are tabs in the same panel, and each has a search. Source shows the project's files as cards.
  • An item opens full size, and Back takes you to where you were.
  • Every screen has less on it. A film has one row under the picture and the timeline is its transport. The timeline only shows the tools you can use right now.
  • Search sits under Everything in the rail. One field finds a project, a piece of work, or any picture or video by what's in it.
  • Publish is one page: your posts and your review links together, the last 30 days' numbers at the top, what needs you (a post to retry, notes to answer, a post ready to schedule) first, and one search across all of it. Share a piece from wherever you find it.
  • An agent wears its AI's icon, or an emoji you give it. The shells and monitors a chat has running sit over its box, so you can see what's still going.
  • A workspace can be renamed or deleted: Edit workspace, in the workspace's menu at the top of the rail. Its folder is renamed to match, and a deleted one goes to the Trash, where you can put it back.

/goal. Type /goal and a condition in any chat, and raea asks the agent to check it each time it stops, up to 40 times, until the agent answers that the goal is met. /goal clear drops it and /goal resume picks it back up. The goal sits over the composer while it's open.

Notes become a plan. A round of notes on an item is one to-do until the agent plans it. Then the agent's own tasks take its place, each pointing back at the note it answers.

A menu on every picture and video. Right-click any picture or video in the app: Show in Finder, Share…, Export… or Copy. Copy works the way Finder's Copy does, and Share opens macOS's share menu.

Replies reach the agent in your terminal. When a Claude Code session is working on a project in its terminal, a message you send from the feed goes to that session instead of starting a copy of it. Two agents no longer edit the same project at once. The message reads "Sent", then "Read" once the session has it, and the answer comes back in its own bubble.

For builders.

  • The raea skill teaches three.js and React Three Fiber. An R3F scene goes in <R3F>, where useFrame, refs and drei work, and a plain three.js page goes in <Three program>. The skill's 3d/r3f.md lists what draws and what doesn't yet.
  • A plain three.js program runs as it is. <Three program={fn}> runs a single-file demo, and the demo's own new THREE.WebGLRenderer() (or three/webgpu's renderer) is raea's, with no raea import. Frames are stepped and seeded, so a frame comes out the same played or rendered alone. Against three.js in Chrome it scores SSIM 0.9921.
  • Three.js scenes run faster on raea's GPU. An InstancedMesh's moving copies are drawn as one set, each copy's colour included: 2,000 moving copies take 25.5 ms a frame (0.1.4: 94 ms) and 20,000 take 41.9 ms (0.1.4: 1,220 ms).
  • Three's TSL node materials are lit by the scene's lights, and TSL compute and storage buffers run in the engine, so a particle system written for three/webgpu runs as it is. Against three.js itself, a lit TSL material scores SSIM 0.9906 and TSL particles 0.9799.
  • Render targets an app draws itself, like a useFBO or GPUComputationRenderer's ping-pong, are drawn on raea's GPU and read back by the materials that use them. Both score above 0.99 SSIM against three.js (in-fbo 0.9941, in-pingpong 0.9949), and drei's MeshReflectorMaterial reflects on raea's GPU (0.9256).
  • Materials patched through onBeforeCompile draw. drei's MeshWobbleMaterial, MeshDistortMaterial and MeshReflectorMaterial, and an app's own vertex displacement, run three's own program with the patch applied, lit by raea: SSIM 0.9717 against three.js.
  • three/webgpu's PostProcessing is drawn as raea's own effects: bloom, GTAO, depth of field, FXAA, SMAA and TRAA, pixelation, film, dot screen, RGB shift, chromatic aberration, LUT, sepia and bleach, with the output's tone mapping and exposure. Against three.js, a bloom chain scores SSIM 0.9737, and sepia with chromatic aberration 0.9233. raea's own <Bloom> gains additive: three's glow added on top, in place of Bevy's energy-conserving mix.
  • A three.js ShaderMaterial under a postprocessing composer matches three's colour. raea drew it darker and more saturated, outside the composer's tone mapping: a test ball's mean colour went from 133,66,33 to 186,135,96, against three's 194,145,110 (SSIM 0.9758 to 0.9848). A composer that raea draws as its own effects no longer draws the scene again either: one bloom added 17 hidden GPU passes, and now adds none.
  • An <R3F>, or raea's own camera, in a box smaller than the page is framed by its own box, as R3F's <Canvas> is in a browser. 0.1.4 drew the page's view cropped to the box (#130). A plane meant to be 80 px in its box now measures 80 px at 600x600 (0.1.4: 120 px) and at 1080x1920 (0.1.4: missing). The new <R3F width height> gives R3F its size, and an orthographic camera the box's.
  • In a live play, an <R3F> gets R3F's pointer events (onClick, onPointerOver and Out, Move, Down, Up, onWheel and onPointerMissed), and drei's <OrbitControls> turn and zoom the camera. They're read from the play's input frame by frame, so raea render --input with the play's log draws the same frames. An <R3F> in a box takes the pointer in its own box when it's given width and height.
  • An edit that uses React Three Fiber now plays live. 0.1.4 couldn't play one at all, and an exported game that uses it plays too.
  • A render no longer waits forever on a frame that never arrives. If one of its runtimes drops a frame, the render asks for it again and says so. If the frame still never comes, the render fails with an error that names the frame and says what the runtime was doing. Thanks @Ragulsundaram (#158).

Known

  • Google Drive goes in by the files you pick (up to 200 at a time; a picked folder isn't opened, so pick the files in it) or through Google Drive for desktop's folder. Reading a whole Drive without picking isn't built.
  • Online-only files in the folders that Google Drive for desktop, Dropbox, OneDrive and Box keep on your Mac are left out and counted.
  • Searching the Index from raea.io or the iPad isn't built yet: on Pro and Team it's searched from your Macs.
  • The Index's fast pass runs on a Mac. On Windows and Linux, raea index add lists a folder and stops there.
  • What you say a library is for stays on this Mac for now: a teammate's agents don't read it yet.
  • R3F's pointer input has no double click yet, and touches don't arrive as pointers.
  • A game whose timeline is baked for the player can't have an <R3F> in that timeline.
  • postprocessing's Outline, TextureEffect, DepthEffect and LUT1D, and TSL's OutlineNode, are skipped, with one warning that names them.
  • A scene's background under an EffectComposer's or PostProcessing's tone mapping is shown as given, where three tone-maps it.
  • Drawn characters (the Character SDK) moved to 0.1.6.
  • The Windows app is unsigned, as before: Windows asks once whether to run it.
  • A proxy set by an automatic configuration (PAC) file isn't read. If yours uses one, start raea with HTTPS_PROXY set.