One Queue, Many Tools

The relay race we only half-deleted

SpaceMusic can already generate hundreds of variations overnight without anyone in the room. They land in a web feed by morning, each with a preview image, waiting for someone to scroll through and pick the good ones. That part is built and running.

What happens next is still a relay race. Someone opens the chosen file in Rhino, runs a command to turn a cloud of points into a solid shape, exports it, opens a slicer, picks a printer profile, and sends the job. The same five motions, every time, for every object. The work is real but it is identical work, which is exactly what makes it worth handing to a machine.

The interesting part is that this is mostly not a problem about Rhino. We have Rhino licences and spare Windows machines; the geometry step turns out to take under two seconds. The problem is that our job queue currently believes every machine in the pool is the same machine. Extending the Factory across several tools is really about teaching one queue that a computer with Rhino on it and a computer running SpaceMusic are different animals, and that some jobs can only go to one of them.

The temptation to build a framework

There is a standard way to approach this, and it is a trap we have already warned ourselves about twice in our own documents. The standard way is to build the general thing first: a tool registry, a plugin interface, a visual editor for chaining steps together, a capability vocabulary rich enough for tools nobody has installed yet. Then you plug in your two actual tools and discover that the abstraction fits neither.

1. Framework first. Design the generic multi-tool platform, then fit the tools into it. Reliable way to spend months on machinery for tools that do not exist. tool registry · plugin API · DAG editor · capability schema
2. One tool, hardcoded. Fast and honest, but the second tool costs a rewrite because nothing was ever separated from anything. the Factory as it stands today
3. Extend the queue that already works. Keep the dispatch engine untouched. Add one column that says what a machine can do, one column that says what a job needs, and one line of matching logic. Prove the second tool on real files before writing any abstraction over it. the position this document argues for

The third option is not modesty for its own sake. What decides whether this ships is not the elegance of an interface — it is whether headless Rhino is reliable, whether a Bambu printer will accept an unattended job after the 2025 firmware change, and whether we are willing to run other people’s code on a machine that holds our login token. A clean plugin API solves none of those.

What is already running

factory.spacemusic.tv is live and doing real work. It shows every result any run has produced, with thumbnails and downloads; it lets you mark favourites; it lists your automations; and since the Workers tab shipped it shows which of your machines are online, what they are doing, and lets you share a whole pool of machines with another user so their jobs can land on your hardware.

Behind the web page, the moving parts are few. The API server holds a small database of projects, results, batches, outcomes and workers. Object storage holds every actual file — the exports, the thumbnails, and soon the printable meshes. A worker asks the queue for a job, is granted a lease on it for two minutes, sends a heartbeat while it works, posts each result as it is produced, and reports completion. If a worker dies mid-job its lease expires and the job goes back in the queue. Six calls, and they already survive network drops, restarts and duplicate workers.

Here is the honest gap. Every worker in that pool can do exactly one thing — run a SpaceMusic scene — and the queue has no way to know or care. There is a column on the workers table called capabilities, it defaults to the text spacemusic, and nothing has ever written to it or read it. The hook was left in place deliberately and has been sitting unused ever since. That column is where this whole extension begins.

A step is a batch

The load-bearing decision is a refusal: we are not building a second dispatch system for multi-step jobs. A pipeline step is realised as an ordinary batch — the same kind of job the queue already hands out. Pipelines live one layer above, as bookkeeping that decides when to create the next batch. The worker never learns the word “pipeline”; it just claims work, does it, and reports back exactly as it does today.

That single choice inherits everything already proven: leases, heartbeats, requeue-on-expiry, per-user ownership, pool sharing, the outcome records, the results feed. None of it has to be rebuilt or reasoned about again.

What has to be added is genuinely small, and every piece has a known address in the code. A machine advertises what it can do by sending a header with each claim, which the server stores on its worker row. A job says what it needs through one new nullable column — empty means “any machine”, so every job that exists today keeps behaving identically. And the query that decides who may claim what gains one more condition alongside the ones it already checks.

Read the figure as one loop. Jobs are created in the cloud and sit in a queue. Machines in the studio reach outward and ask for work, saying what they are capable of — nothing in the studio is reachable from the internet, and no firewall hole is needed. The coral box is the whole extension: the queue already checks that a job is waiting, that no one else holds it, and that you are allowed to run it; the fourth check is new. Files stay in object storage the entire time and are handed over as short-lived signed links, so the database never carries a megabyte of geometry.

Two small details in that box matter more than they look. The capability text is compared with commas around it, so a machine advertising rhino never accidentally satisfies a job asking for rhino-lite. And because the capability check and the ownership check are separate conditions joined by and, sharing your machines with a colleague and restricting jobs to Rhino machines compose without anyone writing code for the combination.

What a tool adapter has to be

With dispatch settled, the remaining question is what it takes to teach the agent a new tool. The answer is three functions: work out whether the tool is installed and usable, check that a given job makes sense for it, and run it. Everything else — downloading inputs, sending heartbeats, honouring a stop request, cleaning up — belongs to the agent and is written once.

Two shapes of tool exist and the distinction is real. SpaceMusic is a resident engine: the agent signals a program that is already running and waits. Rhino and the slicer are spawned children: the agent starts a process, watches it, and must be able to kill it and everything it spawned. As of today’s spike, that second style is not a nicety — see the next section.

What a tool produces is already standardised and, usefully, has nothing to do with which tool made it. When a job finishes it writes the artefact plus a small export.json beside it describing what it is and where it came from. The ingest side has read that format since the scene-automation work; anything that lands a valid one gets a row in the results feed for free. A slicer writing a printable file is, from the feed’s point of view, the same event as SpaceMusic writing an export.

The coral path is the one that carries the design. Marking favourites is not a separate feature bolted on for this pipeline — the favourite flag already exists on every result and the feed already counts them per run. A pipeline step simply says “my inputs are the favourites of step 1”, and that resolves at the moment the step becomes ready into a concrete list of files. Each produced mesh records which favourite it came from, so lineage from printed object back to the scene that generated it is a lookup rather than a guess.

The final step is deliberately not symmetrical with the others. Sending a job to a printer is modelled as a queue a person confirms, because the January 2025 Bambu firmware requires authentication for third-party control and every unattended route around it is either a safety regression or a certificate the vendor can invalidate. That is a constraint we accept rather than engineer around.

What the first new tool taught us

All of the above was design until yesterday. The step that carries the risk is the geometry one, so it was built and measured first, on real exports rather than a toy file. The headline is that it is faster and cheaper than assumed.

The concurrency result changes a planning assumption. One licence served sixteen simultaneous processes without complaint, so a Rhino machine advertises real capacity rather than being a one-job-at-a-time bottleneck, and jobs do not need to be serialised or staggered. Alongside that, three assumptions written into the plan turned out to be wrong: the command-line tool we had named cannot start a headless engine at all (it only attaches to one already running); Grasshopper is not needed for this step, which removes the “we are executing someone else’s code inside our own process” risk from the anchor pipeline entirely; and the open-source server option is the same engine behind a web API, still requiring the same licence, with a request-and-response shape that fits badly against jobs ranging from two seconds to ten minutes.

Two findings constrain the agent rather than the pipeline, and both are the kind that only appear when you actually run the thing.

The first is that the geometry call cannot be cancelled and the process cannot be politely asked to exit. A cancellation request is ignored, and the normal .NET shutdown call reported an exit code to the parent while the child kept consuming a processor core and a gigabyte of memory for a further seven minutes. A supervisor that trusts a child’s exit code will quietly accumulate orphaned processes. The job-object supervision we already use for the video streamer therefore moves from “good practice” to “required”.

The second is that the agent cannot be a Windows service. Run from a scheduled task with no desktop, the engine blocks during startup and never returns — identically whether it runs as the system account or as the signed-in user with full profile and credentials, which rules out licensing as the cause. The practical consequence is concrete: the agent is a per-user application that starts when you log in, the installer must not offer a service option, and setting up a new Rhino machine includes opening Rhino once and signing in. Better still, the agent can check that for itself in nine seconds rather than trusting the person who installed it — the same probe distinguishes “not installed” from “installed but nobody has signed in” from “licensed, as this person”.

Not everything passes. Exports that are open dense meshes, and exports built from tens of thousands of repeated blocks, do not survive the geometry step — the latter expands to forty-one million faces before it begins. The useful discovery is that the dividing line is not file size but whether the shape is already closed, which is a property the adapter can test in a second and reject on cheaply, rather than discovering after ten minutes of grinding.

One last finding came for free and belongs to the ingest side rather than to Rhino. A file that failed to open turned out not to be a Rhino limitation at all: it was a truncated export, recognisable because its folder contained the geometry and none of the sidecar files a finished export always has. The rule that follows is simple and worth adopting now — treat the small export.json as the signal that an export is complete, and ignore a bare geometry file. That also removes a duplicate-entry problem the earlier print-pipeline research flagged.

Why this matters

“The geometry was never the hard part. The hard part was that a human had to be in the room — and then, that the queue could not tell one room from another.”

The ambition here is not “support many tools”. Nobody wants a tool platform. What we want is that someone opens a feed of last night’s output, taps five favourites, and the only human left in the loop is the one lifting a finished object off the print bed. Everything between those two moments is now either running in production or measured on real files.

What makes that reachable is that we are not building a new system. The queue, the leases, the results feed, the ownership rules, the sharing model and the file storage all exist and are in daily use. The extension is one column that says what a machine can do, one column that says what a job needs, one condition in a query, and a small application that sits in the tray and knows how to start two programs. The pipeline layer above it is bookkeeping, not a second engine.

What is left genuinely risky is short and worth naming plainly: sending a job to a Bambu printer unattended, and the day we let one person’s job run on another person’s machine while that machine holds a login token. Neither is solved by architecture. Both are decisions about how much we trust, and they should be made deliberately rather than inherited from a convenient default.

And a smaller lesson, worth carrying into the rest of this build. Two of the spike’s first conclusions were wrong — a tool works fine that was reported as failing, and machines run happily in parallel that were reported as unreliable. In both cases the measuring harness was broken, not the thing being measured, and the false version was confident and specific. For a platform whose entire value is doing work while nobody is watching, being able to tell a real failure from a broken measurement is not a side skill. It is the product.

Glossary

Terms and proper nouns used in this document, in plain language.

batch

One unit of work in the queue — the thing a machine claims and runs. A multi-step pipeline is built out of these rather than replacing them.

capability

A short label a machine advertises for what it can do (spacemusic, rhino, orca-slice), matched against what a job requires.

export.json

A small description file written next to every produced artefact, saying what it is and where it came from. It is what turns a file on disk into a row in the results feed, whichever tool wrote it.

Grasshopper

Rhino’s visual scripting tool. Useful for future recipes, but not needed for the shrink-wrap step.

headless

Running a program with no window and no one watching, driven entirely by a script.

lease

A two-minute reservation on a job. The machine renews it while working; if the machine dies the lease expires and the job returns to the queue.

MinIO

The file storage we run on our own server. Every export, mesh and printable file lives here; the database only ever holds a reference.

OrcaSlicer

A slicing program — turns a 3D mesh into the instructions a printer follows. It has a command-line mode, so it can be automated.

point cloud

A shape stored as a scatter of individual points rather than a solid surface. Needs shrink-wrapping before it can be printed.

Rhino.Inside

The supported way to host Rhino’s engine inside another program, which is how the geometry step runs with no window.

ShrinkWrap

The Rhino operation that turns a loose cloud of points into a solid, watertight mesh. The one genuinely Rhino-only step in the pipeline.

STL / 3MF

File formats a 3D printer’s software understands. STL is a plain triangle mesh; 3MF can also carry printer and material settings.

watertight

A mesh with no holes, so it describes a solid object rather than a surface. A printer needs this.

worker

A machine in your pool that asks the queue for jobs and runs them.