It started with a phone call

Like most of my questionable decisions, this one started on the phone with a mate, talking about technology the way we always do. The conversation drifted, as every technical conversation eventually does now, into AI. And we ended up circling the same question everyone circles: is this stuff actually capable of building real software, or is it just very confident autocomplete?

He was the sceptic. I was less sure. I’d been watching AI coding tools long enough to see signs they could do more than trivial hello-world demos — I’d even written about the right way to work with them more than a year earlier. But I don’t think either of us honestly believed the answer would be a full-featured desktop application.

So I did the thing you should never do on a phone call after 9pm: I made a declaration. I would definitively prove, one way or the other, whether AI could build a real application from scratch.

Not from a single prompt. That’s a stupid idea, and it’s a strawman both camps love: the boosters because it makes a great demo, the sceptics because it makes a great failure. I mean built the way software actually gets built: architecture, iteration, review, tests, releases. Using the pair programming approach I’d already laid out on this blog — human as navigator, AI as driver, with the conversation between them doing the real work.

If it failed, I’d write that post. It would have been an easier post to write, honestly. This is the other one.

Picking a fight worth having

If you’re going to test whether AI can build real software, the project has to be real: something with real complexity, a demanding platform, and a reason to exist beyond the experiment.

I’ve been an on-again, off-again Linux user for the best part of thirty years. The kernel was never the problem. The desktop was never really the problem either, especially lately. The thing that has always dragged me back to commercial operating systems is the application ecosystem. It’s better today than it has ever been, thanks to an enormous number of dedicated open source developers. But there are still categories where the open source options simply don’t compete with their commercial rivals. Photos. Email. The apps your family actually touches.

And that second part matters, because this wasn’t abstract for me. I’ve been trying to move my family away from Apple Photos, and self-hosted Immich is a brilliant back end for that. But my family has spent too many years inside the Apple ecosystem to accept a web interface for something as personal as their photo library. They want a real application. Native, fast, polished. The kind of thing the Linux desktop is chronically short of.

So that was the wager, fully formed: a modern GTK4/libadwaita photo management application for the GNOME desktop, written in Rust. (All the cool kids use Rust. It’s been a long time since I was a kid.)

There was one more reason this experiment appealed to me, and it’s the one I suspect resonates with most working engineers. My background is software engineering. I’ve built some large, complex, performance-critical systems for companies big and small. Building software has never been the hard part. Finding the time has. A career plus a family leaves you time-poor in exactly the way that kills side projects, and it’s why I’d never seriously committed to a sizable open source project before. If AI could compress the gap between the architecture in my head and working code on disk, the economics of contributing to open source change completely — not just for me, for every experienced engineer who’s been sitting on the sidelines for the same reason.

What it was actually like

Five months later I can tell you the answer to the wager, but the honest version of this story is the middle, not the ending. Because the first few weeks were not smooth.

The early problems were all variations on the same theme: an AI will cheerfully solve the problem in front of it with no memory of how it solved the same problem last Tuesday. I’d find duplicate implementations of the same logic in different corners of the codebase. I’d find three different patterns for the same category of problem, each locally reasonable, collectively a mess. I had to learn the right size to chunk features — too big and the AI loses the thread, too small and you spend more time briefing than you save.

But the biggest lesson was about architecture, and it’s the one I’d put on a poster. Claude’s instinct is to reach the outcome in front of it as fast as possible. It is genuinely good at that. What it does not do — unless you make it — is weigh that outcome against the long-term shape of the system. Left unsupervised, you get software that works today and calcifies by next month. Which, now that I write it down, also describes plenty of human teams I’ve worked with.

So the collaboration settled into a division of labour that will look very familiar to anyone who read my pair programming post:

• Mine: application architecture, design, and feature management. Every structural decision — the backend abstraction, the sync model, the storage layout — came from me. I was the navigator, holding the long-term picture.
• Claude’s: the code. The overwhelming majority of the actual Rust was written by the AI, driving under direction.
• Ours: the review loop. Every change went through a pull request, reviewed and audited before merge — the same discipline I’d demand on any professional team. No PR process, no experiment; it’s the review loop that makes the whole thing defensible.

Part way through I added a technique I now consider essential: adversarial review. Don’t let the AI that wrote the code be the only AI that reads it. A second model — or even the same model under a deliberately hostile prompt — catches code smells, duplication and quiet regressions far earlier than a friendly one does. The author is always too fond of its own work. That turns out to be true of machines too.

Somewhere in those months, the experiment stopped feeling like an experiment. That’s the part I didn’t expect. Within days it was apparent that working this way wasn’t just possible — it was preferred. I wasn’t tolerating the AI to prove a point. I was reaching for it because the loop of architect–direct–review–refine was producing better software, faster, than I would have produced alone in the hours I actually have.

The verdict

Here’s what the wager produced:

• ~40,000 lines of Rust, GTK4 and libadwaita
• 898 commits over five months, first commit 22 March 2026
• Every change human-reviewed before merge, through a PR workflow
• Releases through v0.4.1, shipped as Flatpak bundles with checksums, and GPG signatures on signed releases
• Built almost entirely in evenings and spare hours: the time budget of a person with a job and a family, not a funded team

Now overlay the cost lens, because this is where the wager stops being a curiosity and starts being an economic argument. The entire AI side of this project ran on a Claude Code subscription — I started on the Pro plan and very quickly determined it wasn’t enough, so most of the project was built on the Max 5x tier, at roughly AU$150 a month. During heavy development I’d regularly burn through 80–90% of my weekly token allocation, but I never went over it. So call it $150 a month, all in, for a coding partner that helped produce a 40,000-line application. That’s not cheap as subscriptions go, and I won’t pretend it’s nothing — but weighed against what it unlocked, I found it an easy cost to justify. For me, the constraint was never the money. It was only ever the time. I’ll admit I spent more evenings on this than my family would have liked while it was all coming together. Now that we have a functioning app, it’s consuming far less of them.

And here is the verdict, as plainly as I can put it: AI can absolutely build a real application — but not alone, and that’s precisely the point. The single-prompt fantasy is still a fantasy. What is real, right now, is an experienced engineer setting the architecture and standards, an AI doing the driving, and a review process keeping everyone honest. That combination settled the phone argument — and it removed the constraint that has kept me, and I suspect a great many engineers like me, out of open source for years.

I called my mate to concede that neither of us had been right. He’s yet to install a Linux desktop, so I’m calling it a draw.

Meet Moments

Enough about the wager — here’s the app.

Moments is a photo management application for the GNOME desktop. It’s built for exactly the situation my family is in: your photos live either on your own disk or on a self-hosted Immich server, and you want a fast, native way to actually browse them.

It talks to a local folder or to an Immich server. The Immich backend caches metadata and thumbnails locally in SQLite, so browsing, searching and anything you’ve already opened works fully offline; a photo you’ve never viewed before will fetch its full-resolution original on demand, then keep it in a local cache for next time. The grid is keyset-paginated with six zoom levels, and it stays smooth through large libraries.

Beyond that:

• RAW and modern formats — CR2, NEF, ARW, DNG and friends, alongside JPEG, PNG, WebP, HEIC and TIFF
• Video — import and playback via GStreamer
• Albums, People and Favourites — including face data synced from Immich
• EXIF metadata — camera, lens, exposure and GPS in the detail panel
• Non-destructive rotate, flip and crop, with adjustments and filters behind experimental flags

It’s GPL-3.0-or-later, and it’s a real daily-driver application — my family’s photo library runs on it.

Installing it takes a download and two commands. Every release ships a single-file Flatpak bundle (with checksums, and GPG signatures on signed releases):

flatpak install --user moments-<version>-x86_64.flatpak
flatpak run io.github.justinf555.Moments

Grab the latest bundle from the releases page.

The Flathub question

The obvious question is why those two commands aren’t just flatpak install flathub io.github.justinf555.Moments. That deserves a straight answer, because it’s part of this story too.

I submitted Moments to Flathub on 27 March 2026, five days after the project’s first commit — early, deliberately, so the review could run alongside development. The submission worked through the normal review cycle: manifest fixes, portal permissions, test builds passing on both architectures.

Then, at the end of May, Flathub changed its inclusion policy. The new rule is a blanket one: “Applications containing AI-generated or AI-assisted code, documentation, or any other content are not allowed.” Not AI-generated slop with no human behind it — which the previous policy already prohibited, rightly — but AI-assisted anything. Two days after the policy landed, my submission was closed. The full exchange is public, and I’d encourage you to read it and draw your own conclusions rather than take my summary of it. Read closely and you’ll notice I described the AI’s share of the codebase as “parts of the app” in that thread — more of it than I was comfortable admitting mid-argument, if I’m honest. This post is the fuller account.

I withdrew the submission, and I said my piece on the way out, which I’ll stand behind here rather than repeat at length: the old policy drew the correct line — AI-generated code with no human behind it, not allowed; AI-assisted code that a human reviewed, tested and stands behind, allowed. The new policy erases that line. It stops judging software by whether it’s good, tested and maintained, and starts judging it by how it was typed.

I understand where the policy comes from. Maintainers are drowning in low-effort AI submissions, and I have real sympathy for that — I wrote a whole post about the open source community’s fraught relationship with AI before I was ever personally on the receiving end of it. Flathub’s repo, Flathub’s rules. But I think the blanket version of the rule is a mistake, and I think this app is a concrete counterexample: working GPL software, architected and reviewed by a human, in a category the Linux desktop has needed for decades. The policy does allow exceptions for “mature, well-maintained projects”, so perhaps Moments and Flathub will meet again down the track. The door’s open on my side.

In the meantime, the Flatpak bundles work everywhere Flatpak does, and updates are one download away.

What I’d ask of you

Three things, in order:

1. Install it. Point it at a folder of photos or an Immich server and see what you think. It’s a download and two commands.
2. Star the repo. github.com/justinf555/Moments — discovery is the thing losing Flathub actually cost this project, and a star is the closest substitute GitHub offers.
3. Tell me what’s broken or missing. Issues for bugs and feature requests, Discussions for everything else. This is the part of open source no AI can do: real people, with real photo libraries, telling you what actually matters.

And if you’re an experienced engineer who’s been sitting on an idea for years because you couldn’t find the time — that’s the real takeaway of the wager. The time problem is the one that just changed. Go and settle an argument of your own.

The first six weeks of running it were a different story. Four bugs in particular are worth describing, because each one tells you something the diagrams don’t. Two are about the abstraction — assumptions the trait quietly makes about how the world works, and what breaks when the world doesn’t cooperate. Two are about the substrate — the boring queue-and-retry machinery underneath the trait, which has its own opinions about how things should be done. Clean architecture doesn’t relieve you of the substrate’s problems. It just gives you one place to deal with them.

1. The pull-delete loop

The first bug was the most embarrassing one, and the cleanest illustration of a hidden assumption in the design.

The symptom, as a user would see it: trash a photo on your phone via the Immich mobile app. Watch it disappear from the desktop a few seconds later when the next pull cycle arrives. Watch it reappear a few seconds after that. Watch it disappear again. The asset ping-pongs across the network, never settling, every thirty seconds, forever — or at least until you notice and start digging.

The mechanics: the pull-side handler for AssetDeleteV1 called MediaService::delete_permanently, which dutifully recorded an AssetDeleted mutation into the outbox, which the push manager dutifully tried to send back to the server, which had just told us the asset was gone. Server says delete → local deletes → outbox records delete → push tries to delete on server → server returns 404, push gives up. So far so harmless. But the next pull cycle then re-emits the original delete (because the server is still asserting “this asset has been deleted, here’s the event”), and we go around again.

The recorder-based design has a hidden assumption baked into it: mutations have a single source, and that source is the user sitting at the desktop. The whole point of recording mutations is to push them to the server. When the server starts producing mutations — and in a two-way sync, of course it does — the recorder happily reports those back to the server too, and the system feeds on its own output.

The fix (commit f097174) was a Library::delete_permanently_from_sync variant that takes the same code path except it skips the recorder. The _from_sync naming convention now appears wherever pull-side handlers mutate the library — there’s a _from_sync for trash, for restore, for the album mutations, and so on. Each one is a tiny duplicate of its normal counterpart, with the recorder call removed. It feels redundant, and it is, but the redundancy is the whole point: pull is the local equivalent of “don’t echo back what you were just told”, and the duplication makes the don’t-echo explicit at every call site.

There were two alternatives I considered. One was a thread-local “I am inside the sync handler” flag that the recorder would check; that’s a kind of magic that hides the directional question rather than answering it. The other was a parameter on every service method indicating origin; that pushes the question into every signature instead of into a single naming convention. The _from_sync approach won because it makes the asymmetry visible exactly where it matters and nowhere else.

The architectural lesson, stated generally: a recorder-based design implicitly assumes mutations have a single source. The moment a second source appears, you have to mark it in the code, or you get an infinite loop with a polite retry interval.

2. Retry, backoff, and the death of multi-id rows

The second bug is one of substrate, not abstraction. It would have bitten any outbox-based design, regardless of how the recorder was shaped.

The first version of the outbox stored multi-id mutations as a single row. AssetTrashed { ids: [a, b, c] } produced one outbox row with a JSON-encoded array of ids. It was simpler. The mutation type kept its natural shape — “the user trashed three photos” is a single mental event, after all — and the database had three fewer rows to deal with.

It was also wrong, in three independent ways.

The first was partial progress. The push call for that row was, internally, a loop over the ids making per-asset API calls. If the server accepted three of five ids and then the connection dropped, there was no way to record the partial success. The next retry would re-send all five, and the server’s behaviour on the already-trashed ones was implementation-defined enough that I didn’t trust it.

The second was blast radius. If a single id in the batch was the cause of a permanent failure — say, the user trashed an asset, then permanently deleted it elsewhere, then triggered a sync — the batch would fail forever on that one bad asset. The other four perfectly good operations were dragged into the dead-letter queue alongside it.

The third was per-batch backoff. Exponential backoff on the batch row meant one bad asset could starve the entire trash pipeline behind it. Asset X fails three times, the row sleeps for ten minutes, all five operations wait. Asset X fails again, the row sleeps for an hour. Operations Y and Z, which would have succeeded immediately, are blocked on Asset X’s bad behaviour.

The fix (commit 77d9647) rewrote Mutation::to_outbox_rows() to produce one row per entity, and added per-row retry/backoff machinery: attempts and next_attempt_at columns, exponential backoff capped at one hour, a DeadLetter status after ten consecutive failures so a permanently bad row stops being retried. AssetTrashed { ids: [a, b, c] } now produces three rows. Each one is independently retryable, independently dead-letterable, independently backed off.

I want to make this lesson explicit because it took me three iterations to internalise: the unit of retry should be the unit of failure. If a single id can fail independently, each id needs its own row. The temptation to compact the outbox is real — fewer rows feels cleaner, the JSON arrays are tidier in the schema browser — but the cost shows up under exactly the conditions you can’t easily reproduce in development.

This isn’t a recorder bug. It’s a queue-design bug, and would have happened with a channel, a Kafka topic, or a hand-rolled WAL. But it’s still a bug the architecture made me responsible for, because the trait that the architecture promises is only as good as the queue underneath it.

3. Identity across the round-trip

The third bug was the one that taught me the difference between decoupling and ignoring.

MediaId and AlbumId are local UUIDs that Moments generates when an asset or album is first created on the desktop. Immich also generates UUIDs for assets and albums on its end — those land in a column called external_id. The two are not the same. The local id is the row’s primary key forever; the external id is stamped onto the row once the push succeeds and the pull cycle confirms the asset is now visible server-side.

So far so reasonable. Now consider the lifecycle of a freshly imported photo. The user drops a JPEG into the import folder. MediaService::import generates a fresh MediaId, writes a row, records an AssetImported mutation. The push manager picks up the outbox row, uploads the file, receives the Immich asset id from the response, stamps it into media.external_id. Beautiful. Then the next pull cycle, dutifully fetching everything the server knows about, streams an AssetV1 event for that same asset.

The original handler for AssetV1 did INSERT OR REPLACE INTO media (id, external_id, …). The id it inserted was a freshly-generated local UUID — because that’s how the pull handler always created assets. The pull handler had no way to know that this asset already existed locally under a different local id, because the only thing connecting them was the external_id column it was about to overwrite.

The result: a brand-new row, with a brand-new MediaId, replacing the original one. Every album-membership row pointing at the original MediaId now dangled. The asset appeared in the timeline grid but not in any of the albums the user had just added it to. Worse, the original row’s on-disk thumbnail and cached EXIF data were now associated with an id that no longer existed in media.

Commits d43859f and 1692881 fixed this by making MediaId and AlbumId round-trip stable. On inbound sync, every handler looks up the existing row by external_id first. If a row already exists with that external id, the handler reuses its MediaId and updates fields in place, never minting a new one. The local id becomes the durable identity; the external id is the bridge that lets inbound sync find it again.

The general lesson — and this one took me by surprise — is that “an outbox decouples local mutations from remote sync” is technically true and architecturally meaningless if the identity of an entity doesn’t survive the round-trip. The decoupling is between the two flows of operations, not between the two namespaces of identifiers. Those namespaces still have to be reconciled somewhere, and the schema is the natural place. You need a column, or a constraint, or a lookup convention that lets inbound sync recognise “this thing already exists” rather than treating every server-side row as a first-time event.

In retrospect, the lesson is obvious. In practice, it costs you a weekend the first time, because the bug is invisible in any test where the asset is created locally or fetched from the server but never both. The minimum reproducer is a real round-trip, and you don’t usually write tests that span import → push → pull on the same asset.

4. Reset semantics, or: what does the server mean by “start over”

The fourth bug is the one that taught me sync protocols are not what they appear to be.

Immich’s sync protocol can send a SyncResetV1 message when the last checkpoint is more than thirty days stale. The semantics, in the spec, are roughly: “your understanding of my state is too old to be incrementally caught up. Treat what follows as a complete re-emission of everything I know about.”

The naive interpretation is “wipe the local cache and re-fetch everything.” That interpretation is catastrophic for an offline-first store. The user has locally-imported, not-yet-pushed assets sitting in their library. Those have no external_id because the server has never heard of them. A blind wipe would delete the user’s recent imports the moment the sync handshake decided their checkpoint was stale — and “stale” in this protocol can mean as little as a month of laptop-closed time.

The first version of the reset handler tried to be careful. It built a HashSet<MediaId> of every local asset at the start of the reset cycle, removed each id as the stream re-emitted it, and at end-of-stream deleted whatever was left over. That at least filtered to “things the server should have known about”. It still had three latent bugs, and I’ll only name them briefly because the full catalogue is in the design doc.

First, namespace confusion. media.id is the local UUID; the stream’s entity_id is the Immich UUID. The set was being keyed on one and reduced by the other, so the set never actually shrank — every reset cycle wiped the entire library. Second, locally-imported rows (external_id IS NULL) shouldn’t have been candidates at all, but the set didn’t filter them out. Third, albums and asset_faces had no orphan tracking, so they leaked across resets entirely.

Commits e54e037 and 3e24d9f rewrote the whole thing into a heartbeat-based reconciliation, which is the kind of state-machine design that’s interesting on its own merits. On SyncResetV1 the reset handler captures the current timestamp as a “checkpoint” and clears the per-entity-type ack cursors. Nothing gets deleted yet. Every entity-handler — AssetV1, AlbumV1, PersonV1, AssetFaceV1 — has an UPDATE table SET last_seen_at = now() WHERE id = ? baked into it. Any locally-driven server interaction (a push completion, a favorite round-trip) bumps the same column.

On SyncCompleteV1, four targeted sweeps run — on media, albums, people, and asset_faces — each filtering rows where last_seen_at < checkpoint. The media and albums sweeps add external_id IS NOT NULL as a guard, which is the single line of defence between a reset cycle and the user’s local-only imports. people and asset_faces don’t need that guard: nothing local creates them — they only exist once the server emits them. The whole scheme is crash-safe by accident: partial heartbeats persist across a process restart, so an interrupted reset cycle finishes correctly on the next pull rather than starting over.

The lesson is broader than reset. “Reset” in a sync protocol is not “delete everything.” It’s “re-establish ground truth without losing what only you know about.” The set of things only-you-know-about is — for an offline-first store — every row without an external_id plus every locally-mutated field that hasn’t yet been pushed. If your reset handler doesn’t have a name for that set, your reset handler is going to delete it.

This is the bug that has the least to do with the recorder trait itself. It belongs to the broader category of “things you only discover once you have the architecture, the outbox, and a real round-trip running long enough to encounter a thirty-day boundary.” But it earned its place on this list because it follows the same shape as the others: the architecture didn’t cause it, the architecture didn’t prevent it, but the architecture is what gave us one well-defined place to fix it.

What the four have in common

The bugs span the two categories the intro set out — abstraction (1 and 3) and substrate (2 and 4) — and what they share, across both, is that none of them required changing the architecture. Bug 1 added a _from_sync variant alongside the user-facing method, not a new trait or a new flag on the recorder. Bug 2 rewrote to_outbox_rows() and added retry columns to the schema; the recorder trait was untouched. Bug 3 changed how inbound handlers look up existing rows; nothing above the schema cared. Bug 4 added a single column (last_seen_at) and four sweep queries; the trait, the enum, the outbox table layout, the library services — all unchanged.

That’s the dividend the architecture actually pays. The bugs are real, expensive, and embarrassing in their own ways. None of them propagated upward. The library still doesn’t know sync exists. It just describes what it changed, and lets the substrate — and the people maintaining it — pay the round-trip’s price.

That is the entire architecture of how Moments syncs to Immich — and the entire reason the local backend, which has no sync at all, is the same program as the Immich one.

This post is about that trait, the enum it consumes, the outbox table underneath it, and the architectural property they buy together: a library that has no idea it has a sync backend, and a sync backend that’s purely additive to it.

The offline-first problem

Moments has two backends. The local backend stores photos on disk and indexes them in SQLite. The Immich backend talks to a self-hosted Immich server but caches everything in the same SQLite schema — the UI reads from the local DB either way, and an Immich library works fully offline after initial sync.

That means local writes have two possible destinations. Always the local DB. Sometimes — if the backend happens to be Immich — also a remote server, eventually, when the network cooperates. “Trash these three assets” should commit locally, return immediately, and arrange for the server to be told later. “Create this album” should be visible in the UI the moment the row hits the database, regardless of whether the push call has happened, will happen, or will fail and retry six times.

The naive answer is to give every service a reference to the sync layer. MediaService::trash calls sync.enqueue_trash(ids) after writing to the DB. AlbumService::create calls sync.enqueue_album_created(album). It works for the Immich backend. It also pollutes the local backend, which has no sync, with either a second constructor for every service or an Option<SyncHandle> carried everywhere it’s irrelevant. Every service in the library starts to know that there is such a thing as sync.

What I wanted instead was: services produce a value describing what they changed, and someone else decides whether to do anything about it.

The vocabulary

The value is Mutation, in src/library/mutation.rs. It is an enum of state changes the library has just committed:

#[derive(Debug, Clone)]
pub enum Mutation {
    AssetTrashed       { ids: Vec<MediaId> },
    AssetFavorited     { ids: Vec<MediaId>, favorite: bool },
    AssetDeleted       { items: Vec<(MediaId, Option<String>)> },
    AlbumCreated       { id: AlbumId, name: String },
    AlbumMediaAdded    { album_id: AlbumId, media_ids: Vec<MediaId> },
    AssetEditsApplied  { id: MediaId },
    // …
}

Eighteen variants in total — asset lifecycle, album lifecycle, edit application, stacks, tags, people. The shape matters more than the count: every variant names a state change the database has just committed.

The discipline that keeps this enum small is what it deliberately isn’t. There are no *Requested variants — UI intent isn’t a mutation, it’s a method call on a service. There are no *Result variants — results are return values. There are no UI hint variants like ThumbnailReady — those are events on a specific service’s event emitter and don’t belong here.

A Mutation is one thing: a state change the database has just committed, that might need to leave the machine. Nothing more.

The trait

MutationRecorder is in src/library/recorder.rs. It is the entire interface between the library and whatever wants to know what changed:

#[async_trait]
pub trait MutationRecorder: Send + Sync {
    async fn record(&self, mutation: &Mutation) -> Result<(), LibraryError>;
}

That is it. One method, async, takes a borrowed Mutation, returns an error if recording itself failed. Services hold an Arc<dyn MutationRecorder> and call it after a successful write:

pub async fn trash(&self, ids: Vec<MediaId>) -> Result<(), LibraryError> {
    self.repo.trash(&ids).await?;
    self.recorder.record(&Mutation::AssetTrashed { ids }).await?;
    Ok(())
}

A service has no idea what the recorder does with the mutation. It might write a row to a queue table. It might publish to a message broker. It might do absolutely nothing.

Two strategies, one of which is empty

There are exactly two implementations, both in src/sync/outbox/mod.rs.

The first is the entire point of the design:

pub struct NoOpRecorder;

#[async_trait]
impl MutationRecorder for NoOpRecorder {
    async fn record(&self, _mutation: &Mutation) -> Result<(), LibraryError> {
        Ok(())
    }
}

The local backend takes NoOpRecorder. Every service in the library still calls recorder.record(...) after every mutation. Those calls do nothing.

The code path is identical to the Immich one. No if backend == local anywhere in the services. No second constructor. Just a vtable dispatch and an Ok(()).

The second is the implementation that does the actual work:

pub struct QueueWriterOutbox { repo: OutboxRepository }

#[async_trait]
impl MutationRecorder for QueueWriterOutbox {
    async fn record(&self, mutation: &Mutation) -> Result<(), LibraryError> {
        for row in mutation.to_outbox_rows() {
            self.repo.insert(&row).await?;
        }
        Ok(())
    }
}

QueueWriterOutbox is used by the Immich backend. It serialises the mutation into one or more rows in a sync_outbox table — multi-id mutations like AssetTrashed { ids: [a, b, c] } fan out to three rows so retries can be per-entity rather than per-batch. The schema is unsurprising: id, entity_type, entity_id, action, payload (JSON), created_at, plus status, attempts, next_attempt_at, and last_error for retry bookkeeping. This is the transactional outbox pattern, more commonly seen in service-to-service messaging, applied here to client-server sync — with one small adaptation to the trait signature I’ll come to in the next section.

The mapping from Mutation to OutboxRow lives in src/sync/outbox/mutation.rs and is genuinely boring — a match on the variant producing an entity_type string, an entity_id, an action verb, and an optional JSON payload. It is the kind of code that should be boring; the interesting part has already happened.

My favourite test in this codebase

My favourite test in Moments doesn’t assert on database state. It doesn’t assert on side effects. It asserts on a sequence of Mutations the library produced — on what the library said it did, in its own vocabulary, in order. That test works because the third implementation of MutationRecorder lives in src/library/recorder.rs:

#[derive(Default)]
pub(crate) struct CapturingRecorder {
    recorded: Mutex<Vec<Mutation>>,
}

impl CapturingRecorder {
    pub(crate) fn snapshot(&self) -> Vec<Mutation> {
        self.recorded.lock().unwrap().clone()
    }
}

#[async_trait]
impl MutationRecorder for CapturingRecorder {
    async fn record(&self, mutation: &Mutation) -> Result<(), LibraryError> {
        self.recorded.lock().unwrap().push(mutation.clone());
        Ok(())
    }
}

Service tests construct a Library with Arc::new(CapturingRecorder::default()) and then assert on the recorded sequence after exercising a method. “When I trash three assets, then create an album from two of them, the recorder should see AssetTrashed followed by AlbumCreated followed by AlbumMediaAdded.” That assertion is a stronger property than “the database ends up in the right state”, because it pins down the intent surface the library exposes to anything that might eventually act on it — including providers that don’t exist yet.

This is what the trait actually buys: not two production strategies, but three perspectives on the same library code — a no-op that proves the local backend doesn’t care, a queue writer that ships changes upstream, and a capturing fixture that pins down what services promise. None of them required a single change to the services themselves.

Dependency injection: sync as someone else’s problem

The wiring happens in exactly one place. Library::open takes the recorder as a parameter and threads it into every service:

pub async fn open(
    bundle: Bundle,
    mode: LocalStorageMode,
    db: Database,
    recorder: Arc<dyn MutationRecorder>,
    resolver: Arc<dyn OriginalResolver>,
) -> Result<Self, LibraryError> {
    db.open(&bundle.database.join("moments.db")).await?;

    let albums  = AlbumService::new(db.clone(), Arc::clone(&recorder));
    let faces   = FacesService::new(db.clone(), bundle.thumbnails.clone(), Arc::clone(&recorder));
    let editing = EditingService::new(db.clone(), Arc::clone(&recorder));
    let media   = MediaService::new(db.clone(), bundle.originals.clone(), mode,
                                    Arc::clone(&recorder), resolver);
    // …

    Ok(Self { albums, faces, editing, media, /* … */, recorder })
}

The local backend’s caller hands in Arc::new(NoOpRecorder). The Immich backend’s caller hands in Arc::new(QueueWriterOutbox::new(db.clone())), and separately starts a background PushManager that drains the same table.

That single substitution — the choice of recorder at the call site — is the only thing in the system that knows whether sync is happening. Everything below it is identical. From the library’s point of view, sync is someone else’s problem: it dutifully describes what it changed and trusts that someone, somewhere, may care.

The drain side

For completeness: src/sync/providers/immich/push.rs is the PushManager. It is a long-running Tokio task that wakes up on an interval, reads up to a hundred pending rows from sync_outbox, deserialises each back into an OutboxMutation, makes the appropriate Immich API call, and marks the row done or failed. There is exponential backoff on failure, a DeadLetter status after ten consecutive failures, and a MAX_BACKOFF_SECS cap so attempt ten doesn’t sleep for seventeen hours.

It is the side of the system that has to know about HTTP, retries, idempotency, and Immich-specific endpoints. It is also the side that doesn’t exist at all when you’re running the local backend, because nobody constructed it.

The pull side (pull.rs + handlers/) is the reverse direction — POST /sync/stream streams server-side changes back into the local DB. It does not go through the recorder: pull-driven changes are server-originated, and recording them would feed the server’s own news straight back into the outbox heading the other way.

Adapting the pattern

The classical transactional outbox pattern — the one I borrowed the name and the table from — requires the business write and the outbox INSERT to share a single database transaction. That’s the whole point of the pattern: if the entity write commits, the outbox row commits with it; if either fails, both roll back. Atomicity is what guarantees the consumer eventually sees every committed change.

Moments needs that guarantee, non-negotiably. If trash returns success to the caller, the outbox row must be there waiting for the push manager. Otherwise the local state silently drifts ahead of the server: the user trashes a photo, the desktop hides it, the server never hears about it, and the next pull cycle dutifully restores it. A sync layer that loses writes is worse than no sync layer at all — once the user catches it doing that, they stop trusting it for everything else.

The adaptation is small but load-bearing. The trait signature I’ve shown so far is simplified for exposition; the real one takes a transaction handle, and the three implementations all thread it through. Services own the transaction lifecycle, and both the repository write and the recorder call happen inside the same transaction:

#[async_trait]
pub trait MutationRecorder: Send + Sync {
    async fn record(
        &self,
        tx: &mut sqlx::Transaction<'_, Sqlite>,
        mutation: &Mutation,
    ) -> Result<(), LibraryError>;
}

// in MediaService:
pub async fn trash(&self, ids: &[MediaId]) -> Result<(), LibraryError> {
    let mut tx = self.db.begin().await?;
    self.repo.trash_in_tx(&mut tx, ids).await?;
    self.recorder
        .record(&mut tx, &Mutation::AssetTrashed { ids: ids.to_vec() })
        .await?;
    tx.commit().await?;
    Ok(())
}

QueueWriterOutbox::record uses the transaction handle to insert into sync_outbox inside the same transaction the service is about to commit. If the insert fails, the ? propagates and tx.commit() is never called; the business write rolls back with it. The local database and the outbox share a fate by construction — neither can be ahead of the other.

NoOpRecorder ignores the transaction parameter entirely. The local backend pays for opening and committing a transaction the recorder didn’t touch — a small, fixed cost on every mutation, in exchange for the library never having to know which recorder it’s been given. That’s the price of the abstraction, and it’s the only price the abstraction asks the local backend to pay.

An honest aside, because the timing is too good to leave out. I drafted this section assuming the code matched the design above. Then I went to check. It didn’t. The actual service code commits the repository write and then calls the recorder in a second, separate await — and at most call sites the recorder error is logged and swallowed rather than propagated. A recorder failure can silently leave the local state ahead of the outbox, which is exactly the failure mode this section claims is impossible by construction. I filed the bug while finishing this paragraph.

The design above is the design. The code is catching up.

Adding a third backend

The shape of the substitution generalises beyond Immich. Adding a Nextcloud backend tomorrow would touch zero lines in src/library/. The services already produce Mutations. The outbox table is provider-agnostic. The new code lives entirely in src/sync/providers/nextcloud/: an HTTP client, a push manager that drains the outbox into Nextcloud’s API, and a pull manager that streams server-side changes back into the local DB. The wiring in main swaps QueueWriterOutbox in as the recorder, starts the Nextcloud managers, and is done.

That doesn’t make a new backend free — push and pull are still real work, with their own retry semantics and identity gotchas — but it does make the cost proportional to what’s actually different. The 99% of the app that doesn’t care which server it’s talking to doesn’t have to be told.

Why a trait, and not a channel

The same job could be done with an mpsc::UnboundedSender<Mutation> — services hold a sender and push mutations into it; a consumer task somewhere pulls them out. A channel even handles NoOpRecorder naturally: open a channel and never spawn a consumer — the sends just buffer.

But a channel ties the producer and the consumer to a specific lifecycle. Who closes the sender? When? What happens when the buffer fills? Is the consumer guaranteed to run, and if not, are the dropped messages a bug or a feature? The trait sidesteps all of this by making “what to do with a mutation” a synchronous decision at the point of production. NoOpRecorder returns immediately. QueueWriterOutbox writes a row and returns. CapturingRecorder pushes onto a vec and returns. There is no pending state, no backpressure, no “what if the consumer crashed” failure mode lurking in the design. The service doesn’t have to reason about whether its mutation was received, because reception is part of the same await as the rest of the operation.

A Box<dyn Fn(Mutation)> closure could work too, and would even be slightly cheaper. The reason the trait wins isn’t performance; it’s naming. NoOpRecorder, QueueWriterOutbox, CapturingRecorder — each one is a documented, importable type that says what role it plays. A closure is anonymous; it can do anything; you cannot grep for who’s using it. The trait makes the contract a concrete thing that can be referred to in design documents and PR descriptions, and the production implementations a fixed, small set that anyone can enumerate.

The verb “record” matters too. “Emit” or “publish” or “broadcast” would have invited fan-out — and fan-out is exactly the property Mutation has been built to avoid. There is one recipient, statically chosen at Library::open time. “Record” implies durability and report-after-the-fact, which is what services actually do: they write to the database, then they describe what they wrote. The naming makes the contract harder to violate.

What I’d tell myself

If I were starting the Immich backend again, I would write MutationRecorder and NoOpRecorder on day one, before there was anything that needed syncing, and have services call it from the start. The local backend would carry a NoOpRecorder it would never need, and when the Immich backend arrived months later, nothing in the services would change. That’s the value the trait actually provides: not a separation of concerns at the API surface, but the freedom to add an entire concern later without rewriting what’s already there.

The library doesn’t know sync exists. grep -rn 'crate::sync' src/library/ returns six matches: five are #[cfg(test)] imports of NoOpRecorder from test helpers, one a comment pointer. Production imports: zero. From the inside, the local backend and the Immich backend are the same program; the only difference is what an Arc<dyn …> happens to do.

That diagram is a trap. I drew it for Moments, my photo manager, in late March 2026. I had the design reviewed by an external UI architect. I shipped it in April across six phased PRs. And on the third of May I deleted the whole thing in a single commit titled, with some satisfaction:

refactor: delete EventBus, AppEvent, library/commands, MediaClient v1 — closes #580

This is what I learned about why the bus was wrong, what replaced it, and the broader pattern: an event bus is decoupling-by-indirection, and the indirection itself is the cost.

What the bus looked like

The original design was, by the standards of the form, a good one. A single EventBus lived on the GTK main thread. Background Tokio tasks pushed AppEvents into it via a Send + Clone sender. glib::idle_add_once drained the queue and fanned out to subscribers. Subscriptions were RAII-handle-based with re-entrancy-safe deferred drops. It had tests. It had a design doc. It worked.

Here is the central type — the bit that everything else depended on:

#[derive(Debug, Clone)]
pub enum AppEvent {
    Error(String),
    ThumbnailReady { media_id: MediaId },

    // Commands: UI intent → library command handler
    TrashRequested { ids: Vec<MediaId> },
    FavoriteRequested { ids: Vec<MediaId>, state: bool },
    AddToAlbumRequested { album_id: AlbumId, ids: Vec<MediaId> },
    CreateAlbumRequested { name: String, ids: Vec<MediaId> },
    DeleteAlbumRequested { ids: Vec<AlbumId> },
    // ...

    // Results: command handler → subscribers
    MediaTrashed { ids: Vec<MediaId> },
    AlbumCreated { album: Album },
    // ...
}

By the time I deleted it, AppEvent had twenty-one variants in a single file, split into “commands” (UI intent) and “results” (library outcomes). Buttons emitted *Requested events. A CommandHandler trait dispatched each one to a struct in src/commands/. Library backends emitted *Result events. Widgets subscribed to whichever results they cared about and patched their ListStore models.

It was internally consistent, and it was wrong.

Where it started hurting

There was no single bug big enough to justify deleting it. There was just the same small bug, in slightly different costumes, every week.

The enum became a junk drawer. Every new feature added a request variant and a result variant. The file grew. Understanding what could fire meant scanning the whole enum. Worse, dead variants accumulated: refactor: remove dead AppEvent variants (#576) is a real commit, removing variants that had no subscribers anywhere. Nobody had noticed because the bus made the absence of subscribers invisible.

Double-update bugs. When the album-create button fires CreateAlbumRequested, the command handler calls the library, gets back an Album, and emits AlbumCreated. The album grid widget — which initiated the action — also subscribes to AlbumCreated, because how else would it learn? So the widget refreshes from the broadcast. But the widget’s own code path knew the result first and tried to insert the row optimistically. Now you have two writers fighting over the same ListStore. The fix in any individual case is easy — guard with a flag, defer one path. The pattern of needing the fix everywhere is the smell.

Re-entrancy hazards. One commit message that survives in the history reads fix: defer navigate in ImportComplete handler to avoid bus re-entrancy. A subscriber to ImportComplete navigated to a new view. Navigating triggered unrealize on the old view. unrealize dropped a Subscription. Dropping a Subscription mutated the bus’s subscriber list — while the bus was iterating it. The original design had anticipated this and added a “deferred removals” mechanism. That mechanism was a tax on every drop, paid forever, to fix a problem the bus itself created.

O(n) no-op events. During an Immich sync, the face-recognition service updates the face_count on every person row. There can be hundreds. Each update fired a result event. Each event hit every subscriber, even though face_count wasn’t a property any GObject was bound to. The fix was to teach the service to skip emission for update_face_count. So now the service had to know which of its own state changes were “broadcast-worthy” — which is a leaky responsibility for a layer that shouldn’t know who’s listening at all.

You can’t follow it at runtime. Set a breakpoint on bus.send(). Hit it. Where does control end up? Twelve subscribers across four widgets and two background services, in some order, on the next idle tick. Now do that for an interaction that fires three events. The bus had given me decoupling at the cost of being unable to read the program.

The translation layer was the first thing to fall. Library events arrived as a separate LibraryEvent enum and were translated into AppEvent at the boundary. The translation was pure ceremony. It came out in refactor: eliminate LibraryEvent → AppEvent translation loop (#520) — about three weeks after the bus shipped. That should have been the warning sign: the bus’s main job had been routing events from one layer to another, and as soon as the layers started talking directly, the bus’s reason for existing started to thin out.

What replaced it

The replacement is, deliberately, almost boring. Each library service holds its own typed EventEmitter<T> and emits a small enum specific to that service:

#[derive(Debug, Clone)]
pub enum AlbumEvent {
    AlbumAdded(AlbumId),
    AlbumUpdated(AlbumId),
    AlbumRemoved(AlbumId),
    AlbumMediaChanged(AlbumId),
}

Four variants. Owned by the album feature. Nothing about media, faces, sync, errors, or UI intent.

The EventEmitter<T> itself is a single file:

pub struct EventEmitter<T: Clone> {
    senders: Arc<Mutex<Vec<mpsc::UnboundedSender<T>>>>,
}

impl<T: Clone> EventEmitter<T> {
    pub fn subscribe(&self) -> mpsc::UnboundedReceiver<T> {
        let (tx, rx) = mpsc::unbounded_channel();
        self.senders.lock().expect("poisoned").push(tx);
        rx
    }

    pub fn emit(&self, event: T) {
        let mut senders = self.senders.lock().expect("poisoned");
        senders.retain(|tx| tx.send(event.clone()).is_ok());
    }
}

A subscriber gets a fresh receiver. Dead receivers prune themselves on the next emit. Clones share the same subscriber set. That is the entire primitive.

The service uses it in two places — once to expose subscription, once to emit:

pub struct AlbumService {
    db: Arc<Database>,
    events: EventEmitter<AlbumEvent>,
}

impl AlbumService {
    pub fn subscribe(&self) -> mpsc::UnboundedReceiver<AlbumEvent> {
        self.events.subscribe()
    }

    pub async fn delete_album(&self, id: &AlbumId) -> Result<()> {
        self.db.delete_album(id).await?;
        self.events.emit(AlbumEvent::AlbumRemoved(id.clone()));
        Ok(())
    }
}

On the GTK side, the client subscribes once at configure time and spawns a Tokio listener that dispatches model patches back to the main thread:

pub fn configure(&self, library: Arc<Library>, tokio: tokio::runtime::Handle,
                 events_rx: mpsc::UnboundedReceiver<AlbumEvent>) {
    let client_weak: glib::SendWeakRef<AlbumClientV2> = self.downgrade().into();
    tokio.spawn(Self::listen(events_rx, library, client_weak));
}

async fn listen(mut rx: mpsc::UnboundedReceiver<AlbumEvent>,
                library: Arc<Library>,
                client_weak: glib::SendWeakRef<AlbumClientV2>) {
    while let Some(event) = rx.recv().await {
        match event {
            AlbumEvent::AlbumRemoved(id) => {
                let weak = client_weak.clone();
                glib::idle_add_once(move || {
                    if let Some(client) = weak.upgrade() {
                        client.remove_from_models(&id);
                    }
                });
            }
            // ...
        }
    }
}

That is the whole architecture. There is no central registry. There is no AppEvent enum. The album feature defines its events, owns its emitter, and hands receivers to whoever asks. A new feature adds a new service with its own emitter; nothing in the existing code has to change.

The double-update fix that was a design fix

The most interesting thing about the new pattern isn’t the channel itself. It’s a small contract documented on every command method:

Client-initiated mutations don’t emit events — the client patches its model directly in its callback.

When AlbumClientV2::create_album runs, it calls the service, gets back the new album, inserts it into its tracked models itself, and that’s that. The service does not fire AlbumAdded for client-initiated work. The bus-era double-update bug cannot exist, because the broadcast path and the initiator path are no longer the same path.

The bus had pushed me toward “everyone reacts to everything uniformly”, which sounds clean and is a lie — initiators always know more than subscribers. The new design lets the initiator use what it knows. Events are reserved for state changes the initiator didn’t cause: a remote sync arriving, a background scan completing, a peer process writing the database.

What I’d tell myself in 2025

Three things, in order of how useful they would have been.

An event bus is decoupling-by-indirection. It feels like you’ve removed a dependency. You haven’t. You’ve moved it into a giant enum that nobody owns and that everyone has to read to understand the program. The dependency is still there; you’ve just made it untyped, ungrep-able, and impossible to follow at runtime. A direct mpsc::UnboundedSender<AlbumEvent> from the album service to the album client is more coupled than a bus message of type AppEvent::AlbumCreated, and that coupling is exactly what makes the program legible.

The fan-out problem is rarer than you think. I had reached for a bus because “many subscribers might react to one event.” In practice, almost every event in Moments has one or two subscribers, and the subscribers are statically known. A bus’s variable, late-bound subscriber list buys you very little when the real subscriber set is “the album client and nothing else.”

Architecture review can validate internal consistency but not problem fit. The bus design was reviewed and approved. The reviewer was right that, given the bus, the design was sound. They couldn’t have told me — and I couldn’t have asked — whether a bus was the right shape for the problem in the first place. That’s a question only a few weeks of using it can answer, and in this case a few weeks were enough. Build the smallest version of the boring thing first. Earn the right to a bus.

The deletion commit removed about 1,200 lines net. The replacement, including EventEmitter and four service-specific event enums, is closer to 200. The bus had been doing what a thin wrapper around mpsc does, dressed up as architecture.

I don’t regret building it — without the bus there’d be nothing concrete to compare against, and the lessons would have stayed in the realm of opinion. But if I were starting again, I would write pub fn subscribe(&self) -> mpsc::UnboundedReceiver<AlbumEvent> first and not draw the diagram at all.