Schema and Local-first Software
It's been a hot minute since my last newsletter. Allow me to reintroduce myself.
I'm an applied researcher investigating the future of computing. My primary work involves discovering and disseminating generalizable insights into the nature of the computational medium.
I'm currently working with DXOS to create an application framework for building local-first software and developing a malleable software environment called Composer on top of that framework.
Over the past few months, I've been exploring the concept of schemas in local-first software systems. This has led to a related set of realizations:
- There's a generic architectural shape that satisfies the local-first principles that seems to be emerging across different projects: a data store on every client with a generic sync service keeping the data stores synchronized between clients.
- This architecture allows a local-first architecture to unify all of the typical uses of schema with a single schema, which simplifies the DX and creates some interesting scenarios for interoperability and LLM integration.
I gave a talk at Effect Days in February entitled "Beyond Apps: Solving the Schema Problem with Effect Schema." I discussed the architectural shift from client-server architectures to a local-first architecture, showed how we're unifying all the schemas within DXOS using Effect Schema, and showed some fun demos enabled by "same schema everywhere." The talk recording hasn't been posted yet, but I'm working on an article expanding on the ideas in the talk.
DXOS has been transitioning our schema implementation from Protobufs to Effect Schema, and simultaneously extending the use of schema throughout the architecture. We expect to complete the transition and ship Effect Schema in production in the next few weeks. Both the DXOS SDK and Composer itself were already leveraging schema extensively, but the transition to Effect Schema allows us to unify our runtime-defined schemas and statically-declared schemas. A few notes on the implementation for those curious about details:
- All schema, both statically defined in code, and defined at runtime, will be stored in DXOS's ECHO database and will be queryable.
- Effect Schema is the primary representation used to read and write schema.
- You can attach custom annotations to the schema which will get serialized and stored in ECHO.
- We're still debating on the exact API but there will be a way to differentiate statically-defined and runtime-defined schema.
If you're thinking about schema or local-first architecture, I'd love to talk with you. I'm in the thick of it at the moment.
ICYMI: Stuff I've worked on recently
- Orion Reed and I launched the Open Canvas Working Group, focused on establishing interoperability between different infinite canvas tools. The first meeting is the week of March 25.
- The team at DXOS is using Composer every day to collaborate on documents and sketches. Here's a 4 minute overview video of Composer. If you try it out, I'd appreciate any and all feedback.
Recently Read
Glimpses of the future
Localfirst.fm is an excellent new podcast hosted by Johannes Schickling that chronicles the developing field of local-first software through a series of interviews with practitioners, from researchers to builders of local-first products. The editing is tight, the conversations are meaty and technical, and Johannes own experience make for a rich discussion. I can't recommend it highly enough. Normally when I reference a podcast, I point people at a specific episode. I regret to inform you that all of the episodes are excellent. As of today, there are just five. Start listening now before the back catalog piles up!
Relatedly, the first local-first software conference is coming up! Berlin. May 30-31. I'm going. If you're planning to be there, reach out so we can coordinate some time together. It will be a quick trip for me as I was just in Vienna last month (and tacked on a few days of skiing ⛷️) so I have to hurry home once the conference activities conclude.