FEEL: thoughtful, reflective, challenged, stretched
DO: consider how schema is solved within the architecture of software systems they work in
JTBD: When I am starting a new project, I want to understand why local-first architectures are simpler and better, so I can choose DXOS for my next project

See Effect Days 2024 Talk Track - Cleaned Up

Talk Track

Introduce myself, introduce the talk:
- Convince you to rethink core assumptions about software
Apps today have a certain architecture to them
- So common we might think it's immutable
- web, mobile, all the same
- clients (browsers, mobile apps, etc) and servers (backend), an API layer, a database
  - show a diagram of this
- Sometimes we add additional complexity, such as event streams, etc, but effectively all the same
This architecture determines what's possible in apps today - we don't think about it that often, because we paint inside the lines...
- as a simple example, offline support
  - when you assume that the user's data will be stored on a server that they have to access over the internet, it makes supporting offline really hard
  - Sustainable iPad App story?
- realtime is hard
This leads to a dominance of the app, apps have gravity. the easy path is not to integrate, but to add one more feature, the easy path is always-on connectivity, the easy path is the monolith
- and this powers the SaaS business model. effectively, we all become software renters, unable to own or improve our software, to put up a shelf, or change the paint color
- interop sucks
THERE IS ANOTHER WAY (perhaps many, I will show you one)
- DXOS apps are local-first
  - hello.dxos.org
    - guestbook app
    - close your browser
- Works offline, even
  - shut off wifi, keeps on plugging
But Not Local-Only (colloborative)
- Invite everyone to the same guestbook!
  - now try shutting off wifi, adding something, see how everything syncs when you get back
Compare the two architectures
- diagram of client-server
- diagram of local-first
SCHEMA TIME: Focus on one particular area where local-first software struggles but has the potential to shine: SCHEMAS
- What's a schema?
Where is schema in client-server?
- DB, API, front-end
- Ugly!
Where is schema in local-first?
- One schema to rule them all.
our value hypothesis:
- the schema of the db IS the schema of the API and the frontend
- all state lives in strongly typed, validated, reactive objects
- radically improved developer velocity and new scenarios possible
show how we use @effect/schema to make our API type-safe:
- We started to approach the schema problem from the db serialization angle
  - started with protobuf
  - efficient on the wire
- here are the 2-3 reasons
  - typescript - feels good to us
  - has the infer technique to infer the typescript types
  - (huh?) we were using Zod in other parts of the app, but their philosophy / governance model didn't allow for adding metadata
- show guestbook app source
- show declaration of schema
- show dx of inferred types
- show dx of queries, explain how validation happened at runtime
- show dx of mutations, explain how validation happens during mutations
show how schema is important to interop:
- composer can see the same data as the guestbook app - show how it's reactive between the apps
- composer can visualize things in tables by examining the schema
- show ability to infer columns and display guestbook data
- show ability to add/edit records and perform validation trivially (reject invalid input - i.e. schema in the database was used to protect guestbook app from invalid changes in composer, no extra effort on either app, no API layer in between because both apps use the database directly.)
  - explain the problems with this:
    - how do we protect from invalid business rules that aren't expressed as simple schema constraints? (maybe we need to help invent a second schema layer that's more like the API schema on top of the Data schema?)
- show ability to extend schema at runtime without breaking either app: add a column and annotate some user records with custom fields in composer - observe data is retained and guestbook app continues to work
- explain how composer is extensible and can delegate rendering of arbitrary things to plugins, which can use schema to decide how to render things, or delegate parts of rendering to other plugins through more surfaces - surface nesting ftw.
  - demo of contacts plugin
show how schema can be used in AI scenarios:
- show how we can use schema in LLM prompts for augmenting CRM records with more data
- concrete example: show how we can do entity extraction from a plain text document and use schema to format a prompt that extracts tasks and due dates from some meeting notes
conclude with calls to action about composer and dxos.
- prior art: smalltalk, hypercard, xanadu, microsoft access, others
- opportunity: private, open source, developer extensible, local-first knowledge management system
- opportunity: build next generation apps radically faster with dxos
- call to review our resolvers/mutation adapters + schema designs

Some Notes

beyond apps

current world, app architecture
could be different
app architecture, rebooted
what's it like to write a DXOS app?
different trade-offs
schema - who owns the schema?

The Schema Problem

what does the application know about the data based on the schema and how does it know it?
- dynamic schema based on effect/schema
  - runtime validation
  - runtime creation of new schemas
  - also generates typescript types for compile-time typing
- if the application knows the schema of the data... it can do stuff
  - it's the same thing as runtime type introspection / validation
who owns the schema?
how do you make changes to the schema over time (migration)
We're building a unified knowledge environment: Composer
- SmallTalk for the modern internet
- It's open source and fully extensible, come join us!

Project Plan

Demo + Description of DXOS has a different shape of software
Demo of DXOS that runs into the schema problem
Description of the schema problem
Demo of how we use @effect/schema to solve the schema problem
Contact other effect/schema talk-givers to coordinate with them about how we talk about effect/schema
max 1-minute video description for social media
trial run of the talk

Abstract

Title: Beyond Apps: Solving the distributed schema problem with @effect/schema

Rough outline of the talk

Introduce local-first software via DXOS and show an interactive demo
- stores data locally on-device
- replicates across user's devices automatically
- shares with other users via p2p
- synchronizes changes automatically via CRDTs
- users have a single data store that applications write to and read from
Extend the demo by writing a new app that interacts with same data and introduce (run into!) the schema problem
- shared data substrate
- two apps looking at the same piece of data in different ways
- how do they know the shape of the data?
- how do we "migrate" the schema?
Show how DXOS solves the schema problem using @effect/schema
- store the data's schema alongside the data
- introspect the schema
- migrate automatically when the schema changes

Effect Days 2024 Vienna DXOS Talk

Interactive graph

On this page

Effect Days 2024 Vienna DXOS Talk

Talk Track

Some Notes

Project Plan

Abstract