[RFC] Directus Offline First SDK

[ctholho]

Directus offers great tools for building RESTful APIs when your internet connection is great. But, building offline-first applications is still very hard. This RFC suggests creating an offline-first SDK named Indirectus. It can be a community-maintained project and does not depend on upstream changes.

Objectives

The objective of the proposed offline-first SDK, Indirectus, is to help developers create dependable applications that can function offline or in areas with low connectivity:

• SPA Framework agnostic (should work with react, vue and svelte … (maybe angular?))

• Observability of a modifier queue

• Immediate, optimistic updates on the local client

• Safe by default: Due to the inherent insecurity of local storage and IndexedDB, the SDK encrypts data at rest. Optionally you can turn off encryption for public data.

• Offer ways to create custom endpoints for reading, updating and deleting to CRDTs or implement OT conflict merging strategies more easily.

• Allow overwriting HTTPs routes for specific resources if they need special handling in a Directus custom endpoint

  • e.g.: reading a CRDT could be implemented with a custom endpoint GET /custom-crdt/:collection/:id
  • while updating might be at PATCH /items/:collection/:id

• Drop-in replacement for the directus-sdk:

  In existing projects this allows to gradually migrate to offline-first queries. See this example:

  // default call with replaced sdk
  const { InDirectus } = require('indirectus')
  const directus = new InDirectus('https://example.directus.app')
  const articles = directus.items('articles')
  
  // Fail if offline:
  const data = await articles.readByQuery({ id: 42 })
  
  // Use offline first, with retries, fetch status and placeholder data:
  const { isLoading, isError, data, error } = articles
  	.readByQuery({ id: 42 })
  	.indirectus({ 
  		placeholderData: {
  			id: 42,
  			content: 'placeholder article',
  		}
  	})

  Notice:.indirectus() uses tanstack/query and is basically sugar for this query:

  import { useQuery } from '@tanstack/vue-query'
  const { Directus } = require('@directus/sdk')
  const directus = new Directus('https://example.directus.app')
  const articles = directus.items('articles')
  
  const { data } = useQuery({
  	queryKey: [{ id: 42 }],
  	queryFn: articles.readByQuery,
  	placeholderData: { id: 42, content: 'placeholder article' },
  })

Under consideration

• How to handle and enforce idempotent createOne or createMany requests?
  • require a user generated ID on creation to make creation of items idempotent. Consider these implementations
    1. by non-defaulting a primary UUID key and require a user generated value
    2. by adding another unique constrained column idempotent_id. Developers must remember to generate and use idempotent_id on the client to make idempotent createOne calls.
    3. or by adding a custom endpoint, that funnels all createOne requests and add user generated UUIDs to a new deny-lists table.
  • Or use a new timestamp column created_at_client that has to be unique for the user_id and created_at_client. E.g.: in postgres: UNIQUE (user_id, created_at_client)
  • Error or warn if a global config option requireIdempotency is true and developers call createOne without a specific key
  • Consider automatically injecting an idempotency key into the request?
• Indirectus doesn’t offer the necessary reliability for applications that stay offline indefinitely. We’re still talking about a CRUD application with a central server running in a browser.
  • We still could offer an easy API to save query/mutation caches to external files.
  • Is there an API we could offer to fetch “all” data at application startup?
• Integrate with service worker to make sure data not yet uploaded is not lost when reloading after an update.

Non-objectives

• Developers still need to know which conflict resolution is best for their use-case.
  • E.g.: modelling the status of a finished task in a collaborative todo app could be (i) a boolean, (ii) a done_by user id with a simple foreign key or (iii) with a many to many relation or (iv) an enumeration type … among others.
  • The SDK will not help with setting appropriate permissions for version, updated_at, status columns in order to handle conflicts during updates and deletion.
• Indirectus will not help with the Directus Admin App
  • Although it will become trivial to create custom interfaces that call Indirectus

Architecture

Client side:

Basically: Indirectus client is a wrapper for tanstack/query with added client side encryption.

• Indirectus will be a wrapper for tanstack/query, offering the same API
• Clientside encryption is called via tanstack/query hooks
  • This lib has only 5 stars – but an external audit: ❤️
    wear

Backend additions:

• Add custom endpoints to handle a few simple CRDTs
  • e.g. https://github.com/jlongster/crdt-example-app
  • a grow only counter
• Add flows to handle OT

Examples

Create a new item:

By default with retries or saving to an offline mutation cache and automatic cache invalidation:

const { isLoading } = articles
	.createOne({ title: 'Invalidate cache if updated' })
	.indirectus()

// This is basically sugar for tanstack/query mutations
// see https://tanstack.com/query/latest/docs/vue/guides/mutations
//
// const { isLoading, mutate } = useMutation({
//   mutationFn: (newArticle) => articles.updateOne(newArticle),
// })
//
// function addArticle() {
//   mutate({ title: 'Optimistically updated' })
// }

Notice, we’re not explicitly using an idempotency key. In this proposal, we’re generating that automagically.

Optimistic offline updates:

• See how tanstack/query offers optimistic updates

We can offer sugar without disabling the expressibility of tanstack/query

// using vue here, but tanstack/query is available for react, svelte...
import { useQueryClient } from '@tanstack/vue-query'
const queryClient = useQueryClient()

const { isLoading } = articles
	.createOne({ title: 'Optimistically update articles' })
	.indirectus({
		// use your own onMutate, onError and onSettled or ...
	})
	// ... call createOptimistically(queryClient, offlineData)
	// The second argument allows us to merge offline data that we
	// want to use in the optimistic update. In this case, we can display a
	// warning that this article is not yet synced with the server.
	.createOptimistically(queryClient, { 
			id: `offline-${new Date}`
			unsaved: true
		}
	)

Disable client-side storage and encryption if offline:

You’ll be able to set tanstack/query options, directus SDK options and Indirectus options.

// This is the default init object
const config = {
	storage: 'session' // to avoid the persistence layer
	encrypt: 'never' // implicitly set to 'never' if storage === 'session'
	tanstack: { // use tanstack options
		queries: {
	    refetchOnWindowFocus: false,
    },
	},
	directus: { // use directus SDK options (not everything makes sense)
		auth: {
			// ...
 		},
		storage: {
			// ...
		},
		transport: {
			// ...
		},
	}
}

Conflict Resolution Approaches

This section outlines four methods for handling edit conflicts in Directus. Some of these methods can be implemented with Directus custom endpoints or flows. Generally, conflict resolution falls into four categories, with developers selecting the most suitable strategy. Performance and usability with CRDTs have come a long way since their inception 2011. For most small to mid-scale Directus use cases CRDTs will work fine. They will probably still be a rarely used data type.

1. Last Write Wins (LWW): This default method resolves conflicts by prioritizing the most recent write operation in a CRUD app. Earlier edits are overwritten.

   articles.updateOne(15, { content: 'The looser takes it all' })

2. Operational Transformation (OT): This method can be implemented using a Directus flow operating on the article_event data, updating the articles collection accordingly.

   articleEvents.updateOne(15, { content: { index: 23, operation: 'add', value: '!?', timestamp: 1678814119 } })

3. Conflict-free Replicated Data Types (CRDTs): This method employs data structures that automatically merge, resolving conflicts. CRDT implementation would be facilitated by Directus' support for views in relational databases. However, developers should be be able to fully choose their preferred implementation, because developers may utilize resources like Yjs or Automerge.

   articleDeltas.updateOne(15, Y.encodeStateAsUpdate(Y.Doc))

4. Return Conflict List to Last Writer or Arbitrator: This approach requires handling user responses to conflicts, similar to git. It places a significant burden on users and developers, and should only be considered as a last resort for specific use cases.

Conclusion

Offline-first web apps are great for ensuring a consistent user experience in areas with poor network connectivity. Despite the current challenges for application developers, basically all of the groundwork has already been laid.

I'd be glad to hear your comments, experiences, and feedback on the proposed API.

[ctholho]

A small update: I started experimenting in this repo: ctholho/qryo

The readme in that repo provides additional information, challenges and a roadmap – and there is an example Nuxt application.

I'd love some feedback here in discussion. Does my chosen path make any sense at all? I have the feeling it's going towards an opinionated implementation of tanstack/query with some additional stuff thrown in. Because I that means making many choices for devs, I would love some validation and criticism.

[wuservices]

@ctholho love the concepts and discussions. I see it's been almost a year since you were thinking about this, and didn't see any updates in the related repo. Is this still a project you're thinking about and do you have any recent developments?

I've been considering how you could work with anonymous users with Directus for an app that doesn't require authentication at first. Users might start anonymous, then create an account, or start anonymous, then sign in with an existing account, which would require some conflict resolution like you've anticipated - just like an offline scenario.

Also curious if you checked out Dexie.js. I haven't tried it myself yet, but Dexie.Syncable may address some of the sync primitives, and while there are some old an unmaintained add-ons for IndexedDB encryption, the next version of Dexie (5.0) will include an officially supported encryption add-on. I wonder if it'd be easy to write a client sync implementation and Directus extension around this.

[rijkvanzanten]

Love it! I'm wondering if this could "just" be a setting in the regular SDK, instead of having to be a separate package 🙂 I'd love to be able to use something like this for the admin app as well! I'm very curious to hear some more thoughts around the different conflict resolution techniques though. It sounds like that's something that you might have to be able to override method call to method call (eg what happens if you need a different resolution for articles than for authors etc) 🤔

[ctholho]

Thanks for your response! To answer your first question: It would be totally possible. However, there are a few things that made me choose differently:

• It's is a problem for all (open source) BaaS and could be tackled as such.
• I chose tanstack/query because I think it's the superior way to work with CRUD state management as opposed to a central store. Yet, caching and putting stuff into indexedDB is usually what requires devs to use central state management in the first place. Query already solves most of the problems. But, I might be totally mistaken and that's why I'm asking for your feedback.
• While I think tanstack/query's declarative, atomic state management is good for building apps, I don't think it fits every use case. For a quick intro see TK Dodos blog – React Query as a state manager. Implementing your own observers and enabling reactivity in react, vue etc. should not be the concern of directus maintainers. And yet, offline-first apps make things cumbersome because it necessitates a bit of coupling to the GUI. (E.g. showing which items are not saved, which could be outdated, warning before exiting, offering a manual "sync changes" button). Moreover, it requires a "holistic" approach and that's a big reason why offline-first is so expensive/hard.

I'm very curious to hear some more thoughts around the different conflict resolution techniques though. It sounds like that's something that you might have to be able to override method call to method call (eg what happens if you need a different resolution for articles than for authors etc) 🤔

Totally agree. CRDTs are a data model problem. A CRDT – as opposed to a normal data type – might require that you send different (or usually more) data to the server. The proposed API still expects you to be explicit about what you send to the server.

[rijkvanzanten]

It's is a problem for all (open source) BaaS and could be tackled as such.

Very true! On that train of thought, it's basically any REST or GraphQL API for that matter 🤔 If the abstracted solution is architected well, I can imagine using something like this for things like the Stripe API for example as well 🤯

Another brainfart question coming your way: you mentioned encrypting the data at rest in the local store before it's submitted to the database; if this is all client-side logic, how would you properly encrypt that data? Wouldn't the encryption keys etc be available, as it's all client-side logic that does that?

[ctholho]

No, that's not a brainfart question, because you're right. I am overselling the ease to implement that. A username and password could be enough to encrypt and decrypt if a user is offline. For UX reasons you'd want to get another key from the server – which could be a custom endpoint (or a lambda). It would be a fairly hard sell to require re-authentication every time users open the app if they rely on offline data and have to literally type in username and password.

But that is choice for application developers and for some projects it might be totally ok to trash everything every time they open the app. Or it might be preferable to allow startup only when you have a connection or you want users to re-authenticate every time they open the app – or devices give access to a secure enclave and they can skip authenticating manually (e.g. hybrid mobile apps).

[falmanna]

I was exploring RxDB and I think it is a really good piece of software. It supports many backends, one of which is graphql.
Now, there are some requirements before it can work with a graphql endpoint, things as graphql subscription, which is missing in directus.

I think supporting it would be really nice and would solve the need for offline-first. Hasura has a nice write-up about it.

[ctholho]

I think RxDB is an excellent choice for responsive offline-first applications. During my initial research I found multiple points, why I wouldn't want to use it in generalized library or as an abstracted way of building offline-first apps with the minimal amount of overhead.

• It ships a whole db with 45kb gziped
• Interfacing with IndexedDB is via a commercial plugin
• I want to interface with any HTTP endpoint and not rely on GraphQL or other demanding server side requirements

Maybe that's not the right choice for what you are building. Qryo will focus on being very generalizable and broadly applicable at the cost of performance for big datasets or real time applications. Although I don't think it will be noticeable at the scale I would see Qryo being used.

[falmanna]

@ctholho Aside from the size issue, the performance of the free backends is comparatively good . And for using any HTTP endpoints, you can either hit them directly or maybe build a custom remote support instead of using the graphql remote implementation. Someone has done it for supabase and it supports toggling offline support for each table separately.

At the end of the day, it is a tradeoff. RxDB can free you from getting locked down to one specific backend, but it has its drawbacks.

Finally, good luck with Qryo!

[aehlke]

i successfully built a realtime sync layer between rxdb and a swift web app host via postMessage. it's fantastic. I'm running Realm on the swift side which I have syncing to iCloud.

[beebase]

Maybe check out replicache.dev for some offline first ideas.

[aehlke]

rxdb

[flyingwolf1701]

replicache is great... but so expensive that just can't get it approved for work.

[development-vargamarcel]

Could something like this do the job? What would be the drawbacks? urql offline support documentation