Call for Collaboration: Designing a Dgraph offline-first library

[amaster507 (Anthony Master)]

@pbassham, @maaft, @uncle_juniper, @marcown, @verneleem Let’s pick up this topic here to figure out how to better do a better offline first Dgraph client.

Some related posts:

@marcown

Furthermore, there is the use case to use dgraph locally in a standalone application. It is fast, its supported on windows as a single .exe and really easy to setup with the schema.

I think, it should not be on low priority because it has great potential on windows. Moreover, there was no disclaimer on the downloads page and some (i.e. me) have built a application using dgraph.

Also, its not useable for learning purposes if you need > 8GB of RAM to post one schema.

@maaft

Same use-case as with @marcown here. We’re using a dgraph instance integrated into an app that the user can download. That user will be in 90% of cases a windows user. And in 80% of cases this user will have not the experience to configurate WSL or docker or whatever “workaround”.

Let me try to explain our use-case very briefly:

1. The user can do his work offline (which is often a requirement for software that is being used in production facilities due to security/connection issues)
2. His work is saved in a local running dgraph instance
3. As soon as he has internet his instance is synchronized with our dgraph instance and he can use our cloud services on his data

@MichelDiz

Have you thought about using Apollo state management? or pouchdb, lowdb, or even SQLite. Those are DBs for that purpose. Dgraph is meant to be a cluster, therefore a server-based only. I think the Apollo State fits well this case. It can (theoretically) sync with Dgraph’s GraphQL.

@maaft

How does this help with offline-first applications? They can run offline for several weeks without seeing any internet. A cache doesn’t help here. The app will be closed and restarted during that period. We need persistence.
The client uses the same graphql queries supported by dgraph. How do you think will we be able to use pagination, filtering and all the other dgraph stuff on non-graph databases? We decided for dgraph because its “schema-first” and easy to use. Our complete code is build around the fact that the dgraph schema is the single source of truth. Using any other database than dgraph would require:

• to implement a translation layer dgraph-style-graphql-queries ↔ SQL (incl. pagination, filtering, @cascade etc pp) this is a complete nightmare because it basically means to re-implement dgraph from scratch
  or
• use two different react apps that use different APIs a complete nightmare because it will mean double the frontend-dev work and lead to unmaintainable code

Sorry, this is nowhere close to a solution. I’d be honestly happy if it was though.

@MichelDiz

You wanna provide the whole DB? That sounds insecure. Give end-user access to the whole DB. He just needs the basics. A DB commonly used in smartphones is enough. But it requires a business logic to be built.

BTW, I’m pretty sure that others have been in your situation before. You may find better solutions out there. With the experience of others.

@uncle_juniper

My team is working on an electron/react based app (so highly portable by design) and we are trying to find a fitting backend which does not reduce our portability options. We already started a backend in go with prisma/sqlite which is already really nice and also highly portable. But getting all the graphql queries without writing code with dgraph would be a huge thing for us.

So I gave dgraph a try. I was running it on windows and I also had these strange memory issues.

@uncle_juniper

From what I read, dgraph is intended for large-scale, multiserver environments exclusively and not so much for our intended embedded-like use-case. This is really sad because it would fit incredibly good as a persistence layer that we can ship with our software. Seeing the posts above it’s clear that I am not the first with this seemingly unusual idea.

@uncle_juniper

Here is another point: @maaft mentioned something about server synchronisation which we also want to to do in the near future, but probably within our application layer. I imagine this should be pretty easy to do when you can use the same graphql client and the same data types to access the local “embedded” dgraph and the central server.

Idea Phase

What we want to come up with eventually is a thin, embeddable, syncable client. From what we have so far:

• The thin client should have a GraphQL endpoint
• The thin client should have a local data store of some kind
• The thin client should be able to two way sync data when online to the main Dgraph database whether by DQL or GraphQL.
• This will very likely depend upon Query server timestamps or some other means of synce last sync meta data.
• This thin client should be lite and embeddable. (With lowest memory consumption as possible)
• This thin client should answer queries to best of ability when offline and fill in gaps with local cache when online.
• This thin client needs to respect GraphQL authentication rules.
• TBD: will users be authenticated based upon local machines local credentials with a generated JWT, or will users need to authenticate against an online service that responds with a JWT. This discussion will determine if users can authenticate when offline, or if can only use last online authenticated credentials for as long as they last. Maybe allow users to authenticate when online with a long lived JWT. What happens when user logs out, or a different user logs in from a different client on the same thin client.
• In the sync process, it may be possible that the same node was updated by the client when offline and during that same offline time also by the Dgraph online db. The sync process needs to account for this. Storing mutations that a user performed while offline and then pushing those mutations in a sync when a user connects may be a possible solution, but not perfect, as data may have changed on the source of truth and the mutations no longer have the same effect, also the JWT may no longer be valid when reconnected if multiple users are using the same thin-client.

And that is the tip of the iceberg in my opinion.

Research Questions:

• Are there any thin graph dbms like SQLite?
• What about embeddable key-value stores or NoSQL?
• Any references from the web with anyone that has made a GraphQL API client that syncs when online?

Possible Usable Tech

• IndexDB
• pouchdb
• lowdb
• SQLite
• Apollo Local State Management (local resolver deprecated in Apollo Client 3.0)

Client Language

• WASM using Go
• I am a Javascript developer myself, but this is open to suggestions.

Challenges:

• Schema Updates between SSOT and client
• Supporting the same queries and mutations supported on SSOT.
  Example: If when online the client requests queryContact(first: 100)... then when offline the client is qeuried for getContact(id: "0x1")..., This also applies to querying the thin client with filters and arguments not used when online. Apollo State Management cannot handle this very well, and any other local db will require a completely customized GraphQL layer in this thin client.
• overcoming N+1 without a embeddable graph database. Is this even an issue with a local dbms since there are no network trips.

Really Advanced Stuff

• It would be neat if after all of the above is figured out if local LAN clients could communicate with each other to fill in the data from other local thin clients when working together in an offline mode.
  Example: I was working on my Desktop and internet went out, and I am now on my laptop and my Desktop is on the same LAN, I make a query for data that is not locally, but can be obtained on the local network.

Back to work now on my current tasks at hand that I can do without needing an offline client at the moment.