Refactor: The Great Refactor

[carsonfarmer]

The Great Refactor

This is essentially a new Epic that describes a merging of Database, Store, and Client into a single, unified JS ThreadDB API.

Overview

The primary plan is to “shift” support for threads-database in favor of threads-client. This limits our API flexibility in some respects, but what we lose in control, we gain in accessibility and a more streamlined development focus. Additionally, we will deprecate threads-network-client and threads-network, which are low-level Threads protocol libraries. These were never really designed to be publicly consumed modules, so this isn’t likely to be a controversial move. Finally, threads-encoding will be moved to its own repo and maintained for demos and examples that want to build on Threads more directly. The primary change will be to marry threads-database and threads-store on top of threads-client, to provide an offline-first database with fast MongoDB style APIs and queries, and a Realm/MongoDB like local/remote sync interface.

Goals/Outcomes

The goals/outcomes of this refactor are to be able to provide a PouchDB + CouchDB style experience for developers that syncs data to remote peers that can then serve said data over decentralized protocols. Something like, “the dynamic database that syncs to IPFS”. Another analogy would be MongoDB + Realm, where you have a free, mobile/browser database that syncs with a remote MongoDB service. The resultant library be familiar to anyone who has built an application using something like Realm, Pouch, or Mongo. The difference here is everything is available over decentralized protocols such as PubSub/GossipPub, IPFS, IPNS, HTTP gateways, and more.

Roadmap

• Migration
  • Drop threads-network-client and threads-network
  • Move threads-encoding to its own repo (which already exists)
  • Consolidate and combine threads-database with threads-store
  • Import threads-client into threads-database as the primary networking/DB layer
  • Deprecate threads-client in favor of the new threads-database (which is now a nicely wrapped threads-client)
  • Update local (within repo/APIs) documentation to reflect these migrations
• Local-first
  • Upgrade local-first behavior to support threads-client updates
  • Implement required remote sync API(s) in go-threads to support faster local caching
  • Update local (within repo/APIs) documentation to reflect these new features
• P2P
  • Expose Pubsub updates at the database level
  • Expose APIs for accessing database collections and entities directly from the JS clients
  • Return IPFS CIDs for all database updates (rather than Instance IDs alone)
  • Update local (within repo/APIs) documentation to reflect these new features
• DX
  • Refocus primary developer entry-point to js-hub (this is mostly already done)
  • Move from current low level auth access to higher-level, token-based, browser-first implementation. For NodeJS applications, the low-level APIs will remain, or they can utilize a compatibility layer for cookie/header management.
  • Update local (within repo/APIs) documentation to reflect these changes
• Documentation - All of the above will require major documentation updates
  • Update README of js-threads to clearly state primary goals/functionality of the updated library
  • Additional information on Threads inner-workings should be directed to high-level white paper overview or Go Threads
  • Re-working of primary auth flow (now more "browser native") examples and demos
  • Port all existing Database demos to the new Database design (this is likely only the Chat demo)
  • Refocus primary JS client focus to Buckets where appropriate