Marmot
What & Why?
Marmot is a distributed SQLite replicator with leaderless, and eventual consistency. It allows you to build a robust replication
between your nodes by building on top of fault-tolerant NATS JetStream.
If you are running a read heavy website based on SQLite, you should be easily able to scale it out by adding more SQLite replicated nodes.
SQLite is probably the most ubiquitous DB that exists almost everywhere, Marmot aims to make it even more ubiquitous for server
side applications by building a replication layer on top.
Quick Start
Download latest Marmot and extract package using:
tar vxzf marmot-v*.tar.gz
From extracted directory run examples/run-cluster.sh. Make a change in /tmp/marmot-1.db using:
bash > sqlite3 /tmp/marmot-1.db
sqlite3 > INSERT INTO Books (title, author, publication_year) VALUES (‘Pride and Prejudice’, ‘Jane Austen’, 1813);
Now observe changes getting propagated to other database /tmp/marmot-2.db:
bash > sqlite3 /tmp/marmot-2.db
sqlite3 > SELECT * FROM Books;
You should be able to make changes interchangeably and see the changes getting propagated.
Out in wild
Here are some official, and community demos/usages showing Marmot out in wild:
- 2-node HA for edge Kubernetes – Using Marmot
- Scaling Isso with Marmot on Fly.io
- Scaling PocketBase with Marmot on Fly.io
- Scaling PocketBase with Marmot 0.4.x
- Scaling Keystone 6 with Marmot 0.4.x
What is the difference from others?
Marmot is essentially a CDC (Change Data Capture) and replication pipeline running top of NATS. It can automatically configure appropriate
JetStreams making sure those streams evenly distribute load over those shards, so scaling simply boils down to adding more nodes, and
re-balancing those JetStreams (auto rebalancing not implemented yet).
There are a few solutions like rqlite, dqlite, and
LiteFS etc. All of them either are layers on top of SQLite (e.g.
rqlite, dqlite) that requires them to sit in the middle with network layer in order to provide
replication; or intercept physical page level writes to stream them off to replicas. In both
cases they require a single primary node where all the writes have to go, and then these
changes are applied to multiple readonly replicas.
Marmot on the other hand is born different. It’s born to act as a side-car to your existing processes:
- Instead of requiring single primary, there is no primary! Which means any node can make changes to its local DB. Marmot will use triggers to capture your changes, and then stream them off to NATS.
- Instead of being strongly consistent, Marmot is eventually consistent. Which means no locking, or blocking of nodes.
- It does not require any changes to your existing SQLite application logic for reading/writing.
Making these choices has multiple benefits:
- You can read,
