MicroVault:一个简单的分布式二进制存储系统,约1000行代码
我厌倦了阅读成千上万页关于分布式存储系统的文档。我只想在几台服务器上可靠地存储文件。
于是我构建了 MicroVault——一个你可以真正理解的内容可寻址的二进制大对象存储。
*它的功能:*
- 在 N 个卷服务器上存储二进制大对象,并进行复制
- 使用 nginx 进行服务(读取时零开销)
- 使用 SQLite 存储元数据
- 使用一致性哈希进行分布
- 提供简单的 HTTP API
*它不具备的功能:*
- 自定义网络协议
- 分布式共识
- 内置身份验证/加密(使用标准工具)
- 任何复杂的功能
*架构:*
- 主服务器:协调写入操作,将元数据存储在 SQLite 中
- 卷服务器:nginx + 简单的包装器(约 100 行代码)用于 PUT/DELETE
- 存储:磁盘上的常规文件,按哈希前缀组织
- 工具:重建、重新平衡、验证、压缩
*为什么选择 nginx?*
因为它经过数十年的优化,专门用于服务静态文件。对于读取(大多数操作),它是纯粹的 nginx,零开销。
*灵感来源:*
George Hotz 的 minikeyvalue——证明分布式存储不需要复杂。
整个代码库小到可以在一个晚上审计完毕。你可以在一个周末分叉它并将其变成你自己的项目。
GitHub: https://github.com/afonp/microvault
使用 Go、SQLite、nginx 构建,并对复杂性保持健康的不屑态度。
查看原文
I got tired of reading thousands of pages of documentation for distributed storage systems. All I wanted was to store files across a few servers reliably.<p>So I built MicroVault - a content-addressable blob store that you can actually understand.<p>*What it does:*
- Stores blobs across N volume servers with replication
- Uses nginx for serving (zero overhead on reads)
- SQLite for metadata
- Consistent hashing for distribution
- Simple HTTP API<p>*What it doesn't do:*
- Custom network protocols
- Distributed consensus
- Built-in auth/encryption (use standard tools)
- Anything complex<p>*The architecture:*
- Master server: coordinates writes, stores metadata in SQLite
- Volume servers: nginx + tiny wrapper (~100 lines) for PUT/DELETE
- Storage: regular files on disk, organized by hash prefix
- Tools: rebuild, rebalance, verify, compact<p>*Why nginx?*
Because it's been optimized for serving static files for decades. For reads (the majority of operations), it's pure nginx with zero overhead.<p>*Inspired by:*
George Hotz's minikeyvalue - proof that distributed storage doesn't need to be complex.<p>The entire codebase is small enough to audit in an evening. You could fork it and make it your own in a weekend.<p>GitHub: https://github.com/afonp/microvault<p>Built with Go, SQLite, nginx, and a healthy disrespect for complexity.