ZVOLs vs SeaweedFS for VM Storage: Performance and S3 Gateway Considerations

Hey @cronjob26,

We actually pushed the initial code to get it working in FreeBSD, but the performance was a bit shaky given the FUSE layer. That said, we would strongly encourage the community to work it back into the project now that there has been a good amount of development around it since we pushed it.

Let's work together to give our community a lightweight, enterprise-grade distributed storage alternative that honors BSD's philosophy of speed and simplicity!

I was not very impressed when I researched SeaweedFS. Its white paper makes a lot of disingenuous claims. Here are the notes I made at the time. I like MooseFS's design better. Though I still haven't seen anything better than Ceph.

SeaweedFS is a "fast distributed storage system for blobs, objects, and files". It has a FreeBSD port already, which is good. And it boasts some impressive bullet points, like erasure coding, "O(1) disk seek", tiering, "Per-file disk overhead less than 40 bytes", "no SPOF", and redundant metadata servers (filers). However, on closer inspection some of those bullet points aren't as strong as they sound. For one thing, SeaweedFS is written at a very high level. It relies on a COTS database to store all metadata (PostgresQL, Redis, and several others are supported), and POSIX file systems on each volume server to store the file data. So the metadata scaling problem is solved by delegating to a distributed database like Redis (The white paper also says that it's possible to run multiple filers, but doesn't provide details on how that works). And the "O(1) disk seek" neglects the effort to look up metadata in the database, and the effort that the underlying POSIX file system must expend. Even worse, the "O(1) disk seek" criterion only holds for small files. Large files are split into chunks, and very large files' chunk lists are themselves split into multiple chunks, making the seek a multi-step process. For our applications, we're only concerned with very large files, so that's a warning sign. Also, the "40 bytes per-file overhead" completely neglects the inode, which is stored as an empty file on the POSIX file system. It also neglects the database's internal overhead, which might be considerable. But the white paper disingenuously compares that "40 bytes" to XFS's inode size (536 bytes). For directories, the white paper says "The caveat is that the directory listing is not supported (yet)". As for erasure coding, it isn't supported for writes that come from the client. It's only supported afterwards, in a volume compaction process.

In conclusion, I think that SeaweedFS is optimized for small files, intended primarily as a replacement for HDFS/Hadoop, and unlikely to achieve high performance.