Wednesday, April 07, 2010

Cassandra: Fact vs fiction

Cassandra has seen some impressive adoption success over the past months, leading some to conclude that Cassandra is the frontrunner in the highly scalable databases space (a subset of the hot NoSQL category). Among all the attention, some misunderstandings have been propagated, which I'd like to clear up.

Fiction: "Cassandra relies on high-speed fiber between datacenters" and can't reliably replicate between datacenters with more than a few ms of latency between them.

Fact: Cassandra's multi-datacenter replication is one of its earliest features and is by far the most battle-tested in the NoSQL space. Facebook had Cassandra deployed on east and west coast datacenters since before open sourcing it. SimpleGeo's Cassandra cluster spans 3 EC2 availability zones, and Digg is also deployed on both coasts. Claims that this can't possibly work are an excellent sign that you're reading an article by someone who doesn't know what he's talking about.

Fiction: "It’s impossible to tell when [Cassandra] replicas will be up-to-date."

Fact: Cassandra provides consistency when R + W > N (read replica count + write replica count > replication factor), to use the Dynamo vocabulary. If you do writes and reads both with QUORUM, for one example, you can expect data consistency as soon as there are enough reachable nodes for a quorum. Cassandra also provides read repair and anti-entropy, so that even reads at ConsistencyLevel.ONE will be consistent after either of these events.

Fiction: Cassandra has a small community

Fact: Although popularity has never been a good metric for determining correctness, it's true that when using bleeding edge technology, it's good to have company. As I write this late at night (in the USA), there are 175 people in the Cassandra irc channel, 60 in the HBase one, 32 in Riak's, and 15 in Voldemort's. (Six months ago, the numbers were 90, 45, and 12 for Cassandra, HBase, and Voldemort. I did not hang out in #riak yet then.) Mailing list participation tells a similar story.

It's also interesting that the creators of Thrudb and dynomite are both using Cassandra now, indicating that the predicted NoSQL consolidation is beginning.

Fiction: "Cassandra only supports one [keyspace] per install."

Fact: This has not been true for almost a year (June of 2009).

Fiction: Cassandra cannot support Hadoop, or supporting tools such as Pig.

Fact: It has always been straightforward to send the output of Hadoop jobs to Cassandra, and Facebook, Digg, and others have been using Hadoop like this as a Cassandra bulk-loader for over a year. For 0.6, I contributed a Hadoop InputFormat and related code to let Hadoop jobs process data from Cassandra as well, while cooperating with Hadoop to keep processing on the nodes that actually hold the data. Stu Hood then contributed a Pig LoadFunc, also in 0.6.

Fiction: Cassandra achieves its high performance by sacrificing reliability (alternately phrased: Cassandra is only good for data you can afford to lose)

Fact: unlike some NoSQL databases (notably MongoDB and HBase), Cassandra offers full single-server durability. Relying on replication is not sufficient for can't-afford-to-lose-data scenarios; if your data center loses power, you are highly likely to lose data if you are not syncing to disk no matter how many replicas you have, and if you run large systems in production long enough, you will realize that power outages through some combination of equipment failure and human error are not occurrences you can ignore. But with its fsync'd commitlog design, Cassandra can protect you against that scenario too.

What to do after your data is saved, e.g. backups and snapshots, is outside of my scope here but covered in the operations wiki page.

16 comments:

Ashwin Jayaprakash said...

Is FB using the Apache version of Cassandra or their own internal version?

Toby DiPasquale said...

Totally love Cassandra, however, I would mention that all three EC2 availability zones are currently in the same datacenter on the East Coast so that's not such an awesome example of datacenter distribution.

joestump said...

Toby: You're wrong that AZs are in the same data center. AZs are physically different locations that are ~10-15 miles apart.

kryton said...

Fact or fiction: Cassandra can brew my coffee in multiple data centers at the same time?

cloudhead said...

note that in mongo you can now use the --syncdelay option to improve the durability

Anonymous said...

A question that is not related to this blog but I guess you are the right person to ask: why Cassandra is written in Java? Excuse me if this question is over asked. Because writing robust apps would be easy with Java? Or any reason else. I just want to know why if there is a why, thank you

Alexey Lapusta said...

Are there any solutions for using Hadoop with Cassandra?

Platypus said...

For the record, since I've been known to write about this before, I just want to say I've never denied that Cassandra can work across data centers. What I've said is that I don't think the way it works in that environment is the best way, but that's an entirely different kind of statement. Facts are facts, and good enough is good enough. It's just part of my nature to worry about the icebergs we haven't hit yet.

LaggyLuke said...

Just for the record, channel #mongodb on FreeNode has 207 people right now. This makes whole thing look a bit biased.

Anonymous said...

What is the lowest speed boundary?

Rafael Ribeiro said...

Hi Jonathan,

I am in the process of writting an article about Cassandra to one of the Java focused magazines on Brazil. As I am giving a overview of its architecture before covering its usage on Java I'd like to cover a little of its gossip protocol. Do you suggest any readings? Any particular paper/post/whatsoever?

Feel free to contact me over my e-mail that you'll probably see in the post.

best regards,
Rafael Ribeiro

Jonathan Ellis said...

For Gossip, you want to read http://wiki.apache.org/cassandra/ArchitectureGossip and http://www.cs.cornell.edu/home/rvr/papers/flowgossip.pdf

johan said...

So my question is.. can Cassandra replace mysql in all conditions and scenarios?
if it can't which ones that it can't?

Rafael Ribeiro said...

Hi Jonathan!

First of all tks a lot for the article and wiki page link, it clarified a lot of things.
Still... mind if I certify that I got things right? According to wiki and using the naming conventions present on the article Cassandra is a anti-entropy gossip protocol since wiki only mentions it gossips each second and does not mention anything about stopping due to a certain criteira, right? Second wiki says it randomly picks up peers to receive deltas.
Another thing I noticed is that Cassandra employs a protocol similar to the one described on the article, right? And at last... it is a push pull right? Since it pushes deltas upon updates and pulls upon reads (the mentioned read repair thing), right?

tks a lot for all the help!
kindly regards,
Rafael Ribeiro

Robert said...

currently there are no PHP support for using Cassandra.. any info regarding implement Cassandra with PHP (driver) or something?

Jonathan Ellis said...

PHP is supported via Thrift, like everything else. See http://wiki.apache.org/cassandra/ClientOptions