Facebook - Credit: facebook.com
Facebook has released Scribe, its core code under a free license, Apache License version 2.
Here at Facebook, we’re constantly facing scaling challanges because of our enormous growth. One particular problem we encountered a couple of years ago was collection of data from our servers. We were collecting a few billion messages a day (which seemed like a lot at the time) for everything from access logs to performance statistics to actions that went to News Feed. We used a variety of different technologies for the different use cases, and all of them were bursting at the seams. We decided to build a unified system (called Scribe) to handle all of these cases, and do it in a way that would scale with Facebook’s growth. The system we built turned out to be enormously useful, handling over 100 use cases and tens of billions of messages a day. It has also been battle tested by just about anything that can go wrong, so I encourage you to take a look at the newly opened Scribe source and see if it might be useful for you. To give the code some context, I’m going to go through the major design decisions we made to allow the system to scale.
Yay for Facebook, sorta. It’d be nice if they did this before they felt the assurance of having people totally bought, but this is good regardless. I think this is a nice quote from the Open Enterprise Blog: “But in one other respect, there’s something big going on here. The fact that Facebook tosses out its code as open source without making a big ballyhoo about it, and without even bothering to justify the move shows just how mundane – in the best possible sense – free software has become for a modern software company. The benefits of opening up your main code are so obvious, they don’t even need to be enumerated anymore: it’s just the way the Facebook generation of coders does things.”