How does Facebook continuously update without breaking down? (VIDEO)

For a company devoted to making people's lives as transparent as possible, Facebook is extremely secretive.  So it was a rare treat to watch this presentation by Chuck Rossi, a release engineer: How Facebook pushes updates to its site every day.

I thought it was fascinating as Rossi showed screenshots of the myriad tools and processes used by the world's biggest social network.  Intriguingly, he said there were certain things he couldn't show, as they would reveal Facebook's special sauce.

The video is well worth watching for anyone interested in issues of scalability, programming, and database design.  Being a social network is a big challenge.  Users aren't just passive readers.  They're writing status updates, uploading photos, and playing games.  These tasks all consume far more resources than the average blog. 

You can't really control user activity, so the traffic can really spike upwards, regardless of whatever schedule the engineers are on.  Rossi talks about how weekends are especially busy, since that's when everyone is "uploading their drunken party pics." 

In the presentation, Rossi said that Facebook tried to find the balance between a free-for-all hacker culture that encourages innovation, and a careful release process that minimizes the risks of site downtime. 

My favorite parts were where Rossi talked about the human element.  At the beginning and end of the video, he stressed the two lessons he wanted the audience to take away:

1)  Tools won't save you.

2)  You need the right people, in the right culture, at the right company. 

Rossi revealed that Facebook had a system called "Karma," which was sort of a "Like" button for employees.  He explained that every employee starts with 4 stars.  Then they lose stars whenever he clicks on thumbs down, due to whatever errors or screw-ups that inevitably occur.  Half-jokingly, Rossi said he didn't know what the thumbs-up button did, as he's never used it. 

Most managers keep a log of how well their subordinates are doing, even if it's only in their heads and never written down.  Rossi said he uses Karma to quickly get a gauge of an employee's performance at a glance.  It's an external record of the score-keeping he does in his mind anyway.  But when you're managing 500 engineers like Rossi does, a computer app does help.