Time for a Rewrite in Java or Erlang?

Replies to This Discussion

Permalink Reply by Daniel Leuck on February 8, 2010 at 10:27am

Brian Russo: I wonder if their issues are more about architecture or implementation.

This could very well be the case. Most web applications are DB or network constrained rather than being limited by the speed code is executing. Four out of five times when we audit apps with poor performance the issue is unnecessary round trips to the DB.

Twitter is obviously a special case. They do everything in a giant memory pool with a background process writing updates to the DB. I know they do some clever things with caching, but the fact its implemented with a hodgepodge of Ruby, Scala, Java and a bit of C makes me think the system has evolved in a less than ideal manner and is probably due for a rewrite. They should probably pick a language and stick with it :-) I'm sure they have some clever engineers working on it.

Brian Russo: Do media platforms such as twitter really need to survive peak flows that only occur a few times a year? I don't know how often twitter actually goes down, but most other networks aren't that robust and still work fine - telephone grids, road grids, etc.

True, but Twitter is down far more than a few times a year.

Permalink Reply by Cameron Souza on February 8, 2010 at 2:06pm

This is an interesting article about Twitter's architecture: Scaling Twitter: Making Twitter 10000 Percent Faster

Excerpt: Some interesting changes from Twitter's Evan Weaver: everything in RAM now, database is a backup; peaks at 300 tweets/second; every tweet followed by average 126 people; vector cache of tweet IDs; row cache; fragment cache; page cache; keep separate caches; GC makes Ruby optimization resistant so went with Scala; Thrift and HTTP are used internally; 100s internal requests for every external request...