Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why couldn't twitter scale by adding servers the way sites like facebook have?

I have been looking for an explanation for why twitter had to migrate part of its middle ware from Rails to Scala. What prevented them from scaling the way facebook has, by adding servers as its user base expanded. More specifically what about the Ruby/Rails technology prevented the twitter team from taking this approach?

like image 861
Jason Avatar asked Mar 17 '12 06:03

Jason


People also ask

What server does Twitter run on?

Twitter will use AWS Graviton2-based instances on Amazon Elastic Compute Cloud (Amazon EC2) to power its cloud-based workloads as well as AWS container services to develop and deploy new features and applications consistently across its hybrid infrastructure.

Does Twitter have its own servers?

Today, Twitter still runs its main operations on its own servers. But its relationship with Google has evolved and expanded over the last three years. “In some cases, we will move workloads as-is to the cloud.

Does Twitter have datacenters?

Twitter's worldwide network directly interconnects with over 3,000 unique networks in many datacenters worldwide.

Where are the twitter servers located?

After an extensive search in which it considered multiple East Coast sites, Twitter has settled on Atlanta as the location for its next data center. The company will move servers into an enormous data center operated by QTS (Quality Technology Services) in downtown Atlanta, industry sources say.


1 Answers

It's not that Rails doesn't scale, but rather, requests for "live" data in Ruby (or any interpreted language) do not scale, as they are comparatively far more expensive both in terms of CPU & memory utilization than their compiled language counterparts.

Now, were Twitter a different type of service, one that had the same enormous user base, but served data that changed less frequently, Rails could be a viable option via caching; i.e. avoiding live requests to the Rails stack entirely and offloading to front end server and/or in-memory DB cache. An excellent article on this topic:

How Basecamp Next got to be so damn fast

However, Twitter did not ditch Rails for scaling issues alone, they made the switch because Scala, as a language, provides certain built-in guarantees about the state of your application that interpreted languages cannot provide: if it compiles, time wasting bugs such as fat-fingered typos, incorrect method calls, incorrect type declarations, etc. simply cannot exist.

For Twitter TDD was not enough. A quote from Dijkstra in Programming in Scala illustrates this point: "testing can only prove the presence of errors, never their absence". As their application grew, they ran into more and more hard to track down bugs. The magical mystery tour was becoming a hindrance beyond performance, so they made the switch. By all accounts an overwhelming success, Twitter is to Scala what Facebook is to PHP (although Facebook uses their own ultra fast C++ preprocessor so cheating a bit ;-))

To sum up, Twitter made the switch for both performance and reliability. Of course, Rails tends to be on the innovation forefront, so the 99% non-Twitter level trafficked applications of the world can get by just fine with an interpreted language (although, I'm now solidly on the compiled language side of the fence, Scala is just too good!)

like image 74
virtualeyes Avatar answered Oct 14 '22 14:10

virtualeyes