Twitter’s Hadoop clusters migrate to Google Cloud

Today, Twitter revealed that it will be moving a portion of its infrastructure to Google’s Cloud Platform.

The social networking company said that the move is to keep cold data storage, as well as its compute Hadoop clusters.

Twitter’s Engineering CTO, Parag Agrawal, wrote in Twitter’s blog post: Every day, people come to Twitter to find out what’s happening in the world and talk about it. With hundreds of millions of Tweets sent every day, it is critical that our infrastructure and data platforms are able to scale.”

Hadoop clusters

Currently, the companies Hadoop clusters are kept in its own personal data centres, which is a small percentage of its total infrastructure.

“As we have previously discussed, the Hadoop compute system is the core of our data platform, and Twitter runs multiple large Hadoop clusters that are among the biggest in the world,” added Agrawal.

He also added that Hadoop file systems host more than 300PB of data across tens of thousands of servers and that over the past few years, the company has been assessing its platform and infrastructure needs to make sure it’s well positioned to keep up with the growing needs of the service.

Written by Leah Alger