{"id":12647,"date":"2018-05-04T14:49:49","date_gmt":"2018-05-04T13:49:49","guid":{"rendered":"http:\/\/www.devopsonline.co.uk\/?p=12647"},"modified":"2018-05-04T14:49:49","modified_gmt":"2018-05-04T13:49:49","slug":"twitters-hadoop-clusters-migrate-to-google-cloud","status":"publish","type":"post","link":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/","title":{"rendered":"Twitter’s Hadoop clusters migrate to Google Cloud"},"content":{"rendered":"

Today, Twitter revealed that it will be moving a portion of its infrastructure to Google’s Cloud Platform.<\/p>\n

The social networking company said that the move is to keep cold data storage, as well as its compute Hadoop clusters.<\/p>\n

Twitter’s Engineering CTO, Parag Agrawal, wrote in Twitter’s blog post: Every day, people come to Twitter to find out what\u2019s happening in the world and talk about it. With hundreds of millions of Tweets sent every day, it is critical that our infrastructure and data platforms are able to scale.”<\/p>\n

Hadoop clusters<\/h2>\n

Currently, the companies Hadoop clusters are kept in its own personal data centres, which is a small percentage of its total infrastructure.<\/p>\n

“As we have\u00a0previously discussed, the Hadoop compute system is the core of our data platform, and Twitter runs multiple large Hadoop clusters that are among the biggest in the world,” added Agrawal.<\/p>\n

He also added that Hadoop file systems host more than 300PB of data across tens of thousands of servers and that over the past few years, the company has been assessing its platform and infrastructure needs to make sure it’s well positioned to keep up with the growing needs of the service.<\/p>\n

Written by Leah Alger<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Twitter reveals that it will be moving a portion of its infrastructure to Google’s Cloud Platform<\/p>\n","protected":false},"author":12,"featured_media":12648,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","pmpro_default_level":"","footnotes":""},"categories":[2,871],"tags":[1522,1818,2756,1030,2757,1200],"yoast_head":"\nTwitter's Hadoop clusters migrate to Google Cloud<\/title>\n<meta name=\"description\" content=\"Today, Twitter revealed that it will be moving a portion of its infrastructure to Google's Cloud Platform to compute its Hadoop clusters\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Twitter's Hadoop clusters migrate to Google Cloud\" \/>\n<meta property=\"og:description\" content=\"Today, Twitter revealed that it will be moving a portion of its infrastructure to Google's Cloud Platform to compute its Hadoop clusters\" \/>\n<meta property=\"og:url\" content=\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/\" \/>\n<meta property=\"og:site_name\" content=\"DevOps Online North America\" \/>\n<meta property=\"article:published_time\" content=\"2018-05-04T13:49:49+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/05\/iStock-178101320.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"722\" \/>\n\t<meta property=\"og:image:height\" content=\"484\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Leah Alger\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DevOpsAmerica\" \/>\n<meta name=\"twitter:site\" content=\"@DevOpsAmerica\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Leah Alger\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/\"},\"author\":{\"name\":\"Leah Alger\",\"@id\":\"https:\/\/devopsnews.online\/#\/schema\/person\/d705a7d7c56b625b482e9b13a8e49a2c\"},\"headline\":\"Twitter’s Hadoop clusters migrate to Google Cloud\",\"datePublished\":\"2018-05-04T13:49:49+00:00\",\"dateModified\":\"2018-05-04T13:49:49+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/\"},\"wordCount\":210,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/devopsnews.online\/#organization\"},\"image\":{\"@id\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/05\/iStock-178101320.jpg\",\"keywords\":[\"cloud-computing\",\"google cloud platform\",\"hadoop clusters\",\"Infrastructure\",\"tweet\",\"twitter\"],\"articleSection\":[\"Featured\",\"Recruitment\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/\",\"url\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/\",\"name\":\"Twitter's Hadoop clusters migrate to Google Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/devopsnews.online\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/05\/iStock-178101320.jpg\",\"datePublished\":\"2018-05-04T13:49:49+00:00\",\"dateModified\":\"2018-05-04T13:49:49+00:00\",\"description\":\"Today, Twitter revealed that it will be moving a portion of its infrastructure to Google's Cloud Platform to compute its Hadoop clusters\",\"breadcrumb\":{\"@id\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#primaryimage\",\"url\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/05\/iStock-178101320.jpg\",\"contentUrl\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/05\/iStock-178101320.jpg\",\"width\":722,\"height\":484,\"caption\":\"clusters\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/devopsnews.online\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Twitter’s Hadoop clusters migrate to Google Cloud\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/devopsnews.online\/#website\",\"url\":\"https:\/\/devopsnews.online\/\",\"name\":\"DevOps Online North America\",\"description\":\"by 31 Media Ltd.\",\"publisher\":{\"@id\":\"https:\/\/devopsnews.online\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/devopsnews.online\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/devopsnews.online\/#organization\",\"name\":\"DevOps Online North America\",\"url\":\"https:\/\/devopsnews.online\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/devopsnews.online\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2020\/03\/DevOpsOnline_email.png\",\"contentUrl\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2020\/03\/DevOpsOnline_email.png\",\"width\":198,\"height\":64,\"caption\":\"DevOps Online North America\"},\"image\":{\"@id\":\"https:\/\/devopsnews.online\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/DevOpsAmerica\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/devopsnews.online\/#\/schema\/person\/d705a7d7c56b625b482e9b13a8e49a2c\",\"name\":\"Leah Alger\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/devopsnews.online\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/0ac9e809e8ffafd8330210900bac04c0?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/0ac9e809e8ffafd8330210900bac04c0?s=96&d=mm&r=g\",\"caption\":\"Leah Alger\"},\"url\":\"https:\/\/devopsnews.online\/author\/leah-alger\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Twitter's Hadoop clusters migrate to Google Cloud","description":"Today, Twitter revealed that it will be moving a portion of its infrastructure to Google's Cloud Platform to compute its Hadoop clusters","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/","og_locale":"en_US","og_type":"article","og_title":"Twitter's Hadoop clusters migrate to Google Cloud","og_description":"Today, Twitter revealed that it will be moving a portion of its infrastructure to Google's Cloud Platform to compute its Hadoop clusters","og_url":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/","og_site_name":"DevOps Online North America","article_published_time":"2018-05-04T13:49:49+00:00","og_image":[{"width":722,"height":484,"url":"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/05\/iStock-178101320.jpg","type":"image\/jpeg"}],"author":"Leah Alger","twitter_card":"summary_large_image","twitter_creator":"@DevOpsAmerica","twitter_site":"@DevOpsAmerica","twitter_misc":{"Written by":"Leah Alger","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#article","isPartOf":{"@id":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/"},"author":{"name":"Leah Alger","@id":"https:\/\/devopsnews.online\/#\/schema\/person\/d705a7d7c56b625b482e9b13a8e49a2c"},"headline":"Twitter’s Hadoop clusters migrate to Google Cloud","datePublished":"2018-05-04T13:49:49+00:00","dateModified":"2018-05-04T13:49:49+00:00","mainEntityOfPage":{"@id":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/"},"wordCount":210,"commentCount":0,"publisher":{"@id":"https:\/\/devopsnews.online\/#organization"},"image":{"@id":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#primaryimage"},"thumbnailUrl":"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/05\/iStock-178101320.jpg","keywords":["cloud-computing","google cloud platform","hadoop clusters","Infrastructure","tweet","twitter"],"articleSection":["Featured","Recruitment"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/","url":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/","name":"Twitter's Hadoop clusters migrate to Google Cloud","isPartOf":{"@id":"https:\/\/devopsnews.online\/#website"},"primaryImageOfPage":{"@id":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#primaryimage"},"image":{"@id":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#primaryimage"},"thumbnailUrl":"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/05\/iStock-178101320.jpg","datePublished":"2018-05-04T13:49:49+00:00","dateModified":"2018-05-04T13:49:49+00:00","description":"Today, Twitter revealed that it will be moving a portion of its infrastructure to Google's Cloud Platform to compute its Hadoop clusters","breadcrumb":{"@id":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#primaryimage","url":"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/05\/iStock-178101320.jpg","contentUrl":"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/05\/iStock-178101320.jpg","width":722,"height":484,"caption":"clusters"},{"@type":"BreadcrumbList","@id":"https:\/\/devopsnews.online\/twitters-hadoop-clusters-migrate-to-google-cloud\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/devopsnews.online\/"},{"@type":"ListItem","position":2,"name":"Twitter’s Hadoop clusters migrate to Google Cloud"}]},{"@type":"WebSite","@id":"https:\/\/devopsnews.online\/#website","url":"https:\/\/devopsnews.online\/","name":"DevOps Online North America","description":"by 31 Media Ltd.","publisher":{"@id":"https:\/\/devopsnews.online\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/devopsnews.online\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/devopsnews.online\/#organization","name":"DevOps Online North America","url":"https:\/\/devopsnews.online\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/devopsnews.online\/#\/schema\/logo\/image\/","url":"https:\/\/devopsnews.online\/wp-content\/uploads\/2020\/03\/DevOpsOnline_email.png","contentUrl":"https:\/\/devopsnews.online\/wp-content\/uploads\/2020\/03\/DevOpsOnline_email.png","width":198,"height":64,"caption":"DevOps Online North America"},"image":{"@id":"https:\/\/devopsnews.online\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/DevOpsAmerica"]},{"@type":"Person","@id":"https:\/\/devopsnews.online\/#\/schema\/person\/d705a7d7c56b625b482e9b13a8e49a2c","name":"Leah Alger","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/devopsnews.online\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/0ac9e809e8ffafd8330210900bac04c0?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/0ac9e809e8ffafd8330210900bac04c0?s=96&d=mm&r=g","caption":"Leah Alger"},"url":"https:\/\/devopsnews.online\/author\/leah-alger\/"}]}},"_links":{"self":[{"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/posts\/12647"}],"collection":[{"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/comments?post=12647"}],"version-history":[{"count":0,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/posts\/12647\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/media\/12648"}],"wp:attachment":[{"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/media?parent=12647"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/categories?post=12647"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/tags?post=12647"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}