Google builds world fastest machine learning training supercomputer that breaks AI performance records

July 30, 2020
google reuters big
206
Views

Google mentioned that it achieved these outcomes with ML mannequin implementations in TensorFlow, JAX, and Lingvo. 4 of the eight fashions had been educated from scratch in underneath 30 seconds.

Google mentioned it has constructed the world’s quickest machine studying (ML) coaching supercomputer that broke AI efficiency information in six out of eight industry-leading MLPerf benchmarks.

“The most recent outcomes from the industry-standard MLPerf benchmark competitors reveal that Google has constructed the world’s quickest ML coaching supercomputer. Utilizing this supercomputer, in addition to our newest Tensor Processing Unit (TPU) chip, Google set efficiency information in six out of eight MLPerf benchmarks,” a Google weblog mentioned.

Google mentioned that it achieved these outcomes with ML mannequin implementations in TensorFlow, JAX, and Lingvo. 4 of the eight fashions had been educated from scratch in underneath 30 seconds.

Quick coaching of machine studying (ML) fashions is essential for analysis and engineering groups that ship new merchandise, providers, and analysis breakthroughs that had been beforehand out of attaining. Right here at Google, current ML-enabled advances have included extra useful search outcomes and a single ML mannequin that may translate 100 completely different languages.

The most recent outcomes from the industry-standard MLPerf benchmark competitors reveal that Google has constructed the world’s quickest ML coaching supercomputer. Utilizing this supercomputer, in addition to our newest Tensor Processing Unit (TPU) chip, Google set efficiency information in six out of eight MLPerf benchmarks

We achieved these outcomes with ML mannequin implementations in TensorFlow, JAX, and Lingvo. 4 of the eight fashions had been educated from scratch in underneath 30 seconds. To place that in perspective, contemplate that in 2015, it took greater than three weeks to coach considered one of these fashions on probably the most superior {hardware} accelerator accessible. Google’s newest TPU supercomputer can practice the identical mannequin virtually 5 orders of magnitude quicker simply 5 years later.

On this weblog publish we’ll take a look at a number of the particulars of the competitors, how our submissions obtain such excessive-efficiency, and what all of it means on your mannequin coaching velocity.

MLPerf fashions at-a-glance

MLPerf fashions are chosen to be consultant of cutting-edge machine studying workloads which might be widespread all through {industry} and academia. Right here’s the slightly extra element on every MLPerf mannequin within the determine above:

  • DLRM represents rating and advice fashions which might be core to on-line companies from media to journey to e-commerce
  • Transformer is the inspiration of a wave of current advances in pure language processing, together with BERT
  • BERT enabled Google Search’s “greatest leap ahead previously 5 years” 
  • ResNet-50 is an extensively used mannequin for picture classification
  • SSD is an object detection mannequin that’s light-weight sufficient to run on cell units
  • Masks R-CNN is an extensively used picture segmentation mannequin that can be utilized in autonomous navigation, medical imaging, and different domains (you’ll be able to experiment with it in Colab)

Along with the industry-leading outcomes at the most scale above, Google additionally supplied MLPerf submissions utilizing TensorFlow on Google Cloud Platform which might be prepared for enterprises to make use of immediately. You possibly can learn extra about these submissions on this accompanying weblog publish.

The world’s quickest ML coaching supercomputer

The supercomputer Google used for this MLPerf Coaching spherical is 4 instances bigger than the Cloud TPU v3 Pod that set three information within the earlier competitors. The system consists of 4096 TPU v3 chips and a whole bunch of CPU host machines, all linked through an ultra-fast, ultra-large-scale customized interconnect. In whole, this technique delivers over 430 PFLOPs of peak efficiency.

Coaching at scale with TensorFlow, JAX, Lingvo, and XLA

Coaching complicated ML fashions utilizing hundreds of TPU chips required a mix of algorithmic methods and optimizations in TensorFlow, JAX, Lingvo, and XLA. To supply some background, XLA is the underlying compiler expertise that powers all of Google’s MLPerf submissions, TensorFlow is Google’s end-to-end open-source machine studying framework, Lingvo is an excessive degree framework for sequence fashions constructed utilizing TensorFlow, and JAX is a brand new research-focused framework based mostly on composable operate transformations. The record-setting scale above relied on mannequin parallelism, scaled batch normalization, environment-friendly computational graph launches, and tree-based weight initialization.

The entire TensorFlow, JAX, and Lingvo submissions within the desk above—implementations of ResNet-50, BERT, SSD, and Transformer—educated on 2048 or 4096 TPU chips in underneath 33 seconds every.

TPU v4: Google’s fourth-generation Tensor Processing Unit chip

Google’s fourth-generation TPU ASIC provides greater than double the matrix multiplication TFLOPs of TPU v3, a major enhance in reminiscence bandwidth, and advances in interconnect expertise. Google’s TPU v4 MLPerf submissions reap the benefits of this new {hardware} options with complementary compiler and modelling advances. The outcomes reveal a mean enhancement of two.7 instances over TPU v3 efficiency at an analogous scale within the final MLPerf Coaching competitors. Keep tuned, extra info on TPU v4 is coming quickly.

Article Categories:
Technology

We write books, which, considering where you’re reading this, makes perfect sense. We're best known for writing science fiction, political news, technology, entertainment, etc. We also writes non-fiction, on subjects ranging from personal finance to astronomy to film, was the Creative Consultant for the Stargate. We enjoys pie, as should all right-thinking people.

Leave a Reply

Your email address will not be published. Required fields are marked *