NVIDIA Volta GPUs connected by NVSwitch create world’s largest ‘GPU’
The ‘GPU’, which is actually a cluster with 81920 CUDA Cores in total and up to 2K TFLOPs with Tensor Core computing, is actually a cluster of 16 Tesla V100 32GB GPUs, which are connected by NVSwitch.
This system has 512 GB of HBM2 memory and total bandwidth up to 14.4 TB/s. The ‘GPU’ is offered in NVIDIA DGX2 system, which Jensen calls ‘world’s largest graphics card’.
NVSwitch: A Revolutionary Interconnect Fabric
NVSwitch offers 5x higher bandwidth than the best PCIe switch, allowing developers to build systems with more GPUs hyperconnected to each other. It will help developers break through previous system limitations and run much larger datasets. It also opens the door to larger, more complex workloads, including modeling parallel training of neural networks.
NVSwitch extends the innovations made available through NVIDIA NVLink™, the first high-speed interconnect technology developed by NVIDIA. NVSwitch allows system designers to build even more advanced systems that can flexibly connect any topology of NVLink-based GPUs.
NVIDIA DGX-2: World’s First Two Petaflop System
NVIDIA’s new DGX-2 system reached the two petaflop milestone by drawing from a wide range of industry-leading technology advances developed by NVIDIA at all levels of the computing stack.
DGX-2 is the first system to debut NVSwitch, which enables all 16 GPUs in the system to share a unified memory space. Developers now have the deep learning training power to tackle the largest datasets and most complex deep learning models.
Combined with a fully optimized, updated suite of NVIDIA deep learning software, DGX-2 is purpose-built for data scientists pushing the outer limits of deep learning research and computing.
DGX-2 can train FAIRSeq, a state-of-the-art neural machine translation model, in less than two days — a 10x improvement in performance from the DGX-1 with Volta, introduced in September.