We notice you are currently visiting the UK site. Would you like to visit our main site instead?

Yes, please
Servers in a data center

Four things data centre managers can learn from the fastest supercomputers

Jul 2020
Blog Home

If you were to ask a lay person on the street what they thought a supercomputer was, you’d probably get a large percentage citing examples from popular movies – and usually examples with a nefarious reputation. From HAL 9000 (2001: A Space Odyssey) to iRobot’s VIKI and even The Terminator’s Skynet, pop culture often references supercomputers as sentient systems that have evolved and turned against humanity.

Tell that to researchers at Lawrence Livermore National Laboratory, or the National Weather Service, and they’d laugh you out of the room. The truth is that supercomputers today are far from self-aware, and the only AI in use is essentially an overblown search bar that is scanning very large data sets.

Today, supercomputers are powering a multitude of applications that are at the forefront of progress: from oil and gas exploration to weather predictions, financial markets to developing new technologies. Supercomputers are the Lamborghini or Bugatti of the computing world and, at Kingston, we pay a lot of attention to the advancements that are pushing the computing boundaries. From DRAM utilisation and tuning, to firmware advancements in managing storage arrays, to the emphasis on consistency of transfer and latency speeds instead of peak values, our technologies are deeply influenced by the bleeding edge of supercomputing.

Similarly, there are a lot of things that cloud and on-premise data centre managers can learn from supercomputing when it comes to designing and managing their infrastructures, as well as how best to select the components that will be ready for future advancements without huge overhauls.

server with glowing lines representing a network
1. Supercomputers are purpose-built for consistency

Unlike most cloud-computing platforms, such as Amazon Web Services or Microsoft Azure, which are built to power a variety of applications that can utilise shared resources and infrastructures, most supercomputers are purpose-built for specific needs. The most recent update of the TOP500 list of the world’s fastest supercomputers (publicly-known and declassified), notes not only the locations and speeds of installations, but the primary field of application.

Eleven of the top dozen machines are dedicated to energy research, nuclear testing and defence applications. The only outlier is Frontera, a new NSF-funded petascale computing system at the Texas Advanced Computing Centre at the University of Texas, which provides academic resources for science and engineering research partners. Of the next 20 supercomputers on the TOP500 list, almost all are dedicated to Government defence and intelligence applications. Machines between positions 30-50 on the list are largely dedicated to weather predictions. The last 50 of the top 100 are a mix of corporate computing (NVIDIA, Facebook, et.al), midrange weather predictions, space programmes, oil and gas exploration, academic and specific government uses.

These machines aren’t a one-size-fits-all box. They’re custom developed together with manufacturers like Intel, Cray, HP, Toshiba and IBM to perform specific types of calculations on very specific datasets – either in real-time or asynchronous computations.

They have defined acceptable latency thresholds:

      • Preset computing resources leveraging millions of processing cores
      • Deliver clock rates between 18,000 and 200,000 teraFLOPS.

Their storage capacities are measured in exabytes – far beyond the petabytes in modern data warehouses.

Systems like Frontera don’t just have to sprint in a peak compute load, but instead have to consistently read vast amounts of data to arrive at a result. A spike in compute performance could actually cause errors in results, thus the emphasis is on consistency.

Today’s data centre manager must first ask, “What are we doing with the system?” in order to architect, manage resources and build in predictable fail-safes. Managing a data centre that runs a number of virtual desktops is a lot different than a 999 call centre or air-traffic control system. They have different needs, demands, service-level agreements and budgets – and need to be designed accordingly.

Likewise, there needs to be a consideration about how to achieve consistent performance without requiring custom builds. Companies like Amazon, Google and Microsoft have the budgets to engineer custom storage or computing infrastructures, but the majority of service providers have to be more selective with off-the-shelf hardware.

Thus, more data centre managers need to set strict criteria for performance benchmarks that address QoS and ensure the greatest emphasis is not only on compute speed and latency, but also on consistency.

server with glowing lines representing a network
2. Your real-time is not my real-time

With supercomputing applications, most instances of real-time data have major implications. From stopping a nuclear reaction to telemetry data for a rocket launch, compute latency can have catastrophic effects – and the data sets are massive. These streams aren’t just feeding from a single source, but are often delivered from a network of reporting nodes.

But the data is short lived. When working with real-time feeds, most of the data doesn’t get held forever. It’s written and then overwritten with a shelf life for sequential writes and overwrites. Real-time data is always changing, and very few applications would need every bit stored from the beginning of time. The data is processed in batches, computed to create a result (whether an average, statistical model or algorithm) and the result is what’s kept.

Take supercomputer predictions by the National Oceanographic and Atmospheric Administration (NOAA) for example. There are constant changes in meteorological factors, whether precipitation, air and ground temperature, barometric pressure, time of day, solar effects, wind speed and even how wind passes over terrain. These factors change every second and are reported as a real-time stream of information. But NOAA’s National Weather Service (NWS) doesn’t need the raw data forever. It needs the forecasting models. As the global forecasting system (GFS) model takes shape, new data gets pushed through it, forming more accurate and updated predictions.

Moreover, local meteorologists who share and receive data from the NWS don’t need access to the entire global dataset of weather. They limit their models to local areas. This allows them to supplement NWS data with local weather stations, which gives an insight into microclimates and accelerates more accurate local predictions by creating batches, computed to create a result (whether an average, statistical model or algorithm). And the result is what’s kept.

The same could be said for stock trading or financial models, which work with moving averages – each with specific indicators and action triggers built in, based on specific parameters for acceptable market behaviour thresholds. Designing a system that uses “real-time” data doesn’t have to store everything that it ingests – but should leverage non-volatile random access memory (NVRAM) and dynamic random access memory (DRAM) to cache and process data in-flight, then deliver computed output to storage.

flash memory chip illustration with glowing circuit traces
3. Latency thresholds, NAND Flash and tuning DRAM

Most of the latency thresholds are set because of the application demands. In trading scenarios, seconds mean millions, if not billions of dollars. For weather predictions and hurricane tracking, it could mean deciding between evacuating New Orleans or Houston.

Supercomputers operate with the a priori burden of service level – whether latency, computing resources, storage or bandwidth. Most employ fail-aware computing, whereby the system can reroute data streams for optimal latency conditions (based on 𝛱+Δmax clocking) by shifting to asynchronous computing models or prioritising compute resources to deliver sufficient processing power or bandwidth for jobs.

Whether you’re working with high-end workstations, iron servers or HPC and scientific workloads, big computers and Big Data require huge DRAM loadouts. Supercomputers like the Tianhe-2 use huge RAM loadouts combined with specialised accelerator cards. The ways in which supercomputing fine-tunes the hardware and controller framework is unique to the application design. Often, specific computational tasks with disk accesses that create a huge bottleneck with RAM requirements make DRAM impractical but are small enough to fit into NAND flash. The FPGA clusters are also further tuned for each specific workload to ensure large data sets take huge performance hits if they have to use traditional media to retrieve data.

The teams collaborating between the University of Utah, Lawrence Berkeley Lab, the University of Southern California and Argonne National Lab have shown new models for automatic performance tuning (or auto-tuning) as an effective means of providing performance portability between architectures. Rather than depending on a compiler that can deliver optimal performance on more novel multicore architectures, auto-tuned kernels and applications can auto-tune on the target CPU, network and programming model.

helmeted IT working man with a laptop in front of a heads-up display illustration
4. Multiple layers of fail-safes

Energy distribution within the HPC data centre is increasingly challenging – especially with infrastructures that are leveraged as shared resources. In either dedicated or as-a-service provisioned infrastructures, data centres need to ensure continuous operation and reduce the risk of damaging fragile hardware components in the event of a power failure, spike or changes in peak demand.

Architects use a mix of loss-distribution transformers:

  • DC power distribution and UPS backups,
  • Trigeneration (creating electricity through heat to store in backup)
  • Active monitoring
“Save and save often” is the mantra for any application, and the same is true for data centres where “backup” becomes the operative term.

Most data centres today operate with a high-level RAID structure to ensure continuous and near-simultaneous writes across storage arrays. Furthermore, HPC infrastructures leverage a high amount of NVRAM to cache data in process, which are either live streams of data that don’t pull across storage arrays, or parallel-processed information creating a scratch disk-esque usage to free up additional compute resources. The previously mentioned Frontera system leverages 50PB of total scratch capacity. Users with very high bandwidth or IOPS requirements will be able to request an allocation on an all-NVMe (non-volatile memory express) file system with an approximate capacity of 3PB, and bandwidth of ~1.2TB/s.

This constant RAID backup for storage, and consistent caching of NVMe buffers are dependent on the total I/O thresholds for the controllers on the device, and for the total available or provisioned bandwidth for remote storage/backup.

Most HPC infrastructures are also eliminating the chance of hardware failures with spinning drives by moving completely to solid-state arrays and flash storage blocks. These storage solutions provide consistent IOPS and have predictable latencies that fall within the application-specific latency thresholds. Many supercomputers also leverage multiple tape libraries (with capacity scalable to an exabyte or more) in order to have reliable data archival for every bit processed and stored.

Many are also ensuring that, should everything else fail in the chain, there are power-fail (PFail) capacitors (P-Cap) – also labelled as power-loss-protection (PLP) – installed on SSDs and DRAM. P-Caps allow drives (either independent or across an array) to complete writes in-progress, thus reducing the amount of data potentially lost during a catastrophic failure.

Concluding

Again, custom is key in the supercomputing world, but knowing your needs is the first step when building a data centre and identifying how to achieve the most consistent type of performance. No matter the size of the data centre, why not think of it as important or in terms of a supercomputer when it comes to generating, storing or sharing data. By evaluating those factors, architects can design high-performance infrastructures that are ready for future advancements, even with off-the-shelf components.

#KingstonIsWithYou

Was this helpful?

Related Products

Results
Capacity
Filter Filters Applied

Related Articles

Blog Home