Servers in a data center

4 Things Data Center Managers Can Learn from HPC

Jun 2020
Blog Home

If you were to ask a lay person on the street what they thought a supercomputer was, you’d probably get a large percentage citing examples from popular movies ― and, usually examples with a nefarious reputation. From HAL 9000 (2001: A Space Odyssey) to iRobot’s VIKI, and even The Terminator’s Skynet; pop culture often references supercomputers as sentient systems that have evolved and turned against humanity.

Tell that to researchers at Lawrence Livermore National Laboratory, or the National Weather Service, and they’d laugh you out of the room. The truth is that supercomputers today are far from self-aware, and the only AI is essentially an overblown search bar that is scanning very large data sets.

Today, supercomputers are powering a multitude of applications that are at the forefront of progress: from oil and gas exploration to weather predictions, financial markets to developing new technologies. Supercomputers are the Lamborghini or Bugatti of the computing world, and at Kingston, we pay a lot of attention to the advancements that are pushing the computing boundaries. From DRAM utilization and tuning, to firmware advancements in managing storage arrays, to even the emphasis on consistency of transfer and latency speeds instead of peak values, our technologies are deeply influenced the bleeding-edge of supercomputing.

Similarly, there are a lot of things that cloud and on-premise data center managers can learn from supercomputing when it comes to designing and managing their infrastructures, as well as how to best select the components that will be ready for future advancements without huge overhauls.

Server
1. Supercomputers are Purpose-Built for Consistency

Unlike most Cloud-computing platforms, like Amazon Web Services or Microsoft Azure which are built to power a variety of applications that can utilize shared resources and infrastructures, most supercomputers are purpose-built for specific needs. The most recent update of the TOP500 list of the world’s fastest supercomputers (publicly-known and declassified), notes not only the locations and speeds of installations, but the primary field of application.

Eleven of the top dozen machines are dedicated for energy research, nuclear testing and defense applications. The only outlier is Frontera, a new NSF-funded petascale computing system at the Texas Advanced Computing Center at the University of Texas, which provides academic resources for science and engineering research partners. Of the next 20 supercomputers on the TOP500 list, almost all are dedicated for Government defense and intelligence applications. Machines between numbers 30-50 on the list are largely dedicated for weather predictions. The last 50 of the top 100 are a mix of corporate computing (NVIDIA, Facebook, et.al), midrange weather predictions, space programs, oil and gas exploration, academic and specific government uses.

These machines aren’t a one-size-fits-all box. They’re custom developed with manufacturers like Intel, Cray, HP, Toshiba and IBM to perform specific types of calculations on very specific datasets ― either in real-time or asynchronous computations.

They have defined acceptable latency thresholds:

  • Preset computing resources leveraging millions of processing cores
  • Deliver clock rates between 18,000 and 200,000 teraFLOPS.

Their storage capacities are measured in exabytes ― far beyond the petabytes in modern data warehouses.

Systems like Frontera don’t just have to sprint in a peak compute load, but instead have to consistently read vast amounts of data to arrive at a result. A spike in compute performance could actually cause errors in results, thus the emphasis is on consistency.

Today’s data center manager needs to first ask, “What are we doing with the system?,” in order to architect, manage resources and build in predictable fail-safes. Managing a data center that runs a bunch of virtual desktops is a lot different than a 911 call center, or air-traffic control systems. They have different needs, demands, service-level agreements and budgets - and need to be designed accordingly.

Likewise, there needs to be a consideration about how to achieve consistent performance without requiring custom builds. Companies like Amazon, Google and Microsoft have the budgets to engineer custom storage or computing infrastructures, but the majority of service providers have to be more selective with off-the-shelf hardware.

Thus, more data center managers need to set strict criteria for performance benchmarks that address QoS and ensure the greatest emphasis is not only on compute speed and latency, but also consistency.

server with glowing lines representing a network
2. Your Real-Time is Not My Real-Time

With supercomputing applications, most instances of real-time data have major implications. From stopping a nuclear reaction to telemetry data for a rocket launch, compute latency can have catastrophic effects ― and the data sets are massive. These streams aren’t just feeding from a single source; but are often delivered from a network of reporting nodes.

But the data is short-lived. When working with real-time feeds, most of the data doesn’t get held forever. It’s written and then overwritten with a shelf life for sequential writes and overwrites. Real-time data is always changing, and very few applications would need every bit stored from the beginning of time. The data gets processed in batches, computed to create a result (be it an average, statistical model or algorithm) and the result is what’s kept.

Take National Oceanographic and Atmospheric Administration (NOAA) supercomputer predictions for example. There are always constant changes in meteorological factors, be it precipitation, air and ground temperature, barometric pressure, time of day, solar effects, wind and even how it passes over terrain. That changes every second and gets reported as a real-time stream of information. But NOAA’s National Weather Service (NWS) doesn’t need the raw data forever. You need the forecasting models! As the global forecasting system (GFS) model takes shape, new data gets pushed through it, forming more accurate and updated predictions.

Moreover, local meteorologists who share and receive data from the NWS don’t need access to the entire global dataset of weather. They just limit their models to local areas. This allows them to supplement NWS data with local weather stations thus giving insights to microclimates and accelerating more accurate local predictions by creatin batches, computed to create a result (be it an average, statistical model or algorithm) and the result is what’s kept.

The same could be said for stock trading, or financial models, which work with moving averages - each with specific indicators and action triggers built in, based on specific parameters for acceptable market behavior thresholds. Designing a system that uses “real-time” data doesn’t have to store everything that it ingests ― but should leverage non-volatile random access memory (NVRAM) and dynamic random access memory (DRAM) to cache and process data in-flight, then deliver computed output to storage.

flash memory chip illustration with glowing circuit traces
3. Latency Thresholds, NAND Flash and Tuning DRAM

Most of the latency thresholds are set because of the application demands. In trading scenarios, seconds mean millions, if not billions of dollars. For weather predictions and hurricane tracking, it could mean deciding between evacuating New Orleans or Houston.

Supercomputers operate with the a priori burden of service level - be it latency, computing resources, storage or bandwidth. Most employ fail-aware computing, whereby the system can reroute data streams for optimal latency conditions (based on 𝛱+Δmax clocking), shifting to asynchronous computing models, or prioritizing compute resources to deliver sufficient processing power or bandwidth for jobs.

Whether you’re working with high-end workstations, iron servers, or HPC and scientific workloads, big computers and Big Data require huge DRAM loadouts. Supercomputers like the Tianhe-2, use huge RAM loadouts combined with specialized accelerator cards. The ways in which supercomputing fine-tunes the hardware and controller framework is unique to the application design. Often specific computational tasks, where disk access creates a huge bottleneck with RAM requirements, make DRAM impractical but are small enough to fit into NAND flash. The FPGA clusters are also further tuned for each specific workload to ensure large data sets take huge performance hits if they have to use traditional media to retrieve data.

The teams collaborating between the University of Utah, Lawrence Berkeley Lab, the University of Southern California, and Argonne National Lab have shown new models for Automatic Performance Tuning (or Auto-tuning) as an effective means of providing performance portability between architectures. Rather than depending on a compiler that can deliver optimal performance on more novel multicore architectures, auto-tuned kernels and applications can auto-tune on the target CPU, network, and programming model.

helmeted IT working man with a laptop in front of a heads-up display illustration
4. Multiple Layers of Fail-Safes

Energy distribution within the HPC data center is increasingly challenging ― especially with infrastructures that are leveraged as shared resources. In either dedicated or as-a-service provisioned infrastructures, data centers need to ensure continuous operation and reduce the risk of damaging fragile hardware components in the event of a power failure, spike or changes in peak demand.

Architects use a mix of loss-distribution transformers:

  • DC power distribution and UPS backups,
  • Trigeneration (creating electricity through heat to store in backup)
  • Active monitoring
“Save and save often” is the mantra for any application, and the same is true for data centers where “backup” becomes the operative term.

Most data centers today operate with a high-level RAID structure to ensure continuous and near-simultaneous writes across storage arrays. Furthermore, HPC infrastructures leverage a high amount of NVRAM to cache data in process, which are either livestreams of data that don’t pull across storage arrays, or parallel-processed information creating a scratch disk-esque usage to free up additional compute resources. The previously mentioned Frontera system leverages 50PB of total scratch capacity. Users with very high bandwidth or IOPS requirements will be able to request an allocation on an all-NVMe (non-volatile memory express) file system with an approximate capacity of 3PB, and bandwidth of ~1.2TB/s.

This constant RAID backup for storage, and consistent caching of NVMe buffers are dependent on the total I/O thresholds for the controllers on device, and for the total available or provisioned bandwidth for remote storage/backup.

Most HPC infrastructures are also eliminating the chance of hardware failures with spinning drives by moving completely to solid-state arrays and flash storage blocks. These storage solutions provide consistent IOPS and have predictable latencies that fall within the application-specific latency thresholds. Many supercomputers also leverage multiple tape libraries (with capacity scalable to an exabyte or more), in order to have reliable data archival for every bit processed and stored.

Many are also ensuring that should everything else fail in the chain, there are power-fail (PFail) capacitors (P-Cap) also labeled as power-loss-protection (PLP) installed on SSDs and DRAM. P-Caps allow drives (either independent or across an array) to complete writes in-progress, thus reducing the amount of data potentially lost during a catastrophic failure.

Concluding

Again, custom is key in the supercomputing world, but knowing your needs is the first step when building a data center and how to achieve the most consistent type of performance. No matter the size of the data center, why not think of it as important or in terms of a supercomputer when it comes to generating, storing or sharing data. By evaluating those factors, architects can design high performance infrastructures that are ready for future advancements, even with off-the-shelf components.

#KingstonIsWithYou

Related articles

Blog Home