Pitfalls Await When Choosing a System Based on the Datasheet’s Claims

The age of Big Data is placing increasing demands on storage and processing.  Correspondingly, there is a need for faster data transfer speeds.

When customers choose solid-state drives (SSDs), they often go by the datasheet’s performance figures.  However, the numbers in the datasheet don’t accurately represent typical SSD performance.  It is generally understood that the real-world performance obtained from processors and storage devices is very different from that claimed in datasheets and benchmarks. Therefore, choosing a device based on exaggerated figures may lead to a system that does not meet its requirements.

Using CrystalDiskMark to Measure Real-world Performance

How big is the difference between the out-of-the-box claims in datasheets and the real-world, steady-state benchmarks?

To measure the difference between out-of-the-box and steady-state performance, we use the CrystalDiskMark benchmark tool, then run a series of random writes for 72 hours before measuring the CrystalDiskMark performance again.

We have carefully measured the real-world performance of our flash controllers and measured performance against that of competitors.

Real-World Performance versus Datasheet Claims

The difference between real-world and datasheet performance is shocking.

In some devices that we have benchmarked, steady-state speeds can be as little as 1% of the out-of-the-box performance. What is more surprising is how quickly the performance falls off. In nearly all the drives tested, the performance decreased dramatically after about 1.5 minutes of use.

Our finding shows that, in the case of flash memory, in addition to the usual difference between unrealistic peak figures and application performance, even the raw performance numbers may not stand up to scrutiny in practice.

What is the Difference Between Datasheet Claims and Actual Benchmarks?

To increase the storage density of flash memory, manufacturers now store multiple bits per memory cell. Triple-level cells (TLC) memory store three bits, and a more recent quad-level cell (QLC) memory stores four bits per cell.

However, high-density storage comes at a cost. 

As the flash memory becomes filled with data, the flash controller has to execute various background tasks. For example, garbage collection, wear leveling, dynamic data refresh, and calibration occur in the background. The controller can “hide” the effects of these for a short time by prioritizing and scheduling the I/O operations. However, for longer transfers, these tasks inevitably have some impact on performance. Over the longer term, background tasks such as wear leveling and garbage collection become more time-consuming.

The outpour of data into flash memory takes a toll on sequential and random access, too.  The two main measurements used to characterize an SSD are sequential access, where a large chunk of consecutive data is transferred to or from the storage device, and random access, where smaller amounts of data are read or written to non-contiguous addresses. Sequential accesses are typically measured in megabytes per second (MB/s), while random accesses are usually given as input/output operations per second (IOPS).

As the caches become filled with data, there is less free space on the flash. The caches thus start to lose their effectiveness after long sequences of either sequential or random accesses.  Over time, the caches may become increasingly less effective, because some areas may be marked as unusable as a result of non-recoverable errors.

Therefore, as the flash ages, the raw error rate in high-density flashes becomes higher, and memory life shortens. Soft error decoding techniques are used to minimize the increase in error rates as the memory ages. These read the data multiple times and require sophisticated processing to calculate the correct data, both of which reduce the overall performance.

Over time, devices also become more sensitive to cross temperature effects caused by reading data at a different temperature than the temperature at which the data was written. The SSD gets hotter during use, so to guard against overheating “thermal throttling” is employed. Thermal throttling reduces the power consumption and lowers the temperature of the device.  However, thermal throttling also limits device performance. The effects of thermal throttling depend on ambient temperature, the thermal design of the environment, and device workload. The number of variables affecting thermal throttling makes it difficult to predict the real-world level of performance.

In summary, due to many factors, the performance level of an SSD changes over its lifetime, dropping its initial, out-of-the-box level to the steady-state level.

Major Takeaway

Using real-world data is essential when evaluating a device whose performance will affect overall system performance.  Any selection based on exaggerated figures risks a system that falls short of performing as intended.

The actual performance achieved will depend on the patterns of data use, the environment, and the effectiveness of the flash controller. It is essential to understand the factors that can affect performance, but even more critical to use real-world data rather than unrealistic performance claims.

Therefore, when choosing data storage devices, remember that device datasheets will typically show peak figures that can only be achieved by new devices under specific conditions, and only for a short period.

Steffen Allert is heading the European Sales organization of Hyperstone. His responsibilities covers all sales related activities for the EMEA region. Steffen has been involved with the flash industry for over 10 years. Before joining Hyperstone he worked for Daimler Benz Aerospace (now Airbus) and Siemens. Steffen holds a MS in Electronics from the Technical University Darmstadt, Germany and a BBA from GSBA Zurich.