STEC EnhanceIO SSD Caching Software 25th December, 2012 1. WHAT IS ENHANCEIO? EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes throughout this document). EnhanceIO can work with any block device, be it an entire physical disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device. The source volume to SSD mapping is a set-associative mapping based on the source volume sector number with a default set size (aka associativity) of 512 blocks and a default block size of 4 KB. Partial cache blocks are not used. The default value of 4 KB is chosen because it is the common I/O block size of most storage systems. With these default values, each cache set is 2 MB (512 * 4 KB). Therefore, a 400 GB SSD will have a little less than 200,000 cache sets because a little space is used for storing the meta data on the SSD. EnhanceIO supports three caching modes: read-only, write-through, and write-back and three cache replacement policies: random, FIFO, and LRU. Read-only caching mode causes EnhanceIO to direct write IO requests only to HDD. Read IO requests are issued to HDD and the data read from HDD is stored on SSD. Subsequent Read requests for the same blocks are carried out from SSD, thus reducing their latency by a substantial amount. In Write-through mode - reads are handled similar to Read-only mode. Write-through mode causes EnhanceIO to write application data to both HDD and SSD. Subsequent reads of the same data benefit because they can be served from SSD. Write-back improves write latency by writing application requested data only to SSD. This data, referred to as dirty data, is copied later to HDD asynchronously. Reads are handled similar to Read-only and Write-through modes. 2. WHAT HAS ENHANCEIO CHANGED TO FLASHCACHE? 2.1. A new write-back engine The write-back engine in EnhanceiO has been designed from scratch. Several optimizations have been done. IO completion guarantees have been improved. We have defined limits to let a user control the amount of dirty data in a cache. Clean-up of dirty data is stopped by default under a high load; this can be overridden if required. A user can control the extent to which a single cache set can be filled with dirty data. A background thread cleans-up dirty data at regular intervals. Clean-up is also done at regular intevals by identifying cache sets which have been written least recently. 2.2. Transparent cache EnhanceIO does not use device mapper. This enables creation and deletion of caches while a source volume is being used. It's possible to either create or delete cache while a partition is mounted. EnhanceIO also supports creation of a cache for a device which contains partitions. With this feature it's possible to create a cache without worrying about having to create several SSD partitions and many separate caches. 2.3. Large I/O Support Unlike Flashcache, EnhanceIO does not cause source volume I/O requests to be split into cache block size pieces. For the typical SSD cache block size of 4 KB, this means that a write I/O request size of, say, 64 KB to the source volume is not split into 16 individual requests of 4 KB each. This is a performance improvement over Flashcache. IO codepaths have been substantially modified for this improvement. 2.4. Small Memory Footprint Through a special compression algorithm, the meta data RAM usage has been reduced to only 4 bytes for each SSD cache block (versus 16 bytes in Flashcache). Since the most typical SSD cache block size is 4 KB, this means that RAM usage is 0.1% (1/1000) of SSD capacity. For example, for a 400 GB SSD, EnhanceIO will need only 400 MB to keep all meta data in RAM. For an SSD cache block size of 8 KB, RAM usage is 0.05% (1/2000) of SSD capacity. The compression algorithm needs at least 32,768 cache sets (i.e., 16 bits to encode the set number). If the SSD capacity is small and there are not at least 32,768 cache sets, EnhanceIO uses 8 bytes of RAM for each SSD cache block. In this case, RAM usage is 0.2% (2/1000) of SSD capacity for a cache block size of 4K. 2.5. Loadable Replacement Policies Since the SSD cache size is typically 10%-20% of the source volume size, the set-associative nature of EnhanceIO necessitates cache block replacement. The main EnhanceIO kernel module that implements the caching engine uses a random (actually, almost like round-robin) replacement policy that does not require any additional RAM and has the least CPU overhead. However, there are two additional kernel modules that implement FIFO and LRU replacement policies. FIFO is the default cache replacement policy because it uses less RAM than LRU. The FIFO and LRU kernel modules are independent of each other and do not have to be loaded if they are not needed. Since the replacement policy modules do not consume much RAM when not used, both modules are typically loaded after the main caching engine is loaded. RAM is used only after a cache has been instantiated to use either the FIFO or the LRU replacement policy. Please note that the RAM used for replacement policies is in addition to the RAM used for meta data (mentioned in Section 2.1). The table below shows how much RAM each cache replacement policy uses: POLICY RAM USAGE ------ --------- Random 0 FIFO 4 bytes per cache set LRU 4 bytes per cache set + 4 bytes per cache block 2.6. Optimal Alignment of Data Blocks on SSD EnhanceIO writes all meta data and data blocks on 4K-aligned blocks on the SSD. This minimizes write amplification and flash wear. It also improves performance. 2.7. Improved device failure handling Failure of an SSD device in read-only and write-through modes is handled gracefully by allowing I/O to continue to/from the source volume. An application may notice a drop in performance but it will not receive any I/O errors. Failure of an SSD device in write-back mode obviously results in the loss of dirty blocks in the cache. To guard against this data loss, two SSD devices can be mirrored via RAID 1. EnhanceIO identifies device failures based on error codes. Depending on whether the failure is likely to be intermittent or permanent, it takes the best suited action. 2.8. Coding optimizations Several coding optizations have been done to reduce CPU usage. These include removing queues which are not required for write-through and read-only cache modes, splitting of a single large spinlock, and more. Most of the code paths in flashcache have been substantially restructured. 2.9 Sequential I/O bypass EnhanceIO has removed the bypass of sequential IO available in flashcache. The sequential detection logic has a limited use case, espescially in a reasonably multithreaded scenario. 3. EnhanceIO usage 3.1. Cache creation, deletion and editing properties eio_cli utility is used for creating and deleting caches and editing their properties. Manpage for this utility eio_cli(8) provides more information. 3.2. Making a cache configuration persistent It's essential that a cache be resumed before any applications or a filesystem use the source volume during a bootup. If a cache is enabled after a source volume is written to, stale data may be present in the cache. It may cause data corruption. The document Persistent.txt describes how to enable a cache during bootup using udev scripts. In case an SSD does not come up during a bootup, it's ok to allow read and write access to HDD only in the case of a Write-through or a read-only cache. A cache should be created again when SSD becomes available. If a previous cache configuration is resumed, it may cause stale data to be read. 3.3. Using a Write-back cache It's absolutely necessary to make a Write-back cache configuration persistent. This is required particularly in the case of an OS crash or a power failure. A Write-back cache may contain dirty blocks which haven't been written to HDD yet. Reading the source volume without enabling the cache will cause incorrect data to be read. In case an SSD does not come up during a bootup, access to HDD should stopped. It should be enabled only after SSD comes-up and a cache is enabled. Write-back cache needs to perform clean operation in order to flush the dirty data to the source device(HDD). The clean can be either trigerred by the user or automatically initiated, based on preconfigured thresholds. These thresholds are described below. They can be set using sysctl calls. a) Dirty high threshold (%) : The upper limit on percentage of dirty blocks in the entire cache. b) Dirty low threshold (%) : The lower limit on percentage of dirty blocks in the entire cache. c) Dirty set high threshold (%) : The upper limit on percentage of dirty blocks in a set. d) Dirty set low threshold (%) : The lower limit on percentage of dirty blocks in a set. e) Automatic clean-up threshold : An automatic clean-up of the cache will occur only if the number of outstanding I/O requests from the HDD is below the threshold. f) Time based clean-up interval (minutes) : This option allows you to specify an interval between each clean-up process. Clean is trigerred when one of the upper thresholds or time based clean threshold is met and stops when all the lower thresholds are met. 4. ACKNOWLEDGEMENTS STEC acknowledges Facebook and in particular Mohan Srinivasan for the design, development, and release of Flashcache as an open source project. Flashcache, in turn, is based on DM-Cache by Ming Zhao.