5: Hash collisions are a rare possibility Depending on what software is being used, the target storage volume might even temporarily require more space than the uncompressed data consumes on its own, because of workspace required by the deduplication process. Because of the way this process works, there is not initially any space savings on the target volume. A scheduled process performs the deduplication process later on. In this ty pe of architecture, the data is written to the target storage in an uncompressed format. Post process deduplication often (but not always) occurs on a secondary storage target, such as a disk that is used in disk-to-disk-backups. 4: Post process deduplication does not initially save any storage space This isn’t usually a big deal if the deduplication process is offloaded to an appliance or if it occurs on a backup target, but when source deduplication takes place on a production server, the process can sometimes affect the server’s performance. Many deduplication algorithms work by hashing chunks of data and then comparing the hashes for duplicates. Going from 20:1 to 25:1 only yields an extra 1% savings and reduces the data by about 10 GB more than using 20:1. However, a 25:1 compression ratio reduces the size of the data to only 40.96 GB. A 20:1 compression ratio reduces the size of the data from 1 TB to 51.2 GB.

To show you what I mean, consider what happens when you deduplicate 1 TB of data. Hence, higher compression ratios have diminishing returns. It is impossible to deduplicate a file in a way that shrinks the file by 100%. Although higher ratios do convey a higher degree of deduplication, they can be misleading. The effectiveness of data deduplication is measured as a ratio. 2: Higher ratios produce diminishing returns Most backup products that are currently being offered also support deduplication. Compression utilities such as WinZip perform deduplication, but so do many of the WAN optimization solutions. 1: Deduplication is used for a variety of purposesĭeduplication is used in any number of different products. That being the case, I decided to write this blog post as sort of a crash course in data deduplication for those who might not be familiar with the technology. If you're new to the technology, these key facts will help you get up to speed.Īlthough certainly not new, deduplication seems to recently have become a much hotter trend. 10 things you should know about data deduplicationĭeduplication is touted as one of the best ways to manage today's explosive data growth.

File deduplication software mac