PDZ: A Piecewise Differential Zstd Coder for nanopore sequencing raw signals

Abstract

Nanopore sequencing has emerged as a crucial component in the arsenal of genomic technologies, with advances from Oxford Nanopore Technologies (ONT) progressively reducing the costs of DNA sequencing. An ONT nanopore sequencer operates by guiding DNA fragments through a nanopore, partially blocking a flow of electrical current, which is sampled over time. This variation in current is registered as a raw signal, and it allows for the translation of electrical signals into a DNA sequence, a process known as basecalling. As the available algorithms for basecalling continually evolve, it is preferable to retain the raw signal data for future re-analysis. However, the volumes of raw data are massive, being nearly ten times larger than the size of data after basecalling in FASTQ format. Therefore, efficient lossless compression algorithms for raw signals are needed to reduce storage and transmission costs. While recent research has focused on studying nanopore FASTQ data, a thorough study of the methods used in practice for the compression of raw data, such as the state-of-the-art compression algorithm VBZ, is still missing in the scientific literature. In this sense, in this work, we aim to elucidate the mechanisms behind the efficiency of VBZ and introduce a set of variations that further improve its compression performance. Our findings indicate that we can enhance the performance of VBZ by an average of 2.42%, with gains increasing to 3.02% for the latest nanopore flowcells (10.x), using comparable computational resources.