October 5-9, 2014


P1.8 Compression of Visibility Data for MWA

Slava Kitaeff (ICRAR/UWA)

Chen Wu

The Murchison Widefield Array (MWA) is a new low frequency radio telescope operating on the Square Kilometre Array site in Western Australia. MWA is generating tens of terabytes of data daily. The size of the required data storage has become a significant operational limitation and cost. We present a simple binary compression technique and a system for the floating point visibility data developed for Murchison Widefield Array (MWA) interferometer radio telescope. The technique utilises RICE and GZIP binary compression algorithms and a resampling technique. We present the statistics of the impact of such compression on the data with typical compression ratio up to 1:2.5. We also present a system that automatically compresses the data in MWA archive.

Mode of presentation: poster

Applicable ADASS XXIV theme category: Big Data Challenges