October 5-9, 2014


P2.34 Automated and Scalable Data Reduction in the SOFIA Data Processing System

Robert Krzaczek (Rochester Institute of Technology)

Ralph Shuping (Space Science Institute)
Miguel Charcos-Llorens (Universities Space Research Association)
Rosemary Alles (Universities Space Research Association)
William Vacca (Universities Space Research Association)

In order to provide suitable data products to general investigators and other end users in a timely manner, the Stratospheric Observatory for Infrared Astronomy (SOFIA) has developed a framework, the Data Processing System (DPS, see Shuping et. al. for overview), that supports the automated and unattendaed execution of data processing pipelines developed for an increasingly wide range of dissimilar instruments. As the observatory is scheduled for nightly missions, the primary goal is to process all data collected from a flight within eight hours, allowing data quality assessments and inspections to be made the following day. The raw data collected during a flight requires processing by a number of different software packages and tools unique to each combination of instrument and mode of operation, much of it developed in-house, in order to create data products for use by investigators and other end-users. The requirement to deliver these data products, intended for analysis and publication, in a consistent, predictable, and performant manner presents a significant challenge for the observatory. Herein we present aspects of the SOFIA DPS that enable the achievement of these observatory goals. We will discuss how it supports data reduction software written in a variety of languages and environments, as well as its support for new versions and live upgrades to that software and other necessary resources (e.g., calibrations). We will also show how the DPS supports the addition (and eventual removal) of extra computing resources, when large processing loads are anticipated, without requiring any sort of restart or disruption on ongoing activity. Finally, we will report our experiences and performance achievements from the first two observing cycles of SOFIA.

Mode of presentation: poster

Applicable ADASS XXIV theme category: Data Analysis / Pipelines