Database techniques within LOFAR's Transients Key Project

The Transients Key Project Pipeline has to process LOFAR data in nearly real-time and respond quickly to itself and the community. The pipeline is designed to inspect the data on transient and variable sources, to monitor locations of interest, and to store source properties and parameters. Two types of databases using different storage models will assist, one temporary in detecting transient and variable sources in the huge data streams, the other permanent in storing all the measured source properties and relevant observational data, building up the largest ever catalog of radio sources, available for offline data querying and mining. Both are loosely coupled by data transport. Developing search and detection algorithms in the database engine, together with the vertical fragmentation storage model, lead to significant improvements in processing times. Observation modes and surveys benefit from this approach.