The Archiving and Pipeline Procedure
This document gives an overview of the systems of the BIMA Data
Archive and the BIMA Image Pipeline.
About the Data Archive
Datasets written by the BIMA Array telescope at Hat Creek are
transfered in near-real time to the Archive at NCSA. The journey data
take from telescope to archive includes 4 major stps.
- Real-time transfer. When a new experiment is
started at the telescope, the master observering script signals
the real-time archiving system to begin watching the
experiment's output datasets for new data. As the telescope
writes to the new datasets, the new bytes are transfered to the
archive at NCSA. The observing script sends another signal to
tell the system when the experiment is over.
- Data Integrity Checking. When the experiment
is finished, the archiving system at NCSA conducts a test on
the new data to ensure that they are an exact copy of the data
written at the telescope. If any item from a dataset fails
this test, it is retransfered automatically.
- Metadata are extracted. A dataset's metadata
decribes its character. After the dataset has been
successfully copied to NCSA, its metadata are extracted and
converted into XML format.
- Datasets and metadata are loaded into the
Archive. The datasets are tarred up and then copied
along with their metadata onto the archive shelves
where they can be readily accessed by users. The metadata are
loaded into the Archive database to support user searches.
Finally, the data and metadata are also sent to the NCSA Mass
Storage System for long-term storage.
As an important component of the BIMA Image Pipeline, the Data
Archive can also ingest processed data from the pipeline (see below
for details).
About the BIMA Image Pipeline
Under Development
The BIMA Data Archive is
a project of Radio Astronomy Imaging
Team
at the National Center for
Supercomputing Applications on the campus of the
University of Illinois at Urbana-Champaign
Contact the Archivist:
bimadata@ncsa.uiuc.edu
