DMTN-199: Rubin Observatory Data Security Standards Implementation
In this document we describe a set of measures that we plan to take, in order to secure the data taken at Rubin Observatory to the standards set by the US funding agencies.
Engineering topics
Publications and source records attributed to Lim, Kian-Tat.
In this document we describe a set of measures that we plan to take, in order to secure the data taken at Rubin Observatory to the standards set by the US funding agencies.
The Prompt Processing system will be responsible for processing roughly a thousand visits per night, and distributing the results in near real time, for at least ten years of Rubin Observatory operations. As such, it must be highly robust to algorithmic, network, and infrastructure failures, ranging from momentary glitches to extended downtimes. DMTN-219 introduced the initial design for the Prompt Processing framework; this document expands on the design to address expected failure modes and recovery strategies for each.
Special Programs are additional survey areas and/or observing strategies that are driven by specific science goals which build on, or are beyond, the core science pillars of the Wide Fast Deep Main Survey. In order to meet the requirements and enable science related to Special Programs, this document provides recommendations for Rubin Data Management regarding processing and serving data products for Special Programs. Hardware and processing boundaries on the potential diversity of data from Special Programs are discussed along with scenarios in which user-generated processing and data products might be needed to meet Special Programs' science goals.
The Prompt Processing framework has long been a gray area in the overall Rubin Observatory architecture. This document attempts to fill that hole by proposing a concrete design with an accompanying prototype implementation that should meet minimum requirements. Additional features and optimizations that could be incorporated later are also described.
This document describes the data products and processing services to be delivered by the NSF-DOE Vera C. Rubin Observatory whilst performing the Legacy Survey of Space and Time (LSST). LSST will deliver three levels of data products and services. Prompt data products are computed and released within 24 hours of observation, and include images, difference images, catalogs of sources and objects detected in difference images, and catalogs of Solar System objects. Their primary purpose is to enable rapid follow-up of time-domain events. Data Release data products are computed during annual processing campaigns, and include well-calibrated single-epoch images, deep coadds, and catalogs of objects, sources, and forced sources, enabling static sky and precision time-domain science. The Science Platform will allow for the creation of User Generated data products and will enable science cases that greatly benefit from co-location of user processing and/or data within the Rubin Observatory Data Access Center. LSST will also devote 10% of observing time to programs with special cadence. Their data products will be created using the same software and hardware as Prompt and Data Release products. All data products will be made available using user-friendly databases and web services.
The LSST Science Requirements Document (the LSST SRD) specifies a set of data product guidelines, designed to support science goals envisioned to be enabled by the LSST observing program. Following these guidelines, the details of these data products have been described in the LSST Data Products Definition Document (DPDD), and captured in a formal flow-down from the SRD via the LSST System Requirements (LSR), Observatory System Specifications (OSS), to the Data Management System Requirements (DMSR). The LSST Data Management subsystem's responsibilities include the design, implementation, deployment and execution of software pipelines necessary to generate these data products. This document describes the design of the scientific aspects of those pipelines.
The Rubin Observatory LSST Data Management System (DMS) is a set of services employing a variety of software components running on computational and networking infrastructure that combine to deliver science data products to the observatory's users and support observatory operations. This document describes the components, their service instances, and their deployment environments as well as the interfaces among them, the rest of the LSST system, and the outside world.
Here in this article, we describe the optical imaging data processing pipeline developed for the Subaru Telescope’s Hyper Suprime-Cam (HSC) instrument. The HSC Pipeline builds on the prototype pipeline being developed by the Large Synoptic Survey Telescope’s Data Management system, adding customizations for HSC, large-scale processing capabilities, and novel algorithms that have since been reincorporated into the LSST codebase. While designed primarily to reduce HSC Subaru Strategic Program (SSP) data, it is also the recommended pipeline for reducing general-observer HSC data. The HSC pipeline includes high-level processing steps that generate coadded images and science-ready catalogs as well as low-level detrending and image characterizations.
This note provides details of the 5th Extremely Large Databases Conference and Invitational Workshop that were held in 2011 on 18-19 October and 20 October, respectively, at the SLAC National Accelerator Laboratory in Menlo Park, California. The main goals of the conference were: (1) Encourage and accelerate the exchange of ideas between users trying to build extremely large databases worldwide and database solution providers; (2) Share lessons, trends, innovations, and challenges related to building extremely large databases; (3) Facilitate the development and growth of practical technologies for extremely large databases; and (4) Strengthen, expand, and engage the XLDB community.
Array data analytic systems like SciDB hold great potential to accelerate processing data from SLAC's Linac Coherent Light Source and other experiments. SciDB is unique in its ability to integrate storage and processing of array data efficiently, providing both space-efficient storage and out-of-memory efficient parallel array processing. We describe a recent effort to leverage SciDB to store and process LCLS data. The work includes development of software to import data into SciDB, subsequent benchmarks, and interactive manipulation of data in SciDB.