Data management study. Appendix Q - Contractor data requirements advanced missions /AM/ Final report
Contractor data requirements relating to advanced missions and potential follow-up programs of Voyager project
DATA CONTROL SYSTEM↗
SEARCH · Engineering Papers
Results for “Data management”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
"PoliMOR: A Policy Engine \"Made-to-Order\" for Automated and Scalable Data Management in Lustre"
Modern supercomputing systems are increasingly reliant on hierarchical, multi-tiered file and storage system architectures due to cost-performance-capacity trade-offs. Within such multi-tiered systems, data management services are required to maintain healthy utilization, performance, and capacity levels. We present PoliMOR, a pragmatic and reliable policy-driven data management framework. PoliMOR is composed of modular, single-purpose agents that gather file system metadata and enforce policies on storage systems. PoliMOR facilitates automated and scalable data management with customizable agents tailored to HPC facility-specific storage systems and policies. Our evaluations demonstrate the scalability and performance of PoliMOR both by its individual agents and as a collective entity. We believe PoliMOR is widely applicable across HPC facilities with large-scale data management challenges and will garner interest from the HPC community, given its flexible and open-source nature.
George, Anjus↗
LDM-148: Rubin Observatory Data Management System Design
The Rubin Observatory LSST Data Management System (DMS) is a set of services employing a variety of software components running on computational and networking infrastructure that combine to deliver science data products to the observatory's users and support observatory operations. This document describes the components, their service instances, and their deployment environments as well as the interfaces among them, the rest of the LSST system, and the outside world.
79 ASTRONOMY AND ASTROPHYSICS↗
The AmeriFlux Data Activity and Data System: An Evolving Collection of Data Management Techniques, Tools, Products and Services
The Carbon Dioxide Information Analysis Center (CDIAC) at Oak Ridge National Laboratory (ORNL), USA has provided scientific data management support for the U.S. Department of Energy and international climate change science since 1982. Over this period, climate change science has expanded from research focusing on basic understanding of geochemical cycles, particularly the carbon cycle, to integrated research addressing climate change impacts, vulnerability, adaptation, and mitigation. Interests in climate change data and information worldwide have grown remarkably and, as a result, so have demands and expectations for CDIAC’s data systems. To meet the growing demands, CDIAC’s strategy has been to design flexible data systems using proven technologies blended with new, evolving technologies and standards. CDIAC development teams are multidisciplinary and include computer science and information technology expertise, but also scientific expertise necessary to address data quality and documentation issues and to identify data products and system capabilities needed by climate change scientists. CDIAC has learned there is rarely a single commercial tool or product readily available to satisfy long-term scientific data system requirements (i.e., one size does not fit all and the breadth and diversity of environmental data are often too complex for easy use with commercial products) and typically deploys a variety of tools and data products in an effort to provide credible data freely to users worldwide. Like many scientific data management applications, CDIAC’s data systems are highly customized to satisfy specific scientific usage requirements (e.g., developing data products specific for model use) but are also designed to be flexible and interoperable to take advantage of new software engineering techniques, standards (e.g., metadata standards) and tools and to support future Earth system data efforts (e.g., ocean acidification). CDIAC has provided data management support for numerous long-term measurement projects crucial to climate change science. One current example is the AmeriFlux measurement network. AmeriFlux provides continuous measurements from forests, grasslands, wetlands, and croplands in North, Central, and South America and offers important insight about carbon cycling in terrestrial ecosystems. We share our approaches in satisfying the challenges of delivering AmeriFlux data worldwide to benefit others with similar challenges handling climate change data, further heighten awareness and use of an outstanding ecological data resource, and highlight expanded software engineering applications being used for climate change measurement data.
Boden, Thomas A.↗
APS Data Management System
As the capabilities of modern X-ray detectors and acquisition technologies increase, so do the data rates and volumes produced at synchrotron beamlines. This brings into focus a number of challenges related to managing data at such facilities, including data transfer, near real-time data processing, automated processing pipelines, data storage, handling metadata and remote user access to data. The Advanced Photon Source Data Management System software is designed to help beamlines deal with these issues. Finally, this paper presents the system architecture and describes its components and functionality; the system's current usage is discussed, examples of its use have been provided and future development plans are outlined.
97 MATHEMATICS AND COMPUTING↗
Data management for the Skylab program
Skylab program data management systems, discussing onboard data collection, transmission and ground facilities
Disher, J. H.↗
Sparse Data Management in HDF5
The importance of sparse data management is growing with data produced by large-scale experimental and observational facilities that contain small amounts of non-zero values. In this document, we explore different design options to support sparse data in HDF5, one of the most popular highperformance I/O libraries and file formats used for scientific data. We discuss several use cases and requirements. Our main hard design constraint is that any changes to the HDF5 dataset API would be a burden on users and not acceptable. The remaining options are discussed below. We provide rationale for what we believe is the strongest candidate and describe how its potential benefits can be simulated with the existing HDF5 library. We have conducted a set of computational experiments the results of which are reported here. They show that our candidate meets all relevant requirements and gives us a certain degree of confidence for an HDF5 library-native implementation.
Mainzer, John↗
Building a Cyberinfrastructure for Environmental System Science: Modeling Frameworks, Data Management, and Scientific Workflows, Workshop Report, Potomac, Maryland, April 30-May 1, 2015
The Environmental System Science (ESS) activity within the Department of Energy’s (DOE) Office of Biological and Environmental Research (BER) seeks to advance a robust, predictive understanding of terrestrial environments, extending from bedrock to the top of the vegetative canopy and from molecular to global scales, through an iterative cycle of model-driven experimentation and observation dubbed MODEX. Considerable progress has been made toward achieving this overarching goal, but widely recognized is the fragmentation across projects and disciplines of the relevant modeling and simulation capabilities, observational and experimental data, analysis algorithms, and workflow tools. This fragmentation creates significant challenges for studying impacts and feedbacks in these complex multiscale systems. These challenges are further exacerbated by ongoing disruptive changes in high-performance computational architectures and the exponential growth in the types and volume of data that render obsolete the conventional approaches to software development and data management. Overcoming these challenges will require the development of a BER Climate and Environmental Sciences Division (CESD)–wide enabling cyberinfrastructure to support data management, cross-domain modeling, data analysis, and collaborative research. To explore the potential for working groups to initiate and guide a more integrated and community-based cyberinfrastructure, BER held the ESS Workshop on Model-Data Integration: Modeling Frameworks, Data Management, and Scientific Workflows on April 30-May 1, 2015, following the ESS Principal Investigator Meeting in Potomac, Maryland. Participants included model developers, software engineers, and data management specialists from eight national laboratories, which represented a wide range of projects and programs from CESD as well as cross-cutting projects from DOE’s Office of Advanced Scientific Computing Research (ASCR; both BER and ASCR are operated from within DOE’s Office of Science). A series of plenary talks provided background information and clarified the workshop’s three objectives: (1) develop requirements for this community-based cyberinfrastructure to ensure enhanced scientific productivity of the community as a whole; (2) identify challenges associated with developing this new cyberinfrastructure using a phased approach guided by project-driven use cases; and, given these requirements and challenges, (3) chart a path forward for ESS working groups to lead the phased development of the new cyberinfrastructure.
54 ENVIRONMENTAL SCIENCES↗
XML Based Scientific Data Management Facility
The World Wide Web consortium has developed an Extensible Markup Language (XML) to support the building of better information management infrastructures. The scientific computing community realizing the benefits of HTML has designed markup languages for scientific data. In this paper, we propose a XML based scientific data management facility, XDMF. The project is motivated by the fact that even though a lot of scientific data is being generated, it is not being shared because of lack of standards and infrastructure support for discovering and transforming the data. The proposed data management facility can be used to discover the scientific data itself, the transformation functions, and also for applying the required transformations. We have built a prototype system of the proposed data management facility that can work on different platforms. We have implemented the system using Java, and Apache XSLT engine Xalan. To support remote data and transformation functions, we had to extend the XSLT specification and the Xalan package.
Mehrotra, Piyush↗
XML Based Scientific Data Management Facility
The World Wide Web consortium has developed an Extensible Markup Language (XML) to support the building of better information management infrastructures. The scientific computing community realizing the benefits of XML has designed markup languages for scientific data. In this paper, we propose a XML based scientific data management ,facility, XDMF. The project is motivated by the fact that even though a lot of scientific data is being generated, it is not being shared because of lack of standards and infrastructure support for discovering and transforming the data. The proposed data management facility can be used to discover the scientific data itself, the transformation functions, and also for applying the required transformations. We have built a prototype system of the proposed data management facility that can work on different platforms. We have implemented the system using Java, and Apache XSLT engine Xalan. To support remote data and transformation functions, we had to extend the XSLT specification and the Xalan package.
Mehrotra, P.↗
The Virtual Climate Data Server (vCDS): An iRODS-Based Data Management Software Appliance Supporting Climate Data Services and Virtualization-as-a-Service in the NASA Center for Climate Simulation
Scientific data services are becoming an important part of the NASA Center for Climate Simulation's mission. Our technological response to this expanding role is built around the concept of a Virtual Climate Data Server (vCDS), repetitive provisioning, image-based deployment and distribution, and virtualization-as-a-service. The vCDS is an iRODS-based data server specialized to the needs of a particular data-centric application. We use RPM scripts to build vCDS images in our local computing environment, our local Virtual Machine Environment, NASA s Nebula Cloud Services, and Amazon's Elastic Compute Cloud. Once provisioned into one or more of these virtualized resource classes, vCDSs can use iRODS s federation capabilities to create an integrated ecosystem of managed collections that is scalable and adaptable to changing resource requirements. This approach enables platform- or software-asa- service deployment of vCDS and allows the NCCS to offer virtualization-as-a-service: a capacity to respond in an agile way to new customer requests for data services.
Schnase, John L.↗
Design of an Open Smart Energy Gateway for Smart Meter Data Management
With the widespread deployment of electronic interval meters, commonly known as smart meters, came the promise of real-time data on electric energy consumption. Recognizing an opportunity to provide consumers access to their near real-time energy consumption data directly from their installed smart meter, we designed a mechanism for capturing those data for consumer use via an open smart energy gateway (OpenSEG). By design, OpenSEG provides a clearly defined boundary for equipment and data ownership. OpenSEG is an open-source data management platform to enable better data management of smart meter data. Effectively, it is an information architecture designed to work with the ZigBee Smart Energy Profile 1.x (SEP 1.x). It was specifically designed to reduce cyber-security risks and provide secure information directly from smart meters to consumers in near real time, using display devices already owned by the consumers. OpenSEG stores 48 hours of recent consumption data in a circular cache using a format consistent with commonly available archived (not real-time) consumption data such as Green Button, which is based on the Energy Services Provider Interface (ESPI) data standard. It consists of a common XML format for energy usage information and a data exchange protocol to facilitate automated data transfer upon utility customer authorization. Included in the design is an application program interface by which users can acquire data from OpenSEG for further post processing. A sample data display application is included in the initial software product. The data display application demonstrates that OpenSEG can help electricity use data to be retrieved from a smart meter and ported to a wide variety of user-owned devices such as cell phones or a user-selected database. This system can be used for homes, multi-family buildings, or small commercial buildings in California.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
A Brief Introduction to AI/ML Applications of Air Traffic Management Data at NASA Ames
This presentation will give a brief overview of several AI/ML This presentation will give a brief overview of several AI/ML projects that NASA Ames interns are exploring in partnership with NASA Aeronautic Research Institute (NARI) and the FAA. NASA is interested in Natural Language Processing (NLP) of various legacy text and speech data within air traffic management e.g., Notices To Airmen (NOTAMs), Letters of Agreement (LoAs), Standard Operating Procedures (SOPs), and Air Traffic Control Center audio briefings. Since our focus is on applying state of the art AI/ML tools to legacy air traffic management data, we first showcase the different data sources of interest followed by a brief introduction to the techniques and language models used. We present some exciting preliminary results on each topic including both unsupervised learning techniques (e.g., clustering) and other modern language models (e.g., BERT) that help extract useful information from these data sources that are interpretable by both man and machine.
Air Traffic Management↗
Cockpit data management
This study is a continuation of an FAA effort to alleviate the growing problems of assimilating and managing the flow of data and flight related information in the air transport flight deck. The nature and extent of known pilot interface problems arising from new NAS data management programs were determined by a comparative timeline analysis of crew tasking requirements. A baseline of crew tasking requirements was established for conventional and advanced flight decks operating in the current NAS environment and then compared to the requirements for operation in a future NAS environment emphasizing Mode-S data link and TCAS. Results showed that a CDU-based pilot interface for Mode-S data link substantially increased crew visual activity as compared to the baseline. It was concluded that alternative means of crew interface should be available during high visual workload phases of flight. Results for TCAS implementation showed substantial visual and motor tasking increases, and that there was little available time between crew tasks during a TCAS encounter. It was concluded that additional research should be undertaken to address issues of ATC coordination and the relative benefit of high workload TCAS features.
Groce, J. L.↗
GSE, data management system programmers/User' manual
The GSE data management system is a computerized program which provides for a central storage source for key data associated with the mechanical ground support equipment (MGSE). Eight major sort modes can be requested by the user. Attributes that are printed automatically with each sort include the GSE end item number, description, class code, functional code, fluid media, use location, design responsibility, weight, cost, quantity, dimensions, and applicable documents. Multiple subsorts are available for the class code, functional code, fluid media, use location, design responsibility, and applicable document categories. These sorts and how to use them are described. The program and GSE data bank may be easily updated and expanded.
Schlagheck, R. A.↗
Data management, chapter 5, part C
The data management for a spacecraft radar was defined in terms of an end-to-end data system, which performs the following three functions: (1) sampling and compaction of data onboard the spacecraft, (2) manipulation of radar data on the ground and (3) conversion of radar measurements to geophysical quantities by means of pattern recognition and other machine techniques. Data processing for imaging radar onboard the spacecraft was examined with the conclusion that several techniques can be used to compact the data before storage. It is recommended that compaction techniques be studied further and that existing aircraft radars be modified to provide digital data so that these compaction techniques can be tested.
Source record↗
The Design and Usage of the New Data Management Features in NASTRAN
Two new data management features are installed in the April 1984 release of NASTRAN. These two features are the Rigid Format Data Base and the READFILE capability. The Rigid Format Data Base is stored on external files in card image format and can be easily maintained and expanded by the use of standard text editors. This data base provides the user and the NASTRAN maintenance contractor with an easy means for making changes to a Rigid Format or for generating new Rigid Formats without unnecessary compilations and link editing of NASTRAN. Each Rigid Format entry in the data base contains the Direct Matrix Abstraction Program (DMAP), along with the associated restart, DMAP sequence subset and substructure control flags. The READFILE capability allows an user to reference an external secondary file from the NASTRAN primary input file and to read data from this secondary file. There is no limit to the number of external secondary files that may be referenced and read.
Pamidi, P. R.↗
MST radar data-base management
Data management for Mesospheric-Stratospheric-Tropospheric, (MST) radars is addressed. An incoherent-scatter radar data base is discussed in terms of purpose, centralization, scope, and nature of the data base management system.
Wickwar, V. B.↗