Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS1986

ADAMS: AIRLAB data management system user's guide

The AIRLAB Data Management System (ADAMS) is an online environment that supports research at NASA's AIRLAB. ADAMS provides an easy to use interactive interface that eases the task of documenting and managing information about experiments and improves communication among project members. Data managed by ADAMS includes information about experiments, data sets produced, software and hardware available in AIRLAB as well as that used in a particular experiment, and an on-line engineer's notebook. The User's Guide provides an overview of the ADAMS system as well as details of the operations available within ADAMS. A tutorial section takes the user step-by-step through a typical ADAMS session. ADAMS runs under the VAX/VMS operating system and uses the ORACLE database management system and DEC/FMS (the Forms Management System). ADAMS can be run from any VAX connected via DECnet to the ORACLE host VAX. The ADAMS system is designed for simplicity, so interactions within the underlying data management system and communications network are hidden from the user.

Conrad, C. L.↗
DOE OSTI2025

Recent Experience with the CMS Data Management System

The CMS[1] experiment manages a large-scale data infrastructure, currently handling over 200 PB of disk and 500 PB of tape storage and transferring more than 1 PB of data per day on average between various WLCG[2] sites. Utilizing Rucio[3] for high-level data management, FTS[4] for data transfers, and a variety of storage and network technologies at the sites, CMS confronts inevitable challenges due to the system’s growing scale and evolving nature. Key challenges include managing transfer and storage failures, optimizing data distribution across different storages based on production and analysis needs, implementing necessary technology upgrades and migrations, and efficiently handling user requests. The data management team has established comprehensive monitoring to supervise this system and has successfully addressed many of these challenges. The team’s efforts aim to ensure data availability and protection, minimize failures and manual interventions, maximize transfer throughput and resource utilization, and provide reliable user support. This paper details the operational experience of CMS with its data management system in recent years, focusing on the encountered challenges, the effective strategies employed to overcome them and the ongoing challenges as we prepare for future demands.

Öztürk, Hasan [CERN]↗
DOE OSTI2021

SIRIUS: Science-Driven Data Management for Multi-Tiered Storage

The data sets being generated by large applications on very large-scale systems are increasing in both size and complexity. At the same time, there are new ways available to store and access these data sets. The goal in this project is to develop software that applications can use to make use of new and existing storage technologies in more sophisticated ways. One challenge in scientific data management is handling ‘hot’ vs ‘cold’ data. Data that is hot is data that is needed (or will be needed soon) in order for the program to continue progressing, while cold data is either output (and so will not be need further during the life of the program) or will not be needed until significantly later in the program’s run. Hot data should be stored in a way that allows fast access. On most systems, economic factors lead to an inverse relationship between storage performance and storage capacity and so fast access storage is limited. This makes it important to correctly place hot and cold data and avoid cold data unnecessarily consuming precious resources. In this reporting period, we addressed this challenge in various ways and at various levels. Data management frameworks offer only limited control to applications in how data is stored. We have added software capabilities for seamlessly moving data between layers of the storage technology using promote and demote functions to existing software frameworks. This gives direct control to applications in deciding what priority data receives. Additionally, we integrated different storage layer management frameworks in order to allow data to be exchanged and moved between storage layers in a consistent way across the application. Further, applications are not always able to directly decide what storage level makes sense for a given piece of data without an understanding of the underlying storage technologies. Data storage frameworks are often positioned to make these sorts of decisions in service of the application. We have added machine-learning based capabilities to data staging frameworks in order to make intelligent decisions about where data should be stored given learning about patterns in previous usage of similar data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Adding Data Management Services to Parallel File Systems

The objective of this project, called DAMASC for “Data Management in Scientific Computing”, is to coalesce data management with parallel file system management to present a declarative interface to scientists for managing, querying, and analyzing extremely large data sets efficiently and predictably. Managing extremely large data sets is a key challenge of exascale computing. The overhead, energy, and cost of moving massive volumes of data demand designs where computation is close to storage. In current architectures, compute/analysis clusters access data in a physically separate parallel file system and largely leave it scientist to reduce data movement. Over the past decades the high-end computing community has adopted middleware with multiple layers of abstractions and specialized file formats such as NetCDF-4 and HDF5. These abstractions provide a limited set of high-level data processing functions, but have inherent functionality and performance limitations: middleware that provides access to the highly structured contents of scientific data files stored in the (unstructured) file systems can only optimize to the extent that file system interfaces permit; the highly structured formats of these files often impedes native file system performance optimizations. We are developing Damasc, an enhanced high-performance file system with native rich data management services. Damasc will enable efficient queries and updates over files stored in their native byte-stream format while retaining the inherent performance of file system data storage via declarative queries and updates over views of underlying files. Damasc has four key benefits for the development of data-intensive scientific code: (1) applications can use important data-management services, such as declarative queries, views, and provenance tracking, that are currently available only within database systems; (2) the use of these services becomes easier, as they are provided within a familiar file-based ecosystem; (3) common optimizations, e.g., indexing and caching, are readily supported across several file formats, avoiding effort duplication; and (4) performance improves significantly, as data processing is integrated more tightly with data storage. Our key contributions are: SciHadoop which explores changes to MapReduce assumption by taking advantage of semantics of structured data while preserving MapReduce’s failure and resource management; DataMods which extends common abstractions of parallel file systems so they become programmable such that they can be extended to natively support a variety of data models and can be hooked into emerging distributed runtimes such as Stanford’s Legion; and Miso which combines Hadoop and relational data warehousing to minimize time to insight, taking into account the overhead of ingesting data into data warehousing.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
NASA NTRS1988

A design for a ground-based data management system

An initial design for a ground-based data management system which includes intelligent data abstraction and cataloging is described. The large quantity of data on some current and future NASA missions leads to significant problems in providing scientists with quick access to relevant data. Human screening of data for potential relevance to a particular study is time-consuming and costly. Intelligent databases can provide automatic screening when given relevent scientific parameters and constraints. The data management system would provide, at a minimum, information of availability of the range of data, the type available, specific time periods covered together with data quality information, and related sources of data. The system would inform the user about the primary types of screening, analysis, and methods of presentation available to the user. The system would then aid the user with performing the desired tasks, in such a way that the user need only specify the scientific parameters and objectives, and not worry about specific details for running a particular program. The design contains modules for data abstraction, catalog plan abstraction, a user-friendly interface, and expert systems for data handling, data evaluation, and application analysis. The emphasis is on developing general facilities for data representation, description, analysis, and presentation that will be easily used by scientists directly, thus bypassing the knowledge acquisition bottleneck. Expert system technology is used for many different aspects of the data management system, including the direct user interface, the interface to the data analysis routines, and the analysis of instrument status.

Lambird, Barbara A.↗
DOE OSTI2019

Distributed data management on Belle II

The Belle II experiment started taking physics data in April 2018 with an estimated total volume of all files including raw events, Monte-Carlo and skim statistics of 340 petabytes expected by the end of operations in the late-2020s. Originally designed as a fully integrated component of the BelleDIRAC production system, the Belle II distributed data management (DDM) software needs to manage data across about 29 storage elements worldwide for a collaboration of nearly 1000 physicists. By late 2018, this software required significant performance improvements to meet the requirements of physics data taking and was seriously lacking in automation. Rucio, the DDM solution created by ATLAS, was an obvious alternative but required tight integration with BelleDIRAC and a seamless yet non-trivial migration. This contribution describes the work done on both DDM options, the current status of the software running successfully in production and the problems associated with trying to balance long-term operations cost against short term risk. .

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Value of Information and Data Management Platform Promoting Modern Data Analysis Utilization. STTR Phase I, Final Technical Report

The immense growth in weather and river sensor data availability along with increases in computational power and algorithmic sophistication have enabled the widespread application of physical, machine learning, and artificial intelligence algorithms to river forecasting. The forecasts produced by widely varying model approaches are inherently limited in different ways, such as by the number and quality of training observations, producing biased predictions that achieve optimal skill only over specific ranges of weather and river conditions. Thus, despite the growing availability and new sources of forecasts, hydropower owners cannot fully leverage the information because they lack an effective mechanism to determine the “best” forecast for given conditions or objectives. Tools are needed to consolidate forecast information, demonstrate skill, and extract optimal forecast guidance from the range of possibilities. Such tools will increase uptake and beneficial use of the growing wealth of forecast information, and lead to improved operating decisions, optimal power generation, and enhanced system value. This research proposes to improve the value of forecasts by developing a broadly accessible verification framework and applying machine learning techniques to identify the ‘best’ forecast from an ensemble of sources given the current weather, river, and operating conditions.

13 HYDRO ENERGY↗
NASA NTRS1992

Data Management Systems (DMS): Complex data types study. Volume 1: Appendices A-B. Volume 2: Appendices C1-C5. Volume 3: Appendices D1-D3 and E

Two categories were chosen for study: the issue of using a preprocessor on Ada code of Application Programs which would interface with the Run-Time Object Data Base Standard Services (RODB STSV), the intent was to catch and correct any mis-registration errors of the program coder between the user declared Objects, their types, their addresses, and the corresponding RODB definitions; and RODB STSV Performance Issues and Identification of Problems with the planned methods for accessing Primitive Object Attributes, this included the study of an alternate storage scheme to the 'store objects by attribute' scheme in the current design of the RODB. The study resulted in essentially three separate documents, an interpretation of the system requirements, an assessment of the preliminary design, and a detailing of the components of a detailed design.

Leibfried, T. F., Jr.↗
NASA NTRSDate not supplied

HRP Data Management Plan

The purpose of Human Research Program Data Management Plan (DMP) is to define the processes and activities required for the overall management of the research data collected and managed by HRP throughout their life cycle. New updates to the Data Management Plan in 2023 include 1. CAPABILITIES AND SERVICES Data Repositories. Principal Investigators (PIs) funded by HRP may be asked to submit data to one of several NASA data repositories. HRP archives data in the NASA Life Sciences Portal (NLSP) that it considers to be unique and high value. This includes data from human subjects in space flight (ISS and commercial flights) and ground analogs to spaceflight; spaceflight tech demos involving humans; human omics data including the microbiome; parabolic flight studies; and the NASA Space Radiation Laboratory (NSRL). The Open Science Data Repository (OSDR) includes The Ames Life Sciences Data Archive (ALSDA), used to archive non-human biological data (e.g., animal) generated by the Human Research program, and GeneLab, available to HRP PIs to archive non-human omics data. Catalog for search and retrieval. A catalog of non-human HRP life science experiments, with all associated descriptions (mission, payload, hardware, and personnel related information), and biospecimens is provided on the NLSP public web site for search and retrieval. 2. IRB ROLE IN RETURN OF INDIVIDUAL RESEARCH RESULTS The NASA IRB manages the process for incidental findings and for returning results to subjects for studies for which NASA IRB is the IRB of record. Omics data, especially genomics data, may generate information significant to the health of or risk to a research subject. These data potentially hold the keys to understand lifetime risks of chronic diseases, such as cancer, as well as risks associated with exposures common in space flight. 3. UPDATE OF TERMS – IDENTIFIABLE AND ATTRIBUTABLE DATA HRP now follows Federal and NASA policy by using “identifiable” instead of “attributable” for Personally Identifiable Information (PII). 4. POLICY ABOUT INTERNAL NON-RESEARCH USE OF DATA The HRP Chief Scientist grants access to data from HRP-funded research for non-research internal use that includes program management, customer facilitation, strategic planning, and risk research planning. Typical HRP personnel granted access to HRP research data for internal use include the Element Scientist, Subject Matter Experts (SME), and Data/bioinformatics Scientists. If data accessed for Internal Use is provided to an intramural or extramural scientist for hypothesis driven research, all Federal and NASA regulations (e.g., IRB review) regarding human subject research apply.

Data Management Plan↗
NASA NTRS1976

Data management in engineering

An introduction to computer based data management is presented with an orientation toward the needs of engineering application. The characteristics and structure of data management systems are discussed. A link to familiar engineering applications of computing is established through a discussion of data structure and data access procedures. An example data management system for a hypothetical engineering application is presented.

Browne, J. C.↗
NASA NTRS1974

Spacelab data management subsystem phase B study

The Spacelab data management system is described. The data management subsystem (DMS) integrates the avionics equipment into an operational system by providing the computations, logic, signal flow, and interfaces needed to effectively command, control, monitor, and check out the experiment and subsystem hardware. Also, the DMS collects/retrieves experiment data and other information by recording and by command of the data relay link to ground. The major elements of the DMS are the computer subsystem, data acquisition and distribution subsystem, controls and display subsystem, onboard checkout subsystem, and software. The results of the DMS portion of the Spacelab Phase B Concept Definition Study are analyzed.

Source record↗
NASA NTRS1987

SAM 2 and SAGE data management and processing

The data management and processing supplied by ST Systems Corporation (STX) for the Stratospheric Aerosol Measurement 2 (SAM 2) and Stratospheric Aerosol and Gas Experiment (SAGE) experiments for the years 1983 to 1986 are described. Included are discussions of data validation, documentation, and scientific analysis, as well as the archival schedule met by the operational reduction of SAM 2 and SAGE data. Work under this contract resulted in the archiving of the first seven years of SAM 2 data and all three years of SAGE data. A list of publications and presentations supported was also included.

Osborn, M. T.↗
NASA NTRS1976

Lunar motion analysis and laser data management

Work completed in lunar motion analysis and laser data management during the period July 1, 1971 - September 30, 1975 was reported. In this context, analysis refers to theoretical or numerical studies involving real or potential applications of such observations to improvement of the physical model, and data management refers to the process by which observed photon events are turned into observations and are made available to potential users. The data analysis work included: (1) bringing to operational status of computer programs for the numerical integration of the lunar orbit motion and for the application of lunar laser time delays for the improvement of the parameters of the physical model, (2) program improvement and program integrity, (3) three-dimensional ephemeris, and (4) miscellaneous independent studies. The data management work included: (1) data identification, (2) observatory interfaces, and (3) data distribution.

Mulholland, J. D.↗
NASA NTRSDate not supplied

Laying The Foundations for FAIR-ER Science: ISA And LSDA Data Submission Process in NASA’s Evolving Data Management Environment

The Life Sciences Data Archive (LSDA) archives data resulting from research on the effects of spaceflight on humans and the development of countermeasures to mitigate spaceflight hazards. Archivists work with researchers to ensure that unique and high value data products and their metadata are preserved and managed to support current and future research. Currently, LSDA is updating its procedures and data submission requirements in response to the evolving data preservation environment at NASA. LSDA is implementing best practices for research data management through the establishment of clear data submission guidelines, integration of the FAIR (Findability, Accessibility, Interoperability, Reusability) principles, and use of the ISA (Investigation, Study, Assay) research metadata framework for data discoverability and transparency into the data management processes. These changes directly impact LSDA’s requirements for research data submissions. The newly revised Research Data Submission Agreement (RDSA), formerly the Data Submission Agreement (DSA), introduces ISA-compatible metadata collection standards to LSDA’s process. Adherence to LSDA’s data submission guidelines enhances the FAIR-ness of the repository’s collections for future users. This presentation will discuss (1) how submission of research data and associated metadata are impacted by current data management policies, (2) benefits of the adoption of FAIR principles and the ISA metadata framework for retrospective studies utilizing existing LSDA datasets and historic data collections, and (3) the support LSDA will provide to researchers during this transition.

Data submission↗