Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 937 records · Page 52

Earth and Space Science Informatics Perspectives on Integrated, Coordinated, Open, Networked (ICON) Science

Abstract This article is composed of three independent commentaries about the state of Integrated, Coordinated, Open, Networked (ICON) principles (Goldman, et al., 2021b, https://doi.org/10.1029/2021EO153180 ) in Earth and Space Science Informatics (ESSI) and includes discussion on the opportunities and challenges of adopting them. Each commentary focuses on a different topic: (Section 2) Global collaboration, cyberinfrastructure, and data sharing; (Section 3) Machine learning for multiscale modeling; (Section 4) Aerial and satellite remote sensing for advancing Earth system model development by integrating field and ancillary data. ESSI addresses data management practices, computation and analysis, and hardware and software infrastructure. Our role in ICON science therefore involves collaborative work to assess, design, implement, and promote practices and tools that enable effective data management, discovery, integration, and reuse for interdisciplinary work in Earth and space science disciplines. Networks of diverse people with expertise across Earth, space, and data science disciplines are essential for efficient and ethical exchanges of findable, accessible, interoperable, and reusable (FAIR) research products and practices. Our challenge is then to coordinate the development of standards, curation practices, and tools that enable integrating and reusing multiple data types, software, multi‐scale models, and machine learning approaches across disciplines in a way that is as open and/or FAIR as ethically possible. This is a major endeavor that could greatly increase the pace and potential of interdisciplinary scientific discovery.

58 GEOSCIENCES↗

The NINDS Parkinson's disease biomarkers program: The Ninds Parkinson's Disease Biomarkers Program

Background: Neuroprotection for Parkinson Disease (PD) remains elusive. Biomarkers hold the promise of removing roadblocks to therapy development. The National Institute of Neurological Disorders and Stroke (NINDS) has therefore established the Parkinson’s Disease Biomarkers Program (PDBP) to promote discovery of biomarkers for use in phase II-III clinical trials in PD. Methods: The PDBP facilitates biomarker development to improve neuroprotective clinical trial design, essential for advancing therapeutics for PD. To date, eleven consortium projects in the PDBP are focused on the development of clinical and laboratory-based PD biomarkers for diagnosis, progression tracking, and/or the prediction of prognosis. Seven of these projects also provide detailed longitudinal data and biospecimens from PD patients and controls, as a resource for all PD researchers. Standardized operating procedures and pooled reference samples have been created in order to allow cross-project comparisons and assessment of batch effects. A web-based Data Management Resource facilitates rapid sharing of data and biosamples across the entire PD research community for additional biomarker projects. Results: Here we describe the PDBP, highlight standard operating procedures for the collection of biospecimens and data, and provide an interim report with quality control analysis on the first 1082 participants and 1033 samples with quality control analysis collected as of October 2014. Conclusions: By making samples and data available to academics and industry, encouraging the adoption of existing standards, and providing a resource which complements existing programs, the PDBP will accelerate the pace of PD biomarker research, with the goal of improving diagnostic methods and treatment.

biofluids↗

Nationwide Buildings Energy Research enabled through an integrated Data Intensive Scientific Workflow and Advanced Analysis Environment

Modern workflow systems enable scientists to run ensemble simulations at unprecedented scales and levels of complexity, allowing them to study system sizes previously impossible to achieve, due to the inherent resource requirements needed for the modeling work. However as a result of these new capabilities the science teams suddenly also face unprecedented data volumes that they are unable to analyze with their existing tools and methodologies in a timely fashion. In this paper we will describe the ongoing development work to create an integrated data intensive scientific workflow and analysis environment that offers researchers the ability to easily create and execute complex simulation studies and provides them with different scalable methods to analyze the resulting data volumes. The integration of simulation and analysis environments is hereby not only a question of ease of use, but supports fundamental functions in the correlated analysis of simulation input, execution details and derived results for multi-variant, complex studies. To this end the team extended and integrated the existing capabilities of the Velo data management and analysis infrastructure, the MeDICi data intensive workflow system and RHIPE the R for Hadoop version of the well-known statistics package, as well as developing a new visual analytics interface for the result exploitation by multi-domain users. The capabilities of the new environment are demonstrated on a use case that focusses on the Pacific Northwest National Laboratory (PNNL) building energy team, showing how they were able to take their previously local scale simulations to a nationwide level by utilizing data intensive computing techniques not only for their modeling work, but also for the subsequent analysis of their modeling results. As part of the PNNL research initiative PRIMA (Platform for Regional Integrated Modeling and Analysis) the team performed an initial 3 year study of building energy demands for the US Eastern Interconnect domain, which they are now planning to extend to predict the demand for the complete century. The initial study raised their data demands from a few GBs to 400GB for the 3year study and expected tens of TBs for the full century.

99 GENERAL AND MISCELLANEOUS↗

Performance characterization of scientific workflows for the optimal use of Burst Buffers

Scientific discoveries are significantly rely on the analysis of large volumes of data from observations and simulations of complex phenomena. Scientists compose the complex analyses as workflows and execute them on large-scale HPC systems. The workflow structures are in contrast with monolithic single simulations that have often been the primary use case on HPC systems. At the same time, new storage paradigms such as Burst Buffers are becoming available on HPC platforms. In this paper, we analyze the performance characteristics of a Burst Buffer and two representative scientific workflows with the aim of optimizing the usage of a Burst Buffer, extending our previous analyses (Daley et al., 2016). Our key contributions are (a) developing a performance analysis methodology pertinent to Burst Buffers, (b) improving the use of a Burst Buffer in workflows with bandwidth-sensitive and metadata-sensitive I/O workloads, (c) highlighting the key data management challenges when incorporating a Burst Buffer in the studied scientific workflows.

97 MATHEMATICS AND COMPUTING↗

DeepLynx Ecosystem 2025

Poor data integration and governance continue to plague complex engineering projects, resulting in missed cost, schedule, and performance targets. Departments operate in isolated systems with manual data exchange, creating fragmented information that compounds errors and leads to significant delays and cost overruns. The DeepLynx ecosystem addresses these challenges through an open-source, modular data management platform that transforms fragmented project data into an integrated digital thread. Built on a federated microservice architecture, the ecosystem comprises seven specialized tools centered around DeepLynx Nexus, a unified data catalog with hierarchical organization and graph-based navigation capabilities. The ecosystem includes: DeepLynx Stream for real-time timeseries data ingestion from industrial sources; DeepLynx Ingest for governed data uploads with formal review workflows; DeepLynx Lattice for ontology-based entity and relationship extraction; DeepLynx Run for workflow orchestration and secure AI/ML compute; DeepLynx Visualize for 3D digital twin visualization; and DeepLynx Insight for AI-assisted document analysis with traceable, grounded responses. Deployable in cloud, on-premise, or hybrid environments using containerized Docker applications and Helm charts, the DeepLynx ecosystem provides flexible infrastructure that adapts to organizational requirements. By consolidating project data into a unified data lake with role-based access controls and OAuth2 authentication, DeepLynx enables digital thread and digital twin capabilities that improve decision-making, reduce risk, and support complex engineering workflows throughout the project lifecycle.

42 - ENGINEERING↗

Analysis of severe storm data

The Mesoscale Analysis and Space Sensor (MASS) Data Management and Analysis System developed by Atsuko Computing International (ACI) on the MASS HP-1000 Computer System within the Systems Dynamics Laboratory of the Marshall Space Flight Center is described. The MASS Data Management and Analysis System was successfully implemented and utilized daily by atmospheric scientists to graphically display and analyze large volumes of conventional and satellite derived meteorological data. The scientists can process interactively various atmospheric data (Sounding, Single Level, Gird, and Image) by utilizing the MASS (AVE80) share common data and user inputs, thereby reducing overhead, optimizing execution time, and thus enhancing user flexibility, useability, and understandability of the total system/software capabilities. In addition ACI installed eight APPLE III graphics/imaging computer terminals in individual scientist offices and integrated them into the MASS HP-1000 Computer System thus providing significant enhancement to the overall research environment.

Hickey, J. S.↗

U.S. Department of Energy Consequence Management Under the National Response Framework

Dramatic advances in data management have been made as a result of the Paperless FRMAC initiative, sponsored by the DOE’s Office of Emergency Response (NA-42). The FRMAC (Federal Radiological Monitoring and Assessment Center) is the hub for all radiological monitoring and the production of data products that interpret those measurements in terms of protective action guidelines. As such, very large amounts of data must be quickly assimilated from numerous sources and then widely distributed as graphical interpretations as fast as possible. Paperless FRMAC is a broad initiative to move that data faster, farther and better through telemetry, automation, and networking. This discussion reviews for the first time the status of the now two-year-old Paperless FRMAC initiative. Key features of Paperless FRMAC include multipath telemetry of measurements from DOE field teams, 24/7 Internet presence, early data entry by first responders, support for distance collaborations, and data exchange with the EPA’s SCRIBE database. The heart of the enterprise is the RAMS database, which provides seamless interfacing with GIS, LIMS, and TurboFRMAC for calculations. Paperless FRMAC is presented to users via two Internet websites. The first, FRMAC Portal website, is restricted to the emergency responders for data input, analysis, and product development. The second, CMweb website, is the showroom for completed and approved products, which are made much more widely available. In fact, CMweb offers seamless, single log-in access to all data products for any IMAAC, NARAC or CMweb user.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

NASIS data base management system: IBM 360 TSS implementation. Volume 6: NASIS message file

The message file for the NASA Aerospace Safety Information System (NASIS) is discussed. The message file contains all the message and term explanations for the system. The data contained in the file can be broken down into three separate sections: (1) global terms, (2) local terms, and (3) system messages. The various terms are defined and their use within the system is explained.

Source record↗

NASIS data base management system - IBM 360/370 OS MVT implementation. 6: NASIS message file

The message file for the NASA Aerospace Safety Information System (NASIS) is discussed. The message file contains all the message and term explanations for the system. The data contained in the file can be broken down into three separate sections: (1) global terms, (2) local terms, and (3) system messages. The various terms are defined and their use within the system is explained.

Source record↗

AAM NC ATI TechTalk - Aerograph Architecture v1

Aerograph is NASA’s data management system for Advanced Air Mobility. Its mission is to support AAM research by providing a reliable and secure system that collects, stores, protects, and shares AAM data. Its vision is to provide a system that AAM research scientists, aerospace engineers, data scientists, and analysts trust for obtaining NC data and performing key analyses. The types of data Aerograph manages involves data related to flight test events, including: Aircraft Performance and Characterization (e.g., position reports) Airspace (e.g., operation intent, waypoints, and constraints) Environment (e.g., surface and wind weather) Infrastructure (e.g., surveillance coverage) Derivative Analytical Artifacts (e.g., glide path performance chart, 3D position chart, Integrated Data Product)

Aerograph↗

TDRSS data handling and management system study. Ground station systems for data handling and relay satellite control

Results of a two-phase study of the (Data Handling and Management System DHMS) are presented. An original baseline DHMS is described. Its estimated costs are presented in detail. The DHMS automates the Tracking and Data Relay Satellite System (TDRSS) ground station's functions and handles both the forward and return link user and relay satellite data passing through the station. Direction of the DHMS is effected via a TDRSS Operations Control Central (OCC) that is remotely located. A composite ground station system, a modified DHMS (MDHMS), was conceptually developed. The MDHMS performs both the DHMS and OCC functions. Configurations and costs are presented for systems using minicomputers and midicomputers. It is concluded that a MDHMS should be configured with a combination of the two computer types. The midicomputers provide the system's organizational direction and computational power, and the minicomputers (or interface processors) perform repetitive data handling functions that relieve the midicomputers of these burdensome tasks.

Source record↗

Challenges and Opportunities of User-Level File Systemsfor HPC

The performance gap between magnetic disks and data processing on HPC systems has become that huge that an efficient data processing can only be achieved by introducing non-volatile memory (NVRAM) as a new storage tier. Although the benefits of hierarchical storage have been adequately demonstrated to the point that the newest leadership class HPC systems will employ burst buffers, critical questions remain for supporting hierarchical storage systems, including: How should we present hierarchical storage systems to user applications, such that they are easy to use and that application code is portable across systems? How should we manage data movement through a storage hierarchy for best performance and resilience of data? How do the particular I/O use cases mandate the way we manage data? There have been many efforts to explore this space in the form of file systems, with increasingly more implemented at the user level. This is because it is relatively easy to swap in new, specialized user-level file systems for use by applications on a case-by-case basis, as opposed to the current mainstream approach of using general-purpose, system-level file systems which may not be optimized for HPC workloads and must be installed by administrators. In contrast, file systems at the user level can be tailored for specific HPC workloads for high performance and can be used by applications without administrator intervention. Many user-level file system developers have found themselves “having to reinvent the wheel” to implement various optimizations in their file systems. Thus, a main goal of this meeting was to bring together experts in I/O performance, file systems, and storage, and collectively explore the space of current and future problems and solutions for I/O on hierarchical storage systems in order to begin a community effort in enabling user-level file system support for HPC systems. We had a lively week of learning about each other’s approaches as well as unique I/O use cases that can influence the design of a community-driven file and storage system standards. The agenda for this meeting contained talks from participants on the following high level topics: HPC storage and I/O support today; what do HPC users need for I/O; existing userlevel file system efforts; object stores and other alternative storage systems; and components for building user-level file systems. The talks were short and intended to be conversation starters for more in-depth discussions with the whole group. The participants engaged in lengthy discussions on various questions that arose from the talks including: Are we ready to program to a memory hierarchy versus block devices? Are the needs of HPC users reflected in our existing file systems and storage systems? Should we drop or keep POSIX moving forward? What do we mean when we say “user-level file system”? Do we all mean the same thing? How should the IO 500 benchmark be defined so it is fair and useful? and How are stage-in and stage-out actually going to work? The report for this seminar contains a record of the talks from the participants as well as the resulting discussions. Our hope is that the effort initiated during this seminar will result in long-term collaborations that will benefit the HPC community as a whole.

97 MATHEMATICS AND COMPUTING↗

UPS: Unified PMU-Data Storage System to Enhance T+D PMU Data Usability

The emerging distribution-level phasor measurement unit (D-PMU) is expected to play an important role in enhancing distribution system observability and situational-awareness. It is a demanding yet challenging task to develop advanced D-PMU data management and analytics tools to improve D-PMU data usability and further promote D-PMU projects deployment. This paper focuses on D-PMU data processing and storage. It presents a brief review of existing D-PMU data storage systems and points out their limitations on high-performance, flexibility, and scalability. To overcome the limitations, a unified PMU-data storage system (UPS) is proposed. Specifically, a unified I/O interface between storage servers and computing jobs is developed to effectively reduce the overhead of managing various computing jobs and data analytics over multiple storage infrastructures; and PMUCache with PMUCache partitioning and PMUCache replacement algorithms are designed to support in-situ data processing and shared distributed data storage and further serve the computing jobs/queries with different Quality of Service (QoS) requirements. Through a series of experiments, it is demonstrated that UPS achieves high performance on fast and big data processing and storage, and efficiently increases the flexibility and scalability of PMU data management systems.

42 ENGINEERING↗

Development and operational experience of the web based application to collect, manage, and release the alignment and calibration configurations for data processing at CMS

The alignment and calibration workflows at the Compact Muon Solenoid (CMS) experiment are fundamental to provide a high quality physics data and to maintain the design performance of the experiment. To facilitate the operational efforts required by the experiment, the alignment and calibration team has developed and deployed a set of web-based applications to search, navigate and prepare a consistent set of calibrations to be consumed in reconstruction of data for physics, accessible through the Condition DB Browser. The Condition DB Browser hosts also various data management tools, including a visualization tool that allows to easily inspect alignment an calibration contents, an user-defined notification agent for delivering updates on modification to the database, a logging service for the user and the automatic online-to-offline condition uploads. In this paper we report on the operational experience of this web application from 2017 data taking, with focus on new features and tools incorporated during this period.

Md, Hasbuddin↗

Knowledge-based assistance for science visualization and analysis using large distributed databases

Within this decade, the growth in complexity of exploratory data analysis and the sheer volume of space data require new and innovative approaches to support science investigators in achieving their research objectives. To date, there have been numerous efforts addressing the individual issues involved in inter-disciplinary, multi-instrument investigations. However, while successful in small scale, these efforts have not proven to be open and scalable. This proposal addresses four areas of significant need: scientific visualization and analysis; science data management; interactions in a distributed, heterogeneous environment; and knowledge-based assistance for these functions. The fundamental innovation embedded with this proposal is the integration of three automation technologies, namely, knowledge-based expert systems, science visualization and science data management. This integration is based on concept called the DataHub. With the DataHub concept, NASA will be able to apply a more complete solution to all nodes of a distributed system. Both computation nodes and interactive nodes will be able to effectively and efficiently use the data services (address, retrieval, update, etc.) with a distributed, interdisciplinary information system in a uniform and standard way. This will allow the science investigators to concentrate on their scientific endeavors, rather than to involve themselves in the intricate technical details of the systems and tools required to accomplish their work. Thus, science investigators need not be programmers. The emphasis will be on the definition and prototyping of system elements with sufficient detail to enable data analysis and interpretation leading to publishable scientific results. In addition, the proposed work includes all the required end-to-end components and interfaces to demonstrate the completed concept.

Handley, Thomas H., Jr.↗

A comprehensive data acquisition and management system for an ecosystem-scale peatland warming and elevated CO 2 experiment

Ecosystem-scale manipulation experiments represent large science investments that require well-designed data acquisition and management systems to provide reliable, accurate information to project participants and third party users. The SPRUCE project (Spruce and Peatland Responses Under Climatic and Environmental Change, http://mnspruce.ornl.gov ) is such an experiment funded by the Department of Energy's (DOE), Office of Science, Terrestrial Ecosystem Science (TES) Program. The SPRUCE experimental mission is to assess ecosystem-level biological responses of vulnerable, high carbon terrestrial ecosystems to a range of climate warming manipulations and an elevated CO 2 atmosphere. SPRUCE provides a platform for testing mechanisms controlling the vulnerability of organisms, biogeochemical processes, and ecosystems to climatic change (e.g., thresholds for organism decline or mortality, limitations to regeneration, biogeochemical limitations to productivity, and the cycling and release of CO 2 and CH 4 to the atmosphere). The SPRUCE experiment will generate a wide range of continuous and discrete measurements. In order to successfully manage SPRUCE data collection, achieve SPRUCE science objectives, and support broader climate change research, the research staff has designed a flexible data system using proven network technologies and software components. The primary SPRUCE data system components are the following; 1. data acquisition and control system – set of hardware and software to retrieve biological and engineering data from sensors, collect sensor status information, and distribute feedback to control components; 2. data collection system – set of hardware and software to deliver data to a central depository for storage and further processing; and 3. data management plan – set of plans, policies, and practices to control consistency, protect data integrity, and deliver data. This publication presents our approach to meeting the challenges of designing and constructing an efficient data system for managing high volume sources of in situ observations in a remote, harsh environmental location. Finally, the approach covers data flow starting from the sensors and ending at the archival/distribution points, discusses types of hardware and software used, examines design considerations that were used to choose them, and describes the data management practices chosen to control and enhance the value of the data.

54 ENVIRONMENTAL SCIENCES↗

A spatial data handling system for retrieval of images by unrestricted regions of user interest

The Intelligent Data Management (IDM) project at NASA/Goddard Space Flight Center has prototyped an Intelligent Information Fusion System (IIFS), which automatically ingests metadata from remote sensor observations into a large catalog which is directly queryable by end-users. The greatest challenge in the implementation of this catalog was supporting spatially-driven searches, where the user has a possible complex region of interest and wishes to recover those images that overlap all or simply a part of that region. A spatial data management system is described, which is capable of storing and retrieving records of image data regardless of their source. This system was designed and implemented as part of the IIFS catalog. A new data structure, called a hypercylinder, is central to the design. The hypercylinder is specifically tailored for data distributed over the surface of a sphere, such as satellite observations of the Earth or space. Operations on the hypercylinder are regulated by two expert systems. The first governs the ingest of new metadata records, and maintains the efficiency of the data structure as it grows. The second translates, plans, and executes users' spatial queries, performing incremental optimization as partial query results are returned.

Dorfman, Erik↗

NASIS data base management system - IBM 360/370 OS MVT implementation. 5: Retrieval command system reference manual

The retrieval command subsystem reference manual for the NASA Aerospace Safety Information System (NASIS) is presented. The output oriented classification of retrieval commands provides the user with the ability to review a set of data items for verification or inspection as a typewriter or CRT terminal and to print a set of data on a remote printer. Predefined and user-definable data formatting are available for both output media.

Source record↗