Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 847 records · Page 47

NASA Earth Science Data Systems: Open Data, Services and Software

Open, Public, Electronic and Necessary (OPEN) Government Data Act, which requires all non-sensitive government data to be made available in open and machine-readable formats by default is part of the overall Foundations for Evidence-Based Policymaking (FEBP) Act passed in late 2018. This town hall will bring together data officers and policy makers from NOAA, EPA, NASA and others to discuss the impact of the act on data management strategies going forward. In 1994 NASA's Earth Science Division committed to an open data policy for all civilian Earth satellite data. NASA's Earth Observing System Data and Information System (EOSDIS) became the first large scale data system to facilitate public access to global Earth system data and information. This presentation reviews key elements of EOSDIS data policy and data management activities that support the OPEN Government Data Act.

Moses, John F.↗

Maintenance of Maywood Laboratory Operations Support During Contractor Transition and Replacement of the Project Databases - 16329

In December 2013, Cabrera Services (Cabrera) executed transition of the FUSRAP Maywood Superfund Site remediation project operations, including an on-site accredited environmental lab, from a large business that had established technology infrastructure in place for over a decade which required updating to support the expected remediation scope and client needs. As of November 2015, Cabrera has shipped over 116,000 yd{sup 3} of contaminated soil to disposal facilities. During the transition, the USACE FUSRAP Maywood Laboratory (UFML) continued to support project operations, providing the expected timely and reliable results and maintaining their New Jersey Department of Environmental Protection certification for all parameters of concern, while having to cope with new challenges and additional tasks having the potential to disrupt the laboratory's ability to support project operations. This case study examines the challenges with maintaining a certified radiochemistry laboratory and an aggressive remediation schedule during a period of prime contractor transition which included a complete IT infrastructure overhaul, and conversion of the existing Microsoft Access laboratory database and project Electronic Data Management System (EDMS) to a customized commercial solution. Challenges: I. Procurement of a new database subcontractor and working closely with them to achieve the following: a. Migration of all environmental (including radiological, chemical, and quality control), health and safety, and spatial data as well as reports, queries, and analytical instrument software processes from a MS Access data management system to a SQL server system; and b. Verification that the functionality and data integrity of the databases remained intact. II. The onsite laboratory was changed over to a new onsite network and the following tasks were completed with little or no interruption to site operations by January 10, 2014: i. Installation of new instrument personal computers (PCs) and making them fully operational; ii. Configuration of the lab domain and server to host the LIMS (replaced all servers/domain/workstations with Windows 7 and Server 2012 machines). iii. Networking of the laboratory building into the rest of the project site via fiber optic cabling across the site. iv. Upgrade of the wide area network (WAN) data connection to high-speed Fiber Optic service; v. Upgrade of the telephones into voice over internet protocol (VOIP) phones. vi. Splitting of the database into front ends and a centralized back-end hosted on the new server to allow simultaneous access for fixes and improvements. III. Phone service interruption; phone systems were operational by January 10, 2014. Additional Tasks: I. Revision of the USACE FUSRAP Maywood Laboratory (UFML) Quality Systems Manual Rev. 1 Submitted 01/16/14; II. Writing of the Uniform Federal Policy Quality Assurance Project Plan Rev. 1 (UFP-QAPP) submitted 1/21/14. This document replaced the prior project data quality plan, the Chemical Data Quality Management Plan. III. Support of USACE communications to the New Jersey Department of Environmental Protection (NJDEP) on lab management and personnel transition to Cabrera. This work was important since it ensured a smooth transition and shows how critical project records infrastructure such as databases, can be upgraded with minimal interruption of project support. Such work may provide benefit to other similar projects, especially long term projects where database infrastructure upgrades may be beneficial in terms of improved efficiencies in data recording and transfer, and improved security in data storage. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

mzPeak: Designing a Scalable, Interoperable, and Future-Ready Mass Spectrometry Data Format

Advances in mass spectrometry (MS) instrumentation, such as higher resolution, faster scan speeds, and improved sensitivity, have significantly increased the volume and complexity of data. The growing adoption of imaging and ion mobility further amplifies these challenges across MS-based omics fields, including proteomics, metabolomics, and lipidomics. While these technologies unlock new possibilities, they also present significant challenges in data management, storage, and accessibility. Existing open formats, such as the XML-based community standards mzML and imzML, struggle to meet the demands of modern MS workflows due to their large file sizes, slow data access, and limited metadata support. Vendor-specific formats, while optimized for proprietary instruments, lack interoperability, comprehensive metadata support and long-term archival reliability. This white paper lays the groundwork for mzPeak, a next-generation community data format designed to address these challenges and support high-throughput, multi-dimensional MS workflows. By adopting a hybrid model that combines efficient binary storage for numerical data and both human and machine-readable metadata storage, mzPeak will reduce file sizes, accelerate data access, and offer a scalable, adaptable solution for evolving MS technologies. For researchers, mzPeak will enable enhanced interoperability across platforms, seamless support for complex workflows including ion mobility and MS imaging, and faster data access compared to existing community formats such as mzML. Its design will ensure data is managed in compliance with regulatory standards, essential for applications such as precision medicine and chemical safety, where long-term data integrity and accessibility are critical. For vendors, mzPeak provides a streamlined, open alternative to proprietary formats, reducing the burden of regulatory compliance while aligning with the industry's push for transparency and standardization. By offering a high-performance, interoperable solution, mzPeak positions vendors to meet customer demands for sustainable data management tools which will be able to handle emerging and future data types and workflows. mzPeak aspires to become the cornerstone of MS data management, empowering researchers, vendors, and developers to innovate and collaborate more effectively.

data formats↗

Using magnetic tape technology for data migration

Magnetic tape and optical disk library units (jukeboxes) are satisfying the demand for high-capacity cost-effective storage. The choice between optical disk and magnetic tape technology must take into account the cost limitations as well as the performance and reliability requirements of the user environment. Library units require data management software in order to function in an automated and user-transparent way. The most common data management applications are backup and recovery, data migration, and archiving. The medium access patterns that these applications create will be described. Since the most user visible application is data migration, a queue simulator was developed to model its performance against a variety of library units. The major subject of this paper is the design and implementation of this simulator as well as some simulation results. The relative cost and reliability of magnetic tape versus optical disk library units is presented for completeness.

Therrien, David↗

Center for Plasma Edge Simulation (CPES) (Rutgers University Final Report)

The CPES scientific simulations run at scale on leadership class machines, collaborate at runtime and produce and exchange large data sizes, which present multiple I/O and data management challenges. During the CPES project, the Rutgers team worked with the rest of the CPES team to address these challenges at different levels, and specifically (1) at the data transport and communication level through the DART (Decoupled and Asynchronous Remote Data Transfers) framework, and (2) at the data management and services level through the DataSpaces and ActiveSpaces frameworks. These frameworks and their impact are briefly described.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Space Station needs, attributes and architectural options. Volume 2, book 2, part 2, Task 2: Information management system

Missions to be performed, station operations and functions to be carried out, and technologies anticipated during the time frame of the space station were examined in order to determine the scope of the overall information management system for the space station. This system comprises: (1) the data management system which includes onboard computer related hardware and software required to assume and exercise control of all activities performed on the station; (2) the communication system for both internal and external communications; and (3) the ground segment. Techniques used to examine the information system from a functional and performance point of view are described as well as the analyses performed to derive the architecture of both the onboard data management system and the system for internal and external communications. These architectures are then used to generate a conceptual design of the onboard elements in order to determine the physical parameters (size/weight/power) of the hardware and software. The ground segment elements are summarized.

Source record↗

NASA's Role in the North American Carbon Program

NASA's participation in the North American Carbon Program will include a significant effort for quantifying terrestrial and adjacent ocean sources and sinks of carbon using a variety of land, ocean, and atmospheric measurements. This effort will require improved remote sensing algorithms, new measurement technologies, refined model parameterizations of carbon cycle processes, advanced satellite data assimilation methodologies, and a data management strategy. NASA's plan for pursuing these activities will be presented. Examples of NASA data management capabilities such as the SeaWiFS on-line field data system will be discussed.

McClain, Charles R.↗

Uncertainty Analysis of Consequence Management (CM) Data Products.

The goal of this project is to develop and execute methods for characterizing uncertainty in data products that are deve loped and distributed by the DOE Consequence Management (CM) Program. A global approach to this problem is necessary because multiple sources of error and uncertainty from across the CM skill sets contribute to the ultimate p roduction of CM data products. This report presents the methods used to develop a probabilistic framework to characterize this uncertainty and provides results for an uncertainty analysis for a study scenario analyzed using this framework.

97 MATHEMATICS AND COMPUTING↗

Managing Sustainable Data Infrastructures: The Gestalt of EOSDIS

EOSDIS epitomizes a System of Systems, whose many varied and distributed parts are integrated into a single, highly functional organized science data system. A distributed architecture was adopted to ensure discipline-specific support for the science data, while also leveraging standards and establishing policies and tools to enable interdisciplinary research, and analysis across multiple scientific instruments. The EOSDIS is composed of system elements such as geographically distributed archive centers used to manage the stewardship of data. The infrastructure consists of underlying capabilities connections that enable the primary system elements to function together. For example, one key infrastructure component is the common metadata repository, which enables discovery of all data within the EOSDIS system. EOSDIS employs processes and standards to ensure partners can work together effectively, and provide coherent services to users.

remote sensing↗

Uncertainty Analysis of Consequence Management (CM) Data Products: Extended Analyses.

The goal of this project, started in FY17, is to develop and execute methods of characterizing uncertainty in data products that are developed and distributed by the DOE Consequence Management (CM) Program. This report presents the results of uncertainty analyses performed in FY18 for additional scenarios of increased complexity, including different time phases and radionuclide source terms.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

KOPE /Kalendar Oriented Program Efforts/ provides data for management decisions

KOPE /Kalendar Oriented Program Efforts/ is a computer program that establishes control over project efforts to assure management of meeting a specified completion date. With the appropriate input data, KOPE computes the starting and completion dates, the manning level for each activity, and the composite manning level for the program.

Karkainen, T. A.↗

Coupled Aerodynamic-Thermal-Structural (CATS) Analysis

Coupled Aerodynamic-Thermal-Structural (CATS) Analysis is a focused effort within the Numerical Propulsion System Simulation (NPSS) program to streamline multidisciplinary analysis of aeropropulsion components and assemblies. Multidisciplinary analysis of axial-flow compressor performance has been selected for the initial focus of this project. CATS will permit more accurate compressor system analysis by enabling users to include thermal and mechanical effects as an integral part of the aerodynamic analysis of the compressor primary flowpath. Thus, critical details, such as the variation of blade tip clearances and the deformation of the flowpath geometry, can be more accurately modeled and included in the aerodynamic analyses. The benefits of this coupled analysis capability are (1) performance and stall line predictions are improved by the inclusion of tip clearances and hot geometries, (2) design alternatives can be readily analyzed, and (3) higher fidelity analysis by researchers in various disciplines is possible. The goals for this project are a 10-percent improvement in stall margin predictions and a 2:1 speed-up in multidisciplinary analysis times. Working cooperatively with Pratt & Whitney, the Lewis CATS team defined the engineering processes and identified the software products necessary for streamlining these processes. The basic approach is to integrate the aerodynamic, thermal, and structural computational analyses by using data management and Non-Uniform Rational B-Splines (NURBS) based data mapping. Five software products have been defined for this task: (1) a primary flowpath data mapper, (2) a two-dimensional data mapper, (3) a database interface, (4) a blade structural pre- and post-processor, and (5) a computational fluid dynamics code for aerothermal analysis of the drum rotor. Thus far (1) a cooperative agreement has been established with Pratt & Whitney, (2) a Primary Flowpath Data Mapper has been prototyped and delivered to General Electric Aircraft Engines and Pratt & Whitney for evaluation, (3) a collaborative effort has been initiated with the National Institute of Standards and Testing to develop a Standard Data Access Interface, and (4) a blade tip clearance capability has been implemented into the Structural Airfoil Blade Engineering Routine (SABER) program. We plan to continue to develop the data mappers and data management tools. As progress is made, additional efforts will be made to apply these tools to propulsion system applications.

Source record↗

Lightweight Provenance Service for High-Performance Computing

Provenance describes detailed information about the history of a piece of data, containing the relationships among elements such as users, processes, jobs, and workflows that contribute to the existence of data. Provenance is key to supporting many data management functionalities that are increasingly important in operations such as identifying data sources, parameters, or assumptions behind a given result; auditing data usage; or understanding details about how inputs are transformed into outputs. Despite its importance, however, provenance support is largely underdeveloped in highly parallel architectures and systems. One major challenge is the demanding requirements of providing provenance service in situ. The need to remain lightweight and to be always on often conflicts with the need to be transparent and offer an accurate catalog of details regarding the applications and systems. To tackle this challenge, we introduce a lightweight provenance service, called LPS, for high-performance computing (HPC) systems. LPS leverages a kernel instrument mechanism to achieve transparency and introduces representative execution and flexible granularity to capture comprehensive provenance with controllable overhead. Extensive evaluations and use cases have confirmed its efficiency and usability. We believe that LPS can be integrated into current and future HPC systems to support a variety of data management needs.

Dai, Dong↗

Accelerating shared file checkpoint with local burst buffers

A data management system and method for accelerating shared file checkpointing. Written application data is aggregated in an application data file created in a local burst buffer memory at a compute node, and an associated data mapping built index to maintain information related to the offsets into a shared file at which segments of the application data is to be stored in a parallel file system, and where in the buffer those segments are located. The node asynchronously transfers a data file containing the application data and the associated data mapping index to a file server for shared file storage. The data management system and method further accelerates shared file checkpointing in which a shared file, together with a map file that specifies how the shared file is to be distributed, is asynchronously transferred to local burst buffer memories at the nodes to accelerate reading of the shared file.

Gooding, Thomas↗

GHRSST-14 DAS-TAG Report

The DAS-TAG provides the informatics and data management expertise in emerging information technologies for the GHRSST community. It provides expertise in data and metadata formats and standards, fosters improvements for GHRSST data curation, experiments with new data processing paradigms, and evaluates services and tools for data usage. It provides a forum for producer and distributor data management issues and coordination.

data processing↗

An approach for management of geometry data

The strategies for managing Integrated Programs for Aerospace Design (IPAD) computer-based geometry are described. The computer model of geometry is the basis for communication, manipulation, and analysis of shape information. IPAD's data base system makes this information available to all authorized departments in a company. A discussion of the data structures and algorithms required to support geometry in IPIP (IPAD's data base management system) is presented. Through the use of IPIP's data definition language, the structure of the geometry components is defined. The data manipulation language is the vehicle by which a user defines an instance of the geometry. The manipulation language also allows a user to edit, query, and manage the geometry. The selection of canonical forms is a very important part of the IPAD geometry. IPAD has a canonical form for each entity and provides transformations to alternate forms; in particular, IPAD will provide a transformation to the ANSI standard. The DBMS schemas required to support IPAD geometry are explained.

Dube, R. P.↗

Power System Waveform Datasets for Machine Learning

The desire for increased visibility across the electricity grid will necessarily increase the deployment of sensing and measurement devices and associated data management needs to unprecedented levels. For the existing sensing and measurement infrastructure, there remains a great amount of “value” yet to be extracted through advanced data management and analytics. Availability of more data will not, by itself, lead to changes in grid visibility, security, and resiliency. To create the predictive and prescriptive environment required to enable new markets and transactions for customer revenue and a reliable grid, the data must be collected, organized, evaluated, and analyzed using sophisticated algorithms to provide actionable information allowing operators and customers to reliably manage an increasingly complex grid. Progress in artificial intelligence (AI) has been largely driven by large, publicly available datasets that can be used to train AI algorithms such as MNIST, a database of handwritten images of digits, and ImageNet, an image database of everyday objects. These types of publicly available databases of real-world training datasets have been largely credited for advancement of image processing, computer vision, and deep learning algorithms that these use cases deploy. However, in the power systems industry to date, there are few databases with proper event labeling, and data access to a publicly available collection of power system event waveforms that will allow users to interact with grid signature data. Publicly available datasets of power system event waveforms, such as the DOE/EPRI dataset, often lack critical metadata or contain limited examples of each event type, and data formats vary widely across these datasets.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Seismology in the Cloud: A New Streaming Workflow

Data-intensive research in seismology is experiencing a recent boom, driven in part by large volumes of available data and advances in the growing field of data science. However, there are significant barriers to processing large data volumes, such as long retrieval times from data repositories, complex data management, and limited computational resources. New tools and platforms have reduced the barriers to entry for scientific cluster computing, including the maturation of the commercial cloud as an accessible instrument for research. Here, we build a customized research cluster in the cloud to test a new workflow for large-scale seismic analysis, in which data are processed as a stream (retrieved on-the-fly and acted upon without storing), with data from the Incorporated Research Institutions for Seismology Data Management Center. We use this workflow to deploy a spectral peak detection algorithm over 5.6 TB of compressed continuous seismic data from 2074 stations of the USArray Transportable Array EarthScope network. Using a 50-node cluster in the cloud, we completed the noise survey in 80 hr, with an average data throughput of 1.7 GB per minute. By varying cluster sizes, we find the scaling of our analysis to be sublinear, due to a combination of algorithmic limitations and data center response times. The cloud-based streaming workflow represents an order-of-magnitude increase in acquisition and processing speed compared to a traditional download-store-process workflow, and offers the additional benefits of employing a flexible, accessible, and widely used computing architecture. It is limited, however, due to its reliance on Internet transfer speeds and data center service capacity, and may not work well for repeated analyses or those for which even higher data throughputs are needed. These research applications will require a new class of cloud-native approaches in which both data and analysis are in the cloud.

58 GEOSCIENCES↗