Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27

NASA's Earth Observing Data and Information System

NASA's Earth Observing System Data and Information System (EOSDIS) has been a central component of NASA Earth observation program for over 10 years. It is one of the largest civilian science information system in the US, performing ingest, archive and distribution of over 3 terabytes of data per day much of which is from NASA s flagship missions Terra, Aqua and Aura. The system supports a variety of science disciplines including polar processes, land cover change, radiation budget, and most especially global climate change. The EOSDIS data centers, collocated with centers of science discipline expertise, archive and distribute standard data products produced by science investigator-led processing systems. Key to the success of EOSDIS is the concept of core versus community requirements. EOSDIS supports a core set of services to meet specific NASA needs and relies on community-developed services to meet specific user needs. EOSDIS offers a metadata registry, ECHO (Earth Observing System Clearinghouse), through which the scientific community can easily discover and exchange NASA s Earth science data and services. Users can search, manage, and access the contents of ECHO s registries (data and services) through user-developed and community-tailored interfaces or clients. The ECHO framework has become the primary access point for cross-Data Center search-and-order of EOSDIS and other Earth Science data holdings archived at the EOSDIS data centers. ECHO s Warehouse Inventory Search Tool (WIST) is the primary web-based client for discovering and ordering cross-discipline data from the EOSDIS data centers. The architecture of the EOSDIS provides a platform for the publication, discovery, understanding and access to NASA s Earth Observation resources and allows for easy integration of new datasets. The EOSDIS also has developed several methods for incorporating socioeconomic data into its data collection. Over the years, we have developed several methods for determining needs of the user community including use of the American Customer Satisfaction Index and a broad metrics program.

Mitchell, Andrew E.↗

Advanced I/O for large-scale scientific applications.

As scientific simulations scale to use petascale machines and beyond, the data volumes generated pose a dual problem. First, with increasing machine sizes, the careful tuning of IO routines becomes more and more important to keep the time spent in IO acceptable. It is not uncommon, for instance, to have 20% of an application's runtime spent performing IO in a 'tuned' system. Careful management of the IO routines can move that to 5% or even less in some cases. Second, the data volumes are so large, on the order of 10s to 100s of TB, that trying to discover the scientifically valid contributions requires assistance at runtime to both organize and annotate the data. Waiting for offline processing is not feasible due both to the impact on the IO system and the time required. To reduce this load and improve the ability of scientists to use the large amounts of data being produced, new techniques for data management are required. First, there is a need for techniques for efficient movement of data from the compute space to storage. These techniques should understand the underlying system infrastructure and adapt to changing system conditions. Technologies include aggregation networks, data staging nodes for a closer parity to the IO subsystem, and autonomic IO routines that can detect system bottlenecks and choose different approaches, such as splitting the output into multiple targets, staggering output processes. Such methods must be end-to-end, meaning that even with properly managed asynchronous techniques, it is still essential to properly manage the later synchronous interaction with the storage system to maintain acceptable performance. Second, for the data being generated, annotations and other metadata must be incorporated to help the scientist understand output data for the simulation run as a whole, to select data and data features without concern for what files or other storage technologies were employed. All of these features should be attained while maintaining a simple deployment for the science code and eliminating the need for allocation of additional computational resources.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Global Sensitivity Measures from Given Data

Simulation models support managers in the solution of complex problems. International agencies recommend uncertainty and global sensitivity methods as best practice in the audit, validation and application of scientific codes. However, numerical complexity, especially in the presence of a high number of factors, induces analysts to employ less informative but numerically cheaper methods. This work introduces a design for estimating global sensitivity indices from given data (including simulation input–output data), at the minimum computational cost. We address the problem starting with a statistic based on the L1-norm. A formal definition of the estimators is provided and corresponding consistency theorems are proved. The determination of confidence intervals through a bias-reducing bootstrap estimator is investigated. The strategy is applied in the identification of the key drivers of uncertainty for the complex computer code developed at the National Aeronautics and Space Administration (NASA) assessing the risk of lunar space missions. We also introduce a symmetry result that enables the estimation of global sensitivity measures to datasets produced outside a conventional input–output functional framework.

99 GENERAL AND MISCELLANEOUS↗

NASA Giovanni Tool for Visualization and Analysis Support for the YOTC Program

The fundamental challenges to overcoming our shortcomings in understanding and modeling/predicting tropical convection have been two folds: I) the need to represent the broad range of scales applicable to the tropical organization problem (i.e. cumulus to planetary), and II) the lack of observations that adequately and simultaneously characterize this broad range of scales and that also provide three-dimensional information on thermodynamic, radiative and dynamical interactions, including cloud microphysical processes. Because the goal of Overview of Year Of Tropical Convection (YOTC) involves examining a scientifically complex, multi-scale "process", rather than documenting the characteristics of a single parameter (e.g., SST, cloud cover), YOTC has an IOP perspective that targets a period, May 2008 - April 2010, long enough to encompass many cases of tropical convection activity in many of its most challenging yet influential forms. This includes mesoscale and synoptic variability, easterly waves and hurricanes, convectively coupled waves, the MJO, and the culmination of these in terms of the monsoon, their interactions with the extra-tropics, and mean characteristics such as tropical-to-subtropical transitions. The YOTC time period and length are driven in part by the following: 1) keeping the multi-sensor/multi-platform and model-analyses data sets and associated infrastructure manageable, 2) facilitating a focused effort by the research and operational communities on a specific scientific problem, and 3) capitalizing on the recent key additions to the armada of satellites (e.g., CloudSat and CALIPSO).

Ostrenga, Dana↗

Scheduling Multilevel Deadline-Constrained Scientific Workflows on Clouds Based on Cost Optimization

This paper presents a cost optimization model for scheduling scientific workflows on IaaS clouds such as Amazon EC2 or RackSpace. We assume multiple IaaS clouds with heterogeneous virtual machine instances, with limited number of instances per cloud and hourly billing. Input and output data are stored on a cloud object store such as Amazon S3. Applications are scientific workflows modeled as DAGs as in the Pegasus Workflow Management System. We assume that tasks in the workflows are grouped into levels of identical tasks. Our model is specified using mathematical programming languages (AMPL and CMPL) and allows us to minimize the cost of workflow execution under deadline constraints. We present results obtained using our model and the benchmark workflows representing real scientific applications in a variety of domains. The data used for evaluation come from the synthetic workflows and from general purpose cloud benchmarks, as well as from the data measured in our own experiments with Montage, an astronomical application, executed on Amazon EC2 cloud. We indicate how this model can be used for scenarios that require resource planning for scientific workflows and their ensembles.

Malawski, Maciej↗

A2SD: Accelerating Scientific Innovation Through Autonomous Discovery Systems

The 2025 Advancing Autonomous Scientific Discovery (A2SD) workshop convened researchers from academia, national laboratories, and industry to explore the transformative role of autonomy in scientific discovery. The workshop highlighted a convergence of artificial intelligence, robotics, and computational workflows into autonomous systems capable of accelerating the scientific process. Presentations and discussions spanned autonomous experimentation, intelligent workflow orchestration, digital twins, and agent-based systems for managing complex research ecosystems. Key challenges discussed included interoperability across heterogeneous infrastructures, near real-time data management under FAIR principles, reproducibility, and the integration of human oversight. The workshop also emphasized the need for modular software interfaces, federated learning models, and education initiatives to support a next-generation scientific workforce.

Taufer, Michela [University of Tennessee, Knoxvill↗

Data as a Key Resource in Catalysis: A Community Account

The deployment of artificial intelligence (AI) is transforming the scientific fields central to interdisciplinary catalysis research. By enabling more effective use of data, AI (including simpler machine learning and data science tools) holds great promise for accelerating discoveries. However, progress has so far been modest, largely due to the lack of standardized, machine-readable, and openly shared catalysis data. This perspective, accounting for community insights emerging at conferences, analyses the underlying reasons for these challenges and proposes solutions to a future whereFAIR data management becomes an integral part of research in catalysis. In the short-term, we deem that mandatory FAIR data depositing prior to scientific publications along with consensualized top-down guidelines on data sharing powered by ease-to-use tools can make the necessary step change happen to catalyse data as key resource in our community.

36 - MATERIALS SCIENCE↗

Performance Understanding and Analysis for Exascale Data Management Workflows (Collaboration)

The general approach of the MONA project is depicted in Figure 1. The figure shows performance monitoring applied to an I/O workflow associated with a scientific simulation, with online measurements captured from this workflow informing methods for workflow reconfiguration and dynamic adjustment. The goal is to maintain suitable levels of Quality of Service for workflow execution, by understanding the underlying causes of workflow performance and behavior. One outcome will be workflow performance models able to characterize realistic workflows. Another outcome will be performance ‘mini-apps’ implementing such workflow behavior. Out of scope for this project are advanced methods for online workflow control.

97 MATHEMATICS AND COMPUTING↗

Framework for Integrating Science Data Processing Algorithms Into Process Control Systems

A software framework called PCS Task Wrapper is responsible for standardizing the setup, process initiation, execution, and file management tasks surrounding the execution of science data algorithms, which are referred to by NASA as Product Generation Executives (PGEs). PGEs codify a scientific algorithm, some step in the overall scientific process involved in a mission science workflow. The PCS Task Wrapper provides a stable operating environment to the underlying PGE during its execution lifecycle. If the PGE requires a file, or metadata regarding the file, the PCS Task Wrapper is responsible for delivering that information to the PGE in a manner that meets its requirements. If the PGE requires knowledge of upstream or downstream PGEs in a sequence of executions, that information is also made available. Finally, if information regarding disk space, or node information such as CPU availability, etc., is required, the PCS Task Wrapper provides this information to the underlying PGE. After this information is collected, the PGE is executed, and its output Product file and Metadata generation is managed via the PCS Task Wrapper framework. The innovation is responsible for marshalling output Products and Metadata back to a PCS File Management component for use in downstream data processing and pedigree. In support of this, the PCS Task Wrapper leverages the PCS Crawler Framework to ingest (during pipeline processing) the output Product files and Metadata produced by the PGE. The architectural components of the PCS Task Wrapper framework include PGE Task Instance, PGE Config File Builder, Config File Property Adder, Science PGE Config File Writer, and PCS Met file Writer. This innovative framework is really the unifying bridge between the execution of a step in the overall processing pipeline, and the available PCS component services as well as the information that they collectively manage.

Mattmann, Chris A.↗

Integrating science for water security governance

Hydrological extremes are intensifying globally, increasing the complexity of decisions required to ensure water security. Advances in hydrological science, modeling, and data systems have expanded the technical frontier of water research, yet uptake of scientific insights in policy and management decisions remains limited. This persistent science–policy gap is not primarily a failure of knowledge generation or robustness, but an institutional challenge shaped by how scientific and governance systems are organized, coordinated, and connected to support the effective use of scientific knowledge. These challenges are particularly pronounced in multi-level and transboundary water governance, where decisions span jurisdictions and require coordination across institutional and political boundaries. We synthesize research at the science–policy interface and evidence from water security initiatives to show how institutional arrangements, scientific tool development, and research practices enable or constrain the sustained use of scientific knowledge in water-security governance processes. Building on these insights, we develop ‘shared decision infrastructure’ as a framing to describe how scientific knowledge is embedded within the institutional, relational, and procedural arrangements that connect science to decision-making processes over time. We translate this framing into a practical intervention roadmap centered on institutional design, tool translation, sustained co-production, and outcome-oriented evaluation to support the integration of science into ongoing governance processes. By positioning science as shared decision infrastructure, the roadmap clarifies how researchers can design scientific efforts that support more coordinated, accountable, and adaptive water security decisions amid deepening uncertainty.

M whitney, Kristen [NASA Goddard Space Flight Cent↗

MSFC Skylab corollary experiments

The evolution of the development and integration of Skylab experiments from initial concepts through mission operations is documented. All experiment systems are covered as well as management controls which were developed and exercised to assure acceptable operational capability and optimize data acquisition for final scientific results.

Source record↗

High rate science data handling on Space Station Freedom

A study by NASA's User Information System Working Group for Space Station Freedom (SSF) has determined that the proposed onboard Data Management System, as initially configured, will be incapable of handling the data-generation rates typical of numerous scientific sensor payloads; many of these generate data at rates in excess of 10 Mbps, and there are at least four cases of rates in excess of 300 Mbps. The SSF Working Group has accordingly suggested an alternative conceptual architecture based on technology expected to achieve space-qualified status by 1995. The architecture encompasses recorders with rapid data-ingest capabilities and massive storage capabilities, optical delay lines allowing the recording of only the phenomena of interest, and data flow-compressing image processors.

Handley, Thomas H., Jr.↗

Balloon Support Systems Performance for the Cosmic Rays Energetics and Mass Mission

The Ballooncraft Support Systems were developed by NASA Wallops Flight Facility for use on ULDB class balloon missions. The support systems have now flown two missions supporting the Cosmic Rays Energetics and Mass (CREAM) experiment. The first, CREAM I, flown in December 2004, was for a record breaking 41 days, 21 hours, and the second, flown in December 2005, was for 28 days, 9 hours. These support systems provide CREAM with power, telecommunications, command and data handling ioc!uding Plight computers, mechanical structures, thermal management and attitude control to help ensure a successful scientific mission. This paper will address the performance and success of these support systems over the two missions.

Tompson, Linda D.↗

The Orbiting Carbon Observatory (OCO)

CO2 is the principal human generated driver of climate change. Accurate forecasting of future climate requires an improved understanding of the global carbon cycle and its interaction with the climate system. The Orbiting Carbon Observatory (OCO) will make global, space-based observations of atmospheric CO2 with the precision, resolution, and coverage needed to understand sources and sinks. OCO data will provide critical information for decision makers including the scientific basis for policy formulation, guide for carbon management strategies and treaty monitoring.

OCO↗

Impacts of a Changing Climate and Land Use on Reindeer Pastoralism: Indigenous Knowledge and Remote Sensing

The Arctic is home to many indigenous peoples, including those who depend on reindeer herding for their livelihood, in one of the harshest environments in the world. For the largely nomadic peoples, reindeer not only form a substantial part of the Arctic food base and economy, but they are also culturally important, shaping their way of life, mythologies, festivals and ceremonies. Reindeer pastoralism or husbandry has been practiced by numerous peoples all across Eurasia for thousands of years and involves moving herds of reindeer, which are very docile animals, from pasture to pasture depending on the season. Thus, herders must adapt on a daily basis to find optimal conditions for their herds according to the constantly changing conditions. Climate change and variability plus rapid development are increasingly creating major changes in the physical environment, ecology, and cultures of these indigenous reindeer herder communities in the North, and climate changes are occurring significantly faster in the Arctic than the rest of the globe, with correspondingly dramatic impacts (Oskal, 2008). In response to these changes, Eurasian reindeer herders have created the EALAT project, a comprehensive new initiative to study these impacts and to develop local adaptation strategies based upon their traditional knowledge of the land and its uses - in targeted partnership with the science and remote sensing community - involving extensive collaborations and coproduction of knowledge to minimize the impacts of the various changes. This chapter provides background on climate and development challenges to reindeer husbandry across the Arctic and an overview of the EALAT initiative, with an emphasis on indigenous knowledge, remote sensing, Geographic Information Systems (GIS), and other scientific data to 'co-produce' datasets for use by herders for improved decision-making and herd management. It also provides a description of the EALAT monitoring data integration and sharing system and portal being developed for reindeer pastoralism. In addition, the chapter provides some preliminary results from the EALAT Project, including some early remote sensing research results.

Maynard, N. G.↗

Remotely Sensed High‐Resolution Soil Moisture and Evapotranspiration: Bridging the Gap Between Science and Society

This paper reviews the current state of high‐resolution remotely sensed soil moisture (SM) and evapotranspiration (ET) products and modeling, and the coupling relationship between SM and ET. SM downscaling approaches for satellite passive microwave products leverage advances in artificial intelligence and high‐resolution remote sensing using visible, near‐infrared, thermal‐infrared, and synthetic aperture radar sensors. Remotely sensed ET continues to advance in spatiotemporal resolutions from MODIS to ECOSTRESS to Hydrosat and beyond. These advances enable a new understanding of bio‐geo‐physical controls and coupled feedback mechanisms between SM and ET reflecting the land cover and land use at field scale (3–30 m, daily). Still, the state‐of‐the‐science products have their challenges and limitations, which we detail across data, retrieval algorithms, and applications. We describe the roles of these data in advancing 10 application areas: drought assessment, food security, precision agriculture, soil salinization, wildfire modeling, dust monitoring, flood forecasting, urban water, energy, and ecosystem management, ecohydrology, and biodiversity conservation. We discuss that future scientific advancement should focus on developing open‐access, high‐resolution (3–30 m), sub‐daily SM and ET products, enabling the evaluation of hydrological processes at finer scales and revolutionizing the societal applications in data‐limited regions of the world, especially the Global South for socio‐economic development.

54 ENVIRONMENTAL SCIENCES↗

HPDR: High-Performance Portable Scientific Data Reduction Framework

The rapid growth in scientific data generation is outpacing advancements in computing systems necessary for efficient storage, transfer, and analysis, particularly in the context of exascale computing. With the deployment of first-generation exascale computing systems and next-generation experimental facilities, this gap is widening and necessitates effective data reduction techniques to manage enormous data volumes. Over the past decade, various data reduction methods, including lossless compression, error-controlled lossy compression, and data refactoring, have been developed to accelerate I/O in scientific workflows. Despite significant reductions in data volume, these methods introduce considerable computational overhead, which can become the new bottleneck in data processing. To mitigate this, GPU-accelerated data reduction algorithms have been introduced. However, challenges remain in their integration into exascale workflows, including limited portability across different GPU architectures, substantial memory transfer overhead, and reduced scalability on dense multi-GPU systems. To address these challenges, we propose HPDR, a high-performance and portable data reduction framework. HPDR is designed to enable the execution of state-of-the-art reduction algorithms across diverse processor architectures while reducing memory transfer overhead to 2.3 % of the original, resulting in up to 3.5× faster throughput compared to existing solutions. It also achieves up to 96% of the theoretical speedup in multi-GPU settings. In addition, evaluations on accelerating I/O operations at scale up to 1,024 nodes of the Frontier supercomputer demonstrate that HPDR can achieve up to 103 TB/s reduction throughput, providing up to 4× acceleration in parallel I/O performance compared to existing data reduction routines. This work highlights the potential of HPDR to significantly enhance data reduction efficiency in exascale computing environments.

Chen, Jieyang [University of Oregon]↗

ARM Cloud and Precipitation Measurements and Science Group Charter

The Atmospheric Radiation Measurement (ARM) Cloud and Precipitation Measurements and Science Group (CPMSG) brings together members of the ARM instrument operations, engineering, and translator teams with the ARM science community for the purpose of improving the performance and science impact of the ARM measurements of clouds and precipitation. The members of the CPMSG work toward this goal by confronting science needs from the broader research community with operational constraints of the ARM facility. The CPMSG replaces the former radar science and engineering group. The CPMSG will continue to include a significant focus on the application of the ARM radar network; however, the focus of ARM is ultimately not on specific instruments but on providing measurements and associated data products that advance atmospheric science. Providing measurements of cloud properties has always been a core activity for ARM, but increasingly, ARM has striven to couple measurements of cloud properties with precipitation to support the full range of research associated with cloud systems. Radar measurements are often central to cloud and precipitation research but often the retrieval of cloud and precipitation properties requires coupling radar measurements with measurements from other instruments or other instruments may provide information in physical regimes where radars do not do well. Therefore, the CPMSG will address issues related to radar performance and applications but will also consider issues related to other passive and active remote sensors as well as ground-based in situ precipitation measurements. A driving consideration for the group should be how resources can best be applied to measurements of cloud and precipitation properties and the development of associated data products to increase the scientific impact of these measurements.

Charter, Infrastructure Management Board, User Exe↗