Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36

Integrated System and Application Continuous Performance Monitoring and Analysis Capability (Final)

Scientific applications run on high-performance computing (HPC) systems are critical for many national security missions within Sandia and the NNSA complex. However, these applications often face performance degradation and even failures that are challenging to diagnose. To provide unprecedented insight into these issues, the HPC Development, HPC Systems, Computational Science, and Plasma Theory & Simulation departments at Sandia crafted and completed their FY21 ASC Level 2 milestone entitled "Integrated System and Application Continuous Performance Monitoring and Analysis Capability." The milestone created a novel integrated HPC system and application monitoring and analysis capability by extending Sandia’s Kokkos application portability framework, Lightweight Distributed Metric Service (LDMS) monitoring tool, and scalable storage, analysis, and visualization pipeline. The extensions to Kokkos and LDMS enable collection and storage of application data during run time, as it is generated, with negligible overhead. This data is combined with HPC system data within the extended analysis pipeline to present relevant visualizations of derived system and application metrics that can be viewed at run time or post run. This new capability was evaluated using several week-long, 290-node runs of Sandia’s ElectroMagnetic Plasma In Realistic Environments (EMPIRE) modeling and design tool and resulted in 1TB of application data and 50TB of system data. EMPIRE developers remarked this capability was incredibly helpful for quickly assessing application health and performance alongside system state. In short, this milestone work built the foundation for expansive HPC system and application data collection, storage, analysis, visualization, and feedback framework that will increase total scientific output of Sandia’s HPC users.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

The Ultra-scale Visualization Climate Data Analysis Tools (UV-CDAT): Data Analysis and Visualization for Geoscience Data

To support interactive visualization and analysis of complex, large-scale climate data sets, UV-CDAT integrates a powerful set of scientific computing libraries and applications to foster more efficient knowledge discovery. Connected through a provenance framework, the UV-CDAT components can be loosely coupled for fast integration or tightly coupled for greater functionality and communication with other components. This framework addresses many challenges in the interactive visual analysis of distributed large-scale data for the climate community.

97 MATHEMATICS AND COMPUTING↗

Big Data Analaysis and Visualization: What Do LINACS and Tropical Storms Have In Common?

While there is wisdom in the old adage "the two constants in life are death and taxes,'' there are unavoidable truths facing modern experimental and computational science. First is the growing "impedance mismatch'' between our ability to collect and generate data, and our ability to store, manage, and gain understanding from it. The second is the fact that we cannot continue to rely on the same software technologies that have worked well for the past couple of decades for data management, analysis, and visualization. A third is that these complementary activities must be considered in a holistic, rather than Balkanized way. The inseparable interplay between data management, analysis, visualization, and high performance computational infrastructure, are best viewed through the lens of case studies from multiple scientific domains, where teams of computer and scientists combine forces to tackle challenging data understanding problems.

97 MATHEMATICS AND COMPUTING↗

The effects of AVIRIS atmospheric calibration methodology on identification and quantitative mapping of surface mineralogy, Drum Mountains, Utah

The Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) measures reflected light in 224 contiguous spectra bands in the 0.4 to 2.45 micron region of the electromagnetic spectrum. Numerous studies have used these data for mineralogic identification and mapping based on the presence of diagnostic spectral features. Quantitative mapping requires conversion of the AVIRIS data to physical units (usually reflectance) so that analysis results can be compared and validated with field and laboratory measurements. This study evaluated two different AVIRIS calibration techniques to ground reflectance: an empirically-based method and an atmospheric model based method to determine their effects on quantitative scientific analyses. Expert system analysis and linear spectral unmixing were applied to both calibrated data sets to determine the effect of the calibration on the mineral identification and quantitative mapping results. Comparison of the image-map results and image reflectance spectra indicate that the model-based calibrated data can be used with automated mapping techniques to produce accurate maps showing the spatial distribution and abundance of surface mineralogy. This has positive implications for future operational mapping using AVIRIS or similar imaging spectrometer data sets without requiring a priori knowledge.

Kruse, Fred A.↗

The KTOI Ecosystem Project Relational Database : a Report Prepared by Statistical Consulting Services for KTOI Describing the Key Components and Specifications of the KTOI Relational Database.

Data are the central focus of any research project. Their collection and analysis are crucial to meeting project goals, testing scientific hypotheses, and drawing relevant conclusions. Typical research projects often devote the majority of their resources to the collection, storage and analysis of data. Therefore, issues related to data quality should be of foremost concern. Data quality issues are even more important when conducting multifaceted studies involving several teams of researchers. Without the use of a standardized protocol, for example, independent data collection carried out by separate research efforts can lead to inconsistencies, confusion and errors throughout the larger project. A database management system can be utilized to help avoid all of the aforementioned problems. The centralization of data into a common relational unit, i.e. a relational database, shifts the responsibility for data quality and maintenance from multiple individuals to a single database manager, thus allowing data quality issues to be assessed and corrected in a timely manner. The database system also provides an easy mechanism for standardizing data components, such as variable names and values uniformly across all segments of a project. This is particularly an important issue when data are collected on a number of biological/physical response and explanatory variables from various locations and times. The database system can integrate all segments of a large study into one unit, while providing oversight and accessibility to the data collection process. The quality of all data collected is uniformly maintained and compatibility between research efforts ensured. While the physical database would exist in a central location, access will not be physically limited. Advanced database interfaces are created to operate over the internet utilizing a Web-based relational database, allowing project members to access their data from virtually anywhere. These interfaces provide users with the ability to upload, download, edit, and search data remotely, creating a dynamic system that is continually updated with the most recent information. At the same time, data are protected through user access restrictions, by implementing user profiles and password protected security. This accessibility could be set to any combination of read/write/edit abilities from an administrator capacity with full access to all data, to a highly restricted public access capability limited to general project information. Generation of customized summary reports and basic graphical routines could also be obtained through a Web-based interference. Using these types of features, users could produce summary tables, track trends of specified response variables over time or location, and compare results from various disciplines. Exploration of data in this manner can help users to better define and clarify their research goals and provide a means of integrating various aspects of a larger research project.

13 HYDRO ENERGY↗

Visualization Quality Assessment

Understanding how inaccuracies in visualizations affect users’ perception and understanding of scientific data is hard. Inaccuracies in visualizations are quite common and could arise from a range of sources such as errors in the original dataset arising from compression artifacts, errors in the capturing device, noise during transmission of the data, effects due to the algorithm being used to convert data to visualization images, images generated from neural networks, and sources we have yet to discover. Many image quality assessment metrics have been developed to quantify image errors. However, these are usually focused on “natural images” rather than visualizations of scientific data. Common image quality assessment metrics (IQAs) include MSE, PSNR, perceptual metrics such SSIM, FSIM as well as perceptual metrics using deep learning approaches. However, a critical part of understanding how errors are perceived by humans, and subsequently developing more accurate quality assessment metrics, is through user evaluation studies. The goal of this software is to develop a visualization quality assessment (VQA) process that will enable the generation of VQAs that can be used to quantify errors in scientific data visualizations. The VQA development process will include software to support user evaluation experimental design, analysis of visualization differences against standard quality metrics, and the ability to develop additional VQA metrics specific to scientific visualization images.

Grosset, Andre↗

A Multi-Branch Decoder Network Approach to Adaptive Temporal Data Selection and Reconstruction for Big Scientific Simulation Data

A key challenge in scientific simulation is that the simulation outputs often require intensive I/O and storage space to store the results for effective post hoc analysis. This article focuses on a quality-aware adaptive temporal data selection and reconstruction problem where the goal is to adaptively select simulation data samples at certain key timesteps in situ and reconstruct the discarded samples with quality assurance during post hoc analysis. This problem is motivated by the limitation of current solutions that a significant amount of simulation data samples are either discarded or aggregated during the sampling process, leading to inaccurate modeling of the simulated phenomena. Two unique challenges exist: 1) the sampling decisions have to be made in situ and adapted to the dynamics of the complex scientific simulation data; 2) the reconstruction error must be strictly bounded to meet the application requirement. To address the above challenges, we develop DeepSample , an error-controlled convolutional neural network framework, that jointly integrates a set of coherent multi-branch deep decoders to effectively reconstruct the simulation data with rigorous quality assurance. The results on two real-world scientific simulation applications show that DeepSample significantly outperforms other state-of-the-art methods on both sampling efficiency and reconstructed simulation data quality.

Zhang, Yang↗

Analysis of TRMM Microphysical Measurements: Tropical Rainfall Measuring Mission (TRMM)

SPEC Incorporated participated in three of the four TRMM field campaigns (TEFLUN-A, TEFLUN-B and KWAJEX), installing and operating a cloud particle imager (CPI) and a high volume precipitation spectrometer (HVPS) on the SPEC Learjet in TEFLUN-A, the University of North Dakota Citation in TEFLUN-B and KWAJEX, and a CPI on the NASA DC-8 in KWAJEX. This report presents and discusses new software tools and algorithms that were developed to analyze microphysical data collected during these field campaigns, as well as scientific interpretations of the data themselves. Software algorithms were developed to improve the analysis of microphysical measurements collected by the TRMM aircraft during the field campaigns. Particular attention was paid to developing and/or improving algorithms used to compute particle size distributions and ice water content. Software was also developed in support of production of the TRMM Common Microphysical Product (CMP) data files. CMP data files for TEFLUN-A field campaign were produced and submitted to the DAAC. Typical microphysical properties of convective and stratiform regions from TEFLUN-A and KWAJEX clouds were produced. In general, it was found that in the upper cloud region near -20 to -25 C, stratiform clouds contain very high (greater than 1 per cubic centimeter) concentrations of small ice particles, which are suspected to be a residual from homogeneous freezing and sedimentation of small drops in a convective updraft. In the upper cloud region near -20 to -25 C, convective clouds contain aggregates, which are not found lower in the cloud. Stratiform clouds contain aggregates at all levels, with the majority in the lowest levels. Convective cloud regions contain much higher LWC and drop concentrations than stratiform regions at all levels, and higher LWC in the middle and upper regions. Stratiform clouds contain higher IWC than convective clouds only at the lowest level. Irregular shaped ice particles are found in very high concentrations throughout both convective and stratiform cloud regions. A striking difference in particle shape in cirrus formed in situ, cirrus formed from maritime anvils and cirrus formed from continental anvils. Over 50% of the mass of in situ cirrus ice particles is composed of bullet rosettes, while bullet rosettes are virtually non-existent in maritime and tropical anvils. Tropical anvils are composed of mostly singular, plates, capped columns, and blocky irregular shapes, while continental anvils have a much higher percentage of aggregates, some of which are chains of small spheroidal particles that appear to result from homogeneous freezing of drops. A correlation between high electric fields in continental anvils and the formation of aggregates is hypothesized.

Source record↗

Radiosurgery of Glomus Jugulare Tumors: A Meta-Analysis

Purpose: During the past two decades, radiosurgery has arisen as a promising approach to the management of glomus jugulare. In the present study, we report on a systematic review and meta-analysis of the available published data on the radiosurgical management of glomus jugulare tumors. Methods and Materials: To identify eligible studies, systematic searches of all glomus jugulare tumors treated with radiosurgery were conducted in major scientific publication databases. The data search yielded 19 studies, which were included in the meta-analysis. The data from 335 glomus jugulare patients were extracted. The fixed effects pooled proportions were calculated from the data when Cochrane's statistic was statistically insignificant and the inconsistency among studies was <25%. Bias was assessed using the Egger funnel plot test. Results: Across all studies, 97% of patients achieved tumor control, and 95% of patients achieved clinical control. Eight studies reported a mean or median follow-up time of >36 months. In these studies, 95% of patients achieved clinical control and 96% achieved tumor control. The gamma knife, linear accelerator, and CyberKnife technologies all exhibited high rates of tumor and clinical control. Conclusions: The present study reports the results of a meta-analysis for the radiosurgical management of glomus jugulare. Because of its high effectiveness, we suggest considering radiosurgery for the primary management of glomus jugulare tumors.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Kepler Science Operations Center Architecture

We give an overview of the operational concepts and architecture of the Kepler Science Data Pipeline. Designed, developed, operated, and maintained by the Science Operations Center (SOC) at NASA Ames Research Center, the Kepler Science Data Pipeline is central element of the Kepler Ground Data System. The SOC charter is to analyze stellar photometric data from the Kepler spacecraft and report results to the Kepler Science Office for further analysis. We describe how this is accomplished via the Kepler Science Data Pipeline, including the hardware infrastructure, scientific algorithms, and operational procedures. The SOC consists of an office at Ames Research Center, software development and operations departments, and a data center that hosts the computers required to perform data analysis. We discuss the high-performance, parallel computing software modules of the Kepler Science Data Pipeline that perform transit photometry, pixel-level calibration, systematic error-correction, attitude determination, stellar target management, and instrument characterization. We explain how data processing environments are divided to support operational processing and test needs. We explain the operational timelines for data processing and the data constructs that flow into the Kepler Science Data Pipeline.

Middour, Christopher↗

Workshop on Incomplete Network Data Held at Sandia National Labs – Livermore

While network analysis is applied in a broad variety of scientific fields (including physics, computer science, biology, and the social sciences), how networks are constructed and the resulting bias and incompleteness have drawn more limited attention. For example, in biology, gene networks are typically developed via experiment -- many actual interactions are likely yet to be discovered. In addition to this incompleteness, the data-collection processes can introduce significant bias into the observed network datasets. For instance, if you observe part of the World Wide Web network through a classic random walk, then high degree nodes are more likely to be found than if you had selected nodes at random. Unfortunately, such incomplete and biasing data collection methods must be often used.

97 MATHEMATICS AND COMPUTING↗

DOE Repository Metadata Profile (DRMP): A Metadata Framework for Advancing Interoperability and AI Readiness Across Scientific Repositories

The Department of Energy (DOE) funds a diverse and distributed ecosystem of repositories that steward scientific data, publications, and software across its research programs, user facilities, and national laboratories. While significant progress has been made in standardizing dataset-level metadata, the metadata describing repositories themselves (their identity, governance, access interfaces, policies, and technical capabilities) remains inconsistent and fragmented across DOE-funded systems. This variability limits discoverability, interoperability, automated validation, and AI-driven analysis, all of which are increasingly essential for modern scientific workflows. To address this gap, the DOE Data Curation Working Group (DCWG) developed the DOE Repository Metadata Profile (DRMP). The DRMP is a practical, community-driven framework that defines how repositories can describe themselves in a consistent, machine-actionable, and scalable manner. The DRMP is not a new metadata schema. Instead, it is a mapping profile and structured element set capturing the essential characteristics of DOE repositories. It harmonizes repository-level metadata across six widely adopted community schemas: RE3Data; DCAT-US v3; Schema.org; Dublin Core; DataCite 4.6; and PREMIS 3.0. This harmonization eliminates reinvention and enables interoperability within DOE and across the broader scientific ecosystem. A core objective of the DRMP is to reduce burden on repositories by allowing them to reuse their existing metadata through a Rosetta-style crosswalk rather than redesigning local implementations. The profile introduces a three-level conformance model that supports incremental adoption: • Level 1 – Minimum Viable Record (MVR): foundational identification elements required for workflows, project registration, and basic repository presence. • Level 2 – Interoperable: structured metadata enabling alignment with national and international discovery systems. • Level 3 – AI-Ready: enhanced provenance, policy transparency, fixity, semantic context, and capabilities that support automated reasoning, model training governance, and machine-assisted curation. To support implementation, the DRMP includes JSON Schema definitions, OpenAPI patterns, and MCP templates that allow repositories to publish machine-readable metadata directly within existing platforms. These resources are modular and lightweight, enabling adoption without major architectural change. Adopting the DRMP enables repositories to: • Enhance discoverability and interoperability by aligning identifiers, classifications, and descriptive elements across widely used schema standards. • Support federated discovery and cross-registration across DOE systems, Data.gov, and international catalogs. • Enable AI agents and workflow orchestration systems to interpret repository-level metadata within the American Science Cloud (AmSC) through Model Context Protocol (MCP)-based context publication. • Demonstrate alignment with DOE’s open science, stewardship, and FAIR data priorities. This guidance represents a community-driven step forward. Through voluntary adoption and continued feedback, the DRMP advances a cohesive, machine-actionable description of DOE repositories that supports FAIR data practices, preparing the infrastructure for AI-enabled research, and strengthening the discoverability and reuse of DOE’s scientific outputs.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Tigres Workflow Library: Supporting Scientific Pipelines on HPC Systems

The growth in scientific data volumes has resulted in the need for new tools that enable users to operate on and analyze data on large-scale resources. In the last decade, a number of scientific workflow tools have emerged. These tools often target distributed environments, and often need expert help to compose and execute the workflows. Data-intensive workflows are often ad-hoc, they involve an iterative development process that includes users composing and testing their workflows on desktops, and scaling up to larger systems. In this paper, we present the design and implementation of Tigres, a workflow library that supports the iterative workflow development cycle of data-intensive workflows. Tigres provides an application programming interface to a set of programming templates i.e., sequence, parallel, split, merge, that can be used to compose and execute computational and data pipelines. We discuss the results of our evaluation of scientific and synthetic workflows showing Tigres performs with minimal template overheads (mean of 13 seconds over all experiments). We also discuss various factors (e.g., I/O performance, execution mechanisms) that affect the performance of scientific workflows on HPC systems.

97 MATHEMATICS AND COMPUTING↗

Information Power Grid: Distributed High-Performance Computing and Large-Scale Data Management for Science and Engineering

We use the term "Grid" to refer to distributed, high performance computing and data handling infrastructure that incorporates geographically and organizationally dispersed, heterogeneous resources that are persistent and supported. This infrastructure includes: (1) Tools for constructing collaborative, application oriented Problem Solving Environments / Frameworks (the primary user interfaces for Grids); (2) Programming environments, tools, and services providing various approaches for building applications that use aggregated computing and storage resources, and federated data sources; (3) Comprehensive and consistent set of location independent tools and services for accessing and managing dynamic collections of widely distributed resources: heterogeneous computing systems, storage systems, real-time data sources and instruments, human collaborators, and communications systems; (4) Operational infrastructure including management tools for distributed systems and distributed resources, user services, accounting and auditing, strong and location independent user authentication and authorization, and overall system security services The vision for NASA's Information Power Grid - a computing and data Grid - is that it will provide significant new capabilities to scientists and engineers by facilitating routine construction of information based problem solving environments / frameworks. Such Grids will knit together widely distributed computing, data, instrument, and human resources into just-in-time systems that can address complex and large-scale computing and data analysis problems. Examples of these problems include: (1) Coupled, multidisciplinary simulations too large for single systems (e.g., multi-component NPSS turbomachine simulation); (2) Use of widely distributed, federated data archives (e.g., simultaneous access to metrological, topological, aircraft performance, and flight path scheduling databases supporting a National Air Space Simulation systems}; (3) Coupling large-scale computing and data systems to scientific and engineering instruments (e.g., realtime interaction with experiments through real-time data analysis and interpretation presented to the experimentalist in ways that allow direct interaction with the experiment (instead of just with instrument control); (5) Highly interactive, augmented reality and virtual reality remote collaborations (e.g., Ames / Boeing Remote Help Desk providing field maintenance use of coupled video and NDI to a remote, on-line airframe structures expert who uses this data to index into detailed design databases, and returns 3D internal aircraft geometry to the field); (5) Single computational problems too large for any single system (e.g. the rotocraft reference calculation). Grids also have the potential to provide pools of resources that could be called on in extraordinary / rapid response situations (such as disaster response) because they can provide common interfaces and access mechanisms, standardized management, and uniform user authentication and authorization, for large collections of distributed resources (whether or not they normally function in concert). IPG development and deployment is addressing requirements obtained by analyzing a number of different application areas, in particular from the NASA Aero-Space Technology Enterprise. This analysis has focussed primarily on two types of users: the scientist / design engineer whose primary interest is problem solving (e.g. determining wing aerodynamic characteristics in many different operating environments), and whose primary interface to IPG will be through various sorts of problem solving frameworks. The second type of user is the tool designer: the computational scientists who convert physics and mathematics into code that can simulate the physical world. These are the two primary users of IPG, and they have rather different requirements. The results of the analysis of the needs of these two types of users provides a broad set of requirements that gives rise to a general set of required capabilities. The IPG project is intended to address all of these requirements. In some cases the required computing technology exists, and in some cases it must be researched and developed. The project is using available technology to provide a prototype set of capabilities in a persistent distributed computing testbed. Beyond this, there are required capabilities that are not immediately available, and whose development spans the range from near-term engineering development (one to two years) to much longer term R&D (three to six years). Additional information is contained in the original.

Johnston, William E.↗

AutoDrug: fully automated macromolecular crystallography workflows for fragment-based drug discovery

New software has been developed for automating the experimental and data-processing stages of fragment-based drug discovery at a macromolecular crystallography beamline. A new workflow-automation framework orchestrates beamline-control and data-analysis software while organizing results from multiple samples. AutoDrug is software based upon the scientific workflow paradigm that integrates the Stanford Synchrotron Radiation Lightsource macromolecular crystallography beamlines and third-party processing software to automate the crystallography steps of the fragment-based drug-discovery process. AutoDrug screens a cassette of fragment-soaked crystals, selects crystals for data collection based on screening results and user-specified criteria and determines optimal data-collection strategies. It then collects and processes diffraction data, performs molecular replacement using provided models and detects electron density that is likely to arise from bound fragments. All processes are fully automated, i.e. are performed without user interaction or supervision. Samples can be screened in groups corresponding to particular proteins, crystal forms and/or soaking conditions. A single AutoDrug run is only limited by the capacity of the sample-storage dewar at the beamline: currently 288 samples. AutoDrug was developed in conjunction with RestFlow, a new scientific workflow-automation framework. RestFlow simplifies the design of AutoDrug by managing the flow of data and the organization of results and by orchestrating the execution of computational pipeline steps. It also simplifies the execution and interaction of third-party programs and the beamline-control system. Modeling AutoDrug as a scientific workflow enables multiple variants that meet the requirements of different user groups to be developed and supported. A workflow tailored to mimic the crystallography stages comprising the drug-discovery pipeline of CoCrystal Discovery Inc. has been deployed and successfully demonstrated. This workflow was run once on the same 96 samples that the group had examined manually and the workflow cycled successfully through all of the samples, collected data from the same samples that were selected manually and located the same peaks of unmodeled density in the resulting difference Fourier maps.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

Efficient Feature-Driven Visualization of Large-Scale Scientific Data

Very large, complex scientific data acquired in many research areas creates critical challenges for scientists to understand, analyze, and organize their data. The objective of this project is to expand the feature extraction and analysis capabilities to develop powerful and accurate visualization tools that can assist domain scientists with their requirements in multiple phases of scientific discovery. We have recently developed several feature-driven visualization methods for extracting different data characteristics of volumetric datasets. Our results verify the hypothesis in the proposal and will be used to develop additional prototype systems.

97 MATHEMATICS AND COMPUTING↗

An Integrative Computational Framework for Hypotheses-Driven Systems Biology Research in Proteomics and Genomics

Systems biology research is sometimes categorized as either discovery science or hypothesis-driven science. However, we believe that hypotheses are always used regardless, and that explicit recognition that hypothesis testing underlies all high-throughput data analysis leads to better experimental designs, data analysis and interpretation of the data. We outline the current use of hypothesis testing for proteomics data analysis in systems biology research for several projects at the Pacific Northwest National Laboratory, and provide examples of where scientific principles can be used to formulate the hypotheses used to analyze the data. We additionally discuss the data infrastructure is required to (1) track the data from different projects and diverse assays, (2) pull the data together in a congruent manner, (3) analyze the data with respect to cellular networks, and (4) visualize the resulting networks and contrast those with information from bioinformatics databases.

59 BASIC BIOLOGICAL SCIENCES↗