SEARCH · Engineering Papers

Results for “Scientific Data Analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

All sources NASA NTRS DOE OSTI

NASA NTRS1991

High altitude aircraft direction using real-time scientific analysis of telemetered data

Microscale and some mesoscale scientific targets are often too small and fast moving in a dynamic atmosphere. In this environment the ER-2 may spend only a small time sampling the phenomena of interest. The ER-2 scientists currently rely on post-flight analysis to determine the success of the mission and to provide input for planning the next flight. Real time analysis, combined with realistic modeling, can provide important clues to significantly enhance the probability of success in the investigation of specific scientific targets.

Wegener, Steven↗

DOE OSTI2021

Hierarchical Analysis of Halo Center in Cosmology

Ever-increasing data size raises many challenges for scientific data analysis. Particularly in cosmological N-body simulation, finding the center of a dark matter halo suffers heavily from the large computational cost associated with the large number of particles (up to 20 million). In this work, we exploit the latent structure embed in a halo, and we propose a hierarchical approach to approximate the exact gravitational potential calculation for each particle in order to more efficiently find the halo center. Tests of our method on data from N-body simulations show that in many cases the hierarchical algorithm performs significantly faster than existing methods with a desirable accuracy.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1974

Apollo data analysis experiment S-211. Low brightness image data analysis

The scientific analyses of photographic data in astronomy obtained by the Apollo Orbital Science Photographic Team during Apollo flights 14 through 17 are described. Some of the tasks are the collection and correlation of Quick-Look listings, working prints for further study, the generation of post-mission reports, and project reports. Postflight calibrations of the flight photographic systems are carried out, and lens vignetting effects are corrected. The data handling is described and includes photodigitization of photographic scenes, calibrated step-wedges, and vignetting patterns, which are followed by computer program processing of the film density readings.

Mercer, R. D.↗

NASA NTRS1993

Dynamics Explorer 2: Continued FPI and NACS instrument data analysis and associated scientific activity at the University of Michigan

The grant entitled 'Dynamics Explorer 2 - continued FPI and NACS instrument data analysis and associated scientific activity at the University of Michigan' is a continuation of a grant that began with instrument development for the Dynamics Explorer 2 (DE 2) satellite. Over the years, many publications and presentations at scientific meetings have occurred under the aegis of this grant. This present report details the progress that has been made in the final three years of the grant. In these last 4 years of the grant 26 papers have been published or are in press and about 10 more are in preparation or have been submitted. A large number of presentations have been made in the same time span: 36 are listed in Appendix 2. Evidence of the high educational utility of this research is indicated by the list of Ph. D. and M. S. theses that have been completed in the last 3 years that have involved work connected with NAG5-465. The structure of this report is as follows: a brief synopsis of the aims of the grant NAG5-465 is given in the next section; then there is a summary of the scientific accomplishments that have occurred over the grant period; last, we make some brief concluding remarks. Reprints of articles that have recently appeared in refereed journals are appended to the end of this document.

Burns, Alan↗

DOE OSTI2025

Eureka: Enabling Fine-Grained Access and Range Queries on Compressed Scientific Data via Data-Index Co-Compression

Handling large-scale scientific data in high-performance computing (HPC) environments poses significant challenges, including excessive I/O, high storage costs, and slow query performance. Traditional approaches often require full data decompression and scans, making them impractical for real-time or interactive analysis. To address these limitations, we introduce Eureka, a unified data-index co-compression framework that enables fine-grained access and efficient range queries on compressed scientific datasets. Eureka integrates spatial domain decomposition with block-wise error-bounded lossy compression to support selective decompression. It constructs a hierarchical AVL-tree index during compression to capture block-level value ranges, enabling fast pruning during query execution. To reduce metadata overhead, the index itself is also compressed while ensuring recall-preserving results. Experiments on six diverse HPC simulation datasets show that Eureka achieves up to 25x data compression and over 300x index compression, surpassing state-of-the-art compressors such as SZ3 and ZFP in rate-distortion performance. Additionally, Eureka delivers over 30x speedup for low-selectivity range queries, making it a scalable and efficient solution for modern scientific data analysis.

Yan, Ning↗

NASA NTRS1985

International Ultraviolet Explorer Observatory operations

This volume contains the final report for the International Ultraviolet Explorer IUE Observatory Operations contract. The fundamental operational objective of the International Ultraviolet Explorer (IUE) program is to translate competitively selected observing programs into IUE observations, to reduce these observations into meaningful scientific data, and then to present these data to the Guest Observer in a form amenable to the pursuit of scientific research. The IUE Observatory is the key to this objective since it is the central control and support facility for all science operations functions within the IUE Project. In carrying out the operation of this facility, a number of complex functions were provided beginning with telescope scheduling and operation, proceeding to data processing, and ending with data distribution and scientific data analysis. In support of these critical-path functions, a number of other significant activities were also provided, including scientific instrument calibration, systems analysis, and software support. Routine activities have been summarized briefly whenever possible.

Source record↗

DOE OSTI2019

A Software Defined Network Design for Analyzing Streaming Data in Transit

Year after year, network traffic keeps reaching newhighs as unprecedented volumes of data flow between an ever-increasing number of sources and destinations. As part of the Analysis on the Wire project, we aim to develop a network-centric approach for streaming data processing to facilitate scientific data analysis and reduce the overhead in sending big data to a data center. This work discusses our design for programmable network devices to augment network capabilities for streaming data processing, including efforts in progress at Brookhaven National Laboratory and the challenges faced to date.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

A Strategy for NACS investment in Machine Learning

The Nuclear and Chemical Sciences (NACS) Division furnishes the expertise in the scientific areas of chemical, nuclear and isotopic sciences that are foundational in the Laboratory’s national security missions. This expertise is maintained and advanced through identification, development and application of state-of-the-art theoretical, computational and experimental methods and tools. Recent developments in artificial intelligence and machine learning (AI/ML) techniques enabled by advances in computing capabilities and widespread availability of powerful software implementations have made use of these techniques ubiquitous across both science and industry. While the scope of AI/ML applications is incredibly large and evolves very rapidly, the topics most relevant to NACS missions fall into the general category of detecting, categorizing or identifying features in large, complex datasets using either supervised or unsupervised learning. This covers both basic scientific data analysis and the development of efficient surrogate models of real-life technological systems, experimental detectors, or theoretical models. To remain at the forefront of its core scientific disciplines, NACS must both cultivate ML expertise as well as continuously explore applying this expertise to new problems or utilizing new methods. This document identifies the key areas where this support is critical and provides a strategy for investing in them.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗

NASA NTRS1992

PCIPS 2.0: Powerful multiprofile image processing implemented on PCs

Over the years, the processing power of personal computers has steadily increased. Now, 386- and 486-based PC's are fast enough for many image processing applications, and inexpensive enough even for amateur astronomers. PCIPS is an image processing system based on these platforms that was designed to satisfy a broad range of data analysis needs, while requiring minimum hardware and providing maximum expandability. It will run (albeit at a slow pace) even on a 80286 with 640K memory, but will take full advantage of bigger memory and faster CPU's. Because the actual image processing is performed by external modules, the system can be easily upgraded by the user for all sorts of scientific data analysis. PCIPS supports large format lD and 2D images in any numeric type from 8-bit integer to 64-bit floating point. The images can be displayed, overlaid, printed and any part of the data examined via an intuitive graphical user interface that employs buttons, pop-up menus, and a mouse. PCIPS automatically converts images between different types and sizes to satisfy the requirements of various applications. PCIPS features an API that lets users develop custom applications in C or FORTRAN. While doing so, a programmer can concentrate on the actual data processing, because PCIPS assumes responsibility for accessing images and interacting with the user. This also ensures that all applications, even custom ones, have a consistent and user-friendly interface. The API is compatible with factory programming, a metaphor for constructing image processing procedures that will be implemented in future versions of the system. Several application packages were created under PCIPS. The basic package includes elementary arithmetics and statistics, geometric transformations and import/export in various formats (FITS, binary, ASCII, and GIF). The CCD processing package and the spectral analysis package were successfully used to reduce spectra from the Nordic Telescope at La Palma. A photometry package is also available, and other packages are being developed. A multitasking version of PCIPS that utilizes the factory programming concept is currently under development. This version will remain compatible (on the source code level) with existing application packages and custom applications.

Smirnov, O. M.↗

DOE OSTI2014

A Scientific Data Processing Framework for Time Series NetCDF Data

ARM Data Integrator (ADI) is a framework to streamline the development of scientific algorithms that analyze time-series NetCDF data, and to improve the content and consistency of the output data products produced by these algorithms. ADI achieves these goals by automating the process of retrieving and preparing data for analysis, supporting the definition of output data products through a graphical interface, and providing a modular, flexible software development architecture. The input data, preprocessing, and output data specifications are defined through a graphical interface and stored in a database. ADI also includes a workflow for data integration, a library of software modules to support the workflow, and a source code generator that produces C, IDL and Python templates. Data preparation support includes automated retrieval of data from input files, merging the retrieved data into appropriately sized chunks, and transformation of the data onto a common coordinate system grid. Through the graphical interface, users can view the details of both their data products and those in the ARM catalog. The variable and attribute definitions of the existing data products can be used to build new output data products. In addition, the rules that make up the ARM archive’s data standards are laid on top of the view of the new data product providing the user with a visual cue indicating where their output violates an archive standard. The necessary configurations are stored in a database that is accessed by the ADI libraries. This paper discusses the ADI framework, its supporting components, and how ADI can significantly decrease the time and cost of implementing scientific algorithms while improving the ability of scientists to disseminate their results.

Atmospheric Science↗

NASA NTRS2002

The Intelligent Data Understanding Element of NASA's Intelligent Systems Program

Within the NASA Intelligent Systems Program, the Intelligent Data Understanding (IDU) element develops techniques for transforming data into scientific understanding. Automating such tools is critical for space science, space-based earth science, and planetary exploration with onboard scientific data analysis. Intelligent data understanding (IDU) is about extracting meaning from large, diverse science and engineering databases, via autonomous techniques that transform very large datasets into understanding. The earth science community in particular needs new tools for analyzing multi-formatted and geographically distributed datasets and for identifying cause-effect relationships in the complex data. Research within the IDU program element seeks to automate data analysis tasks so that humans can focus on creative hypothesis generation and knowledge synthesis. It may also enable NASA space missions in which autonomous agents must generate knowledge and take actions, and missions where limited bandwidth permits transmission of only the most interesting scientific observations, summaries, and conclusions. Twenty-seven research projects are-currently funded.

Coughlan, Joseph C.↗

DOE OSTI2024

Probabilistic data fusion and physics-informed machine learning: A new paradigm for modeling under uncertainty, and its application to accelerating the discovery of new materials

In this report we summarize the work conducted by PI Perdikaris and his group under this Early Career project DE–SC0019116 during the period of 09/01/2018 – 08/31/2023. The central aim of the work was to introduce a new paradigm for scientific data analysis that can seamlessly synthesize rigorous mathematical modeling with data of variable fidelity (e.g., measurements at multiple scales/resolutions or predictions of variable fidelity models) and multiple modalities (e.g., images, time–series, or scattered measurements). The setting we are interested in involves complex systems that are partially observed and whose dynamical behavior could be hard to model or totally unknown. The inherent uncertainty associated with this setting necessitates a departure from the classical deterministic realm of modeling and scientific computation, and, consequently, our main building blocks can no longer be crisp deterministic numbers and governing laws, but instead we must operate with probabilistic models.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Assessing data change in scientific datasets

Summary Scientific datasets are growing rapidly and becoming critical to next‐generation scientific discoveries. The validity of scientific results relies on the quality of data used and data are often subject to change, for example, due to observation additions, quality assessments, or processing software updates. The effects of data change are not well understood and difficult to predict. Datasets are often repeatedly updated and recomputing derived data products quickly becomes time consuming and resource intensive and may in some cases not even be necessary, thus delaying scientific advance. Despite its importance, there is a lack of systematic approaches for best comparing data versions to quantify the changes, and ad‐hoc or manual processes are commonly used. In this article, we propose a novel hierarchical approach for analyzing data changes, including real‐time (online) and offline analyses. We employ a variety of fast‐to‐compute numerical analyses, graphical data change representations, and more resource‐intensive recomputations of a subset of the data product. We illustrate the application of our approach using three scientific diverse use cases, namely, satellite, cosmological, and x‐ray data. The results show that a variety of data change metrics should be employed to enable a comprehensive representation and qualitative evaluation of data changes.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1994

BATSE data analysis

The Burst and Transient Source Experiment (BATSE) is one of four instruments on the Compton Observatory which was launched by the space shuttle Atlantis on 5 Apr. 1991. As of the end of Nov. 1993, BATSE detected more than 830 cosmic gamma ray bursts and more than 690 solar flares. Pulsed gamma rays have been detected from at least 16 sources and emission from at least 28 sources (including most of the pulsed sources) has been detected by the earth occultation technique. The daily BATSE operations tasks represent a substantial level of effort and involve a large team which includes MSFC personnel as well as contractors such as UAH. The effort is naturally divided into several areas: data operations, burst operations, occultation operations, and pulsar operations. UAH personnel have been involved to some extent in all of these as well as contributing to various areas of scientific data analysis.

Paciesas, William S.↗

NASA NTRS1997

The Next Generation Space Telescope: Visiting a Time When Galaxies Were Young

In the spring and summer of 1996, three independent teams studied the feasibility of a large aperture space telescope to follow the Hubble Space Telescope. The scientific goals for the new telescope had been laid out in a report by the HST & Beyond Committee, a group appointed by the Association of Universities for Research in Astronomy to consider the needs of the astronomical community after the nominal end of the HST mission in 2005. The technical capabilities and constraints on the new observatory were daunting: the telescope optics should be at least 4 meters in diameter and passively cooled to achieve optimum sensitivity in the near-infrared portion of the spectrum. Moreover, the costs should be kept within a fraction of those for the HST: approximately $500M for construction and $900M for lifetime costs, not including support for scientific data analysis. In their presentations to NASA on 19-21 August, 1996, the teams led by Lockheed Martin, TRW, and the Goddard Space Flight Center concluded that a Next Generation Space Telescope (NGST) was not only feasible and affordable, but that it could be made more powerful using recent breakthroughs in space technologies. Coming on the heels of breathtaking HST observations of distant galaxies in the process of formation, such an NGST could bridge the gap in our understanding of the earliest origins of stars, galaxies, and the elements that are the foundations of Life. This report presents the findings of the three teams and the technological roadmap which will guide us to the successful development of the NGST over the next decade. We have made liberal use of the written material, tables, and diagrams prepared by the three study teams. We have also taken advantage of the knowledge and ideas of our colleagues in government, industry and academia. In Appendix A, we list the members of the three study groups, the NGST Science Working Group and the NGST Scientific Oversight Committee. We deeply appreciate their assistance, advice, and enthusiasm. The scientific and technological goals of NGST are part of the Origins initiative in the Office of Space Science, NASA Headquarters. We are pleased to acknowledge the support and leadership of Edward Weiler, a steadfast friend of HST, Harley Thronson, a proponent of all things infrared, and Mike Kaplan, a tireless advocate of new technology. John Campbell, Project Manager for HST, initiated the NGST study at GSFC and we deeply appreciate his formative efforts and continued support. We are also grateful for the foresight of Riccardo Giacconi, Director General of the European Southern Observatory (ESO), and the efforts of his staff. We note that European Space Agency (ESA) staff at the Space Telescope Science Institute and ESO played important roles in the NGST scientific and technical studies. We look forward to future collaboration with ESA, ESO, and other international partners.

H.S. Stockman↗

DOE OSTI2016

πScope: Python based scientific workbench with MDSplus data visualization tool

In this work, a newly developed python based scientific data analysis and visualization tool, πScope (http://piscope.psfc.mit.edu), is reported. The primary motivation is 1) to provide an updated tool to browse the MDSplus data beyond existing dwscope/jScope and 2) to realize a universal foundation to construct interface tools to perform computer modeling from experimental data. To visualize MDSplus data, πScope has many features including overplotting different signals and discharges, generating various plot types (line, contour, image, etc.), performing in-panel data analysis using python scripts, and producing publication quality graphics. The logic to generate multi-panel plots is designed to be backward compatible with dwscope, enabling smooth migration for users. πScope uses multi-threading in data loading, and is easy to modify and expand due to its object-oriented design. Furthermore, A user can access the data structure both from a GUI and a script, enabling relatively complex data analysis workflow built quickly on πScope.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2016

On-demand data analytics in HPC environments at leadership computing facilities: Challenges and experiences

Scaling-up scientific data analysis and machine learning algorithms for data-driven discovery is a grand challenge that we face today. Despite the growing need for analysis from science domains that are generating `Big Data' from instruments and simulations, building high-performance analytical workflows of data-intensive algorithms have been daunting because: (i) the `Big Data' hardware and software architecture landscape is constantly evolving, (ii) newer architectures impose new programming models, and (iii) data-parallel kernels of analysis algorithms and their performance facets on different architectures are poorly understood. To address these problems, we have: (i) identified scalable data-parallel kernels of popular data analysis algorithms, (ii) implemented `Mini-Apps' of those kernels using different programming models (e.g. Map Reduce, MPI, etc.), (iii) benchmarked and validated the performance of the kernels in diverse architectures. In this paper, we discuss two of those Mini-Apps and show the execution of principal component analysis built as a workflow of the Mini-Apps. We show that Mini-Apps enable scientists to (i) write domain-specific data analysis code that scales on most HPC hardware and (ii) and offers the ability (most times with over a 10x speed-up) to analyze data sizes 100 times the size of what off-the-shelf desktop/workstations of today can handle.

Harney, John↗

DOE OSTI2024

To Derive or Not to Derive: I/O Libraries Take Charge of Derived Quantities Computation

The ever-increasing volume of data produced by HPC simulations necessitates scalable methods for data exploration and knowledge extraction. Scientific data analysis often involves complex queries across distributed datasets, requiring manipulation of multiple primary variables and generating derived data that needs to be handled efficiently, creating challenges for applications that need to parse many large datasets. Relying on individual applications to handle all intermediate data generally leads to redundant computations across studies and unnecessary data transfers. In this paper, we investigate the performance of different approaches where applications define derived variables as quantities of interest (QoIs) and offload the computation and transfer of these QoIs to the I/O library. This significantly reduces redundancy and optimizes data movement across the distributed storage and processing infrastructure by allowing control over when and where derived variables are computed. We present a detailed analysis of the performance-storage trade-offs associated with different solutions and showcase results for our study on two large-scale datasets created from climate and combustion simulations.

Gainaru, Ana↗