Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

On-demand data analytics in HPC environments at leadership computing facilities: Challenges and experiences

Scaling-up scientific data analysis and machine learning algorithms for data-driven discovery is a grand challenge that we face today. Despite the growing need for analysis from science domains that are generating `Big Data' from instruments and simulations, building high-performance analytical workflows of data-intensive algorithms have been daunting because: (i) the `Big Data' hardware and software architecture landscape is constantly evolving, (ii) newer architectures impose new programming models, and (iii) data-parallel kernels of analysis algorithms and their performance facets on different architectures are poorly understood. To address these problems, we have: (i) identified scalable data-parallel kernels of popular data analysis algorithms, (ii) implemented `Mini-Apps' of those kernels using different programming models (e.g. Map Reduce, MPI, etc.), (iii) benchmarked and validated the performance of the kernels in diverse architectures. In this paper, we discuss two of those Mini-Apps and show the execution of principal component analysis built as a workflow of the Mini-Apps. We show that Mini-Apps enable scientists to (i) write domain-specific data analysis code that scales on most HPC hardware and (ii) and offers the ability (most times with over a 10x speed-up) to analyze data sizes 100 times the size of what off-the-shelf desktop/workstations of today can handle.

Harney, John↗
DOE OSTI2024

To Derive or Not to Derive: I/O Libraries Take Charge of Derived Quantities Computation

The ever-increasing volume of data produced by HPC simulations necessitates scalable methods for data exploration and knowledge extraction. Scientific data analysis often involves complex queries across distributed datasets, requiring manipulation of multiple primary variables and generating derived data that needs to be handled efficiently, creating challenges for applications that need to parse many large datasets. Relying on individual applications to handle all intermediate data generally leads to redundant computations across studies and unnecessary data transfers. In this paper, we investigate the performance of different approaches where applications define derived variables as quantities of interest (QoIs) and offload the computation and transfer of these QoIs to the I/O library. This significantly reduces redundancy and optimizes data movement across the distributed storage and processing infrastructure by allowing control over when and where derived variables are computed. We present a detailed analysis of the performance-storage trade-offs associated with different solutions and showcase results for our study on two large-scale datasets created from climate and combustion simulations.

Gainaru, Ana↗
DOE OSTI2019

Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence

Scientific Machine Learning (SciML) and Artificial Intelligence (AI) will have broad use and transformative effects across the Department of Energy. Accordingly, the January 2018 Basic Research Needs workshop identified six Priority Research Directions (PRDs). The first three PRDs describe foundational research themes that correspond to the need for domain-awareness (PRD #1), interpretability (PRD #2), and robustness (PRD #3). The other three PRDs describe capability research themes and correspond to the three major use cases for massive scientific data analysis (PRD #4), machine learning-enhanced modeling and simulation (PRD #5), and intelligent automation and decision-support for complex systems (PRD #6). The Priority Research Directions provide a sound basis for a coherent, long-term research and development strategy in SciML and AI. Over the last decade, DOE investments in applied mathematics have laid the groundwork for the type of basic research that will underpin key advances in the six PRDs. Such advances will build on the work from leading researchers in optimization, linear algebra, high-performance solvers and algorithms, multiscale modeling and simulation, complex systems research, uncertainty quantification, and the new basic research areas that will emerge from the pursuit of transformative technologies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Brochure on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence [Slides]

Scientific Machine Learning (SciML) and Artificial Intelligence (AI) will have broad use and transformative effects across the Department of Energy. Accordingly, the January 2018 Basic Research Needs workshop identified six Priority Research Directions (PRDs). The first three PRDs describe foundational research themes that correspond to the need for domain-awareness (PRD #1), interpretability (PRD #2), and robustness (PRD #3). The other three PRDs describe capability research themes and correspond to the three major use cases for massive scientific data analysis (PRD #4), machine learning-enhanced modeling and simulation (PRD #5), and intelligent automation and decision-support for complex systems (PRD #6). The Priority Research Directions provide a sound basis for a coherent, long-term research and development strategy in SciML and AI. Over the last decade, DOE investments in applied mathematics have laid the groundwork for the type of basic research that will underpin key advances in the six PRDs. Such advances will build on the work from leading researchers in optimization, linear algebra, high-performance solvers and algorithms, multiscale modeling and simulation, complex systems research, uncertainty quantification, and the new basic research areas that will emerge from the pursuit of transformative technologies.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1983

U.S. data processing for the IRAS project

The JPL's Scientific Data Analysis System (SDAS), which will process IRAS data and produce a catalogue of perhaps a million infrared sources in the sky, as well as other information for astronomical records, is described. The purposes of SDAS are discussed, and the major SDAS processors are shown in block diagram. The catalogue processing is addressed, mentioning the basic processing steps which will be applied to raw detector data. Signal reconstruction and conversion to astrophysical units, source detection, source confirmation, data management, and survey data products are considered in detail.

Duxbury, J. H.↗
NASA NTRS1991

AIAA/NASA International Symposium on Space Information Systems, 2nd, Pasadena, CA, Sept. 17-19, 1990, Proceedings. Vols. 1 & 2

These proceedings overview major space information system projects and lessons learned from current missions. Other topics include the science information system requirements for the 1990s, an information systems design approach for major programs, the technology needs and projections, the standards for space data information systems, the artificial intelligence technology and applications, international interoperability, and spacecraft data systems and architectures advanced communications. Other topics include the software engineering technology and applications, the multimission multidiscipline information system architectures, the distributed planning and scheduling systems and operations, and the computer and information systems architectures. Paper presented include prospects for scientific data analysis systems for solar-terrestrial physics in the 1990s, the Columbus data management system, data storage technologies for the future, the German aerospace research establishment, and launching artificial intelligence in NASA ground systems.

Tavenner, Leslie A.↗
NASA NTRS1998

Investigation of Coronal Large Scale Structures Utilizing Spartan 201 Data

Two telescopes aboard Spartan 201, a small satellite has been launched from the Space Shuttles, on April 8th, 1993, September 8th, 1994, September 7th, 1995 and November 20th, 1997. The main objective of the mission was to answer some of the most fundamental unanswered questions of solar physics-What accelerates the solar wind and what heats the corona? The two telescopes are 1) Ultraviolet Coronal Spectrometer (UVCS) provided by the Smithsonian Astrophysical Observatory which uses ultraviolet emissions from neutral hydrogen and ions in the corona to determine velocities of the coronal plasma within the solar wind source region, and the temperature and density distributions of protons and 2) White Light Coronagraph (WLC) provided by NASA's Goddard Space Flight Center which measures visible light to determine the density distribution of coronal electrons within the same region. The PI has had the primary responsibility in the development and application of computer codes necessary for scientific data analysis activities, end instrument calibration for the white-light coronagraph for the entire Spartan mission. The PI was responsible for the science output from the WLC instrument. PI has also been involved in the investigation of coronal density distributions in large-scale structures by use of numerical models which are (mathematically) sufficient to reproduce the details of the observed brightness and polarized brightness distributions found in SPARTAN 201 data.

Guhathakurta, Madhulika↗
NASA NTRS2003

Lessons Learned From Developing A Streaming Data Framework for Scientific Analysis

We describe the development and usage of a streaming data analysis software framework. The framework is used for three different applications: Earth science hyper-spectral imaging analysis, Electromyograph pattern detection, and Electroencephalogram state determination. In each application the framework was used to answer a series of science questions which evolved with each subsequent answer. This evolution is summarized in the form of lessons learned.

Wheeler. Kevin R.↗
NASA NTRS2003

A Step Beyond Simple Keyword Searches: Services Enabled by a Full Content Digital Journal Archive

The problems of managing and searching large archives of scientific journal articles can potentially be addressed through data mining and statistical techniques matured primarily for quantitative scientific data analysis. A journal paper could be represented by a multivariate descriptor, e.g., the occurrence counts of a number key technical terms or phrases (keywords), perhaps derived from a controlled vocabulary ( e . g . , the American Meteorological Society's Glossary of Meteorology) or bootstrapped from the journal archive itself. With this technique, conventional statistical classification tools can be leveraged to address challenges faced by both scientists and professional societies in knowledge management. For example, cluster analyses can be used to find bundles of "most-related" papers, and address the issue of journal bifurcation (when is a new journal necessary, and what topics should it encompass). Similarly, neural networks can be trained to predict the optimal journal (within a society's collection) in which a newly submitted paper should be published. Comparable techniques could enable very powerful end-user tools for journal searches, all premised on the view of a paper as a data point in a multidimensional descriptor space, e.g.: "find papers most similar to the one I am reading", "build a personalized subscription service, based on the content of the papers I am interested in, rather than preselected keywords", "find suitable reviewers, based on the content of their own published works", etc. Such services may represent the next "quantum leap" beyond the rudimentary search interfaces currently provided to end-users, as well as a compelling value-added component needed to bridge the print-to-digital-medium gap, and help stabilize professional societies' revenue stream during the print-to-digital transition.

Boccippio, Dennis J.↗
NASA NTRS2012

JPL Genesis and Rapid Intensification Processes (GRIP) Portal

Satellite observations can play a very important role in airborne field campaigns, since they provide a comprehensive description of the environment that is essential for the experiment design, flight planning, and post-experiment scientific data analysis. In the past, it has been difficult to fully utilize data from multiple NASA satellites due to the large data volume, the complexity of accessing NASA s data in near-real-time (NRT), as well as the lack of software tools to interact with multi-sensor information. The JPL GRIP Portal is a Web portal that serves a comprehensive set of NRT observation data sets from NASA and NOAA satellites describing the atmospheric and oceanic environments related to the genesis and intensification of the tropical storms in the North Atlantic Ocean. Together with the model forecast data from four major global atmospheric models, this portal provides a useful tool for the scientists and forecasters in planning and monitoring the NASA GRIP field campaign during the 2010 Atlantic Ocean hurricane season. This portal uses the Google Earth plug-in to visualize various types of data sets, such as 2D maps, wind vectors, streamlines, 3D data sets presented at series of vertical cross-sections or pointwise vertical profiles, and hurricane best tracks and forecast tracks. Additionally, it allows users to overlap multiple data sets, change the opacity of each image layer, generate animations on the fly with selected data sets, and compare the observation data with the model forecast using two independent calendars. The portal also provides the capability to identify the geographic location of any point of interest. In addition to supporting the airborne mission planning, the NRT data and portal will serve as a very rich source of information during the post-field campaign analysis stage of the airborne experiment. By including a diverse set of satellite observations and model forecasts, it provides a good spatial and temporal context for the high-resolution, but limited in space and time, airborne observations.

Knosp, Brian W.↗
DOE OSTI2023

The U.S. CMS HL-LHC R&D Strategic Plan

The HL-LHC run is anticipated to start at the end of this decade and will pose a significant challenge for the scale of the HEP software and computing infrastructure. The mission of the U.S. CMS Software & Computing Operations Program is to develop and operate the software and computing resources necessary to process CMS data expeditiously and to enable U.S. physicists to fully participate in the physics of CMS. We have developed a strategic plan to prioritize R&D efforts to reach this goal for the HL-LHC. This plan includes four grand challenges: modernizing physics software and improving algorithms, building infrastructure for exabyte-scale datasets, transforming the scientific data analysis process and transitioning from R&D to operations. We are involved in a variety of R&D projects that fall within these grand challenges. In this talk, we will introduce our four grand challenges and outline the R&D program of the U.S. CMS Software & Computing Operations Program.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRS2003

SOHO Ultraviolet Coronagraph Spectrometer (UVCS) Mission Operations and Data Analysis

The scientific goal of UVCS is to obtain detailed empirical descriptions of the extended solar corona as it evolves over the solar cycle and to use these descriptions to identify and understand the physical processes responsible for coronal heating, solar wind acceleration, coronal mass ejections (CMEs), and the phenomena that establish the plasma properties of the solar wind as measured by 'in situ' solar wind instruments. This report covers the period from 01 February 2002 to 15 February 2003. During that time, UVCS observations have consisted of three types: 1) standard synoptic observations comprising, primarily, the H I Ly alpha line profile and the O VI 103.2 and 103.7 nm intensity over a range of heights from 1.5 to about 3.0 solar radii and covering 360 degrees about the sun, 2) sit and stare watches for CMEs, and 3) special observations designed by the UVCS Lead Observer of the Week for a specific scientific purpose. The special observations are often coordinated with those of other space-based and ground-based instruments and they often are part of SOHO joint observation programs and campaigns. Lead observers have included UVCS Co-Investigators, scientists from the solar physics community and several graduate and undergraduate level students. UVCS has continued to achieve its purpose of using powerful spectroscopic diagnostic techniques to obtain a much more detailed description of coronal structures and dynamic phenomena than existed before the SOHO mission. The new descriptions of coronal mass ejections (CMEs) and coronal structures from UVCS have inspired a large number of theoretical studies aimed at identifying the physical processes responsible for CMEs and solar wind acceleration in coronal holes and streamers. UVCS has proven to be a very stable instrument. Stellar observations have demonstrated its stability. UVCS has required no flight software modifications and all mechanisms are operational. The UVCS O VI Channel with its redundant optical path for wavelengths near H I Ly alpha is capable of observing the entire UVCS wavelength range. Since December 1998, the O VI Channel has been used for all UVCS observations. Although the H I Ly alpha Channel and detector are still operational, increases in the dark count up to about 5 x 10(exp -4) counts/sec/pixel and an increase in high voltage current to within a factor of two of the maximum used in the laboratory before flight led to the decision to not use that detector at the present time. There is no significant decrease in the scientific capability of UVCS owing to the O VI channel redundant optical path. UVCS data, data analysis software, calibration files and the mission log are available from the SOHO archive and SAO. All UVCS data is now available to scientists and the general public via the SOHO Data Archive and SAO within three months of the observations. UVCS has resulted in 46 scientific papers in 2002. There were numerous presentations at scientific meetings. All requests for observation time by qualified outside users have been granted.

Kohl, John L.↗
NASA NTRS1971

Scientific data reduction and analysis plan: PI services

This plan comprises two parts. The first concerns the real-time data display to be provided by MSC during the mission. The prime goal is to assess the operation of the UVS and to identify any problem areas that could be corrected during the mission. It is desirable to identify any possible observations of unusual scientific interest in order to repeat these observations at a later point in the mission, or to modify the time line with respect to the operating modes of the UVS. The second part of the plan discusses the more extensive postflight analysis of the data in terms of the scientific objectives of this experiment.

Feldman, P. D.↗
NASA NTRS2004

SOHO Ultraviolet Coronagraph Spectrometer (UVCS) Mission Operations and Data Analysis

The scientific goal of UVCS is to obtain detailed empirical descriptions of the extended solar corona as it evolves over the solar cycle and to use these descriptions to identify and understand the physical processes responsible for coronal heating, solar wind acceleration, coronal mass ejections (CMEs), and the phenomena that establish the plasma properties of the solar wind as measured by "in situ" solar wind instruments. This report covers the period from 15 February 2003 to 14 April 2004. During that time, UVCS observations have consisted of three types: 1) standard synoptic observations comprising, primarily, the H I Lyalpha line profile and the 0 VI 103.2 and 103.7 nm intensity over a range of heights from 1.5 to about 3.0 solar radii and covering 360 degrees about the Sun, 2) sit and stare observations for major flare watches, and 3) special observations designed by the UVCS Lead Observer of the Week for a specific scientific purpose. The special observations are often coordinated with those of other space-based and ground-based instruments and they often are part of SOHO joint observation programs and campaigns. Lead observers have included UVCS Co-Investigators, scientists from the solar physics community and several graduate and undergraduate level students. UVCS has continued to achieve its purpose of using powerful spectroscopic diagnostic techniques to obtain a much more detailed description of coronal structures and dynamic phenomena than existed before the SOHO mission. The new descriptions of coronal mass ejections (CMEs) and coronal structures from UVCS have inspired a large number of theoretical studies aimed at identifying the physical processes responsible for CMEs and solar wind acceleration in coronal holes and streamers. UVCS has proven to be a very stable instrument. Stellar observations have demonstrated its radiometric stability. UVCS has not required any flight software modifications and all mechanisms are operational. The UVCS 0 VI Channel with its redundant optical path for wavelengths near H I Lyalpha is capable of observing the entire UVCS wavelength range. The regions of the detector currently being used require different grating angles for direct OVI observations and redundant path H I Lyalpha observations, and so those can no longer be observed simultaneously. Since December 1998, the 0 VI Channel has been used for all UVCS observations. Although the H I Lyalpha Channel and detector are still operational, increases in the dark count up to about 5x10(exp 4) counts/sec/pixel and an increase in high voltage current to within a factor of two of the maximum used in the laboratory before flight led to the decision to not use that detector after 1998. The visible light channel functioned nominally during the reporting period. UVCS data, data analysis software, calibration files and the mission log are available from the SOHO archive and SAO. All UVCS data are now available within three months of the observations to scientists and the general public via the SOHO Data Archive and SAO. UVCS has resulted in 33 scientific papers in 2003. There were numerous presentations at scientific meetings. UVCS Education and Public Outreach activities involved nine members of the UVCS team. During the reporting period, there were over a dozen events directed at students and teachers, museum audiences, and public audiences via the mass media, internet and educational literature.

Gurman, Joseph↗
NASA NTRS1998

SOHO Ultraviolet Coronagraph Spectrometer (UVCS) Mission Operations and Data Analysis

The scientific goal of UVCS is to obtain detailed empirical descriptions of the extended solar corona as it evolves from a period of minimum activity to maximum and to use these descriptions to identify and understand the physical processes responsible for coronal heating, solar wind acceleration, coronal mass ejections (CME's), and the phenomena that establish the plasma properties of the solar wind as measured by the CELIAS/SOHO experiment. We must observe a wide range of conditions (e.g., heating rate, impulsive mass ejections, magnetic field opening) in order to choose among competing theoretical explanations of the observed coronal structures and dynamics. During the first year of scientific observations (1 April 1996 through 31 March 1997), the LTVCS made the first extensive ultraviolet spectroscopic observations above the base of the solar corona. Those observations revealed extremely high most probable speeds along the line of sight for O(5+) in polar coronal holes above 2 solar radii from sun center and smaller such speeds for H(0) and protons. Doppler dimming and pumping diagnostics indicated that O(5+) and H(0) I reached supersonic outflow velocities within 2 solar radii. The observations appeared to be consistent with O95+) heating and direct momentum transfer by high frequency MHD waves through ion cyclotron resonance. Equatorial streamers were found to appear much different when observed in O VI, Fe XII and Mg X than when observed in H I Lyman-alpha or visible light. This led G. Noci to suggest that the elemental abundance of O, Fe, and Mg are lower in the dark center region of the O VI streamer image than it is in the brighter outer legs of the streamer. The O VI line ratio indicated a much smaller outflow speed in streamers than in the surrounding coronal holes. Spectroscopic observations of CME's in June and December 1996 showed large Doppler shifts indicating macroscopic flows of up to 200 km/s transverse to the apparent motion of the CME. The December event displayed factor of 500 intensity increases in H I Lyman lines and bright emissions in cool ion lines like C III 977 A indicating that the origin of some of the gas was a prominence. UVCS also observed two comets, several stars and interplanetary hydrogen. In the current reporting period, the initial results were published and detailed analyses including self consistent empirical models were developed. The research results were reported, published or submitted/accepted for publication. New observations were made to uncover additional information about the surprising results of the first year and other new observations revealed more surprises. A UVCS Tutorial was developed to familiarize potential users of UVCS data with the data products, analysis techniques and coronal models needed to analyze the data.

Kohl, John L.↗
NASA NTRS1990

Scientific study of data analysis

We present a comparison between two numerical methods for the extrapolation of nonlinear force-free magnetic fields, the Iterative Method (IM) and the Progressive Extension Method (PEM). The advantages and disadvantages of these two methods are summarized and the accuracy and numerical instability are discussed. On the basis of this investigation, we claim that the two methods do resemble each other qualitatively.

Wu, S. T.↗