Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27

Performance Understanding and Analysis for Exascale Data Management Workflows

The goal of the Exascale Data Management Workflows project is to create tools to support techniques for in-situ data analysis and data management. Online data management co-running with simulations accelerates the scientific processes being carried out, provides rapid and timely scientific insights, and can help avoid unnecessary and scientifically invalid simulation computations, particularly when combined with or able to utilize data from experimental instrument and/or from past simulation runs. The University of Oregon team is contributing to the performance instrumentation and analysis aspect of this project. Our focus has been on efficient performance monitoring of data management at scale. The general approach of the MONA project is depicted in Figure 1 below. The figure shows performance monitoring applied to an I/O workflow associated with a scientific simulation, with online measurements captured from this workflow informing methods for workflow reconfiguration and dynamic adjustment. The goal is to maintain suitable levels of Quality of Service for workflow execution, by understanding the under lying causes of workflow performance and behavior. One outcome is workflow performance models able to characterize realistic workflows. Another outcome is performance ’mini-apps’ implementing such workflow behavior. Out of scope for this project are advanced methods for online workflow control.

97 MATHEMATICS AND COMPUTING↗

Analysis and interpretation of high energy cosmic rays measured on Spacelab-2

Under the contract with NASA's Marshall Space Flight Center (NAS8-32828) the University of Chicago designed, built and delivered the CRN instrument for flight in the Spacelab-2 configuration. The instrument was flown from July 29 to August 5, 1985 on the Space Shuttle Challenger. The performance of our experiment was entirely successful and we reached our scientific goals. The contract included funds for the first year of data analysis. Further data analysis was carried out under the grant NAGW-1311 which ran from January 1, 1988 through December 31, 1990, and on which we report. The final products of this grant are the published scientific papers with the results from the experiment. We attach copies of the papers that have been published to date, and which we consider the most important part of this report. However, in order to put them into context we give a brief account of the project as a whole for which the grant covered the final phase.

Mueller, Dietrich↗

Planet X probe: A fresh new look at an old familiar place

Planet X Probe utilizes a Get Away Special (GAS) payload to provide a large student population with a remote Earth sensing experimental package. To provide a cooperative as well as a competitive environment, the effort is targeted at all grade levels and at schools in different geographical regions. LANDSAT capability allows students to investigate the Earth, its physical makeup, its resources, and the impact of man. This project also serves as an educational device to get students to stand back and take a fresh look at their home planet. The key element is to treat the familiar Earth as an unknown planet with knowledge based only on what is observable and provable from the images obtained. Through participation, a whole range of experiences will include: (1) mission planning; (2) research and pilot projects to train teams; (3) identification and recruitment of scientific mentors and dialogue; (4) selection of a student advisory team to be available during the mission; (5) analysis of data and compilation of findings; (6) report preparation, constucted along sound scientific principles; and (7) presentation and defense of findings before a meeting of competitive student groups and scientist in the field.

Nicholson, James↗

Combining In-situ and In-transit Processing to Enable Extreme-Scale Scientific Analysis

With the onset of extreme-scale computing, I/O constraints make it increasingly difficult for scientists to save a sufficient amount of raw simulation data to persistent storage. One potential solution is to change the data analysis pipeline from a post-process centric to a concurrent approach based on either in-situ or in-transit processing. In this context computations are considered in-situ if they utilize the primary compute resources, while in-transit processing refers to offloading computations toa set of secondary resources using asynchronous data transfers. In this paper we explore the design and implementation of three common analysis techniques typically performed on large-scale scientific simulations: topological analysis, descriptive statistics, and visualization. We summarize algorithmic developments, describe a resource scheduling system to coordinate the execution ofvarious analysis workflows, and discuss our implementation using the DataSpaces and ADIOS frameworks that support efficient data movement between in-situ and in-transit computations. Wedemonstrate the efficiency of our lightweight, flexible framework by deploying it on the Jaguar XK6 to analyze data generated by S3D, a massively parallel turbulent combustion code. Our framework allows scientists dealing with the data deluge at extreme scale to perform analyses at increased temporal resolutions, mitigate I/O costs, and significantly improve the time to insight.

Bennett, Janine C.↗

Provenance infrastructure for multimodal x-ray experiments and reproducible analysis

Exploring relationships between material structures and desired properties for material discovery and synthesis requires mining machine-readable databases and linking experimental data with sufficiently rich annotations. There are vast gaps both in these data sources and in the connections between them. We present a provenance-based data management and analysis framework that enables capturing, persisting, and reanalyzing experimental data from the BNL National Synchrotron Light Source II (NSLS-II). Our system leverages NSLS-II Bluesky for experimental data acquisition, captures analysis parameters, and enriches the search space with results from the scientific literature. We describe in detail the challenges and benefits of data-driven approaches when comparing computational and experimental data, such as the multiple possible scenarios of interest in often ill-posed inverse problems and missing data. Our infrastructure utilizes the NSLS-II event model and includes discovery mechanisms across multiple beamlines. A text and data mining module using natural language processing classification techniques extracts relevant information from the literature that is added to data sources. We describe a provenance acquisition and analysis rerun workflow based on a directed acyclic graph that keeps track of process and data provenance and allows to replay analysis.

97 MATHEMATICS AND COMPUTING↗

Improving NASA Earth Science Data and Information Access Through Natural Language Processing Based Data Analysis and Visualization

NASA: The Research Access initiative is part of the agency's framework for increasing public access to scientific publications and digital scientific data. The initiative follows the release of White House Office of Science and Technology Policy's (OSTP) memorandum "Increasing Access to the Results of Federally Funded Research," to ensure federally funded research is available to the public within one year of publication. NASA answered the mandate by creating an agency plan entitled "NASA Plan for Increasing Access to the Results of Scientific Research" and associated policy, NPD 2230.1, Research Data and Publication Access. Principles in NASA SMD Strategic Plan for Scientific Data and Computing: Continued free and open access to scientific data for any use. Improved ease of use and discoverability. Enhanced science applications and new use cases. Incorporates best practices and "state of the art" through partnerships. Earth Data and Systems are Evolving: Increasing archive and file sizes. More complicated data structures. More user-friendly and data services. What is the future direction?

Liu, Zhong↗

FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework

Scientific applications are increasingly using cloud resources for their data analysis workflows. However, managing data effectively and efficiently over these cloud resources is challenging due to the myriad storage choices with different performance, cost trade-offs, complex application choices and complexity associated with elasticity, failure rates in these environments. The different data access patterns for data-intensive scientific applications require a more flexible and robust data management solution than the ones currently in existence. FRIEDA is a Flexible Robust Intelligent Elastic Data Management framework that employs a range of data management strategies in cloud environments. FRIEDA can manage storage and data lifecycle of applications in cloud environments. There are four different stages in the data management lifecycle of FRIEDA – (i) storage planning, (ii) provisioning and preparation, (iii) data placement, and (iv) execution. FRIEDA defines a data control plane and an execution plane. The data control plane defines the data partition and distribution strategy, whereas the execution plane manages the execution of the application using a master-worker paradigm. FRIEDA also provides different data management strategies, either to partition the data in real-time, or predetermine the data partitions prior to application execution.

97 MATHEMATICS AND COMPUTING↗

Integrating Scientific Argumentation to Improve Undergraduate Writing and Learning in a Global Environmental Change Course

We present a strategy for using scientific argumentation in an early undergraduate laboratory course to teach disciplinary writing practices and to promote critical thinking, knowledge transformation, and understanding of the scientific method. The approach combines targeted writing instruction; data analysis and interpretation; formulation of a hypothesis; and construction of an argument. Students submit and receive feedback on two drafts of two different argumentation essays, providing the opportunity for guided practice. Each written argument is intended to draw on several weeks' course material, including short lectures, discussions, readings, and problem sets. Thus our aim with these writing assignments is to help students synthesize content and concepts, deepening their learning. We have found that this inquiry-based approach to writing engages students in course material, and significantly improves both writing and learning. We observed the greatest improvement among students with the lowest initial scores, suggesting that lower-achieving students benefitted disproportionately from this approach. Students have responded positively to the use of writing in the course, many stating on course evaluations that this is the first time they have received instruction in scientific writing. They have also pointed to a greater 'big-picture' understanding of the course gained through writing. We describe the course and our curriculum, and provide suggestions for implementation as well as rubrics used to evaluate problem sets and student argumentation essays.

99 GENERAL AND MISCELLANEOUS↗

Toward designing effective exascale scientific computing workflows: experiences and best practices

Many fields within scientific computing have embraced advances in big-data analysis and machine learning, which often requires the deployment of large, distributed and complicated workflows that may combine training neural networks, performing simulations, running inference, and performing database queries and data analysis in asynchronous, parallel and pipelined execution frameworks. Such a shift has brought into focus the need for scalable, efficient workflow management solutions with reproducibility, error and provenance handling, traceability, and checkpoint-restart capabilities, among other needs. Here, we discuss challenges and best-practices for deploying exascale-generation computational science workflows on resources at the Oak Ridge Leadership Computing Facility (OLCF). We present our experiences with large-scale deployment of distributed workflows on the Summit supercomputer, including for bioinformatics and computational biophysics, materials science, and deep learning model optimization. We also present problems and solutions created by working within a Python-centric software base on traditional HPC systems, and discuss steps that will be required before the convergence of HPC, AI, and data science can be fully realized. Our results point to a wealth of exciting new possibilities for harnessing this convergence to tackle new scientific challenges.

Coletti, Mark↗

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific ap- plications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Net- works (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate band- widths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer perfor- mance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application- level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher pre- diction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

Panorama 360 (Final Report)

This is the final technical report for the DOE-funded Panorama 360 project. Panorama 360 provided a resource for the collection, analysis, and sharing of performance data about end-to-end scientific workflows executing on DOE facilities. The work focused on workflows that include experimental data generation at DOE facilities. The main activities of Panorama 360 include the development of: 1. A distributed repository that stores different types of workflow execution data (e.g., point and time series performance traces at fine- and coarse-grained levels); 2. A set of open-source data capture, curation, and publishing tools fully integrated with a state-of-the-art workflow management system that automates data ingestion to the repository and enables users to discover, query, and process data from the repository; 3. A set of analysis algorithms and machine learning based tools to perform analysis and characterization of the gathered data, which can be used to detect anomalous performance or system faults; and 4. Best practices and recommendations for workflow evaluation, analysis, execution, and architectures.

97 MATHEMATICS AND COMPUTING↗

DASL-Data and Activities for Solar Learning

DASL-Data and Activities for Solar Learning Data and Activities for Solar Learning (DASL) provides a classroom learning environment based on a twenty-five year record of solar magnetograms from the National Solar Observatory (NSO) at Kitt Peak, AZ. The data, together with image processing software for Macs or PCs, can be used to learn basic facts about the Sun and astronomy at the middle school level. At the high school level, students can study properties of the Sun's magnetic cycle with classroom exercises emphasizing data and error analysis and can participate in a new scientific study, Research in Active Solar Longitudes (RASL), in collaboration with classrooms throughout the country and scientists at NSO and NASA. We present a half-day course to train teachers in the scientific content of the project and its classroom use. We will provide a compact disc with the data and software and will demonstrate software installation and use, classroom exercises, and participation in RASL with computer projection.

Jones, Harrison P.↗

Analysis of AVHRR, CZCS and historical in situ data off the Oregon Coast

The original scientific objectives of this grant were to: (1) characterize the seasonal cycles and interannual variability for phytoplankton concentrations and sea surface temperature (SST) in the California Current using satellite data; and (2) to explore the spatial and temporal relationship between these variables and surface wind forcing. An additional methodological objective was to develop statistical methods for forming mean fields, which minimize the effects of random data gaps and errors in the irregularly sampled CZCS (Coastal Zone Color Scanner) and AVHRR (Advanced Very High Resolution Radiometer) satellite data. A final task was to evaluate the level of uncertainty in the wind fields used for the statistical analysis. Funding in the first year included part of the cost of an image processing system to enable this and other projects to process and analyze satellite data. This report consists of summaries of the major projects carried out with all or partial support from this grant. The appendices include a list of papers and professional presentations supported by the grant, as well as reprints of the major papers and reports.

Strub, P. Ted↗

Physics in the Machine: Integrating Physical Knowledge in Autonomous Phase-Mapping

Application of artificial intelligence (AI), and more specifically machine learning, to the physical sciences has expanded significantly over the past decades. In particular, science-informed AI, also known as scientific AI or inductive bias AI, has grown from a focus on data analysis to now controlling experiment design, simulation, execution and analysis in closed-loop autonomous systems. The CAMEO (closed-loop autonomous materials exploration and optimization) algorithm employs scientific AI to address two tasks: learning a material system’s composition-structure relationship and identifying materials compositions with optimal functional properties. By integrating these, accelerated materials screening across compositional phase diagrams was demonstrated, resulting in the discovery of a best-in-class phase change memory material. Key to this success is the ability to guide subsequent measurements to maximize knowledge of the composition-structure relationship, or phase map. In this work we investigate the benefits of incorporating varying levels of prior physical knowledge into CAMEO’s autonomous phase-mapping. This includes the use of ab-initio phase boundary data from the AFLOW repositories, which has been shown to optimize CAMEO’s search when used as a prior.

97 MATHEMATICS AND COMPUTING↗

AXAF user interfaces for heterogeneous analysis environments

The AXAF Science Center (ASC) will develop software to support all facets of data center activities and user research for the AXAF X-ray Observatory, scheduled for launch in 1999. The goal is to provide astronomers with the ability to utilize heterogeneous data analysis packages, that is, to allow astronomers to pick the best packages for doing their scientific analysis. For example, ASC software will be based on IRAF, but non-IRAF programs will be incorporated into the data system where appropriate. Additionally, it is desired to allow AXAF users to mix ASC software with their own local software. The need to support heterogeneous analysis environments is not special to the AXAF project, and therefore finding mechanisms for coordinating heterogeneous programs is an important problem for astronomical software today. The approach to solving this problem has been to develop two interfaces that allow the scientific user to run heterogeneous programs together. The first is an IRAF-compatible parameter interface that provides non-IRAF programs with IRAF's parameter handling capabilities. Included in the interface is an application programming interface to manipulate parameters from within programs, and also a set of host programs to manipulate parameters at the command line or from within scripts. The parameter interface has been implemented to support parameter storage formats other than IRAF parameter files, allowing one, for example, to access parameters that are stored in data bases. An X Windows graphical user interface called 'agcl' has been developed, layered on top of the IRAF-compatible parameter interface, that provides a standard graphical mechanism for interacting with IRAF and non-IRAF programs. Users can edit parameters and run programs for both non-IRAF programs and IRAF tasks. The agcl interface allows one to communicate with any command line environment in a transparent manner and without any changes to the original environment. For example, the authors routinely layer the GUI on top of IRAF, ksh, SMongo, and IDL. The agcl, based on the facilities of a system called Answer Garden, also has sophisticated support for examining documentation and help files, asking questions of experts, and developing a knowledge base of frequently required information. Thus, the GUI becomes a total environment for running programs, accessing information, examining documents, and finding human assistance. Because the agcl can communicate with any command-line environment, most projects can make use of it easily. New applications are continually being found for these interfaces. It is the authors' intention to evolve the GUI and its underlying parameter interface in response to these needs - from users as well as developers - throughout the astronomy community. This presentation describes the capabilities and technology of the above user interface mechanisms and tools. It also discusses the design philosophies guiding the work, as well as hopes for the future.

Mandel, Eric↗

The Solar-B Mission: First Light, Future Plans and Community Participation

The Solar-B spacecraft was launched from the Uchinoura Space Center into a circular, sun-synchronous, polar orbit by the Japanese Aerospace Exploration Agency in late September 2006. The spacecraft carries thee scientific instruments designed to follow the flow of magnetic energy from the photosphere to the corona to improve our understanding of both steady state and transient energy release. This goal will be achieved through coordinated observations of three highly advanced solar telescopes developed cooperatively by teams from Japan, the United States and the United Kingdom. The three telescopes are a 0.5m aperture, diffraction limited, solar optical telescope (SOT), an X-ray telescope (XRT) designed for full sun imaging with 1.0 arcsec pixels and an EUV imaging spectrometer (EIS) with an order of magnitude improvement in sensitivity over past instruments. The SOT focal plane contains three instruments, a spectropolarimeter for measuring vector magnetic fields, a broadband filter imager for recording images of the photosphere and chromosphere at the highest resolution the telescope is capable of, and a narrow band filter imager that will record Doppler grams and vector magnetograms. The XRT has broad temperature coverage and a spatial a resolution three times as high as Yohkoh. EIS covers a broad range of transition region and coronal temperatures in two spectral bands. Both XRT and EIS have 2 arcsec spatial resolution (1 arcsec pixels). Instrument first light occurred after five weeks on orbit to allow for out gassing and the opening of the telescopes doors. The initial observation sequences are designed to test the functionality of the different operating modes and for calibration. After this commissioning phase is complete a series of observations are planned to demonstrate the ability of the instruments to meet NASA's mission minimum success criteria. Data is downloaded every orbit to the Norwegian high latitude ground station at Svalbard. The data are transmitted to ISAS where they are reformatted into FITS files and archived as Level 0 data on the ISAS DARTS system. Once the initial observation period is complete, approximately six months after launch, the mission data will be open and freely available to researchers shortly after receipt at the DARTS data archive hosted in Japan and at NASA s Solar Data Analysis Center at the Goddard Space Flight Center. Scientific operations will be conducted from the ISAS facility in Sagamihara, Japan and the observatory will become available for performing joint operations with both ground and space based instruments and for conducting observing programs proposed by non-team members. This process will be described together with a status report from the initial operation of the observatory, showing examples of the first observations.

Davis, John M.↗

Towards Exascale Scientific Metadata Management

Advances in technology and computing hardware are enabling scientists from all areas of science to produce massive amounts of data using large-scale simulations or observational facilities. In this era of data deluge, effective coordination between the data production and the analysis phases hinges on the availability of metadata that describe the scientific datasets. Existing workflow engines have been capturing a limited form of metadata to provide provenance information about the identity and lineage of the data. However, much of the data produced by simulations, experiments, and analyses still need to be annotated manually in an ad hoc manner by domain scientists. Systematic and transparent acquisition of rich metadata becomes a crucial prerequisite to sustain and accelerate the pace of scientific innovation. Yet, ubiquitous and domain-agnostic metadata management infrastructure that can meet the demands of extreme-scale science is notable by its absence. To address this gap in scientific data management research and practice, we present our vision for an integrated approach that (1) automatically captures and manipulates information-rich metadata while the data is being produced or analyzed and (2) stores metadata within each dataset to permeate metadata-oblivious processes and to query metadata through established and standardized data access interfaces. We motivate the need for the proposed integrated approach using applications from plasma physics, climate modeling and neuroscience, and then discuss research challenges and possible solutions.

97 MATHEMATICS AND COMPUTING↗