Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “scientific data change analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Assessing data change in scientific datasets

Summary Scientific datasets are growing rapidly and becoming critical to next‐generation scientific discoveries. The validity of scientific results relies on the quality of data used and data are often subject to change, for example, due to observation additions, quality assessments, or processing software updates. The effects of data change are not well understood and difficult to predict. Datasets are often repeatedly updated and recomputing derived data products quickly becomes time consuming and resource intensive and may in some cases not even be necessary, thus delaying scientific advance. Despite its importance, there is a lack of systematic approaches for best comparing data versions to quantify the changes, and ad‐hoc or manual processes are commonly used. In this article, we propose a novel hierarchical approach for analyzing data changes, including real‐time (online) and offline analyses. We employ a variety of fast‐to‐compute numerical analyses, graphical data change representations, and more resource‐intensive recomputations of a subset of the data product. We illustrate the application of our approach using three scientific diverse use cases, namely, satellite, cosmological, and x‐ray data. The results show that a variety of data change metrics should be employed to enable a comprehensive representation and qualitative evaluation of data changes.

97 MATHEMATICS AND COMPUTING↗

Randomized Algorithms for Scientific Computing (RASC)

Randomized algorithms have propelled advances in artificial intelligence (AI) and represent a foundational research area in advancing AI for Science. Future advancements in DOE Office of Science priority areas such as climate science, astrophysics, fusion, advanced materials, combustion, and quantum computing all require randomized algorithms for surmounting challenges of complexity, robustness, and scalability. Advances in data collection and numerical simulation have changed the dynamics of scientific research and motivate the need for randomized algorithms. For instance, advances in imaging technologies such as X-ray ptychography, electron microscopy, electron energy loss spectroscopy, or adaptive optics lattice light-sheet microscopy collect hyperspectral imaging and scattering data in terabytes, at breakneck speed enabled by state-of-the-art detectors. The data collection is exceptionally fast compared with its analysis. Likewise, advances in high-performance architectures have made exascale computing a reality and changed the economies of scientific computing in the process. Floating-point operations that create data are essentially free in comparison with data movement. Thus far, most approaches have focused on creating faster hardware. Ironically, this faster hardware has exacerbated the problem by making data still easier to create. Under such an onslaught, scientists often resort to heuristic deterministic sampling schemes (e.g., low-precision arithmetic, sampling every nth element) and sacrifice potentially valuable accuracy. Dramatically better results can be achieved via randomized algorithms, reducing the data size as much as or more than naive deterministic subsampling can achieve, while retaining the high accuracy of computing on the full data set. By randomized algorithms we mean those algorithms that employ some form of randomness in internal algorithmic decisions to accelerate time to solution, increase scalability, or improve reliability. Examples include matrix sketching for solving large-scale least-squares problems (see Figure 1) and stochastic gradient descent for training machine learning models. We are not recommending heuristic methods but rather randomized algorithms that have certificates of correctness and probabilistic guarantees of optimality and near-optimality. Such approaches can be useful beyond acceleration, for example, in understanding how to avoid measure zero worst-case scenarios that plague methods such as QR matrix factorization.

97 MATHEMATICS AND COMPUTING↗

Finding Tropical Cyclones on a Cloud Computing Cluster: Using Parallel Virtualization for Large-Scale Climate Simulation Analysis

Extensive computing power has been used to tackle issues such as climate changes, fusion energy, and other pressing scientific challenges. These computations produce a tremendous amount of data; however, many of the data analysis programs currently only run a single processor. In this work, we explore the possibility of using the emerging cloud computing platform to parallelize such sequential data analysis tasks. As a proof of concept, we wrap a program for analyzing trends of tropical cyclones in a set of virtual machines (VMs). This approach allows the user to keep their familiar data analysis environment in the VMs, while we provide the coordination and data transfer services to ensure the necessary input and output are directed to the desired locations. This work extensively exercises the networking capability of the cloud computing systems and has revealed a number of weaknesses in the current cloud system software. In our tests, we are able to scale the parallel data analysis job to a modest number of VMs and achieve a speedup that is comparable to running the same analysis task using MPI. However, compared to MPI based parallelization, the cloud-based approach has a number of advantages. The cloud-based approach is more flexible because the VMs can capture arbitrary software dependencies without requiring the user to rewrite their programs. The cloud-based approach is also more resilient to failure; as long as a single VM is running, it can make progress while as soon as one MPI node fails the whole analysis job fails. In short, this initial work demonstrates that a cloud computing system is a viable platform for distributed scientific data analyses traditionally conducted on dedicated supercomputing systems.

97 MATHEMATICS AND COMPUTING↗

Big Data, Big Solutions

Data—lots of data—generated in seconds and piling up on the internet, streaming and stored in countless databases. Big data is important for commerce, society and our nation’s security. Yet the volume, velocity, variety and veracity of data is simply too great for any single analyst to make sense of alone. It requires advanced, data-intensive computing. Simply put, data-intensive computing is the use of sophisticated computers to sort through mounds of information and present analysts with solutions in the form of graphics, scenarios, formulas, new hypotheses and more. This scientific capability is foundational to PNNL’s energy, environment and security missions. Senior Scientist and Division Director Bill Pike and his team are developing analytic tools that are used to solve important national challenges, including cyber systems defense, power grid control systems, intelligence analysis, climate change and scientific exploration.

97 MATHEMATICS AND COMPUTING↗

Non-stationary precipitation design standards for stormwater infrastructure modernization at USAF installations

The resilience of defense infrastructure systems to a changing climate is critical for national security. Climate induced recurrent flooding is already impacting over 20 U.S. Air Force installations, underscoring the urgency of revisiting precipitation standards and stormwater infrastructure design. Despite growing scientific knowledge and an expanding set of tools for updating outdated precipitation standards based on the assumption of climate stationarity, the adoption of climate informed analyses remain limited in practice. This study utilizes an existing framework to update Intensity (or Depth)-Duration-Frequency (DDF) curves using an ensemble of future climate projections. Change factors in precipitation estimates are derived and applied to six USAF installations across the U.S. The analysis is further extended to evaluate the implications of climate-informed DDFs on stormwater infrastructure performance and flood analysis at Tyndall AFB. Results indicate that the current design precipitation estimates are likely to become obsolete in all six USAF bases by the end of the century. The wide range of change factors across 32 GCM ensembles highlights the need to integrate uncertainty and evolving scientific data into infrastructure planning. The study also finds that the impacts of a changing climate vary spatially and temporally, emphasizing the value of localized analysis for infrastructure decision-making. The work advances ongoing DoD and societal efforts to implement adaptation strategies aimed at enhancing infrastructure resilience.

Intensity-duration-frequency curves↗

A Topological Framework for the Interactive Exploration of Large Scale Turbulent Combustion

The advent of highly accurate, large scale volumetric simulations has made data analysis and visualization techniques an integral part of the modern scientific process. To develop new insights from raw data, scientists need the ability to define features of interest in a flexible manner and to understand how changes in the feature definition impact the subsequent analysis of the data. Therefore, simply exploring the raw data is not sufficient. This paper presents a new topological framework for the analysis of large scale, time-varying, turbulent combustion simulations. It allows the scientists to explore interactively the complete parameter space of fuel consumption thresholds for an entire time-dependent combustion simulation. By computing augmented merge trees and their corresponding data segmentations, the system allows the user complete flexibility to segment, select, and track burning cells through time thanks to a linked view interface. We developed this technique in the context of low-swirl turbulent pre-mixed flame simulation analysis, where the topological abstractions enable an efficient tracking through time of the burning cells and provide new qualitative and quantitative insights into the dynamics of the combustion process.

99 GENERAL AND MISCELLANEOUS↗

Community Organizations: Changing the Culture in Which Research Software Is Developed and Sustained

Software is the key crosscutting technology that enables advances in mathematics, computer science, and domain-specific science and engineering to achieve robust simulations and analysis for science, engineering, and other research fields. However, software itself has not traditionally received focused attention from research communities; rather, software has evolved organically and inconsistently, with its development largely as by-products of other initiatives. Moreover, challenges in scientific software are expanding due to disruptive changes in computer hardware, increasing scale and complexity of data, and demands for more complex simulations involving multiphysics, multiscale modeling and outer-loop analysis. In recent years, community members have established a range of grass-roots organizations and projects to address these growing technical and social challenges in software productivity, quality, reproducibility, and sustainability. This article provides an overview of such groups and discusses opportunities to leverage their synergistic activities while nurturing work toward emerging software ecosystems.

97 MATHEMATICS AND COMPUTING↗

Scientific Analysis Is Essential to Assess Biofuel Policy Effects

Land-use change (LUC) estimated by economic models has sparked intense international debate. Models estimate how much LUC might be induced under prescribed scenarios and rely on assumptions to generate LUC values. It is critical to test and validate underlying assumptions with empirical evidence. Furthermore, this modeling approach cannot answer if any specific indirect effects are actually caused by biofuel policy. The best way to resolve questions of causation is via scientific methods. Kim and Dale attempt to address the question of if, rather than how much, market-induced land-use change is currently detectable based on the analysis of historic evidence, and in doing so, explore some modeling assumptions behind the drivers of change. Given that there is no accepted approach to estimate the global effects of biofuel policy on land-use change, it is critical to assess the actual effects of policies through careful analysis and interpretation of empirical data. Decision makers need a valid scientific basis for policy decisions on energy choices.

54 ENVIRONMENTAL SCIENCES↗

Issues in using landscape indicators to assess land changes

Landscape indicators, when combined with information about environmental conditions (such as habitat potential, biodiversity, carbon and nutrient cycling, and erosion) and socioeconomic forces, can provide insights about changing ecosystem services. They also provide information about opportunities for improving natural resources management. Landscape indicators rely on data regarding land cover, land management and land functionality. Challenges in using landscape indicators to assess change and effects include (1) measures of land management and attributes that are reliable, robust and consistent for all areas on the Earth do not exist, and thus land cover is more frequently utilized; (2) multiple types of land cover and management are often found within a single landscape and are constantly changing, which complicates measurement and interpretation; and (3) while causal analysis is essential for understanding and interpreting changes in indicator values, the interactions among multiple causes and effects over time make accurate attribution among many drivers of change particularly difficult. Because of the complexity, sheer number of variables, and limitations of empirical data on land changes, models are often used to illustrate and estimate values for landscape indicators, and those models have several problems. Recommendations to improve our ability to assess the effects of changes in land management include refinement of questions to be more consistent with available information and the development of data sets based on systematic measurement over time of spatially explicit land qualities such as carbon and nutrient stocks, water and soil quality, net primary productivity, habitat and biodiversity. Well-defined and consistent land-classification systems that are capable of tracking changes in these and other qualities that matter to society need to be developed and deployed. Because landscapes are so dynamic, it is crucial to develop ways for the scientific community to work together to collect data and develop tools that will enable better analysis of causes and effects and to develop robust management recommendations that will increases land s capacity to meet societal needs in a changing world.

Kline, Keith L↗

Combining In-situ and In-transit Processing to Enable Extreme-Scale Scientific Analysis

With the onset of extreme-scale computing, I/O constraints make it increasingly difficult for scientists to save a sufficient amount of raw simulation data to persistent storage. One potential solution is to change the data analysis pipeline from a post-process centric to a concurrent approach based on either in-situ or in-transit processing. In this context computations are considered in-situ if they utilize the primary compute resources, while in-transit processing refers to offloading computations toa set of secondary resources using asynchronous data transfers. In this paper we explore the design and implementation of three common analysis techniques typically performed on large-scale scientific simulations: topological analysis, descriptive statistics, and visualization. We summarize algorithmic developments, describe a resource scheduling system to coordinate the execution ofvarious analysis workflows, and discuss our implementation using the DataSpaces and ADIOS frameworks that support efficient data movement between in-situ and in-transit computations. Wedemonstrate the efficiency of our lightweight, flexible framework by deploying it on the Jaguar XK6 to analyze data generated by S3D, a massively parallel turbulent combustion code. Our framework allows scientists dealing with the data deluge at extreme scale to perform analyses at increased temporal resolutions, mitigate I/O costs, and significantly improve the time to insight.

Bennett, Janine C.↗

NASA GISS Surface Temperature (GISTEMP) Analysis

The NASA GISS Surface Temperature (GISTEMP) analysis provides a measure of the changing global surface temperature with monthly resolution for the period since 1880, when a reasonably global distribution of meteorological stations was established. The input data that the GISTEMP Team use for the analysis, collected by many national meteorological services around the world, are the adjusted data of the Global Historical Climatology Network (GHCN) Vs. 3 (this represents a change from prior use of unadjusted Vs. 2 data) (Peterson and Vose, 1997 and 1998), United States Historical Climatology Network (USHCN) data, and SCAR (Scientific Committee on Antarctic Research) data from Antarctic stations. Documentation of the basic analysis method is provided by Hansen et al. (1999), with several modifications described by Hansen et al. (2001). The GISS analysis is updated monthly, however CDIAC's presentation of the data here is updated annually.For access to the data files, click this link to the CDIAC data transition website: http://cdiac.ess-dive.lbl.gov/trends/temp/hansen/hansen.html

(GISTEMP)↗

Scaling the Earth System Grid to 100Gbps Networks

The SC11 demonstration, titled Scaling the Earth System Grid to 100Gbps Networks, showed the ability to use underlying infrastructure for the movement of climate data over 100Gbps network. Climate change research is one of the critical data intensive sciences, and the amount of data is continuously growing. Climate simulation data is geographically distributed over the world, and it needs to be accessed from many sources for fast and efficient analysis and inter-comparison of simulations. We used a 100Gbps link connecting National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory (LBNL), Argonne National Laboratory (ANL) and Oak Ridge National Laboratory (ORNL). In the demo, the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report (AR4) phase 3 of the Coupled Model Intercomparison Project (CMIP-3) dataset was staged into the memory of computing nodes at ANL and ORNL from NERSC over the 100Gbps network for analysis and visualization. In general, climate simulation data consists of relatively small and large files with irregular file size distribution in each dataset. In this demo, we addressed challenges on data management in terms of high bandwidth networks, usability of existing protocols and middleware tools, and how applications can adapt and benefit from next generation networks.

97 MATHEMATICS AND COMPUTING↗

Western Regional Center of the National Institute for Climatic Change Research

The major goal of this project was fostering, integrating, synthesizing, and disseminating experimental, observational, and modeling research on predicted climate change in the western region of the U.S. and the impacts of that change on the structure, productivity, and climatic interactions of the region's natural and managed ecological systems. This was accomplished through administering a competitive grants program developed in collaboration with the other four regional centers of the NICCR. The activities supported included efforts to synthesize research on climate change in the western U.S. through meta-analysis studies, model comparisons, and data synthesis workshops. Results from this work were disseminated to the scientific and public media. This project also supported the development of the NICCR web site, hosted at NAU, which was used as the means to accept pre-proposal and proposal submissions for each funding cycle, and served as a clearing house for public outreach for results from NICCR-funded research

54 ENVIRONMENTAL SCIENCES↗

Tropospheric Ozone Assessment Report: Database and Metrics Data of Global Surface Ozone Observations

In support of the first Tropospheric Ozone Assessment Report (TOAR) a relational database of global surface ozone observations has been developed and populated with hourly measurement data and enhanced metadata. A comprehensive suite of ozone data products including standard statistics, health and vegetation impact metrics, and trend information, are made available through a common data portal and a web interface. These data form the basis of the TOAR analyses focusing on human health, vegetation, and climate relevant ozone issues, which are part of this special feature. Cooperation among many data centers and individual researchers worldwide made it possible to build the world's largest collection of in-situ hourly surface ozone data covering the period from 1970 to 2015. By combining the data from almost 10,000 measurement sites around the world with global metadata information, new analyses of surface ozone have become possible, such as the first globally consistent characterisations of measurement sites as either urban or rural/remote. Exploitation of these global metadata allows for new insights into the global distribution, and seasonal and long-term changes of tropospheric ozone and they enable TOAR to perform the first, globally consistent analysis of present-day ozone concentrations and recent ozone changes with relevance to health, agriculture, and climate. Considerable effort was made to harmonize and synthesize data formats and metadata information from various networks and individual data submissions. Extensive quality control was applied to identify questionable and erroneous data, including changes in apparent instrument offsets or calibrations. Such data were excluded from TOAR data products. Limitations of a posteriori data quality assurance are discussed. As a result of the work presented here, global coverage of surface ozone data for scientific analysis has been significantly extended. Yet, large gaps remain in the surface observation network both in terms of regions without monitoring, and in terms of regions that have monitoring programs but no public access to the data archive. Therefore future improvements to the database will require not only improved data harmonization, but also expanded data sharing and increased monitoring in data-sparse regions.

54 ENVIRONMENTAL SCIENCES↗

The AmeriFlux Data Activity and Data System: An Evolving Collection of Data Management Techniques, Tools, Products and Services

The Carbon Dioxide Information Analysis Center (CDIAC) at Oak Ridge National Laboratory (ORNL), USA has provided scientific data management support for the U.S. Department of Energy and international climate change science since 1982. Over this period, climate change science has expanded from research focusing on basic understanding of geochemical cycles, particularly the carbon cycle, to integrated research addressing climate change impacts, vulnerability, adaptation, and mitigation. Interests in climate change data and information worldwide have grown remarkably and, as a result, so have demands and expectations for CDIAC’s data systems. To meet the growing demands, CDIAC’s strategy has been to design flexible data systems using proven technologies blended with new, evolving technologies and standards. CDIAC development teams are multidisciplinary and include computer science and information technology expertise, but also scientific expertise necessary to address data quality and documentation issues and to identify data products and system capabilities needed by climate change scientists. CDIAC has learned there is rarely a single commercial tool or product readily available to satisfy long-term scientific data system requirements (i.e., one size does not fit all and the breadth and diversity of environmental data are often too complex for easy use with commercial products) and typically deploys a variety of tools and data products in an effort to provide credible data freely to users worldwide. Like many scientific data management applications, CDIAC’s data systems are highly customized to satisfy specific scientific usage requirements (e.g., developing data products specific for model use) but are also designed to be flexible and interoperable to take advantage of new software engineering techniques, standards (e.g., metadata standards) and tools and to support future Earth system data efforts (e.g., ocean acidification). CDIAC has provided data management support for numerous long-term measurement projects crucial to climate change science. One current example is the AmeriFlux measurement network. AmeriFlux provides continuous measurements from forests, grasslands, wetlands, and croplands in North, Central, and South America and offers important insight about carbon cycling in terrestrial ecosystems. We share our approaches in satisfying the challenges of delivering AmeriFlux data worldwide to benefit others with similar challenges handling climate change data, further heighten awareness and use of an outstanding ecological data resource, and highlight expanded software engineering applications being used for climate change measurement data.

Boden, Thomas A.↗

Benchmark Dose Analysis of DNA Damage Biomarker Responses Provides Compound Potency and Adverse Outcome Pathway Information for the Topoisomerase II Inhibitor Class of Compounds

Genetic toxicology data have traditionally been utilized for hazard identification to provide a binary call for a compound's risk. Recent advances in the scientific field, especially with the development of high‐throughput methods to quantify DNA damage, have influenced a change of approach in genotoxicity assessment. The in vitro MultiFlow® DNA Damage Assay is one such method which multiplexes γH2AX, p53, phospho‐histone H3 biomarkers into a single‐flow cytometric analysis (Bryce et al., [2016]: Environ Mol Mutagen 57:546–558). This assay was used to study human TK6 cells exposed to each of eight topoisomerase II poisons for 4 and 24 hr. Using PROAST v65.5, the Benchmark Dose approach was applied to the resulting flow cytometric datasets. With “compound” serving as covariate, all eight compounds were combined into a single analysis, per time point and endpoint. The resulting 90% confidence intervals, plotted in Log scale, were considered as the potency rank for the eight compounds. The in vitro MultiFlow data showed a maximum confidence interval span of 1Log, which indicates data of good quality. Patterns observed in the compound potency rank were scrutinized by using the expert rule‐based software program Derek Nexus, developed by Lhasa Limited. Compound sub‐classification and structural alerts were considered contributory to the potencies observed for the topoisomerase II poisons studied herein. The Topo II poison Adverse Outcome Pathway was evaluated with MultiFlow endpoints serving as Key Events. The step‐wise approach described herein can be considered as a foundation for risk assessment of compounds within a specific mode of action of interest. Environ. Mol. Mutagen. 2020. © 2020 Wiley Periodicals, Inc.

Wheeldon, Ryan P.↗

Measurement Uncertainty in One-Of-A-Kind Event Data Analysis

A golden standard in science is to repeat an experiment a statistically significant number of times, recording data using the same set of detectors and the same data analysis methodology. In such case experimental error includes both the range of true values generated by repetitions of the experiment, and measurement uncertainty caused by the detector. They are independent. It is a huge and too frequently used simplification, to assume that one can measure multiple repetitions of an identical experiment, resulting in identical true experimental value. Repetitions, as similar is it is experimentally achievable, have unavoidable built-in differences resulting in a range of the true values rather than in a single value. When modern, very sensitive and well calibrated measurement systems are used, this range is not negligible, and sometimes dominates over the measurement uncertainty. Range of true values depends on built-in differences in physics of the experiment. Stochastic physical processes result typically in a broader range of true values than non-stochastic processes do. Measurement uncertainty depends on a measurement method (properties of the detector not of the experiment). Modern measurement methods, including digital ones, frequently make the measurement uncertainty very small. When data from one–of –a kind experiment are analyzed, only the measurement uncertainty is reported. It provides no information about the range of true experimental values, neither about reliability of a reported data point. Reliability of a data point is in general independent from its measurement uncertainty. However, in practice reliable measurement methods frequently have high measurement uncertainty, while low reliability methods are applied to limit measurement uncertainty. Comparison of reliable data with high measurement uncertainty to not so reliable data measured with low uncertainty is discussed – in different scenarios different data analysis methods are applicable. Methods for data analysis from an experiment repeated statistically significant number of times are very well developed. They do not require a detailed expertise in physics of an experiment, nor in the properties of the measurement system used, and meaning of the reported uncertainty is well understood in any scientific community. It all changes when data from one-of-a-kind experiment is analyzed. Analyst’s expertise is required both in the physics of the experiment and in all aspects of the measurement system, all possible malfunctions. Data users must remember that only measurement uncertainty is reported from any one-of-a-kind experiment. Theory with simulations may provide estimation of expected built-in differences in the experiment, and by this of expected range of true values for a given experiment; yet measurement uncertainty can never be used in place of the range of true experimental values.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗