Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “scientific data change analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Assessing data change in scientific datasets

Summary Scientific datasets are growing rapidly and becoming critical to next‐generation scientific discoveries. The validity of scientific results relies on the quality of data used and data are often subject to change, for example, due to observation additions, quality assessments, or processing software updates. The effects of data change are not well understood and difficult to predict. Datasets are often repeatedly updated and recomputing derived data products quickly becomes time consuming and resource intensive and may in some cases not even be necessary, thus delaying scientific advance. Despite its importance, there is a lack of systematic approaches for best comparing data versions to quantify the changes, and ad‐hoc or manual processes are commonly used. In this article, we propose a novel hierarchical approach for analyzing data changes, including real‐time (online) and offline analyses. We employ a variety of fast‐to‐compute numerical analyses, graphical data change representations, and more resource‐intensive recomputations of a subset of the data product. We illustrate the application of our approach using three scientific diverse use cases, namely, satellite, cosmological, and x‐ray data. The results show that a variety of data change metrics should be employed to enable a comprehensive representation and qualitative evaluation of data changes.

97 MATHEMATICS AND COMPUTING↗

Anticipated Changes in Conducting Scientific Data-Analysis Research in the Big-Data Era

A Big-Data environment is one that is capable of orchestrating quick-turnaround analyses involving large volumes of data for numerous simultaneous users. Based on our experiences with a prototype Big-Data analysis environment, we anticipate some important changes in research behaviors and processes while conducting scientific data-analysis research in the near future as such Big-Data environments become the mainstream. The first anticipated change will be the reduced effort and difficulty in most parts of the data management process. A Big-Data analysis environment is likely to house most of the data required for a particular research discipline along with appropriate analysis capabilities. This will reduce the need for researchers to download local copies of data. In turn, this also reduces the need for compute and storage procurement by individual researchers or groups, as well as associated maintenance and management afterwards. It is almost certain that Big-Data environments will require a different "programming language" to fully exploit the latent potential. In addition, the process of extending the environment to provide new analysis capabilities will likely be more involved than, say, compiling a piece of new or revised code.We thus anticipate that researchers will require support from dedicated organizations associated with the environment that are composed of professional software engineers and data scientists. A major benefit will likely be that such extensions are of higherquality and broader applicability than ad hoc changes by physical scientists. Another anticipated significant change is improved collaboration among the researchers using the same environment. Since the environment is homogeneous within itself, many barriers to collaboration are minimized or eliminated. For example, data and analysis algorithms can be seamlessly shared, reused and re-purposed. In conclusion, we will be able to achieve a new level of scientific productivity in the Big-Data analysis environments.

Kuo, Kwo-Sen↗

Earth Science Data and Models for Improved Targeting of Humanitarian Aid

Humanitarian assistance to developing countries has long focused on countries that have political, economic and strategic interest to the United States. Recent changes in global security concerns have heightened the perception that humanitarian action is becoming increasingly politicized. This is seen to be largely driven by the 'global war on terror' along with a push by donors and the United Nations for closer integration between humanitarian action and diplomatic, military and other spheres of engagement in conflict and crisis-affected states (HPG 2010). As we enter an era of rising commodity prices and increasing uncertainty in global food production due to a changing climate, scientific data and analysis will be increasingly important to improve the targeting of humanitarian assistance. Earth science data enables appropriate humanitarian response to complex food emergencies that arise in regions outside the areas of current strategic and security focus. As the climate changes, new places will become vulnerable to food insecurity and will need emergency assistance. Earth science data and multidisciplinary models will enable an information-based comparison of need that goes beyond strategic and political considerations to identify new hotspots of food insecurity as they emerge. These analyses will improve aid targeting and timeliness while reducing strategic risk by highlighting new regions at risk of crisis in a rapidly changing world. Improved targeting with respect to timing and location could reduce cost while increasing the likelihood that those who need aid get it.

Brown, Molly E.↗

Randomized Algorithms for Scientific Computing (RASC)

Randomized algorithms have propelled advances in artificial intelligence (AI) and represent a foundational research area in advancing AI for Science. Future advancements in DOE Office of Science priority areas such as climate science, astrophysics, fusion, advanced materials, combustion, and quantum computing all require randomized algorithms for surmounting challenges of complexity, robustness, and scalability. Advances in data collection and numerical simulation have changed the dynamics of scientific research and motivate the need for randomized algorithms. For instance, advances in imaging technologies such as X-ray ptychography, electron microscopy, electron energy loss spectroscopy, or adaptive optics lattice light-sheet microscopy collect hyperspectral imaging and scattering data in terabytes, at breakneck speed enabled by state-of-the-art detectors. The data collection is exceptionally fast compared with its analysis. Likewise, advances in high-performance architectures have made exascale computing a reality and changed the economies of scientific computing in the process. Floating-point operations that create data are essentially free in comparison with data movement. Thus far, most approaches have focused on creating faster hardware. Ironically, this faster hardware has exacerbated the problem by making data still easier to create. Under such an onslaught, scientists often resort to heuristic deterministic sampling schemes (e.g., low-precision arithmetic, sampling every nth element) and sacrifice potentially valuable accuracy. Dramatically better results can be achieved via randomized algorithms, reducing the data size as much as or more than naive deterministic subsampling can achieve, while retaining the high accuracy of computing on the full data set. By randomized algorithms we mean those algorithms that employ some form of randomness in internal algorithmic decisions to accelerate time to solution, increase scalability, or improve reliability. Examples include matrix sketching for solving large-scale least-squares problems (see Figure 1) and stochastic gradient descent for training machine learning models. We are not recommending heuristic methods but rather randomized algorithms that have certificates of correctness and probabilistic guarantees of optimality and near-optimality. Such approaches can be useful beyond acceleration, for example, in understanding how to avoid measure zero worst-case scenarios that plague methods such as QR matrix factorization.

97 MATHEMATICS AND COMPUTING↗

Finding Tropical Cyclones on a Cloud Computing Cluster: Using Parallel Virtualization for Large-Scale Climate Simulation Analysis

Extensive computing power has been used to tackle issues such as climate changes, fusion energy, and other pressing scientific challenges. These computations produce a tremendous amount of data; however, many of the data analysis programs currently only run a single processor. In this work, we explore the possibility of using the emerging cloud computing platform to parallelize such sequential data analysis tasks. As a proof of concept, we wrap a program for analyzing trends of tropical cyclones in a set of virtual machines (VMs). This approach allows the user to keep their familiar data analysis environment in the VMs, while we provide the coordination and data transfer services to ensure the necessary input and output are directed to the desired locations. This work extensively exercises the networking capability of the cloud computing systems and has revealed a number of weaknesses in the current cloud system software. In our tests, we are able to scale the parallel data analysis job to a modest number of VMs and achieve a speedup that is comparable to running the same analysis task using MPI. However, compared to MPI based parallelization, the cloud-based approach has a number of advantages. The cloud-based approach is more flexible because the VMs can capture arbitrary software dependencies without requiring the user to rewrite their programs. The cloud-based approach is also more resilient to failure; as long as a single VM is running, it can make progress while as soon as one MPI node fails the whole analysis job fails. In short, this initial work demonstrates that a cloud computing system is a viable platform for distributed scientific data analyses traditionally conducted on dedicated supercomputing systems.

97 MATHEMATICS AND COMPUTING↗

Visualization: A pathway to enhanced scientific productivity in the expanding missions of Space and Earth Sciences

The movement toward the solution of problems involving large-scale system science, the ever-increasing capabilities of three-dimensional, time-dependent numerical models, and the enhanced capabilities of 'in situ' and remote sensing instruments bring a new era of scientific endeavor that requires an important change in our approach to mission planning and the task of data reduction and analysis. Visualization is at the heart of the requirements for a much-needed enhancement in scientific productivity as we face these new challenges. This article draws a perspective on the problem as it crosses discipline boundaries from solar physics to atmospheric and ocean sciences. It also attempts to introduce visualization as a new approach to scientific discovery and a tool which expedites and improves our insight into physically complex problems. A set of simple illustrations demonstrates a number of visualization techniques and the discussion emphasizes the trial-and-error and search-and-discover modes that are necessary for the techniques to reach their full potential. Further discussions also point to the importance of integrating data access, management, mathematical operations, and visualization into a single system. Some of the more recent developments in this area are reviewed.

Szuszczewicz, E. P.↗

How the Open Data Policy of the Landsat Program Has Advanced Our Understanding of Environmental Change

A time series is a sequence of observations of a phenomenon taken sequentially in time. A crucial characteristic of a time series is the dependence among adjacent observations – techniques for analyzing this dependence are referred to as time series analysis. This analytical approach enables us to predict or forecast future values of a time series, study the impact of various inputs on the observed phenomenon, and examine interrelationships among related time series variables. Within the geographical sciences, time series analysis has historically been limited to coarse-resolution satellite data, as constructing time series of data suitable for studying land cover and land use dynamics, such as Landsat data, were prohibitively costly. A transformative shift occurred in 2008 when the U.S. Government decided to make free and open all past and future data collected by the Landsat satellite program. The decision brought about a paradigm shift away from analyzing individual images or observations to continuous monitoring in time. Of particular relevance to environmental remote sensing is the ability to forecast observations – if we can predict how future observations should behave, we can infer information about how the land surface is changing. In this presentation, we will examine literature examples that showcase scientific gains enabled by time series analysis of satellite data. We will delve into how the analysis of dense time series of satellite data revealed that overall rate of forest disturbance in the Amazon has increased despite a reduction in deforestation; how different types of forest degradation, previously unquantified, are now being accurately assessed in the Caucasus region; and how we now can study the highly dynamic and intricate patterns of shifting cultivation in Southeast Asia.

Pontus Olofsson↗

Big Data, Big Solutions

Data—lots of data—generated in seconds and piling up on the internet, streaming and stored in countless databases. Big data is important for commerce, society and our nation’s security. Yet the volume, velocity, variety and veracity of data is simply too great for any single analyst to make sense of alone. It requires advanced, data-intensive computing. Simply put, data-intensive computing is the use of sophisticated computers to sort through mounds of information and present analysts with solutions in the form of graphics, scenarios, formulas, new hypotheses and more. This scientific capability is foundational to PNNL’s energy, environment and security missions. Senior Scientist and Division Director Bill Pike and his team are developing analytic tools that are used to solve important national challenges, including cyber systems defense, power grid control systems, intelligence analysis, climate change and scientific exploration.

97 MATHEMATICS AND COMPUTING↗

Linking Indigenous Knowledge and Observed Climate Change Studies

We present indigenous knowledge narratives and explore their connections to documented temperature and other climate changes and observed climate change impact studies. We then propose a framework for enhancing integration of these indigenous narratives of observed climate change with global assessments. Our aim is to contribute to the thoughtful and respectful integration of indigenous knowledge with scientific data and analysis, so that this rich body of knowledge can inform science, and so that indigenous and traditional peoples can use the tools and methods of science for the benefit of their communities if they choose to do so. Enhancing ways of understanding such connections are critical as the Intergovernmental Panel on Climate Change Fifth Assessment process gets underway.

Alexander, Chief Clarence↗

Collaborative Data Curation to Support the Multi-Mission Algorithm and Analysis Platform (MAAP)

Upcoming space-borne missions will offer unprecedented data about Earth but will also feature exponentially high data volumes. These high data volumes will change the way the scientific community works with data and will also create a unique need for improved data sharing and collaboration. NASA and ESA are working together to address these issues by collaboratively developing the Multi-Mission Algorithm and Analysis Platform (MAAP) to improve the understanding of global aboveground terrestrial carbon dynamics. The MAAP will support ESA’s BIOMASS mission, NASA’s GEDI mission and NASA/ISRO’s NISAR mission. The MAAP will be developed in two phases: a pilot phase and a full production phase. The pilot phase will demonstrate collaboration and basic capabilities. The pilot phase will focus on biomass relevant airborne and field campaign data. Two NASA teams are supporting the development of the MAAP. The MAAP engineering team is responsible for the development, maintenance and operations of the MAAP system while the MAAP data team ensures the ongoing quality of the data, metadata and other information provided in the MAAP. The MAAP data team also supports the ingest and archive of identified data to the MAAP platform. This poster describes the use case development process for the pilot MAAP and the data curated in support of those use cases. Additionally, this presentation will outline the pilot MAAP data ingest process and metadata curation effort along with efforts to ensure interoperability between ESA and NASA data and metadata.

Bugbee, Kaylin↗

Non-stationary precipitation design standards for stormwater infrastructure modernization at USAF installations

The resilience of defense infrastructure systems to a changing climate is critical for national security. Climate induced recurrent flooding is already impacting over 20 U.S. Air Force installations, underscoring the urgency of revisiting precipitation standards and stormwater infrastructure design. Despite growing scientific knowledge and an expanding set of tools for updating outdated precipitation standards based on the assumption of climate stationarity, the adoption of climate informed analyses remain limited in practice. This study utilizes an existing framework to update Intensity (or Depth)-Duration-Frequency (DDF) curves using an ensemble of future climate projections. Change factors in precipitation estimates are derived and applied to six USAF installations across the U.S. The analysis is further extended to evaluate the implications of climate-informed DDFs on stormwater infrastructure performance and flood analysis at Tyndall AFB. Results indicate that the current design precipitation estimates are likely to become obsolete in all six USAF bases by the end of the century. The wide range of change factors across 32 GCM ensembles highlights the need to integrate uncertainty and evolving scientific data into infrastructure planning. The study also finds that the impacts of a changing climate vary spatially and temporally, emphasizing the value of localized analysis for infrastructure decision-making. The work advances ongoing DoD and societal efforts to implement adaptation strategies aimed at enhancing infrastructure resilience.

Intensity-duration-frequency curves↗

A Topological Framework for the Interactive Exploration of Large Scale Turbulent Combustion

The advent of highly accurate, large scale volumetric simulations has made data analysis and visualization techniques an integral part of the modern scientific process. To develop new insights from raw data, scientists need the ability to define features of interest in a flexible manner and to understand how changes in the feature definition impact the subsequent analysis of the data. Therefore, simply exploring the raw data is not sufficient. This paper presents a new topological framework for the analysis of large scale, time-varying, turbulent combustion simulations. It allows the scientists to explore interactively the complete parameter space of fuel consumption thresholds for an entire time-dependent combustion simulation. By computing augmented merge trees and their corresponding data segmentations, the system allows the user complete flexibility to segment, select, and track burning cells through time thanks to a linked view interface. We developed this technique in the context of low-swirl turbulent pre-mixed flame simulation analysis, where the topological abstractions enable an efficient tracking through time of the burning cells and provide new qualitative and quantitative insights into the dynamics of the combustion process.

99 GENERAL AND MISCELLANEOUS↗

Community Organizations: Changing the Culture in Which Research Software Is Developed and Sustained

Software is the key crosscutting technology that enables advances in mathematics, computer science, and domain-specific science and engineering to achieve robust simulations and analysis for science, engineering, and other research fields. However, software itself has not traditionally received focused attention from research communities; rather, software has evolved organically and inconsistently, with its development largely as by-products of other initiatives. Moreover, challenges in scientific software are expanding due to disruptive changes in computer hardware, increasing scale and complexity of data, and demands for more complex simulations involving multiphysics, multiscale modeling and outer-loop analysis. In recent years, community members have established a range of grass-roots organizations and projects to address these growing technical and social challenges in software productivity, quality, reproducibility, and sustainability. This article provides an overview of such groups and discusses opportunities to leverage their synergistic activities while nurturing work toward emerging software ecosystems.

97 MATHEMATICS AND COMPUTING↗

Scientific Analysis Is Essential to Assess Biofuel Policy Effects

Land-use change (LUC) estimated by economic models has sparked intense international debate. Models estimate how much LUC might be induced under prescribed scenarios and rely on assumptions to generate LUC values. It is critical to test and validate underlying assumptions with empirical evidence. Furthermore, this modeling approach cannot answer if any specific indirect effects are actually caused by biofuel policy. The best way to resolve questions of causation is via scientific methods. Kim and Dale attempt to address the question of if, rather than how much, market-induced land-use change is currently detectable based on the analysis of historic evidence, and in doing so, explore some modeling assumptions behind the drivers of change. Given that there is no accepted approach to estimate the global effects of biofuel policy on land-use change, it is critical to assess the actual effects of policies through careful analysis and interpretation of empirical data. Decision makers need a valid scientific basis for policy decisions on energy choices.

54 ENVIRONMENTAL SCIENCES↗

Issues in using landscape indicators to assess land changes

Landscape indicators, when combined with information about environmental conditions (such as habitat potential, biodiversity, carbon and nutrient cycling, and erosion) and socioeconomic forces, can provide insights about changing ecosystem services. They also provide information about opportunities for improving natural resources management. Landscape indicators rely on data regarding land cover, land management and land functionality. Challenges in using landscape indicators to assess change and effects include (1) measures of land management and attributes that are reliable, robust and consistent for all areas on the Earth do not exist, and thus land cover is more frequently utilized; (2) multiple types of land cover and management are often found within a single landscape and are constantly changing, which complicates measurement and interpretation; and (3) while causal analysis is essential for understanding and interpreting changes in indicator values, the interactions among multiple causes and effects over time make accurate attribution among many drivers of change particularly difficult. Because of the complexity, sheer number of variables, and limitations of empirical data on land changes, models are often used to illustrate and estimate values for landscape indicators, and those models have several problems. Recommendations to improve our ability to assess the effects of changes in land management include refinement of questions to be more consistent with available information and the development of data sets based on systematic measurement over time of spatially explicit land qualities such as carbon and nutrient stocks, water and soil quality, net primary productivity, habitat and biodiversity. Well-defined and consistent land-classification systems that are capable of tracking changes in these and other qualities that matter to society need to be developed and deployed. Because landscapes are so dynamic, it is crucial to develop ways for the scientific community to work together to collect data and develop tools that will enable better analysis of causes and effects and to develop robust management recommendations that will increases land s capacity to meet societal needs in a changing world.

Kline, Keith L↗

Evaluation and optimization of lidar temperature analysis algorithms using simulated data

The middle atmosphere (20 to 90 km altitude) ha received increasing interest from the scientific community during the last decades, especially since such problems as polar ozone depletion and climatic change have become so important. Temperature profiles have been obtained in this region using a variety of satellite-, rocket-, and balloon-borne instruments as well as some ground-based systems. One of the more promising of these instruments, especially for long-term high resolution measurements, is the lidar. Measurements of laser radiation Rayleigh backscattered, or Raman scattered, by atmospheric air molecules can be used to determine the relative air density profile and subsequently the temperature profile if it is assumed that the atmosphere is in hydrostatic equilibrium and follows the ideal gas law. The high vertical and spatial resolution make the lidar a well adapted instrument for the study of many middle atmospheric processes and phenomena as well as for the evaluation and validation of temperature measurements from satellites, such as the Upper Atmosphere Research Satellite (UARS). In the Network for Detection of Stratospheric Change (NDSC) lidar is the core instrument for measuring middle atmosphere temperature profiles. Using the best lidar analysis algorithm possible is therefore of crucial importance. In this work, the JPL and CNRS/SA lidar analysis software were evaluated. The results of this evaluation allowed the programs to be corrected and optimized and new production software versions were produced. First, a brief description of the lidar technique and the method used to simulate lidar raw-data profiles from a given temperature profile is presented. Evaluation and optimization of the JPL and CNRS/SA algorithms are then discussed.

Leblanc, Thierry↗

Combining In-situ and In-transit Processing to Enable Extreme-Scale Scientific Analysis

With the onset of extreme-scale computing, I/O constraints make it increasingly difficult for scientists to save a sufficient amount of raw simulation data to persistent storage. One potential solution is to change the data analysis pipeline from a post-process centric to a concurrent approach based on either in-situ or in-transit processing. In this context computations are considered in-situ if they utilize the primary compute resources, while in-transit processing refers to offloading computations toa set of secondary resources using asynchronous data transfers. In this paper we explore the design and implementation of three common analysis techniques typically performed on large-scale scientific simulations: topological analysis, descriptive statistics, and visualization. We summarize algorithmic developments, describe a resource scheduling system to coordinate the execution ofvarious analysis workflows, and discuss our implementation using the DataSpaces and ADIOS frameworks that support efficient data movement between in-situ and in-transit computations. Wedemonstrate the efficiency of our lightweight, flexible framework by deploying it on the Jaguar XK6 to analyze data generated by S3D, a massively parallel turbulent combustion code. Our framework allows scientists dealing with the data deluge at extreme scale to perform analyses at increased temporal resolutions, mitigate I/O costs, and significantly improve the time to insight.

Bennett, Janine C.↗

Western Regional Center of the National Institute for Climatic Change Research

The major goal of this project was fostering, integrating, synthesizing, and disseminating experimental, observational, and modeling research on predicted climate change in the western region of the U.S. and the impacts of that change on the structure, productivity, and climatic interactions of the region's natural and managed ecological systems. This was accomplished through administering a competitive grants program developed in collaboration with the other four regional centers of the NICCR. The activities supported included efforts to synthesize research on climate change in the western U.S. through meta-analysis studies, model comparisons, and data synthesis workshops. Results from this work were disseminated to the scientific and public media. This project also supported the development of the NICCR web site, hosted at NAU, which was used as the means to accept pre-proposal and proposal submissions for each funding cycle, and served as a clearing house for public outreach for results from NICCR-funded research

54 ENVIRONMENTAL SCIENCES↗