SEARCH · Engineering Papers

Results for “big data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 721 records · Page 40

All sources NASA NTRS DOE OSTI

DOE OSTI2018

Use of computational modeling combined with advanced visualization to develop strategies for the design of crop ideotypes to address food security

Sustainable crop production is a contributing factor to current and future food security. Innovative technologies are needed to design strategies that will achieve higher crop yields on less land and with fewer resources. Computational modeling coupled with advanced scientific visualization enables researchers to explore and interact with complex agriculture, nutrition, and climate data to predict how crops will respond to untested environments. These virtual observations and predictions can direct the development of crop ideotypes designed to meet future yield and nutritional demands. This review surveys modeling strategies for the development of crop ideotypes and scientific visualization technologies that have led to discoveries in “big data” analysis. Combined modeling and visualization approaches have been used to realistically simulate crops and to guide selection that immediately enhances crop quantity and quality under challenging environmental conditions. Lastly, this survey of current and developing technologies indicates that integrative modeling and advanced scientific visualization may help overcome challenges in agriculture and nutrition data as large-scale and multidimensional data become available in these fields.

3D crop modeling↗

DOE OSTI2015

Toward Interactive Scenario Analysis and Exploration

As Modeling and Simulation (M&S) tools have matured, their applicability and importance have increased across many national security challenges. In particular, they provide a way to test how something may behave without the need to do real world testing. However, current and future changes across several factors including capabilities, policy, and funding are driving a need for rapid response or evaluation in ways that many M&S tools cannot address. Issues around large data, computational requirements, delivery mechanisms, and analyst involvement already exist and pose significant challenges. Furthermore, rising expectations, rising input complexity, and increasing depth of analysis will only increase the difficulty of these challenges. In this study we examine whether innovations in M&S software coupled with advances in ''cloud'' computing and ''big-data'' methodologies can overcome many of these challenges. In particular, we propose a simple, horizontally-scalable distributed computing environment that could provide the foundation (i.e. ''cloud'') for next-generation M&S-based applications based on the notion of ''parallel multi-simulation''. In our context, the goal of parallel multi- simulation is to consider as many simultaneous paths of execution as possible. Therefore, with sufficient resources, the complexity is dominated by the cost of single scenario runs as opposed to the number of runs required. We show the feasibility of this architecture through a stable prototype implementation coupled with the Umbra Simulation Framework [6]. Finally, we highlight the utility through multiple novel analysis tools and by showing the performance improvement compared to existing tools.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Physics-informed machine learning

Despite great progress in simulating multiphysics problems using the numerical discretization of partial differential equations (PDEs), one still cannot seamlessly incorporate noisy data into existing algorithms, mesh generation remains complex, and high-dimensional problems governed by parameterized PDEs cannot be tackled. Moreover, solving inverse problems with hidden physics is often prohibitively expensive and requires different formulations and elaborate computer codes. Machine learning has emerged as a promising alternative, but training deep neural networks requires big data, not always available for scientific problems. Instead, such networks can be trained from additional information obtained by enforcing the physical laws (for example, at random points in the continuous space-time domain). Such physics-informed learning integrates (noisy) data and mathematical models, and implements them through neural networks or other kernel-based regression networks. Moreover, it may be possible to design specialized network architectures that automatically satisfy some of the physical invariants for better accuracy, faster training and improved generalization. Furthermore, we review some of the prevailing trends in embedding physics into machine learning, present some of the current capabilities and limitations and discuss diverse applications of physics-informed learning both for forward and inverse problems, including discovering hidden physics and tackling high-dimensional problems.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2017

Exploiting analytics techniques in CMS computing monitoring

The CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been done, but are of crucial importance for a better understanding of how CMS did successful operations, and to reach an adequate and adaptive modelling of the CMS operations, in order to allow detailed optimizations and eventually a prediction of system behaviours. These data are now streamed into the CERN Hadoop data cluster for further analysis. Specific sets of information (e.g. data on how many replicas of datasets CMS wrote on disks at WLCG Tiers, data on which datasets were primarily requested for analysis, etc) were collected on Hadoop and processed with MapReduce applications profiting of the parallelization on the Hadoop cluster. We present the implementation of new monitoring applications on Hadoop, and discuss the new possibilities in CMS computing monitoring introduced with the ability to quickly process big data sets from mulltiple sources, looking forward to a predictive modeling of the system.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

NASA NTRSDate not supplied

Investigation and Evaluation of Advanced Spectrum Management Concepts for Aeronautical Communications

With the emergence of new aerial vehicles into the airspace and the continued growth of aviation operations, there will be an increasing demand for voice and data communications within the National Airspace System (NAS). The continued use of existing VHF and UHF frequency allocations is not a sustainable approach, and as a result, a new spectrum management solution is required to support future mission needs. The proposed spectrum management concepts leverage modern advancements such as artificial intelligence (AI) and big data to dynamically optimize the spectrum utilization based on the predicted communications demand throughout the airspace. This technical investigation considers both air-ground and air-air communications networks, and can be applied to both existing applications such as the air traffic control (ATC) system, as well as future applications, such as the emerging Advanced Air Mobility (AAM). To support the evaluation of the proposed concepts and technologies, a modeling and simulation capability is currently under development and will continue to evolve to support new and advanced airspace applications. It is anticipated that this proposed spectrum concept will better serve the spectrum needs of future NAS applications.

Eric J Knoblock↗

NASA NTRSDate not supplied

Investigation and Evaluation of Advanced Spectrum Management Concepts for Aeronautical Communications

Eric J Knoblock↗

DOE OSTI2021

Observational Capabilities to Capture Water Cycle Event Dynamics and Impacts in the Age of AI

This whitepaper is responsive to focal area (1) Data acquisition and assimilation enabled by machine learning (ML), Artificial Intelligence (AI), and advanced methods. Here we describe how Earth observations specific to water cycle disturbances can be collected in parallel with and integrated into future model development, and make use of the latest technologies other than AI/ ML such as 5G/satellite, edge computing, big data technologies, and cloud computing.

58 GEOSCIENCES↗

DOE OSTI2018

Coupled Approximation of U.S. Driving Speed and Volume Statistics using Spatial Conflation and Temporal Disaggregation

The advent of mobile devices with embedded global positioning systems has allowed commercial providers of real-time traffic data to develop highly accurate estimates of network-level vehicle speeds. Traffic speed data have far outpaced the availability and accuracy of real-time traffic volume information. Limited to a relatively small number of permanent and temporary traffic counters in any city, traffic volumes typically only cover a handful of roadways, with inconsistent temporal resolution. This work addressed this data gap by coupling a commercial data set of typical traffic speeds (by roadway and time of week) from TomTom to the U.S. Federal Highway Administration's Highway Performance Monitoring System database of annual average daily traffic (AADT) counts by roadway. This work is technically novel in its solution for establishing a national crosswalk between independent network geometries using spatial conflation and big data techniques. The resulting product is a national data set providing traffic speed and volume estimates under typical conditions for all U.S. roadways with AADT values.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2023

The disCO2ver Platform: Curating Data and Tools for Geologic Carbon Sequestration and Deep Subsurface Research Systems

The U.S. DOE National Energy Technology Laboratory has invested 12+ years of development into the data repository and digital laboratory, the Energy Data eXchange (EDX, edx.netl.doe.gov). Supporting a variety of research areas across the DOE Office of Fossil Energy and Carbon Management, the platform has successfully curated and preserved thousands of data products from DOE research. The Carbon Storage Program has successfully supported data curation, upload, and publishing of data products on EDX for many years, demonstrating a success story of how resources like EDX can effectively help with long term preservation and publishing of DOE data products. EDX continues to shift towards cloud-supported infrastructure, taking a hybrid approach combining on-premises compute and storage integrated with cloud-hosted services. The integration of cloud compute and hybrid architecture enables the development of EDX-hosted platforms that tailor the data and tools hosted on them to a specific community, enables implementation of machine learning tools for data discovery and filtering, and enables the hosting of virtual (online user interface) tools. Geologic carbon sequestration (GCS) research continues to scale up in response to the current administration goals to reduce greenhouse gas emissions and transition the energy economy. Over the last year, EDX’s disCO2ver platform has been developed in response to the need for access to data products and tools to support the scaling up of GCS research. disCO2ver provides access to data resources and tools, produced by DOE and outside authoritative external resources. The platform also provides a user-access control component for the virtualization and cloud hosting of tools. Tools that need to be virtualized, to eliminate the need for users to download the tool and use local compute resources, is essential to supporting big-data analysis and machine learning that is becoming common place in carbon storage modeling, risk analysis, and data publishing practices. This talk will review the EDX’s disCO2ver platform and the current work ongoing to curate data and tools to support GCS and deep subsurface systems research.

Morkner, Paige↗

DOE OSTI2018

Methods and Models for Analyzing Methane Sorption Capacity of Coal Based on Its Physicochemical Characteristics

The authors study the influence of physicochemical parameters on methane adsorption capacity of coal and offer the analytical method for the methane adsorption capacity for three-phased condition of methane. It is found that in the depth interval to 300 m below surface, methane adsorption capacity measured in lab can exceed natural gas content of coal obtained from geological exploration data by 30%, and the change in the thermodynamic condition of coal–methane system brings irreversible physicochemical consequences in terms of the altered ratios of physical states of the main components. There is no linear connection between natural gas content of a coal bed and its methane adsorption capacity with respect to occurrence depth. The application of Big Data in treatment and interpretation of large data flows is described. The theoretical data predicted using the proposed method and the experimental data on methane content of Kuzbass coal agree.

01 COAL, LIGNITE, AND PEAT↗

DOE OSTI2020

Database Management Module Framework for Dynamic Contingency Analysis and Visualization

The Dynamic Contingency Analysis Tool (DCAT) is being developed to mimic the response of generation dynamics, protection, and load components to extreme contingencies in power systems. Currently, DCAT generates a huge data set, with the analysis results stored in multiple subfolders using a great many files in different formats (power flow cases, time series, figures, spreadsheets). To help power system engineers thoroughly understand and analyze system behavior under many scenarios and contingencies, we developed a database management module (DBM) that enables future integration of DCAT with industry-grade Big Data analytical and interactive visualization platforms. In this paper, we demonstrate the DBM framework with interactive analytics and visualization for DCAT on the synthetic South Carolina 500-bus power system model using hurricane contingencies that capture possibilities for an extreme contingency. DCAT’s advanced data management and interactive visualization can help planning engineers to visually explore and analyze system behavior, all in one view.

Cascading, dynamic simulations, database managemen↗

DOE OSTI2018

Coupled Approximation of U.S. Driving Speed and Volume Statistics Using Spatial Conflation and Temporal Disaggregation

The advent of mobile devices with embedded global positioning systems has allowed commercial providers of real-time traffic data to develop highly accurate estimates of network-level vehicle speeds. Traffic speed data have far outpaced the availability and accuracy of real-time traffic volume information. Limited to a relatively small number of permanent and temporary traffic counters in any city, traffic volumes typically only cover a handful of roadways with inconsistent temporal resolution. This work addressed this data gap by coupling a commercial dataset of typical traffic speeds (by roadway and time of week) from TomTom to the U.S. FHWA's Highway Performance Monitoring System database of annual average daily traffic (AADT) counts by roadway. This work is technically novel in its solution for establishing a national crosswalk between independent network geometries using spatial conflation and big data techniques. The resulting product is a national dataset providing traffic speed and volume estimates under typical conditions for all U.S. roadways with AADT values.

30 DIRECT ENERGY CONVERSION↗

NASA NTRS2014

The Importance of Data Visualization: Incorporating Storytelling into the Scientific Presentation

From its inception in 2000, one of the primary tasks of the Biomedical Data Reduction Analysis (BDRA) group has been translation of large amounts of data into information that is relevant to the audience receiving it. BDRA helps translate data into an integrated model that supports both operational and research activities. This data integrated model and subsequent visual data presentations have contributed to BDRA's success in delivering the message (i.e., the story) that its customers have needed to communicate. This success has led to additional collaborations among groups that had previously not felt they had much in common until they worked together to develop solutions in an integrated fashion. As more emphasis is placed on working with "big data" and on showing how NASA's efforts contribute to the greater good of the American people and of the world, it becomes imperative to visualize the story of our data to communicate the greater message we need to share. METHODS To create and expand its data integrated model, BDRA has incorporated data from many different collaborating partner labs and other sources. Data are compiled from the repositories of the Lifetime Surveillance of Astronaut Health and the Life Sciences Data Archive, and from the individual laboratories at Johnson Space Center that support collection of data from medical testing, environmental monitoring, and countermeasures, as designated in the Medical Requirements Integration Documents. Ongoing communication with the participating collaborators is maintained to ensure that the message and story of the data are retained as data are translated into information and visual data presentations are delivered in different venues and to different audiences. RESULTS We will describe the importance of storytelling through an integrated model and of subsequent data visualizations in today's scientific presentations and discuss the collaborative methods used. We will illustrate the discussion with examples of graphs from BDRA's past work supporting operations and/or research efforts.

Babiak-Vazquez, A.↗

DOE OSTI2020

Scanning Probe Microscopy in the Information Age

Scanning probe microscopy (SPM) has brought scientific exploration, visualization, and control to nanoscale. A plethora of measurement approaches and techniques devised around SPM enabled probing structure and functionalities crucial for virtually all areas of science. While high-resolution electron microscopy techniques preceded scanning probes, SPM effectively opened the door to probing the structure–property relationship on length scales spanning 100s of micrometers down to a single atom. As the complexity of the material systems and the scientific questions increased, so too did the SPM techniques. Substantial efforts have been dedicated toward the development of better instrumentation platforms, probes, and novel spectroscopic measurements; however, many opportunities related to data processing and acquisition in SPM are just being realized now. This chapter will describe the recent progress in data-driven SPM by giving a broad overview of the field, the foundation of the scanning tunneling microscope (STM) and the atomic force microscope (AFM), as well as a description behind the operating principles and dynamic modes. A comprehensive overview of the multi-frequency techniques, Band excitation, and general acquisition mode SPM is given, setting the context for the application of big data, machine learning, and artificial intelligence in the field of dynamic SPM.

Collins, Liam↗

DOE OSTI2023

DLIO: A DATA-CENTRIC BENCHMARK FOR DEEP LEARNING APPLICATIONS

SF-22-136 Deep learning has been shown as a successful method for various tasks, and its popularity results in numerous open-source deep learning software tools. Deep learning has been applied to a broad spectrum of scientific domains such as cosmology, particle physics, computer vision, fusion, and astrophysics. Scientists have performed a great deal of work to optimize the computational performance of deep learning frameworks. However, the same cannot be said for I/O performance. As deep learning algorithms rely on big-data volume and variety to effectively train neural networks accurately, I/O is a significant bottleneck on large-scale distributed deep learning training. DLIO, is a novel representative benchmark suite built based on the I/O profiling of the selected workloads. DLIO can be utilized to accurately emulate the I/O behavior of modern deep learning applications. Using DLIO, application developers and system software solution architects can identify potential I/O bottlenecks in their applications and guide optimizations to boost the I/O performance leading to lower training times. The storage vendor can also use DLIO as a guide for designing and optimize the storage and filesystem targeting at deep learning application.

ZHENG, HUIHUO↗

DOE OSTI2018

Locality-aware and load-balanced static task scheduling for MapReduce

Task scheduling for MapReduce jobs has been an active area of research with the objective of decreasing the amount of data transferred during the shuffle phase via exploiting data locality. In the literature, generally only the scheduling of reduce tasks is considered with the assumption that scheduling of map tasks is already determined by the input data placement. However, in cloud or HPC deployments of MapReduce, the input data is located in a remote storage and scheduling map tasks gains importance. Here, we propose models for simultaneous scheduling of map and reduce tasks in order to improve data locality and balance the processors’ loads in both map and reduce phases. Our approach is based on graph and hypergraph models which correctly encode the interactions between map and reduce tasks. Partitions produced by these models are decoded to schedule map and reduce tasks. A two-constraint formulation utilized in these models enables balancing processors’ loads in both map and reduce phases. The partitioning objective in the hypergraph models correctly encapsulates the minimization of data transfer when a local combine step is performed prior to shuffle, whereas the partitioning objective in the graph models achieve the same feat when a local combine is not performed. Here, we show the validity of our scheduling on the MapReduce parallelizations of two important kernel operations – sparse matrix–vector multiplication (SpMV) and generalized sparse matrix–matrix multiplication (SpGEMM) – that are widely encountered in big data analytics and scientific computations. Compared to random scheduling, our models lead to tremendous savings in data transfer by reducing data traffic from several hundreds of megabytes to just a few megabytes in the shuffle phase and consequently leading up to 2.6x and 4.2x speedup for SpMV and SpGEMM, respectively.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2017

Parallel Multivariate Spatio-Temporal Clustering of Large Ecological Datasets on Hybrid Supercomputers

A proliferation of data from vast networks of remote sensing platforms (satellites, unmanned aircraft systems (UAS), airborne etc.), observational facilities (meteorological, eddy covariance etc.), state-of-the-art sensors, and simulation models offer unprecedented opportunities for scientific discovery. Unsupervised classification is a widely applied data mining approach to derive insights from such data. However, classification of very large data sets is a complex computational problem that requires efficient numerical algorithms and implementations on high performance computing (HPC) platforms. Additionally, increasing power, space, cooling and efficiency requirements has led to the deployment of hybrid supercomputing platforms with complex architectures and memory hierarchies like the Titan system at Oak Ridge National Laboratory. The advent of such accelerated computing architectures offers new challenges and opportunities for big data analytics in general and specifically, large scale cluster analysis in our case. Although there is an existing body of work on parallel cluster analysis, those approaches do not fully meet the needs imposed by the nature and size of our large data sets. Moreover, they had scaling limitations and were mostly limited to traditional distributed memory computing platforms. We present a parallel Multivariate Spatio-Temporal Clustering (MSTC) technique based on k-means cluster analysis that can target hybrid supercomputers like Titan. We developed a hybrid MPI, CUDA and OpenACC implementation that can utilize both CPU and GPU resources on computational nodes. We describe performance results on Titan that demonstrate the scalability and efficacy of our approach in processing large ecological data sets.

Sreepathi, Sarat↗

DOE OSTI2015

Evolving the Land Information System into a Cloud Computing Service

The Land Information System (LIS) was developed to use advanced flexible land surface modeling and data assimilation frameworks to integrate extremely large satellite- and ground-based observations with advanced land surface models to produce continuous high-resolution fields of land surface states and fluxes. The resulting fields are extremely useful for drought and flood assessment, agricultural planning, disaster management, weather and climate forecasting, water resources assessment, and the like. We envisioned transforming the LIS modeling system into a scientific cloud computing-aware web and data service that would allow clients to easily setup and configure for use in addressing large water management issues. The focus of this Phase 1 project was to determine the scientific, technical, commercial merit and feasibility of the proposed LIS-cloud innovations that are currently barriers to broad LIS applicability. We (a) quantified the barriers to broad LIS utility and commercialization (high performance computing, big data, user interface, and licensing issues); (b) designed the proposed LIS-cloud web service, model-data interface, database services, and user interfaces; (c) constructed a prototype LIS user interface including abstractions for simulation control, visualization, and data interaction, (d) used the prototype to conduct a market analysis and survey to determine potential market size and competition, (e) identified LIS software licensing and copyright limitations and developed solutions, and (f) developed a business plan for development and marketing of the LIS-cloud innovation. While some significant feasibility issues were found in the LIS licensing, overall a high degree of LIS-cloud technical feasibility was found.

54 ENVIRONMENTAL SCIENCES↗