Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “big data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

Cloud Giovanni: Reining in Costs and Improving Performance with Analytical Data Stores Using Scalable Serverless Architecture

Giovanni is the Geospatial Interactive Online Visualization ANd aNalysis Infrastructure developed at NASA GES DISC which provides a simple and intuitive way to visualize, analyze, and access vast amounts of Earth science data. It receives large number of user requests each day for a variety of analysis and visualization services, which leads to the big data challenge of serving gradually increasing large data volumes with diverse statistical algorithms. We hereby propose a multi-dimensional accumulation method which provides fast and cost-efficient cloud analysis for diverse services including both area averaging and time averaging. This method involves the weighted volume integration over multiple variable dimensions (time and space), and is implemented in AWS using Athena providing serverless and highly scalable data analysis. Compared to the standard method, this approach dramatically reduced the computational time by order of magnitude with a minimal AWS cost incurred. For example, for a benchmark of 10-year area averaging over the 1x1 degree daily variable, the computational time was reduced from minutes to seconds, and the Athena cost is only $5 for 100,000 requests.

Zhang, Hailiang↗
NASA NTRS2018

Enabling Analytics in the Cloud for Earth Science Data

The purpose of this workshop was to hold interactive discussions where providers, users, and other stakeholders could explore the convergence of three main elements in the rapidly developing world of technology: Big Data, Cloud Computing, and Analytics, [for earth science data].

Analytics↗
DOE OSTI2019

Accelerating Climate Resilient Plant Breeding by Applying Next-Generation Artificial Intelligence

Breeding crops for high yield and superior adaptability to new and variable climates is imperative to ensure continued food security, biomass production, and ecosystem services. Advances in genomics and phenomics are delivering insights into the complex biological mechanisms that underlie plant functions in response to environmental perturbations. However, linking genotype to phenotype remains a huge challenge and is hampering the optimal application of high-throughput genomics and phenomics to advanced breeding. Critical to success is the need to assimilate large amounts of data into biologically meaningful interpretations. Here, we present the current state of genomics and field phenomics, explore emerging approaches and challenges for multiomics big data integration by means of next-generation (Next-Gen) artificial intelligence (AI), and propose a workable path to improvement.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Facility Cybersecurity Framework Best Practices

Federal facilities are increasingly adopting automation and connecting to the Internet creating an energy-internet-of-things environment that converges operational technology (OT) and information technology (IT). Today's buildings increasingly weave together networked sensors and cyber and physical systems that enable data to be collected, aggregated, exchanged, stored and monetized in new ways. Building technological advances have created new energy technology, services, markets and value creation opportunities (e.g. transactive energy, two-way grid communications, machine learning, and increased use of renewable and distributed energy resources). But as larger data sets are being exchanged at faster speeds between an increasing number of OT systems, it becomes more difficult to protect the security of the data lifecycle and the physical equipment it interacts with. These challenges are especially difficult to overcome because the economic and environmental gain (interoperability, big data, social networks and ubiquitous information sharing) are driving these prominent trends in the digital age. Often cybersecurity is an afterthought. The U.S. Department of Energy’s (DOE) Federal Energy Management Program (FEMP) funded the Pacific Northwest National Laboratory (PNNL) to develop various cybersecurity tools, trainings, and reports to aid federal facility managers – and other building owners and operators – in better applying frameworks and lessons learned from the National Institute of Standards and Technology (NIST) Cybersecurity Framework (CSF), risk management framework (RMF), DOE’s cybersecurity capability maturity model (C2M2), and a wide variety of industry best practices and guidance documents (i.e., NIST 800 series, Department of Defense United Facilities Criteria). This set of tools, collectively known as the FEMP Facility-Related Control System Cyber Toolkit (FRCS Cyber Toolkit)2, is focused on cybersecurity concerns from facility-related control systems and other operational technology (OT), such as industrial control systems (ICS). The FRCS Cyber Toolkit can be applied across six of the sixteen critical infrastructure sectors designated by the Department of Homeland Security, including government facilities, healthcare and public health, commercial facilities (e.g., public assembly, offices, lodging), financial services (e.g., banking and insurance), emergency services (e.g., fire and police stations), and information technology. With increasingly converged IT and OT systems, it is crucial to address OT cybersecurity considerations and assess how the seam of these two systems could impact the overall cybersecurity posture of a facility. The objective of this report is to provide an overview of the best possible method to use FRCS Cyber Toolkit (section 2.0) and distilled cybersecurity best practices for the federal facilities to address growing non-linear cyber threats (section 3.0). Recommendations in this document are aggregated from several NIST and other documents (see Appendix A for additional details).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Facility Cybersecurity Framework Best Practices Version 2.0

Federal facilities are increasingly adopting automation and connecting to the Internet creating an energy-internet-of-things environment that converges operational technology (OT) and information technology (IT). Today's buildings increasingly weave together networked sensors and cyber and physical systems that enable data to be collected, aggregated, exchanged, stored and monetized in new ways. Building technological advances have created new energy technology, services, markets and value creation opportunities (e.g. transactive energy, two-way grid communications, machine learning, and increased use of renewable and distributed energy resources). But as larger data sets are being exchanged at faster speeds between an increasing number of OT systems, it becomes more difficult to protect the security of the data lifecycle and the physical equipment it interacts with. These challenges are especially difficult to overcome because the economic and environmental gain (interoperability, big data, social networks and ubiquitous information sharing) are driving these prominent trends in the digital age. Often cybersecurity is an afterthought.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

Facility Cybersecurity Framework Best Practices Version 2.0

Federal facilities are increasingly adopting automation and connecting to the Internet creating an energy-internet-of-things environment that converges operational technology (OT) and information technology (IT). Today's buildings increasingly weave together networked sensors and cyber and physical systems that enable data to be collected, aggregated, exchanged, stored and monetized in new ways. Building technological advances have created new energy technology, services, markets and value creation opportunities (e.g. transactive energy, two-way grid communications, machine learning, and increased use of renewable and distributed energy resources). But as larger data sets are being exchanged at faster speeds between an increasing number of OT systems, it becomes more difficult to protect the security of the data lifecycle and the physical equipment it interacts with. These challenges are especially difficult to overcome because the economic and environmental gain (interoperability, big data, social networks and ubiquitous information sharing) are driving these prominent trends in the digital age. Often cybersecurity is an afterthought.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS2018

Interactive Multi-Instrument Database of Solar Flares

The fundamental motivation of the project is that the scientific output of solar research can be greatly enhanced by better exploitation of the existing solar/heliosphere space-data products jointly with ground-based observations. Our primary focus is on developing a specific innovative methodology based on recent advances in "big data" intelligent databases applied to the growing amount of high-spatial and multi-wavelength resolution, high-cadence data from NASA's missions and supporting ground-based observatories. Our flare database is not simply a manually searchable time-based catalog of events or list of web links pointing to data. It is a preprocessed metadata repository enabling fast search and automatic identification of all recorded flares sharing a specifiable set of characteristics, features, and parameters. The result is a new and unique database of solar flares and data search and classification tools for the Heliophysics community, enabling multi-instrument/multi-wavelength investigations of flare physics and supporting further development of flare-prediction methodologies.

Heliophysics↗
NASA NTRSDate not supplied

Pyroscopegridding: Geo-Leo Aerosol Data Fusion (an Open-Source Package)

The retrieval of aerosol optical depths (AODs) from sun-synchronous polar orbiting satellites, such as MODISs, VIIRSs, OMI, TROPOMI, etc., has been widely adopted as a method for obtaining information regarding particulate matter (PM) and related atmospheric processes. However, the advent of recently launched geostationary satellites, such as GOES-16/17/18, Himawari-8/9, and Meteosat Third Generation (MTG), has led to an increased temporal resolution of AOD observations (order of 10 minutes), resulting in typically one or more images per hour during daylight hours, compared to the once-per-day observations obtained from LEO satellites. By integrating these observations, the diurnal cycle of global AOD can be characterized at local, regional, and global scales. The scientific community is still examining the novel data from geostationary satellite observations and evaluating methods for effectively merging these observations with differing spatial and temporal resolutions. This presents a significant ""Big Data"" challenge, encompassing not only data storage, but also data discoverability, accessibility, and migration within cloud computing environments. This study presents our attempts at fusing Level 2 aerosol data from six satellites, three of which are geostationary (GOES-16/17 and Himawari-8) and three of which are polar orbiting (TERRA/MODIS, AQUA/MODIS, and SNPP-VIIRS), using the Dark Target aerosol retrieval algorithm. The ability to fuse remote sensing products on demand into desired temporal and spatial domains empowers researchers and practitioners to more efficiently work with satellite and sensor data. It is our hope that through making our open-source package and accompanying functionality available, the scientific community will have improved access to aerosol data processing resources.

Jennifer Wei↗
DOE OSTI2022

The Data Synergy Effects of Time-Series Deep Learning Models in Hydrology

When fitting statistical models to variables in geoscientific disciplines such as hydrology, it is a customary practice to stratify a large domain into multiple regions (or regimes) and study each region separately. Traditional wisdom suggests that models built for each region separately will have higher performance because of homogeneity within each region. However, each stratified model has access to fewer and less diverse data points. Here, through two hydrologic examples (soil moisture and streamflow), we show that conventional wisdom may no longer hold in the era of big data and deep learning (DL). We systematically examined an effect we call data synergy, where the results of the DL models improved when data were pooled together from characteristically different regions. The performance of the DL models benefited from modest diversity in the training data compared to a homogeneous training set, even with similar data quantity. Moreover, allowing heterogeneous training data makes eligible much larger training datasets, which is an inherent advantage of DL. A large, diverse data set is advantageous in terms of representing extreme events and future scenarios, which has strong implications for climate change impact assessment. The results here suggest the research community should place greater emphasis on data sharing.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

Zodiac: A Comprehensive Depiction of Genetic Interactions in Cancer by Integrating TCGA Data

Background: Genetic interactions play a critical role in cancer development. Existing knowledge about cancer genetic interactions is incomplete, especially lacking evidences derived from large-scale cancer genomics data. The Cancer Genome Atlas (TCGA) produces multimodal measurements across genomics and features of thousands of tumors, which provide an unprecedented opportunity to investigate the interplays of genes in cancer. Methods: We introduce Zodiac, a computational tool and resource to integrate existing knowledge about cancer genetic interactions with new information contained in TCGA data. It is an evolution of existing knowledge by treating it as a prior graph, integrating it with a likelihood model derived by Bayesian graphical model based on TCGA data, and producing a posterior graph as updated and data-enhanced knowledge. In short, Zodiac realizes “Prior interaction map + TCGA data → Posterior interaction map.” Results: Zodiac provides molecular interactions for about 200 million pairs of genes. All the results are generated from a big-data analysis and organized into a comprehensive database allowing customized search. In addition, Zodiac provides data processing and analysis tools that allow users to customize the prior networks and update the genetic pathways of their interest. Zodiac is publicly available at www.compgenome.org/ZODIAC. Conclusions: Zodiac recapitulates and extends existing knowledge of molecular interactions in cancer. It can be used to explore novel gene-gene interactions, transcriptional regulation, and other types of molecular interplays in cancer.

Zhu, Yitan↗
DOE OSTI2016

SU-F-P-10: A Web-Based Radiation Safety Relational Database Module for Regulatory Compliance

Purpose: Maintaining compliance with Radioactive Materials Licenses is inherently a time-consuming task requiring focus and attention to detail. Staff tasked with these responsibilities, such as the Radiation Safety Officer and associated personnel must retain disparate records for eventual placement into one or more annual reports. Entering results and records in a relational database using a web browser as the interface, and storing that data in a cloud-based storage site, removes procedural barriers. The data becomes more adaptable for mining and sharing. Methods: Web-based code was written utilizing the web framework Django, written in Python. Additionally, the application utilizes JavaScript for front-end interaction, SQL, HTML and CSS. Quality assurance code testing is performed in a sequential style, and new code is only added after the successful testing of the previous goals. Separate sections of the module include data entry and analysis for audits, surveys, quality management, and continuous quality improvement. Data elements can be adapted for quarterly and annual reporting, and for immediate notification of user determined alarm settings. Results: Current advances are focusing on user interface issues, and determining the simplest manner by which to teach the user to build query forms. One solution has been to prepare library documents that a user can select or edit in place of creation a new document. Forms are being developed based upon Nuclear Regulatory Commission federal code, and will be expanded to include State Regulations. Conclusion: Establishing a secure website to act as the portal for data entry, storage and manipulation can lead to added efficiencies for a Radiation Safety Program. Access to multiple databases can lead to mining for big data programs, and for determining safety issues before they occur. Overcoming web programming challenges, a category that includes mathematical handling, is providing challenges that are being overcome.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2021

The clustering of the SDSS-IV extended Baryon Oscillation Spectroscopic Survey quasar sample: testing observational systematics on the Baryon Acoustic Oscillation measurement

ABSTRACT Baryon Acoustic Oscillations (BAO) are considered to be a very robust standard ruler against various systematics. This premise has been tested against observational systematics, but not to the level required for the next generation of galaxy surveys such as the Dark Energy Spectroscopic Instrument (DESI) and Euclid. In this paper, we investigate the effect of observational systematics on the BAO measurement of the final sample of quasars from the extended Baryon Oscillation Spectroscopic Survey Data Release 16 in order to prepare and hone a similar analysis for upcoming surveys. We employ catalogues with various treatments of imaging systematic effects using linear and neural network-based non-linear approaches and consider how the BAO measurement changes. We also test how the variations to the BAO fitting model respond to the observational systematics. As expected, we confirm that the BAO measurements obtained from the DR16 quasar sample are robust against imaging systematics well within the statistical error, while reporting slightly modified constraints that shift the line-of-sight BAO signal by less than 1.1 per cent. We use realistic simulations with similar redshift and angular distributions as the DR16 sample to conduct statistical tests for validating the pipeline, quantifying the significance of differences, and estimating the expected bias on the BAO scale in future high-precision data sets. Although we find a marginal impact for the eBOSS QSO data, the work presented here is of vital importance for constraining the nature of dark energy with the BAO feature in the new era of big data cosmology.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2015

Research on data from the ATLAS experiment at CERN

In this report senior investigator Prof. Milind V. Purohit describes research done with data from the ATLAS experiment at CERN. This includes preparing papers on the performance of the CSC detector, searches for SUSY using a new modern ''big data'' technique, and a search for supersymmetry (SUSY) using the "zero leptons razor" (0LRaz) technique. The prediction of the W=Z+jets background processes by the ATLAS simulation prior to the fit is found to be overestimated in the phase space of interest. In all new signal regions presented in this analysis the number of events observed is consistent with the post-fit SM expectations. Assuming R-parity conservation, the limit on the gluino mass exceeds 1150 GeV at 95% confidence level, for an LSP mass smaller than 100 GeV. Other USC personnel who participated in this project during the period of this grant were a graduate student, Anton Kravchenko.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRS2022

An Advanced Open-Source Platform for Air Quality Analysis, Visualization, and Prediction

Ambient air pollution is the largest environmental health risk factor, leading to several million premature deaths globally per year. The challenge of combating poor air quality is exacerbated by growing urban populations, changing emissions, and a warming climate. While there have been many advances monitoring and modeling of atmospheric composition, reflected in the dramatic increase in archived Earth Observations, there is no single measurement or method that alone can provide an accurate depiction of the entire atmosphere. The rapidly growing collections of observational and modeling data require us to be smarter about what data to include, and how such data is used. In recent years, NASA has invested significantly in advancing the concepts for Analytics Collaborative Framework (ACF) [5] and New Observing Strategies (NOS) [4] to tackle our software infrastructure need for harmonized data management and dynamic acquisition of diverse measurements for on-demand, interactive, multivariate analysis, and access [3]. It is not enough to have a big data, standalone analytics solution; it is critical that we start integrating data from remote sensing, modeling, and in-situ networks in a harmonized manner that enables timely and data-driven decision-making for air quality management. This work presents the design and development of an Air Quality Analytics Collaborative Framework (AQ ACF), as part of NASA’s Advanced Information Systems Technology (AIST) effort, to establish a data, machine-learning, and numerically driven platform for air quality analysis, visualization, and prediction.

Liu, Qian↗
DOE OSTI2021

Hardening DOE R&D Software Tools for Web-based Visualization SBIR Phase I Final Report

Ubiquitous web-based visualization is essential to delivering large-scale data visualization to various stakeholders, from the scientist to the board member. These stakeholders will not tolerate a stalled application or a pop-up window asking them to wait for the processing to complete. They require a responsive and interactive visualization environment with high-quality imagery suitable for detailed analysis and boardroom presentations. At Kitware, Inc., we have accomplished web visualization to this point, leveraging state-of-the-art tools like HTML5, CSS3, SVG, Canvas, and WebGL. Solutions that leverage a combination of these technologies are necessary to handle workloads that vary significantly in data size efficiently. However, it is not always practical to move large data to the web client for visualization. Kitware's ParaView as a Service combines client-side visualization using both distributed processing and remote rendering on big data impractical to move. Existing distributed processing and remote rendering solution's interactivity is below the expectations of web-based applications. Our project examined proposed solutions to the areas outlined above in ParaView as a Service. We have investigated concurrent pipelines, streaming images, progressive rendering, and optimization of algorithms and data movement to address these concerns. For the Phase I project, we completed the proposed work plan. As a result, the project produced three prototypes of essential importance for web visualization and the ParaView as a Service community. We created a simple desktop application for an interactive streamline placement prototype, a web-based interactive streamline placement prototype, and a web-based progressive rendering utilizing raytracing prototype. These prototypes relied on the hardening of emerging software toolkits funded by the Department of Energy (DOE) Advanced Scientific Computing Research (ASCR) program (such as ParaView, VTK-m, and Mochi). We blended these components into web-based visualization prototypes that meet the industry's expectations for interactivity and responsiveness. The Phase I project had four essential focus areas: 1. Develop prototype ParaView as a Service backend server using asynchronous, non-blocking design principles. 2. Develop a prototype web application that uses the ParaView as a Service backend server for remote data visualization. 3. Implement image streaming with encoding/compression and progressive rendering capabilities in the proposed platform. 4. Evaluate the prototype developed and summarize observations, including the challenges and pitfalls of our approach. After our successful completion of Phase I, we are strongly positioned to propose a successful Phase II project.

Geveci, Berk↗
DOE OSTI2022

Discovery of Signatures, Anomalies, and Precursors in Synchrophasor Data with Matrix Profile and Deep Recurrent Neural Networks (Final Project Report)

The widespread deployment of phasor measurement unit (PMU) across the U.S. together with the burgeoning machine learning technology made it possible to develop data-driven PMU data analytics to improve grid security and reliability in a more insightful and effective manner. Although PMU applications have been explored for over a decade, the representative PMU usage is limited to the bulk power system monitoring mainly due to the data integrity issues associated with PMUs (typically missing, fragmented, and wrongly amplified data). To forge a breakthrough on this stalemate and embrace PMUs for power system control and protection as well, we applied various advanced machine learning and big data analysis technology to the power system event detection and classification as the first step toward the power system control and protection pertaining to grid security enhancement.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

Spatiotemporal Pattern Recognition in the PMU Signals in the WECC system

Phasor measurement unit (PMU) data has been used by multiple power system applications, including state estimation, post event analysis, oscillation detection, model validation, and many others. Still, due to its big data nature and availability to general research institutions, comprehensive understanding of the spatiotemporal patterns and underlying mechanisms are incomplete. This study applies a set of signal processing and machine learning approaches aiming at deciphering the characteristic behaviors of multiple phasor measurement units (PMUs) attributes (e.g., voltage, frequency, rate of change of frequency, phase angle), including their auto-correlation, cross-dependence, similarities and discrepancies across units and temporal scales, and distributions of anomalies and their linkages to potential external factors such as weather events. Data analytics are applied to PMUs from the U.S. Western Electricity Coordinating Council (WECC) system. The PMU measurements, recorded events, outages, and weather extremes are all from real world datasets. The findings from the study and mechanistic understanding of the PMU dynamics help provide guidance on system control or preventing blackouts. The derived metrics can be directly used for adjusting or filtering simulated PMU data used for advanced algorithm development.

Hou, Zhangshuan↗