SEARCH · Engineering Papers

Results for “big data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,063 records · Page 59

All sources NASA NTRS DOE OSTI

DOE OSTI2014

Final Report - Phylogenomic tools and web resources for the Systems Biology Knowledgebase

The major advance during this last reporting period (8/15/12 to present) is our release of data on the PhyloFacts website: phylogenetic trees, multiple sequence alignments and other data for protein families are now available for download from http://phylogenomics.berkeley.edu/data/. This project as a whole aimed to develop high-throughput functional annotation systems that exploit information from protein 3D structure and evolution to provide highly precise inferences of various aspects of gene function, including molecular function, biological process, pathway association, Pfam domains, cellular localization and so on. We accomplished these aims by developing and testing different systems on a database of protein family trees: the PhyloFacts Phylogenomic Encyclopedia (at http://phylogenomics.berkeley.edu/phylofacts/ ).

60 APPLIED LIFE SCIENCES↗

DOE OSTI2016

Large-Scale Public Transcriptomic Data Mining Reveals a Tight Connection between the Transport of Nitrogen and Other Transport Processes in Arabidopsis

Movement of nitrogen to the plant tissues where it is needed for growth is an important contribution to nitrogen use efficiency. However, we have very limited knowledge about the mechanisms of nitrogen transport. Loading of nitrogen into the xylem and/or phloem by transporter proteins is likely important, but there are several families of genes that encode transporters of nitrogenous molecules (collectively referred to as N transporters here), each comprised of many gene members. In this study, we leveraged publicly available microarray data of Arabidopsis to investigate the gene networks of N transporters to elucidate their possible biological roles. First, we showed that tissue-specificity of nitrogen (N) transporters was well reflected among the public microarray data. Then, we built coexpression networks of N transporters, which showed relationships between N transporters and particular aspects of plant metabolism, such as phenylpropanoid biosynthesis and carbohydrate metabolism. Furthermore, genes associated with several biological pathways were found to be tightly coexpressed with N transporters in different tissues. Our coexpression networks provide information at the systems-level that will serve as a resource for future investigation of nitrogen transport systems in plants, including candidate gene clusters that may work together in related biological roles.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2014

Scaling Semantic Graph Databases in Size and Performance

In this paper we present SGEM, a full software system for accelerating large-scale semantic graph databases on commodity clusters. Unlike current approaches, SGEM addresses semantic graph databases by only employing graph methods at all the levels of the stack. On one hand, this allows exploiting the space efficiency of graph data structures and the inherent parallelism of graph algorithms. These features adapt well to the increasing system memory and core counts of modern commodity clusters. On the other hand, however, these systems are optimized for regular computation and batched data transfers, while graph methods usually are irregular and generate fine-grained data accesses with poor spatial and temporal locality. Our framework comprises a SPARQL to data parallel C compiler, a library of parallel graph methods and a custom, multithreaded runtime system. We introduce our stack, motivate its advantages with respect to other solutions and show how we solved the challenges posed by irregular behaviors. We present the result of our software stack on the Berlin SPARQL benchmarks with datasets up to 10 billion triples (a triple corresponds to a graph edge), demonstrating scaling in dataset size and in performance as more nodes are added to the cluster.

Aggregation↗

DOE OSTI2016

Imaging via complete cantilever dynamic detection: General dynamic mode imaging and spectroscopy in scanning probe microscopy

We develop and implement a multifrequency spectroscopy and spectroscopic imaging mode, referred to as general dynamic mode (GDM), that captures the complete spatially- and stimulus dependent information on nonlinear cantilever dynamics in scanning probe microscopy (SPM). GDM acquires the cantilever response including harmonics and mode mixing products across the entire broadband cantilever spectrum as a function of excitation frequency. GDM spectra substitute the classical measurements in SPM, e.g. amplitude and phase in lock-in detection. Here, GDM is used to investigate the response of a purely capacitively driven cantilever. We use information theory techniques to mine the data and verify the findings with governing equations and classical lock-in based approaches. We explore the dependence of the cantilever dynamics on the tip–sample distance, AC and DC driving bias. This approach can be applied to investigate the dynamic behavior of other systems within and beyond dynamic SPM. In conclusion, GDM is expected to be useful for separating the contribution of different physical phenomena in the cantilever response and understanding the role of cantilever dynamics in dynamic AFM techniques.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2015

Query Optimization for Graph Analytics on Linked Data Using SPARQL

Triplestores that support query languages such as SPARQL are emerging as the preferred and scalable solution to represent data and meta-data as massive heterogeneous graphs using Semantic Web standards. With increasing adoption, the desire to conduct graph-theoretic mining and exploratory analysis has also increased. Addressing that desire, this paper presents a solution that is the marriage of Graph Theory and the Semantic Web. We present software that can analyze Linked Data using graph operations such as counting triangles, finding eccentricity, testing connectedness, and computing PageRank directly on triple stores via the SPARQL interface. We describe the process of optimizing performance of the SPARQL-based implementation of such popular graph algorithms by reducing the space-overhead, simplifying iterative complexity and removing redundant computations by understanding query plans. Our optimized approach shows significant performance gains on triplestores hosted on stand-alone workstations as well as hardware-optimized scalable supercomputers such as the Cray XMT.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2015

Enabling Graph Appliance for Genome Assembly

In recent years, there has been a huge growth in the amount of genomic data available as reads generated from various genome sequencers. The number of reads generated can be huge, ranging from hundreds to billions of nucleotide, each varying in size. Assembling such large amounts of data is one of the challenging computational problems for both biomedical and data scientists. Most of the genome assemblers developed have used de Bruijn graph techniques. A de Bruijn graph represents a collection of read sequences by billions of vertices and edges, which require large amounts of memory and computational power to store and process. This is the major drawback to de Bruijn graph assembly. Massively parallel, multi-threaded, shared memory systems can be leveraged to overcome some of these issues. The objective of our research is to investigate the feasibility and scalability issues of de Bruijn graph assembly on Cray s Urika-GD system; Urika-GD is a high performance graph appliance with a large shared memory and massively multithreaded custom processor designed for executing SPARQL queries over large-scale RDF data sets. However, to the best of our knowledge, there is no research on representing a de Bruijn graph as an RDF graph or finding Eulerian paths in RDF graphs using SPARQL for potential genome discovery. In this paper, we address the issues involved in representing a de Bruin graphs as RDF graphs and propose an iterative querying approach for finding Eulerian paths in large RDF graphs. We evaluate the performance of our implementation on real world ebola genome datasets and illustrate how genome assembly can be accomplished with Urika-GD using iterative SPARQL queries.

Big data↗

DOE OSTI2018

Occupant comfort and behavior: High-resolution data from a 6-month field study of personal comfort systems with 37 real office workers

Personal Comfort Systems (PCS) provide individual occupants local heating and cooling to meet their comfort needs without affecting others in the same space. It saves energy by relaxing ambient temperature requirements for the HVAC system. Aside from these benefits, PCS offers a wealth of data that can describe how individuals interact with heating/cooling devices in their own environment. Recently developed Internet-connected PCS chairs have unlocked this opportunity by generating continuous streams of heating and cooling usage data, along with occupancy status and environmental measurements via embedded sensors. The data allow individuals' comfort and behavior to be learned, and can inform centralized systems to provide ‘just the right’ amount of conditioning to meet occupant needs. In summer 2016, we carried out a study with PCS chairs involving 37 occupants in an office building in California. During the study period, we collected >5 million chair usage data-points and 4500 occupant survey responses, as well as continuous measurements of environmental and HVAC system conditions. The data analysis shows that (1) local temperatures experienced by individual occupants vary quite widely across different parts of the building, even within the same thermal zone; (2) occupants often have different thermal preferences even under the same thermal conditions; (3) PCS control behavior can dynamically describe individuals' thermal preferences; (4) PCS chairs produce much higher comfort satisfaction (96%) than typically achieved in buildings. We conclude that PCS not only provide personalized comfort solutions but also offer individualized feedback that can improve comfort analytics and control decisions in buildings.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2026

Ensemble Federated Machine Learning‐Based Cybersecurity Situational Awareness in Microgrid Network

Cyber-physical microgrids are vulnerable to stealthy cybersecurity threats that disguise their actions through the exploitation of system knowledge. Such actions can severely impacts microgrids deployed in defense bases, slowing the response time of military forces during national emergencies. Several machine-learning algorithms have been proposed to detect intrusions in the grid networks; however, these traditional machine-learning algorithms lack data privacy and are subject to several adversarial machine-learning threats. This paper proposes a novel federated machine learning (FML)-based three-model framework to detect and identify stealthy data-integrity attacks while ensuring data privacy in microgrid networks. The proposed architecture uses a variational mode decomposition technique to extract derived features from incoming measurement and control datasets. The extraction of these derived features allows FML models to learn minute variations in data patterns that allow them to perform significantly better than the models trained with generic datasets consisting of raw features. Our experimental results show the efficient performance of the proposed methodology against different types of data integrity attacks while considering primary and secondary controllers in microgrids. Further, the applied FML-integrated random forest ensemble algorithm outperforms the existing generic FML algorithms during noisy and noise-free datasets with prediction latencies of only 91–134 µs per sample within the 0.1 s sampling interval and requires communication bandwidth of around ∼8.25 KB/s at the control center and ∼2.7 KB/s per edge client for communication.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2016

Simplification of femtosecond transient absorption microscopy data from CH 3 NH 3 PbI 3 perovskite thin films into decay associated amplitude maps

Our work aims to simplify multi-dimensional femtosecond transient absorption microscopy (TAM) data into decay associated amplitude maps that describe the spatial distributions of dynamical processes occurring on various characteristic timescales. Application of this method to TAM data obtained from a model methyl-ammonium lead iodide (CH 3 NH 3 PbI 3 ) perovskite thin film allows us to simplify the dataset consisting of a 68 time-resolved images into 4 decay associated amplitude maps. Furthermore, these maps provide a simple means to visualize the complex electronic excited-state dynamics in this system by separating distinct dynamical processes evolving on characteristic timescales into individual spatial images. This approach provides new insight into subtle aspects of ultrafast relaxation dynamics associated with excitons and charge carriers in the perovskite thin film, which have recently been found to coexist at spatially distinct locations.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2022

Learning Global Proliferation Expertise Evolution Using AI-Driven Analytics and Public Information

Detecting and anticipating global proliferation expertise and capability evolution from unstructured, noisy, and incomplete public data streams is a highly desired, but extremely challenging task. Here, in this article, we present our pioneering data-driven approach to support the non-proliferation mission to detect and explain the evolution of proliferation expertise and capability development globally from terabytes of publicly available information (PAI), focusing on our knowledge extraction pipeline and descriptive analytics. We first discuss how we fuse nine open-source data streams, including multilingual data, to convert 4 TB of unstructured data to structured knowledge and encode dynamically evolving proliferation expertise representations—content and context graphs. For this, we rely on natural language processing (NLP) and deep learning (DL) models to perform information extraction, topic modeling, and distributed text representation (aka embedding) learning. We then present interactive, usable, and explainable descriptive analytics to refine domain knowledge and present it in a human-understandable form. Finally, we introduce future work avenues that will leverage our dynamic knowledge representations and descriptive analytics to enable predictive and prescriptive inferences to achieve real-time domain understanding and contextual reasoning about global proliferation expertise and capability evolution.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

DOE OSTI2022

A Comparison of Infectious Disease Forecasting Methods across Locations, Diseases, and Time

Accurate infectious disease forecasting can inform efforts to prevent outbreaks and mitigate adverse impacts. This study compares the performance of statistical, machine learning (ML), and deep learning (DL) approaches in forecasting infectious disease incidences across different countries and time intervals. We forecasted three diverse diseases: campylobacteriosis, typhoid, and Q-fever, using a wide variety of features (n = 46) from public datasets, e.g., landscape, climate, and socioeconomic factors. We compared autoregressive statistical models to two tree-based ML models (extreme gradient boosted trees [XGB] and random forest [RF]) and two DL models (multi-layer perceptron and encoder–decoder model). The disease models were trained on data from seven different countries at the region-level between 2009–2017. Forecasting performance of all models was assessed using mean absolute error, root mean square error, and Poisson deviance across Australia, Israel, and the United States for the months of January through August of 2018. The overall model results were compared across diseases as well as various data splits, including country, regions with highest and lowest cases, and the forecasted months out (i.e., nowcasting, short-term, and long-term forecasting). Overall, the XGB models performed the best for all diseases and, in general, tree-based ML models performed the best when looking at data splits. There were a few instances where the statistical or DL models had minutely smaller error metrics for specific subsets of typhoid, which is a disease with very low case counts. Feature importance per disease was measured by using four tree-based ML models (i.e., XGB and RF with and without region name as a feature). The most important feature groups included previous case counts, region name, population counts and density, mortality causes of neonatal to under 5 years of age, sanitation factors, and elevation. This study demonstrates the power of ML approaches to incorporate a wide range of factors to forecast various diseases, regardless of location, more accurately than traditional statistical approaches.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2020

Detect and correct bias in multi-site neuroimaging datasets

The desire to train complex machine learning algorithms and to increase the statistical power in association studies drives neuroimaging research to use ever-larger datasets. The most obvious way to increase sample size is by pooling scans from independent studies. However, simple pooling is often ill-advised as selection, measurement, and confounding biases may creep in and yield spurious correlations. In this work, we combine 35,320 magnetic resonance images of the brain from 17 studies to examine bias in neuroimaging. In the first experiment, Name That Dataset, we provide empirical evidence for the presence of bias by showing that scans can be correctly assigned to their respective dataset with 71.5% accuracy. Given such evidence, we take a closer look at confounding bias, which is often viewed as the main shortcoming in observational studies. In practice, we neither know all potential confounders nor do we have data on them. Hence, we model confounders as unknown, latent variables. Kolmogorov complexity is then used to decide whether the confounded or the causal model provides the simplest factorization of the graphical model. Finally, we present methods for dataset harmonization and study their ability to remove bias in imaging features. In particular, we propose an extension of the recently introduced ComBat algorithm to control for global variation across image features, inspired by adjusting for unknown population stratification in genetics. Overall, our results demonstrate that harmonization can reduce dataset-specific information in image features. Further, confounding bias can be reduced and even turned into a causal relationship. However, harmonization also requires caution as it can easily remove relevant subject-specific information. Code is available at https://github.com/ai-med/Dataset-Bias.

42 ENGINEERING↗

DOE OSTI2017

Obtaining and Managing Answer Quality for Online Data-Intensive Services

Online data-intensive (OLDI) services use anytime algorithms to compute over large amounts of data and respond quickly. Interactive response times are a priority, so OLDI services parallelize query execution across distributed software components and return best effort answers based on the data so far processed. Omitted data from slow components could lead to better answers, but tracing online how much better the answers could be is difficult. Here, we introduce Ubora, a design method to measure the effect of slow-running components on the quality of answers. Ubora randomly samples online queries and executes them a second time. The first online execution omits data from slow components and provides interactive answers. The second execution uses mature results from intermediate components completed after the online execution finishes. Ubora uses memoization to speed up mature executions by replaying network messages exchanged between components. Our systems-level implementation works for a wide range of services, including Hadoop/Yarn, Apache Lucene, the EasyRec Recommendation Engine, and the OpenEphyra question-answering system. Ubora computes answer quality with more mature executions per second than competing approaches that do not use memoization. With Ubora, we illustrate that answer quality is effective at guiding online admission control. While achieving the same answer quality on high-priority queries, our adaptive controller had 55% higher peak throughput on low-priority queries than a competing controller guided by the rate of timeouts.

answer quality↗

DOE OSTI2024

Machine Learning-Enhanced Multiphase CFD for Carbon Capture Modeling Run Data

Repository for the data generated as part of the 2023-2024 ALCC project "Machine Learning-Enhanced Multiphase CFD for Carbon Capture Modeling." The data was generated with MFIX-Exa's CFD-DEM model. The problem of interest is gravity driven, particle-laden, gas-solid flow in a triply-periodic domain of length 2048 particle diameters with an aspect ratio of 4. The mean particle concentration ranges from 1% to 40% and the Archimedes number ranges from 18 to 90. The particle-to-fluid density ratio, particle-particle restitution and friction coefficients and domain aspect ratio are held constant at values of 1000, 0.9, 0.25 and 4, respectively. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231 using NERSC award ALCC-ERCAP0025948.

AMReX↗

DOE OSTI2016

Separation of distinct photoexcitation species in femtosecond transient absorption microscopy

Femtosecond transient absorption microscopy is a novel chemical imaging capability with simultaneous high spatial and temporal resolution. Although several powerful data analysis approaches have been developed and successfully applied to separate distinct chemical species in such images, the application of such analysis to distinguish different photoexcited species is rare. In this paper, we demonstrate a combined approach based on phasor and linear decomposition analysis on a microscopic level that allows us to separate the contributions of both the excitons and free charge carriers in the observed transient absorption response of a composite organometallic lead halide perovskite film. We found spatial regions where the transient absorption response was predominately a result of excitons and others where it was predominately due to charge carriers, and regions consisting of signals from both contributors. Lastly, quantitative decomposition of the transient absorption response curves further enabled us to reveal the relative contribution of each photoexcitation to the measured response at spatially resolved locations in the film.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2018

Deep data analysis via physically constrained linear unmixing: universal framework, domain examples, and a community-wide platform

Abstract Many spectral responses in materials science, physics, and chemistry experiments can be characterized as resulting from the superposition of a number of more basic individual spectra. In this context, unmixing is defined as the problem of determining the individual spectra, given measurements of multiple spectra that are spatially resolved across samples, as well as the determination of the corresponding abundance maps indicating the local weighting of each individual spectrum. Matrix factorization is a popular linear unmixing technique that considers that the mixture model between the individual spectra and the spatial maps is linear. Here, we present a tutorial paper targeted at domain scientists to introduce linear unmixing techniques, to facilitate greater understanding of spectroscopic imaging data. We detail a matrix factorization framework that can incorporate different domain information through various parameters of the matrix factorization method. We demonstrate many domain-specific examples to explain the expressivity of the matrix factorization framework and show how the appropriate use of domain-specific constraints such as non-negativity and sum-to-one abundance result in physically meaningful spectral decompositions that are more readily interpretable. Our aim is not only to explain the off-the-shelf available tools, but to add additional constraints when ready-made algorithms are unavailable for the task. All examples use the scalable open source implementation from https://github.com/ramkikannan/nmflibrary that can run from small laptops to supercomputers, creating a user-wide platform for rapid dissemination and adoption across scientific disciplines.

47 OTHER INSTRUMENTATION↗

DOE OSTI2021

MindSynchro

This report presents the developments and results of MindSynchro project as part of DOE OE FOA 1861. DOE and Pacific Northwest National Laboratory (PNNL) have made available to FOA awardees datasets containing years of real historical data recorded from various phasor measurement units (PMUs) which are installed in three large US interconnections: Texas (IC A), Western (IC B), and Eastern (IC C). The main goal of the project, which was successfully achieved, was to develop methods for detection and identification of events which are relevant for power grid operation. Tasks performed for achieving the project goals included data exploration and pre-processing, the development and application of physics-based features, data analysis and labeling based on unsupervised learning approaches, training and testing of DSSL models for classification of events which are relevant for power grid operation, and deployment of solutions to cloud environments. The methods developed in the project can potentially provide relevant benefits to power grid asset owners/operators in general in terms of situational awareness. Two main types of outcomes can be provided by these tools: Identification of specific relevant power grid event types: Semi-supervised ML methods developed in the project can adequately employ not only the relatively scarce labeled data but also the large amount of available unlabeled data to train models for detection of specific event types. Such methods enable the application of trained models for the detection of events in a population of PMUs much larger than that associated to the labeled events. Support in data labeling / label validation: Labels are critical for training of models for identification of specific types of events. However, labeling large amounts of data is a manual and tedious process. This means that such process is error prone and is not scalable. Methods developed in the project, based on ensembles of clustering models, have been successfully employed for turning manual labeling into a scalable process. Accurate identification of specific relevant events can provide the operators with immediate situational awareness that could otherwise require hours or days of analysis from domain experts. We envision that such methods could be initially employed in support of post-mortem analysis of events and, as confidence is gained, they could be employed for online/real-time support, providing, among other benefits, insights for avoiding major events which could happen due to a combination of smaller ones. On the longer term, related methods could potentially be employed to improve protection and control.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2021

Subfield crop yields and temporal stability in thousands of US Midwest fields

Understanding subfield crop yields and temporal stability is critical to better manage crops. Several algorithms have proposed to study within-field temporal variability but they were mostly limited to few fields. In this study, a large dataset composed of 5520 yield maps from 768 fields provided by farmers was used to investigate the influence of subfield yield distribution skewness on temporal variability. The data are used to test two intuitive algorithms for mapping stability: one based on standard deviation and the second based on pixel ranking and percentiles. The analysis of yield monitor data indicates that yield distribution is asymmetric, and it tends to be negatively skewed (p < 0.05) for all of the four crops analyzed, meaning that low yielding areas are lower in frequency but cover a larger range of low values. The mean yield difference between the pixels classified as high-and-stable and the pixels classified as low-and-stable was 1.04 Mg ha –1 for maize, 0.39 Mg ha –1 for cotton, 0.34 Mg ha –1 for soybean, and 0.59 Mg ha —1 for wheat. The yield of the unstable zones was similar to the pixels classified as low-and-stable by the standard deviation algorithm, whereas the two-way outlier algorithm did not exhibit this bias. Furthermore, the increase in the number years of yield maps available induced a modest but significant increase in the certainty of stability classifications, and the proportion of unstable pixels increased with the precipitation heterogeneity between the years comprising the yield maps.

59 BASIC BIOLOGICAL SCIENCES↗