Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23
All sourcesNASA NTRSDOE OSTI
DOE OSTI2013

Efficient Aho-Corasick String Matching on Emerging Multicore Architectures

String matching algorithms are critical to several scientific fields. Beside text processing and databases, emerging applications such as DNA protein sequence analysis, data mining, information security software, antivirus, ma- chine learning, all exploit string matching algorithms [3]. All these applica- tions usually process large quantity of textual data, require high performance and/or predictable execution times. Among all the string matching algorithms, one of the most studied, especially for text processing and security applica- tions, is the Aho-Corasick algorithm. 1 2 Book title goes here Aho-Corasick is an exact, multi-pattern string matching algorithm which performs the search in a time linearly proportional to the length of the input text independently from pattern set size. However, depending on the imple- mentation, when the number of patterns increase, the memory occupation may raise drastically. In turn, this can lead to significant variability in the performance, due to the memory access times and the caching effects. This is a significant concern for many mission critical applications and modern high performance architectures. For example, security applications such as Network Intrusion Detection Systems (NIDS), must be able to scan network traffic against very large dictionaries in real time. Modern Ethernet links reach up to 10 Gbps, and malicious threats are already well over 1 million, and expo- nentially growing [28]. When performing the search, a NIDS should not slow down the network, or let network packets pass unchecked. Nevertheless, on the current state-of-the-art cache based processors, there may be a large per- formance variability when dealing with big dictionaries and inputs that have different frequencies of matching patterns. In particular, when few patterns are matched and they are all in the cache, the procedure is fast. Instead, when they are not in the cache, often because many patterns are matched and the caches are continuously thrashed, they should be retrieved from the system memory and the procedure is slowed down by the increased latency. Efficient implementations of string matching algorithms have been the fo- cus of several works, targeting Field Programmable Gate Arrays [4, 25, 15, 5], highly multi-threaded solutions like the Cray XMT [34], multicore proces- sors [19] or heterogeneous processors like the Cell Broadband Engine [35, 22]. Recently, several researchers have also started to investigate the use Graphic Processing Units (GPUs) for string matching algorithms in security applica- tions [20, 10, 32, 33]. Most of these approaches mainly focus on reaching high peak performance, or try to optimize the memory occupation, rather than looking at performance stability. However, hardware solutions supports only small dictionary sizes due to lack of memory and are difficult to customize, while platforms such as the Cell/B.E. are very complex to program.

Tumeo, Antonino↗
DOE OSTI2011

Universal squash model for optical communications using linear optics and threshold detectors

Transmission of photons through open-air or optical fibers is an important primitive in quantum-information processing. Theoretical descriptions of this process often consider single photons as information carriers and thus fail to accurately describe experimental implementations where any number of photons may enter a detector. It has been a great challenge to bridge this big gap between theory and experiments. One powerful method for achieving this goal is by conceptually squashing the received multiphoton states to single-photon states. However, until now, only a few protocols admit a squash model; furthermore, a recently proven no-go theorem appears to rule out the existence of a universal squash model. Here we show that a necessary condition presumed by all existing squash models is in fact too stringent. By relaxing this condition, we find that, rather surprisingly, a universal squash model actually exists for many protocols, including quantum key distribution, quantum state tomography, Bell's inequality testing, and entanglement verification.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2018

Classification of machine learning frameworks for data-driven thermal fluid models

Thermal fluid processes are inherently multi-physics and multi-scale, involving mass-momentum-energy transport phenomena at multiple scales. Thermal fluid simulation (TFS) is based on solving conservative equations, for which – except for “first-principles” direct numerical simulation – closure relations (CRs) are required to provide microscopic interactions or so-called sub-grid-scale physics. In practice, TFS is realized through reduced-order modeling, and its CRs as low-fidelity models can be informed by observations and data from relevant and adequately evaluated experiments and high-fidelity simulations. This paper is focused on data-driven TFS models, specifically on their development using machine learning (ML). Five ML frameworks are introduced including physics-separated ML (PSML or Type I ML), physics-evaluated ML (PEML or Type II ML), physics-integrated ML (PIML or Type III ML), physics-recovered (PRML or Type IV ML), and physics-discovered ML (PDML or Type V ML). The frameworks vary in their performance for different applications depending on the level of knowledge of governing physics, source, type, amount and quality of available data for training. Notably, outlined for the first time in this paper, Type III models present stringent requirements on modeling, substantial computing resources for training, and high potential in extracting value from “big data” in thermal fluid research. Here the current paper demonstrates and investigates ML frameworks in three examples. First, we utilize the heat diffusion equation with a nonlinear conductivity model formulated by convolutional neural networks (CNNs) and feedforward neural networks (FNNs) to illustrate the applications of Type I, Type II, Type III, and Type V ML. The results indicate a preference for Type II ML under deficient data support. Type III ML can effectively utilize field data, potentially generating more robust predictions than Type I and Type II ML. CNN-based closures exhibit more predictability than FNN-based closures, but CNN-based closures require more training data to obtain accurate predictions. Second, we illustrate how to employ Type I ML and Type II ML frameworks for data-driven turbulence modeling using reference works. Third, we demonstrate Type I ML by building a deep FNN-based slip closure for two-phase flow modeling. The results show that deep FNN-based closures exhibit a bounded error in the prediction domain.

42 ENGINEERING↗
DOE OSTI2016

True beam commissioning experience at Nordland Hospital Trust, Norway

To evaluate the measured of all photon beam data of first Varian True Beam version 2.0 slim model, recently commissioned at Nordland Hospital Trust, Bodø. To compare and evaluate the possibility of beam matching with the Clinac2300, for the energies of 6MV and 15 MV. Materials/Methods: Measurements of PDD, OAR, and Output factors were realized with the IBA Blue-phantom with different detectors and evaluated between them for all photon energies: 6MV, 15MV, 6MV FFF and 10MV FFF. The ionization chambers used were Pin Point CC01, CC04, Semiflex CC13 and photon diode by Iba dosimetry. The data were processed using Beizer algorithm with a resolution of 1 mm. The measured depth dose curves, diagonals, OAR, and output factors were imported into Eclipse in order to calculate beam data for the anisotropic analytical algorithm (AAA version 10.0.28) for both the dataset measured with CC04 and CC13 and compared. The model head of 23EX was selected as the most near model to True Beam as a restriction of our version of Aria. It was seen that better results were achieved with the CC04 measured data as a result of better resolution. For the biggest field after 10 cm depth a larger difference is seen between measured and calculated for both dataset, but it is within the criteria for acceptance. Results: The Beam analysis criteria of 2 mm at 50% dose is achieved for all the fields accept for 40x40 that is within 3%. Depth difference at maximum dose is within 1 mm for all the fields and dose difference at 100 mm and 200 mm is lower than 1% for or all the fields. The PDD between two machines for all the fields differ after Dmax with less than 1%. For profiles in the field zone and outside field the difference is within 1% for all the fields. In the penumbra region the difference is from 2% up to 12% for big fields. As for diagonals they differ as a result of the head construction at the edge of the field and the penumbra region. The output factors differ for big fields within 5% and for the small fields within 3%. MU and dose distribution does not change for plans recalculated with the new modeled machine.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2018

Planning of distributed data production for High Energy and Nuclear Physics

Modern experiments in High Energy and Nuclear Physics heavily rely on distributed computations using multiple computational facilities across the world. One of the essential types of the computations is a distributed data production where petabytes of raw files from a single source has to be processed once (per production campaign) using thousands of CPUs at distant locations and the output has to be transferred back to that source. The data distribution over a large system does not necessary match the distribution of storage, network and CPU capacity. Therefore, bottlenecks may appear and lead to increased latency and degraded performance. In this paper we propose a new scheduling approach for distributed data production which is based on the network flow maximization model. In our approach a central planner defines how much input and output data should be transferred over each network link in order to maximize the computational throughput. Such plans are created periodically for a fixed planning time interval using up-to-date information on network, storage and CPU resources. The centrally created plans are executed in a distributed manner by dedicated services running at participating sites. In conclusion, our simulations based on the log records from the data production framework of the experiment STAR (Solenoid Tracker at RHIC) have shown that the proposed model systematically provides a better performance compared to the simulated traditional techniques.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2015

An adaptive filtered back-projection for photoacoustic image reconstruction

Purpose: The purpose of this study is to develop an improved filtered-back-projection (FBP) algorithm for photoacoustic tomography (PAT), which allows image reconstruction with higher quality compared to images reconstructed through traditional algorithms. Methods: A rigorous expression of a weighting function has been derived directly from a photoacoustic wave equation and used as a ramp filter in Fourier domain. The authors’ new algorithm utilizes this weighting function to precisely calculate each photoacoustic signal’s contribution and then reconstructs the image based on the retarded potential generated from the photoacoustic sources. In addition, an adaptive criterion has been derived for selecting the cutoff frequency of a low pass filter. Two computational phantoms were created to test the algorithm. The first phantom contained five spheres with each sphere having different absorbances. The phantom was used to test the capability for correctly representing both the geometry and the relative absorbed energy in a planar measurement system. The authors also used another phantom containing absorbers of different sizes with overlapping geometry to evaluate the performance of the new method for complicated geometry. In addition, random noise background was added to the simulated data, which were obtained by using an arc-shaped array of 50 evenly distributed transducers that spanned 160° over a circle with a radius of 65 mm. A normalized factor between the neighbored transducers was applied for correcting measurement signals in PAT simulations. The authors assumed that the scanned object was mounted on a holder that rotated over the full 360° and the scans were set to a sampling rate of 20.48 MHz. Results: The authors have obtained reconstructed images of the computerized phantoms by utilizing the new FBP algorithm. From the reconstructed image of the first phantom, one can see that this new approach allows not only obtaining a sharp image but also showing the correct signal strength of the absorbers. The reconstructed image of the second phantom further demonstrates the capability to form clear images of the spheres with sharp borders in the overlapping geometry. The smallest sphere is clearly visible and distinguishable, even though it is surrounded by two big spheres. In addition, image reconstructions were conducted with randomized noise added to the observed signals to mimic realistic experimental conditions. Conclusions: The authors have developed a new FBP algorithm that is capable for reconstructing high quality images with correct relative intensities and sharp borders for PAT. The results demonstrate that the weighting function serves as a precise ramp filter for processing the observed signals in the Fourier domain. In addition, this algorithm allows an adaptive determination of the cutoff frequency for the applied low pass filter.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2021

Deep learning for Alzheimer's disease: Mapping large-scale histological tau protein for neuroimaging biomarker validation

Abnormal tau inclusions are hallmarks of Alzheimer's disease and predictors of clinical decline. Several tau PET tracers are available for neurodegenerative disease research, opening avenues for molecular diagnosis in vivo. However, few have been approved for clinical use. Understanding the neurobiological basis of PET signal validation remains problematic because it requires a large-scale, voxel-to-voxel correlation between PET and (immuno) histological signals. Large dimensionality of whole human brains, tissue deformation impacting co-registration, and computing requirements to process terabytes of information preclude proper validation. We developed a computational pipeline to identify and segment particles of interest in billion-pixel digital pathology images to generate quantitative, 3D density maps. The proposed convolutional neural network for immunohistochemistry samples, IHCNet, is at the pipeline's core. We have successfully processed and immunostained over 500 slides from two whole human brains with three phospho-tau antibodies (AT100, AT8, and MC1), spanning several terabytes of images. Our artificial neural network estimated tau inclusion from brain images, which performs with ROC AUC of 0.87, 0.85, and 0.91 for AT100, AT8, and MC1, respectively. Introspection studies further assessed the ability of our trained model to learn tau-related features. We present an end-to-end pipeline to create terabytes-large 3D tau inclusion density maps co-registered to MRI as a means to facilitate validation of PET tracers.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2021

Identification and Online Updating of Dynamic Models for Demand Response of an Industrial Air Separation Unit

Demand-response operation of air separation units requires frequent changes in production rate(s), and scheduling calculations must explicitly consider process dynamics to ensure feasibility of the solutions. To this end, scale-bridging models (SBMs) approximate the scheduling-relevant dynamics of a process and its controller in a low-order representation. In contrast to previous works that have employed nonlinear SBMs, this paper proposes linear SBMs, developed using time-series analysis, to facilitate online scheduling computations. Using a year-long industrial dataset, we nd that compact linear SBMs are suitable approximations over typical scheduling horizons, but that their accuracies are unpredictable over time. We introduce a strategy for online updating of the SBMs, based on Kalman ltering schemes for online parameter estimation. The approach greatly improves the accuracy of SBM predictions and will enable the use of linear SBM-based demand-response scheduling in the future.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Reconstructing large-scale temperature profiles around z ∼ 6 quasars

ABSTRACT High-redshift quasars ionize He ii into He iii around them, heating the intergalactic medium in the process and creating large regions with elevated temperature. In this work, we demonstrate a method based on a convolutional neural network (CNN) to recover the spatial profile for T0, the temperature at the mean cosmic density, in quasar proximity zones. We train the neural network with synthetic spectra drawn from a Cosmic Reionization on Computers simulation. We discover that the simple CNN is able to recover the temperature profile with an accuracy of ≈1400 K in an idealized case of negligible observational uncertainties. We test the robustness of the CNN and discover that it is robust against the uncertainties in quasar host halo mass, quasar continuum, and ionizing flux. We also find that the CNN has good generality with regard to the hardness of quasar spectra. This shows that with noiseless spectra, one could use a simple CNN to distinguish gas inside or outside the He iii region created by the quasar. Because the size of the He iii region is closely related to the total quasar lifetime, this method has great potential in constraining the quasar lifetime on ∼Myr time-scales. However, noise poses a big problem for accuracy and could downgrade the accuracy to ≈2340 K even for very high signal-to-noise (≳50) spectra. Future studies are needed to reduce the error associated with noise to constrain the lifetimes of reionization epoch quasars with currently available data.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2015

Transportation Database Development Using Floating Car Data

In support of projects funded by the U.S. Department of Transportation and the U.S. Department of Energy, the National Renewable Energy Laboratory (NREL) has received on average 4 million floating car data (FCD) points each day over a 3-year period. To handle data sets of this size, the Transportation & Hydrogen Systems Center at NREL has developed a series of transportation databases (Transportation Secure Data Center and Fleet DNA projects) to support real-world vehicle and transportation infrastructure analysis on a national scale. To ensure the quality of the FCD received by NREL and to ensure that data can be accessed and applied quickly and easily across transportation research disciplines, a customized data processing routine and database were required. 'Floating car data' is a general term for location data recorded by a vehicle or by a person traveling in a vehicle. FCD are used to support real-time traffic information systems and have proven to be beneficial in a large number of research applications. For large-scale traffic information systems, FCD are logged from smart phones using either the GPS included in the device, or if GPS is disabled, triangulation from the network. Additionally, in scientific applications FCD are oftentimes collected on a smaller scale using onboard GPS-enabled data logging systems. Through the fusion of real-time data and historically logged results, up-to-date and accurate vehicle operating information are provided in near real-time to support routing and operation decisions. The approach outlined within this paper builds upon methods developed for advanced traffic information systems to link FCD to the street of travel and further extends the link to data recorded by components within a vehicle. Linking vehicle component performance to the infrastructure a vehicle is operating within provides the path to assess vehicle and infrastructure interactions, going far beyond providing information on traffic flow alone.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2024

Convergence in simulating global soil organic carbon by structurally different models after data assimilation

Abstract Current biogeochemical models produce carbon–climate feedback projections with large uncertainties, often attributed to their structural differences when simulating soil organic carbon (SOC) dynamics worldwide. However, choices of model parameter values that quantify the strength and represent properties of different soil carbon cycle processes could also contribute to model simulation uncertainties. Here, we demonstrate the critical role of using common observational data in reducing model uncertainty in estimates of global SOC storage. Two structurally different models featuring distinctive carbon pools, decomposition kinetics, and carbon transfer pathways simulate opposite global SOC distributions with their customary parameter values yet converge to similar results after being informed by the same global SOC database using a data assimilation approach. The converged spatial SOC simulations result from similar simulations in key model components such as carbon transfer efficiency, baseline decomposition rate, and environmental effects on carbon fluxes by these two models after data assimilation. Moreover, data assimilation results suggest equally effective simulations of SOC using models following either first‐order or Michaelis–Menten kinetics at the global scale. Nevertheless, a wider range of data with high‐quality control and assurance are needed to further constrain SOC dynamics simulations and reduce unconstrained parameters. New sets of data, such as microbial genomics‐function relationships, may also suggest novel structures to account for in future model development. Overall, our results highlight the importance of observational data in informing model development and constraining model predictions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

MPEX AI Digital Twins

All magnetically confined plasma fusion power plant concepts (Tokamak, Spherical Tokamak, Stellarator, Mirror, ...) must exhaust the heat and plasma from the core confinement region to the material walls. The primary channel for this exhaust is through a plasma divertor which directs plasma along open magnetic field lines to a material target. The Material Plasma Exposure eXperiment (MPEX) illustrated in Figure 1, is a high-power, steady-state linear plasma device designed to produce the plasma material interaction (PMI) conditions of the divertor of future magnetic confinement fusion power plants: energy flux 20MW/m 2 , ion fluence 1031/m 2 , pulse duration 106 sec. These goals of plasma exposure in MPEX are well beyond those achieved in magnetic fusion experimental devices. Successfully achieving these high power steady state conditions for long pulses requires operational control of the heating and particle sources and the plasma flux to the walls and target. The MPEX AI Hot Spot Controller, proposed in this project, will help achieve the operational milestones of MPEX. The MPEX device will begin commissioning at the end of FY26. A smaller proto-MPEX was operated for 14,666 plasma discharges and will resume operation in September of 2025 as proto-MPEX-lite, with reduced capability, to test a new window for the Helicon plasma source. The proto-MPEX data has undergone surrogate modeling with machine learning methods (R. Archibald, 2022 IEEE International Conference on Big Data). This proto-MPEX data will be used to begin development of the AI digital twins described in this white paper. The scientific mission of MPEX is to qualify materials of different composition for use in the high energy and plasma flux conditions of a fusion power plant. The materials exposed in MPEX will in some cases be exposed to high neutron fluxes at other ORNL facilities to measure the changes to their PMI properties. The targets exposed in MPEX will be transported under vacuum to a Surface Analysis Station (SAS). The SAS will be equipped with the following diagnostics: Focused Ion Beam (FIB) for trench milling, 100-400 angstrom resolution scanning electron microscope (SEM), surface mapping x-ray spectrometer, high resolution camera, and a future upgrade to a laser induced breakdown spectroscopy quadruple mass spectrometer (LIBS-QMS). The MPEX experiments will generate diverse pre- and post-exposure measurement data of detailed material properties down to the crystal grain level in 3D for post-exposure assessment of PMI damage (e.g. cracking, melting, erosion and redeposition of the material). Physics models for the PMI, and how the material composition and manufacturing impact its performance under high energy plasma exposure, need to be validated with MPEX data to guide the selection of new candidate materials. Our vision for the MPEX AI Digital Twins project is to supply experimental and physics model simulation data to train Artificial Intelligence (AI) models for data processing, analysis, operational control, PMI and materials simulation to maximize the scientific output of the MPEX device. Ultimately, an AI digital twin of MPEX material assessment metrics for tested and synthetic material types with simulated PMI will be trained by the AI Modeling Teams on the experimental and physics simulation data submitted to the American Science Cloud by this project. A purely empirical search for the best material is inefficient given the finite number of samples that can be tested on MPEX. In order to expand the material properties database for training the MPEX Material Assessment AI Digital Twin, and to gain physics understanding of the PMI processes, physics models of the material properties and PMI processes are required. The physics simulations provide detailed simulation data, like impact angles for plasma ions, sputtering yields, transport of the ionized sputtered target material in the plasma, and redeposition locations. This simulation data expands the measurement data for deeper physics understanding. The experimental data is essential to validate the PMI and material structure simulation models. The validated models can then be used to generate new simulation data of MPEX material assessments for synthetic material compositions that have not been exposed in MPEX. These predictive simulations, plus the whole experimental dataset, will be used to train the MPEX Material Assessment AI Digital Twin allowing a rapid generative AI search for new materials with reduced PMI damage by interpolating the domain of the training set. These new optimum materials can be simulated with the physics codes and/or tested in MPEX. The ability of AI neural networks to interpolate multi-dimensional parameter spaces and generate virtual data is exploited for a more efficient search for optimum materials. The advent of the Transformational AI Models Consortium (TAIMC) is an opportunity to engage with state of the art private and public AI developers to achieve the goals of the AI digital twins and AI accelerated physics models proposed in this project. Our partners at ORNL from the Advance Scientific Computing Research (ASCR) organization will collaborate in accelerating the integrated plasma material interaction simulation framework. This simulation framework will provide a platform for generating simulation data across a range of physical fidelities, including hybrid methods that produce multi-fidelity results. This data will be leveraged for AI model development, both for generation of surrogates and the automation of simulation campaigns. A part of the research below will include collaborative efforts with the TAIMC to (i) adapt data storage approaches to ensure AI-readiness, (ii) provide a protypical exemplar to inform and exercise constructed workflows, and (iii) generate and share data, using the TAIMC unified AI data standard, for foundational models that will be trained from multiple sources across the DOE complex. We will also collaborate with the TAIMC, as well as the planned AI modeling teams, to develop approaches for reducing the cost of data generation. These include tailored multi-fidelity approaches as well as fine-tuning strategies to augment general, large-scale foundational models.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2016

Large-Scale Public Transcriptomic Data Mining Reveals a Tight Connection between the Transport of Nitrogen and Other Transport Processes in Arabidopsis

Movement of nitrogen to the plant tissues where it is needed for growth is an important contribution to nitrogen use efficiency. However, we have very limited knowledge about the mechanisms of nitrogen transport. Loading of nitrogen into the xylem and/or phloem by transporter proteins is likely important, but there are several families of genes that encode transporters of nitrogenous molecules (collectively referred to as N transporters here), each comprised of many gene members. In this study, we leveraged publicly available microarray data of Arabidopsis to investigate the gene networks of N transporters to elucidate their possible biological roles. First, we showed that tissue-specificity of nitrogen (N) transporters was well reflected among the public microarray data. Then, we built coexpression networks of N transporters, which showed relationships between N transporters and particular aspects of plant metabolism, such as phenylpropanoid biosynthesis and carbohydrate metabolism. Furthermore, genes associated with several biological pathways were found to be tightly coexpressed with N transporters in different tissues. Our coexpression networks provide information at the systems-level that will serve as a resource for future investigation of nitrogen transport systems in plants, including candidate gene clusters that may work together in related biological roles.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2015

Prediction Models for Dynamic Demand Response

As Smart Grids move closer to dynamic curtailment programs, Demand Response (DR) events will become necessary not only on fixed time intervals and weekdays predetermined by static policies, but also during changing decision periods and weekends to react to real-time demand signals. Unique challenges arise in this context vis-a-vis demand prediction and curtailment estimation and the transformation of such tasks into an automated, efficient dynamic demand response (D 2 R) process. While existing work has concentrated on increasing the accuracy of prediction models for DR, there is a lack of studies for prediction models for D 2 R, which we address in this paper. Our first contribution is the formal definition of D 2 R, and the description of its challenges and requirements. Our second contribution is a feasibility analysis of very-short-term prediction of electricity consumption for D 2 R over a diverse, large-scale dataset that includes both small residential customers and large buildings. Our third, and major contribution is a set of insights into the predictability of electricity consumption in the context of D 2 R. Specifically, we focus on prediction models that can operate at a very small data granularity (here 15-min intervals), for both weekdays and weekends - all conditions that characterize scenarios for D 2 R. We find that short-term time series and simple averaging models used by Independent Service Operators and utilities achieve superior prediction accuracy. We also observe that workdays are more predictable than weekends and holiday. Also, smaller customers have large variation in consumption and are less predictable than larger buildings. Key implications of our findings are that better models are required for small customers and for non-workdays, both of which are critical for D 2 R. Also, prediction models require just few days’ worth of data indicating that small amounts of historical training data can be used to make reliable predictions, simplifying the complexity of big data challenge associated with D 2 R.

Aman, Saima↗
DOE OSTI2014

Big Fish in Small Ponds: massive stars in the low-mass clusters of M83

We have used multi-wavelength Hubble Space Telescope WFC3 data of the starbursting spiral galaxy M83 in order to measure variations in the upper end of the stellar initial mass function (uIMF) using the production rate of ionizing photons in unresolved clusters with ages ≤ 8 Myr. As in earlier papers on M51 and NGC 4214, the uIMF in M83 is consistent with a universal IMF, and stochastic sampling of the stellar populations in the ∼<10{sup 3} M {sub ☉} clusters are responsible for any deviations in this universality. The ensemble cluster population, as well as individual clusters, also imply that the most massive star in a cluster does not depend on the cluster mass. In fact, we have found that these small clusters seem to have an over-abundance of ionizing photons when compared to an expected universal or truncated IMF. This also suggests that the presence of massive stars in these clusters does not affect the star formation in a destructive way.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2015

Enabling Graph Appliance for Genome Assembly

In recent years, there has been a huge growth in the amount of genomic data available as reads generated from various genome sequencers. The number of reads generated can be huge, ranging from hundreds to billions of nucleotide, each varying in size. Assembling such large amounts of data is one of the challenging computational problems for both biomedical and data scientists. Most of the genome assemblers developed have used de Bruijn graph techniques. A de Bruijn graph represents a collection of read sequences by billions of vertices and edges, which require large amounts of memory and computational power to store and process. This is the major drawback to de Bruijn graph assembly. Massively parallel, multi-threaded, shared memory systems can be leveraged to overcome some of these issues. The objective of our research is to investigate the feasibility and scalability issues of de Bruijn graph assembly on Cray s Urika-GD system; Urika-GD is a high performance graph appliance with a large shared memory and massively multithreaded custom processor designed for executing SPARQL queries over large-scale RDF data sets. However, to the best of our knowledge, there is no research on representing a de Bruijn graph as an RDF graph or finding Eulerian paths in RDF graphs using SPARQL for potential genome discovery. In this paper, we address the issues involved in representing a de Bruin graphs as RDF graphs and propose an iterative querying approach for finding Eulerian paths in large RDF graphs. We evaluate the performance of our implementation on real world ebola genome datasets and illustrate how genome assembly can be accomplished with Urika-GD using iterative SPARQL queries.

Big data↗
DOE OSTI2020

Software-defined network for end-to-end networked science at the exascale

Domain science applications and workflow processes are currently forced to view the network as an opaque infrastructure into which they inject data and hope that it emerges at the destination with an acceptable Quality of Experience. There is little ability for applications to interact with the network to exchange information, negotiate performance parameters, discover expected performance metrics, or receive status/troubleshooting information in real time. The work presented here is motivated by a vision for a new smart network and smart application ecosystem that will provide a more deterministic and interactive environment for domain science workflows. The Software-Defined Network for End-to-end Networked Science at Exascale (SENSE) system includes a model-based architecture, implementation, and deployment which enables automated end- to-end network service instantiation across administrative domains. An intent based interface allows applications to express their high-level service requirements, an intelligent orchestrator and resource control systems allow for custom tailoring of scalability and real-time responsiveness based on individual application and infrastructure operator requirements. This allows the science applications to manage the network as a first-class schedulable resource as is the current practice for instruments, compute, and storage systems. Deployment and experiments on production networks and testbeds have validated SENSE functions and performance. Emulation based testing verified the scalability needed to support research and education infrastructures. Key contributions of this work include an architecture definition, reference implementation, and deployment. This provides the basis for further innovation of smart network services to accelerate scientific discovery in the era of big data, cloud computing, machine learning and artificial intelligence.

47 OTHER INSTRUMENTATION↗
DOE OSTI2017

Obtaining and Managing Answer Quality for Online Data-Intensive Services

Online data-intensive (OLDI) services use anytime algorithms to compute over large amounts of data and respond quickly. Interactive response times are a priority, so OLDI services parallelize query execution across distributed software components and return best effort answers based on the data so far processed. Omitted data from slow components could lead to better answers, but tracing online how much better the answers could be is difficult. Here, we introduce Ubora, a design method to measure the effect of slow-running components on the quality of answers. Ubora randomly samples online queries and executes them a second time. The first online execution omits data from slow components and provides interactive answers. The second execution uses mature results from intermediate components completed after the online execution finishes. Ubora uses memoization to speed up mature executions by replaying network messages exchanged between components. Our systems-level implementation works for a wide range of services, including Hadoop/Yarn, Apache Lucene, the EasyRec Recommendation Engine, and the OpenEphyra question-answering system. Ubora computes answer quality with more mature executions per second than competing approaches that do not use memoization. With Ubora, we illustrate that answer quality is effective at guiding online admission control. While achieving the same answer quality on high-priority queries, our adaptive controller had 55% higher peak throughput on low-priority queries than a competing controller guided by the rate of timeouts.

answer quality↗