Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

Measuring Filament Orientation: A New Quantitative, Local Approach

The relative orientation between filamentary structures in molecular clouds and the ambient magnetic field provides insight into filament formation and stability. To calculate the relative orientation, a measurement of filament orientation is first required. We propose a new method to calculate the orientation of the one-pixel-wide filament skeleton that is output by filament identification algorithms such as filfinder. We derive the local filament orientation from the direction of the intensity gradient in the skeleton image using the Sobel filter and a few simple post-processing steps. We call this the “Sobel-gradient method.” The resulting filament orientation map can be compared quantitatively on a local scale with the magnetic field orientation map to then find the relative orientation of the filament with respect to the magnetic field at each point along the filament. It can also be used for constructing radial profiles for filament width fitting. The proposed method facilitates automation in analyses of filament skeletons, which is imperative in this era of “big data.”.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2018

Machine Learning to Improve Retrieval by Category in Big Volunteered Geodata

Nowadays, Volunteered Geographic Information (VGI) is commonly used in research and practical applications. However, the quality assurance of such a geographic data remains a problem. In this study we use machine learning and natural language processing to improve record retrieval by category (e.g. restaurant, museum, etc.) from Wikimapia Points of Interest data. We use textual information contained in VGI records to evaluate its ability to determine the category label. The performance of the trained classifier is evaluated on the complete dataset and then is compared with its performance on regional subsets. Preliminary analysis shows significant difference in the classifier performance across the regions. Such geographic differences will have a significant effect on data enrichment efforts such as labeling entities with missing categories.

Sorokine, Alexandre↗
DOE OSTI2017

Why wait? Let's start computing while data is still on the wire

In this era of Big Data, computing useful information from data is becoming increasingly complicated, particularly due to the ever increasing volumes of data that need to travel over the network to data centers to be stored and processed, all highly expensive operations in the long haul. In this paper we suggest that we can do computing and analysis of data "on the wire," i.e., while data is still in transit. The nature of these computations include analysis, visualization, pattern recognition, and prediction, or forecasting, on the streaming data. We follow a service function chaining architecture to implement this, assuming that the data packets arrive within a single network administrative domain. As a demonstration of this new computing paradigm, we present three examples. Firstly, we demonstrate pattern recognition and data visualization on streaming forex data, which can be used for lucrative trading in the forex market. In our second example, we analyze and learn user buying patterns from clickstream data streaming from multiple websites. Finally, we monitor solar sensors for a zero reading while the packets are still on their way to the data center, to schedule any maintenance and requisite repairs with no time delay.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Distributed Machine Learning Workflow with PanDA and iDDS in LHC ATLAS

Machine Learning (ML) has become one of the important tools for High Energy Physics analysis. As the size of the dataset increases at the Large Hadron Collider (LHC), and at the same time the search spaces become bigger and bigger in order to exploit the physics potentials, more and more computing resources are required for processing these ML tasks. In addition, complex advanced ML workflows are developed in which one task may depend on the results of previous tasks. How to make use of vast distributed CPUs/GPUs in WLCG for these big complex ML tasks has become a popular research area. In this paper, we present our efforts enabling the execution of distributed ML workflows on the Production and Distributed Analysis (PanDA) system and intelligent Data Delivery Service (iDDS). First, we describe how PanDA and iDDS deal with large-scale ML workflows, including the implementation to process workloads on diverse and geographically distributed computing resources. Next, we report real-world use cases, such as HyperParameter Optimization, Monte Carlo Toy confidence limits calculation, and Active Learning. Finally, we conclude with future plans.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Leveraging AMI data for distribution system model calibration and situational awareness

The many new distributed energy resources being installed at the distribution system level require increased visibility into system operations that will be enabled by distribution system state estimation (DSSE) and situational awareness applications. Reliable and accurate DSSE requires both robust methods for managing the big data provided by smart meters and quality distribution system models. This paper presents intelligent methods for detecting and dealing with missing or inaccurate smart meter data, as well as the ways to process the data for different applications. It also presents an efficient and flexible parameter estimation method based on the voltage drop equation and regression analysis to enhance distribution system model accuracy. Finally, it presents a 3-D graphical user interface for advanced visualization of the system state and events. Moreover, we demonstrate this paper for a university distribution network with the state-of-the-art real-time and historical smart meter data infrastructure.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2019

Strong SUSY at ATLAS and CMS [PowerPoint]

Introduction points: LHC performed very well, delivering > 150 fb -1 ; Huge challenge to quickly process this amount of data; Understand detector & object performance; pT miss tails very important for SUSY searches; Most results on full Run 2 data set will come in Summer; SUSY searches are expanding; Big inclusive searches complemented by dedicated searches to close gaps in coverage; Machine learning for H, b, c, top,… tagging; and Improved analysis techniques

43 PARTICLE ACCELERATORS↗
DOE OSTI2013

Simulation and big data challenges in tuning building energy models

EnergyPlus is the flagship building energy simulation software used to model whole building energy consumption for residential and commercial establishments. A typical input to the program often has hundreds, sometimes thousands of parameters which are typically tweaked by a buildings expert to “get it right”. This process can sometimes take months. “Autotune” is an ongoing research effort employing machine learning techniques to automate the tuning of the input parameters for an EnergyPlus input description of a building. Even with automation, the computational challenge faced to run the tuning simulation ensemble is daunting and requires the use of supercomputers to make it tractable in time. In this paper, we describe the scope of the problem, particularly the technical challenges faced and overcome, and the software infrastructure developed/in development when taking the EnergyPlus engine, which was primarily designed to run on desktops, and scaling it to run on shared memory supercomputers (Nautilus) and distributed memory supercomputers (Frost and Titan). The parametric simulations produce data in the order of tens to a couple of hundred terabytes. We describe the approaches employed to streamline and reduce bottlenecks in the workflow for this data, which is subsequently being made available for the tuning effort as well as made available publicly for open-science.

Sanyal, Jibonananda↗
DOE OSTI2013

Simulation and Big Data Challenges in Tuning Building Energy Models

EnergyPlus is the flagship building energy simulation software used to model whole building energy consumption for residential and commercial establishments. A typical input to the program often has hundreds, sometimes thousands of parameters which are typically tweaked by a buildings expert to get it right . This process can sometimes take months. Autotune is an ongoing research effort employing machine learning techniques to automate the tuning of the input parameters for an EnergyPlus input description of a building. Even with automation, the computational challenge faced to run the tuning simulation ensemble is daunting and requires the use of supercomputers to make it tractable in time. In this proposal, we describe the scope of the problem, the technical challenges faced and overcome, the machine learning techniques developed and employed, and the software infrastructure developed/in development when taking the EnergyPlus engine, which was primarily designed to run on desktops, and scaling it to run on shared memory supercomputers (Nautilus) and distributed memory supercomputers (Frost and Titan). The parametric simulations produce data in the order of tens to a couple of hundred terabytes.We describe the approaches employed to streamline and reduce bottlenecks in the workflow for this data, which is subsequently being made available for the tuning effort as well as made available publicly for open-science.

Sanyal, Jibonananda↗
DOE OSTI2017

A Data Analysis Framework for Earth System Simulation within an In-Situ Infrastructure

This paper presents a generic procedure to implement a scalable and high performance data analysis framework for large-scale scientific simulation within an in-situ infrastructure. It demonstrates a unique capability for global Earth system simulations using advanced computing technologies (i.e ., automated code analysis and instrumentation), in-situ infrastructure (i.e ., ADIOS) and big data analysis engines (i.e ., SciKit-learn). This paper also includes a useful case that analyzes a globe Earth System simulations with the integration of scalable in-situ infrastructure and advanced data processing package. Finally, the in-situ data analysis framework can provides new insights on scientific discoveries in multiscale modeling paradigms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Big data health physics: a case study in operational health physics

This paper, written as a two part project as required by Illinois Institute of Technology’s Phys597 Reading & Special Problems class and health physics professional master’s program, focuses on the results of a prototype data collection process implemented at a Los Alamos National Laboratory facility. In particular, this paper examines the data in a statistical context in an effort to optimize operational health physics processes at the Los Alamos National Laboratory.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2015

Sterile neutrinos with secret interactions — lasting friendship with cosmology

Sterile neutrinos with mass ≃1 eV and order 10% mixing with active neutrinos have been proposed as a solution to anomalies in neutrino oscillation data, but are tightly constrained by cosmological limits. It was recently shown that these constraints are avoided if sterile neutrinos couple to a new MeV-scale gauge boson A{sup ′}. However, even this scenario is restricted by structure formation constraints when A{sup ′}-mediated collisional processes lead to efficient active-to-sterile neutrino conversion after neutrinos have decoupled. In view of this, we reevaluate in this paper the viability of sterile neutrinos with such “secret” interactions. We carefully dissect their evolution in the early Universe, including the various production channels and the expected modifications to large scale structure formation. We argue that there are two regions in parameter space — one at very small A{sup ′} coupling, one at relatively large A{sup ′} coupling — where all constraints from big bang nucleosynthesis (BBN), cosmic microwave background (CMB), and large scale structure (LSS) data are satisfied. Interestingly, the large A{sup ′} coupling region is precisely the region that was previously shown to have potentially important consequences for the small scale structure of dark matter halos if the A{sup ′} boson couples also to the dark matter in the Universe.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

Melt pool temperature measurement and monitoring during laser powder bed fusion based additive manufacturing via single-camera two-wavelength imaging pyrometry (STWIP)

Melt pool (MP) temperature is one of the determining factors and a key signature for evaluating the properties of printed components in metal additive manufacturing (AM). The state-of-the-art measurement systems are hindered, primarily by the large-scale data acquisition and processing demands. In this work, we introduce a novel coaxial, high-speed, single-camera two-wavelength imaging pyrometer (STWIP) system as opposed to the typical utilization of multiple cameras for measuring MP temperature profiles in laser powder bed fusion (LPBF) processes. Developed on a commercial LPBF machine (EOS M290), the STWIP system demonstrated its ability to quantitatively monitor the MP temperature and its variation for 50 layers at high framerates (>30,000 fps) for a real-world application (standard fatigue specimens) print. High performance computing is employed to analyze the acquired big data (MP images), for determining each MP's average temperature and 2D temperature profile. The MP temperature evolution in the gage section of a fatigue specimen is also examined at a temporal resolution of 1 ms, by evaluating the MP temperatures in the samples' first, middle, and last layers. This report is the first of its kind on monitoring MP temperature distribution and evolution at such a large, detailed scale for longer durations in practical applications.

42 ENGINEERING↗
DOE OSTI2016

Accelerating Science Impact through Big Data Workflow Management and Supercomputing

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. ATLAS, one of the largest collaborations ever assembled in the the history of science, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. To manage the workflow for all data processing on hundreds of data centers the PanDA (Production and Distributed Analysis)Workload Management System is used. An ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF), is realizing within BigPanDA and megaPanDA projects. These projects are now exploring how PanDA might be used for managing computing jobs that run on supercomputers including OLCF’s Titan and NRC-KI HPC2. The main idea is to reuse, as much as possible, existing components of the PanDA system that are already deployed on the LHC Grid for analysis of physics data. The next generation of PanDA will allow many data-intensive sciences employing a variety of computing platforms to benefit from ATLAS experience and proven tools in highly scalable processing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

ZMPY3D: accelerating protein structure volume analysis through vectorized 3D Zernike moments and Python-based GPU integration

Abstract Motivation Volumetric 3D object analyses are being applied in research fields such as structural bioinformatics, biophysics, and structural biology, with potential integration of artificial intelligence/machine learning (AI/ML) techniques. One such method, 3D Zernike moments, has proven valuable in analyzing protein structures (e.g., protein fold classification, protein–protein interaction analysis, and molecular dynamics simulations). Their compactness and efficiency make them amenable to large-scale analyses. Established methods for deriving 3D Zernike moments, however, can be inefficient, particularly when higher order terms are required, hindering broader applications. As the volume of experimental and computationally-predicted protein structure information continues to increase, structural biology has become a “big data” science requiring more efficient analysis tools. Results This application note presents a Python-based software package, ZMPY3D, to accelerate computation of 3D Zernike moments by vectorizing the mathematical formulae and using graphical processing units (GPUs). The package offers popular GPU-supported libraries such as CuPy and TensorFlow together with NumPy implementations, aiming to improve computational efficiency, adaptability, and flexibility in future algorithm development. The ZMPY3D package can be installed via PyPI, and the source code is available from GitHub. Volumetric-based protein 3D structural similarity scores and transform matrix of superposition functionalities have both been implemented, creating a powerful computational tool that will allow the research community to amalgamate 3D Zernike moments with existing AI/ML tools, to advance research and education in protein structure bioinformatics. Availability and implementation ZMPY3D, implemented in Python, is available on GitHub (https://github.com/tawssie/ZMPY3D) and PyPI, released under the GPL License.

Lai, Jhih-Siang (ORCID:0000000156775890)↗
DOE OSTI2019

A communication-avoiding 3D algorithm for sparse LU factorization on heterogeneous systems

We propose a new algorithm to improve the strong scalability of right-looking sparse LU factorization on distributed memory systems. Our 3D algorithm for sparse LU uses a three-dimensional MPI process grid, exploits elimination tree parallelism, and trades off increased memory for reduced per-process communication. We also analyze the asymptotic improvements for planar graphs (e.g., those arising from 2D grid or mesh discretizations) and certain non-planar graphs (specifically for 3D grids and meshes). For a planar graph with $n$ vertices, our algorithm reduces communication volume asymptotically in $n$ by a factor of $\mathscr{O}$ $\Big(\sqrt{log \ n}\Big)$ and latency by a factor of $\mathscr{O}$ $(log \ n)$. For nonplanar cases, our algorithm can reduce the per-process communication volume by 3× and latency by $\mathscr{O}$ $\Big(n^\frac13\Big)$ times. In all cases, the memory needed to achieve these gains is a constant factor. We implemented our algorithm by extending the 2D data structure used in SuperLU_DIST. Our new 3D code achieves empirical speedups up to 27× for planar graphs and up to 3.3× for non-planar graphs over the baseline 2D SuperLU_DIST when run on 24,000 cores of a Cray XC30. We extend the 3D algorithm for heterogeneous architectures by adding the Highly Asynchronous Lazy Offload (Halo) algorithm for co-processor offload. On 4096 nodes of a Cray XK7 with 32,768 CPU cores and 4096 Nvidia K20x GPUs, the 3D algorithm achieves empirical speedups up to 24× for planar graphs and 3.5× for non-planar graphs over the baseline 2D SuperLU_DIST with co-processor acceleration.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2011

ARE THERE ECHOES FROM THE PRE-BIG-BANG UNIVERSE? A SEARCH FOR LOW-VARIANCE CIRCLES IN THE COSMIC MICROWAVE BACKGROUND SKY

In a recent analysis of Wilkinson Microwave Anisotropy Probe (WMAP) seven-year temperature maps, Gurzadyan and Penrose claim to find concentric circular patterns in the sky with anomalously low variances. These circles are presented as observational evidence for violent processes in a universe preceding our big bang as predicted by Penrose's Conformal Cyclic Cosmology. We reassess the statistical significance of the detection of the claimed concentric low-variance circles by comparing the WMAP data with Monte Carlo simulations of the cosmic microwave background (CMB) sky plus realistic modeling of WMAP's anisotropic noise. We find no anomaly in the variances compared with the {Lambda}CDM cosmological model. The observed variances in the data are consistent with a Gaussian CMB sky as predicted by the inflationary cosmology model at better than 3{sigma}.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2012

Clinical evaluation of a commercial orthopedic metal artifact reduction tool for CT simulations in radiation therapy

Purpose: Severe artifacts in kilovoltage-CT simulation images caused by large metallic implants can significantly degrade the conspicuity and apparent CT Hounsfield number of targets and anatomic structures, jeopardize the confidence of anatomical segmentation, and introduce inaccuracies into the radiation therapy treatment planning process. This study evaluated the performance of the first commercial orthopedic metal artifact reduction function (O-MAR) for radiation therapy, and investigated its clinical applications in treatment planning. Methods: Both phantom and clinical data were used for the evaluation. The CIRS electron density phantom with known physical (and electron) density plugs and removable titanium implants was scanned on a Philips Brilliance Big Bore 16-slice CT simulator. The CT Hounsfield numbers of density plugs on both uncorrected and O-MAR corrected images were compared. Treatment planning accuracy was evaluated by comparing simulated dose distributions computed using the true density images, uncorrected images, and O-MAR corrected images. Ten CT image sets of patients with large hip implants were processed with the O-MAR function and evaluated by two radiation oncologists using a five-point score for overall image quality, anatomical conspicuity, and CT Hounsfield number accuracy. By utilizing the same structure contours delineated from the O-MAR corrected images, clinical IMRT treatment plans for five patients were computed on the uncorrected and O-MAR corrected images, respectively, and compared. Results: Results of the phantom study indicated that CT Hounsfield number accuracy and noise were improved on the O-MAR corrected images, especially for images with bilateral metal implants. The {gamma} pass rates of the simulated dose distributions computed on the uncorrected and O-MAR corrected images referenced to those of the true densities were higher than 99.9% (even when using 1% and 3 mm distance-to-agreement criterion), suggesting that dose distributions were clinically identical. In all patient cases, radiation oncologists rated O-MAR corrected images as higher quality. Formerly obscured critical structures were able to be visualized. The overall image quality and the conspicuity in critical organs were significantly improved compared with the uncorrected images: overall quality score (1.35 vs 3.25, P= 0.0022); bladder (2.15 vs 3.7, P= 0.0023); prostate and seminal vesicles/vagina (1.3 vs 3.275, P= 0.0020); rectum (2.8 vs 3.9, P= 0.0021). The noise levels of the selected ROIs were reduced from 93.7 to 38.2 HU. On most cases (8/10), the average CT Hounsfield numbers of the prostate/vagina on the O-MAR corrected images were closer to the referenced value (41.2 HU, an average measured from patients without metal implants) than those on the uncorrected images. High {gamma} pass rates of the five IMRT dose distribution pairs indicated that the dose distributions were not significantly affected by the CT image improvements. Conclusions: Overall, this study indicated that the O-MAR function can remarkably reduce metal artifacts and improve both CT Hounsfield number accuracy and target and critical structure visualization. Although there was no significant impact of the O-MAR algorithm on the calculated dose distributions, we suggest that O-MAR corrected images are more suitable for the entire treatment planning process by offering better anatomical structure visualization, improving radiation oncologists' confidence in target delineation, and by avoiding subjective density overrides of artifact regions on uncorrected images.

61 RADIATION PROTECTION AND DOSIMETRY↗