Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Intensive Computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

A Bayesian Approach for Quantifying Data Scarcity when Modeling Human Behavior via Inverse Reinforcement Learning

Computational models that formalize complex human behaviors enable study and understanding of such behaviors. However, collecting behavior data required to estimate the parameters of such models is often tedious and resource intensive. Thus, estimating dataset size as part of data collection planning (also known as Sample Size Determination) is important to reduce the time and effort of behavior data collection while maintaining an accurate estimate of model parameters. In this paper, we present a sample size determination method based on Uncertainty Quantification (UQ) for a specific Inverse Reinforcement Learning (IRL) model of human behavior, in two cases: 1) pre-hoc experiment design—conducted in the planning stage before any data is collected, to guide the estimation of how many samples to collect; and 2) post-hoc dataset analysis—performed after data is collected, to decide if the existing dataset has sufficient samples and whether more data is needed. Here, we validate our approach in experiments with a realistic model of behaviors of people with Multiple Sclerosis (MS) and illustrate how to pick a reasonable sample size target. Our work enables model designers to perform a deeper, principled investigation of effects of dataset size on IRL.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

Improving Large-scale Storage System Performance via Topology-aware and Balanced Data Placement

With the advent of big data, the I/O subsystems of large-scale compute clusters are becoming a center of focus, with more applications putting greater demands on end-to-end I/O performance. These subsystems are often complex in design. They comprise of multiple hardware and software layers to cope with the increasing capacity, capability and scalability requirements of data intensive applications. The sharing nature of storage resources and the intrinsic interactions across these layers make it to realize user-level, end-to-end performance gains a great challenge. We propose a topology-aware resource load balancing strategy to improve per-application I/O performance. We demonstrate the effectiveness of our algorithm on an extreme-scale compute cluster, Titan, at the Oak Ridge Leadership Computing Facility (OLCF). Our experiments with both synthetic benchmarks and a real-world application show that, even under congestion, our proposed algorithm can improve large-scale application I/O performance significantly, resulting in both the reduction of application run times and higher resolution simulation runs.

Wang, Feiyi↗
DOE OSTI2023

Computing rank‐revealing factorizations of matrices stored out‐of‐core

This paper describes efficient algorithms for computing rank-revealing factorizations of matrices that are too large to fit in main memory (RAM), and must instead be stored on slow external memory devices such as disks (out-of-core or out-of-memory). Traditional algorithms for computing rank-revealing factorizations (such as the column pivoted QR factorization and the singular value decomposition) are very communication intensive as they require many vector-vector and matrix-vector operations, which become prohibitively expensive when data is not in RAM. Randomization allows to reformulate new methods so that large contiguous blocks of the matrix are processed in bulk. The paper describes two distinct methods. The first is a blocked version of column pivoted Householder QR, organized as a “left-looking” method to minimize the number of the expensive write operations. The second method results employs a UTV factorization. It is organized as an algorithm-by-blocks to overlap computations and I/O operations. As it incorporates power iterations, it is much better at revealing the numerical rank. Numerical experiments on several computers demonstrate that the new algorithms are almost as fast when processing data stored on slow memory devices as traditional algorithms are for data stored in RAM.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Breaking Down the Computational Barriers to Real-Time Urban Flood Forecasting

Flooding impacts are on the rise globally, and concentrated in urban areas. Currently, there are no operational systems to forecast flooding at spatial resolutions that can facilitate emergency preparedness and response actions mitigating flood impacts. We present a framework for real-time flood modeling and uncertainty quantification that combines the physics of fluid motion with advances in probabilistic methods. The framework overcomes the prohibitive computational demands of high-fidelity modeling in real-time by using a probabilistic learning method relying on surrogate models that are trained prior to a flood event. This shifts the overwhelming burden of computation to the trivial problem of data storage, and enables forecasting of both flood hazard and its uncertainty at scales that are vital for time-critical decision-making before and during extreme events. The framework has the potential to improve flood prediction and analysis and can be extended to other hazard assessments requiring intense high-fidelity computations in real-time.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Multi phenomena melt pool sensor data fusion for enhanced process monitoring of laser powder bed fusion additive manufacturing

Finding actionable trends in laser-based metal additive manufacturing process monitoring data is challenging owing to the diversity and complexity of the underlying physical interactions. A single monitoring solution that captures a particular process phenomenon, such as a photodiode that tracks melt pool intensity, is not alone capable of evaluating process stability or detecting flaw formation with sufficient precision for routine application in industry. In this work, to improve flaw detection performance, we adopted a data fusion approach that captures multiple process phenomena. To demonstrate this, we acquired data from laser powder bed fusion (LPBF) builds of cylindrical specimens produced with different laser spot sizes, emulating defocusing due to process faults such as thermal lensing. The resulting specimens had porosity of varying types and severity, quantified by post-build non-destructive X-ray computed tomography, Archimedes density measurements, and destructive metallographic characterization. During the build, the melt pool state was monitored with two coaxial high-speed video cameras and a temperature field imaging system. Physically intuitive low-level melt pool signatures, such as melt pool temperature, shape and size, and spatter intensity were extracted from this high-dimensional, image-based sensor data. These process signatures were subsequently used as input features in relatively simple machine learning models, such as a support vector machine, which were trained to detect laser defocusing, and in addition, predict porosity type and severity. The results show that the data fusion approach significantly enhanced system performance by reducing the overall false positive rate from ~ 0.1 to ~ 0.001 without sacrificing the true positive rate (~0.90). These results were at par with a black-box, deep machine learning approach (convolutional neural network).

36 MATERIALS SCIENCE↗
DOE OSTI2014

Improving large-scale storage system performance via topology-aware and balanced data placement

With the advent of big data, the I/O subsystems of large-scale compute clusters are becoming a center of focus. More applications are putting greater demands on end-to-end I/O performance. These subsystems are often complex in design. They comprise of multiple hardware and software layers to cope with the increasing capacity, capability, and scalability requirements of data intensive applications. However, the sharing nature of storage resources and the intrinsic interactions across these layers make it a great challenge to realize end-to-end performance gains. This paper proposes a topology-aware strategy to balance the load across resources, to improve the per-application I/O performance. We demonstrate the effectiveness of our algorithm on an extreme-scale compute cluster, Titan, at the Oak Ridge Leadership Computing Facility (OLCF). Our experiments with both synthetic benchmarks and a real-world application show that, even under congestion, our proposed algorithm can improve large-scale application I/O performance significantly, resulting in both a reduction in application run time as well as a higher resolution of simulation run.

Wang, Feiyi↗
DOE OSTI2011

Towards Composing Data Aware Systems Biology Workflows on Cloud Platforms: A MeDICi-based Approach

Cloud computing is being increasingly adopted for deploying systems biology scientific workflows. Scientists developing these workflows use a wide variety of fragmented and competing data sets and computational tools of all scales to support their research. To this end, the synergy of client side workflow tools with cloud platforms is a promising approach to share and reuse data and workflows. In such systems, the location of data and computation is essential consideration in terms of quality of service for composing a scientific workflow across remote cloud platforms. In this paper, we describe a cloud-based workflow for genome annotation processing that is underpinned by MeDICi - a middleware designed for data intensive scientific applications. The workflow implementation incorporates an execution layer for exploiting data locality that routes the workflow requests to the processing steps that are colocated with the data. We demonstrate our approach by composing two workflowswith the MeDICi pipelines.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2026

Advancing specialized biofoundries via automated adaptive laboratory evolution

Adaptive laboratory evolution (ALE) is a powerful strategy for improving microbial phenotypes by harnessing natural selection under defined environmental conditions. Through applying selection regimes, beneficial mutations accumulate, enabling the generation of strains with enhanced properties. However, conventional ALE is labor-intensive and difficult to scale, limiting reproducibility and broader discovery of evolutionary principles. Recent advances in robotics, automation, and computational infrastructure are transforming ALE into a scalable, data-rich experimental paradigm. Automated platforms enable standardized and complex protocols, real-time monitoring, and highly parallel evolution campaigns, improving consistency while generating longitudinal datasets that reveal convergent adaptive mechanisms. Here, we discuss the role of specialized biofoundries in advancing automated ALE and enabling large-scale evolutionary engineering. We review major automated ALE formats and outline key design principles for effective ALE biofoundries, highlighting how automated ALE can support autonomous experimentation and AI-guided strain engineering.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2026

Conditional diffusion machine-learning framework for mapping valence electron distribution from convergent beam electron diffraction

Quantitative convergent beam electron diffraction (CBED) enables determination of aspherical valence electron distributions through refinement of low-order structure factors, which are highly sensitive to chemical bonding and charge density variations. However, conventional quantitative CBED (QCBED) requires solving a highly nonlinear inverse problem with many coupled parameters, and computationally intensive dynamical diffraction calculations, making it time-consuming and difficult to apply to complex systems. More broadly, reconstructing charge density and orbital electron distribution from diffraction data has long been a central challenge in both x-ray and electron crystallography. Here, in this study, we introduce an artificial-intelligence (AI)-based framework that replaces traditional refinement with a data-driven inverse solver. Using a large synthetic CBED dataset generated by Bloch-wave simulations, we train a conditional diffusion model to directly infer crystal structural parameters and multipole density formalism parameters, and hence valence electron distributions, from CBED patterns alone. By learning from forward simulations across realistic parameter space, the model effectively solves the inverse problem. Compared with direct regression approaches, the diffusion-based framework provides posterior parameter distributions for rigorous uncertainty quantification while preserving quantitative fidelity and reducing analysis time by orders of magnitude. By eliminating the need for external single-crystal x-ray diffraction data and complex nonlinear refinement, this approach enables practical, high-throughput, and in situ quantitative CBED, enabling real-time mapping of valence electron distributions and their correlation with functional responses in quantum and energy materials.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2013

An Efficient Deterministic Parallel Algorithm for Adaptive Multidimensional Numerical Integration on GPUs

Recent development in Graphics Processing Units (GPUs) has enabled a new possibility for highly efficient parallel computing in science and engineering. Their massively parallel architecture makes GPUs very effective for algorithms where processing of large blocks of data can be executed in parallel. Multidimensional integration has important applications in areas like computational physics, plasma physics, computational fluid dynamics, quantum chemistry, molecular dynamics and signal processing. The computationally intensive nature of multidimensional integration requires a high-performance implementation. In this study, we present an efficient deterministic parallel algorithm for adaptive multidimensional numerical integration on GPUs. Various optimization techniques are applied to maximize the utilization of the GPU. GPU-based implementation outperforms the best known sequential methods and achieves a speed-up of up to 100. It also shows good scalability with the increase in dimensionality.

Arumugam, Kamesh↗
DOE OSTI2018

CAT: computer aided triage improving upon the Bayes risk through ε-refusal triage rules

Manual extraction of information from electronic pathology (epath) reports to populate the Surveillance, Epidemiology, and End Result (SEER) database is labor intensive. Systematizing the data extraction automatically using machine-learning (ML) and natural language processing (NLP) is desirable to reduce the human labor required to populate the SEER database and to improve the timeliness of the data. This enables scaling up registry efficiency and collection of new data elements. To ensure the integrity, quality, and continuity of the SEER data, the misclassification error of ML and NPL algorithms needs to be negligible. Current algorithms fail to achieve the precision of human experts who can bring additional information in their assessments. Differences in registry format and the desire to develop a common information extraction platform further complicate the ML/NLP tasks. The purpose of our study is to develop triage rules to partially automate registry workflow to improve the precision of the auto-extracted information. This paper presents a mathematical framework to improve the precision of a classifier beyond that of the Bayes classifier by selectively classifying item that are most likely to be correct. This results in a triage rule that only classifies a subset of the item. We characterize the optimal triage rule and demonstrate its usefulness in the problem of classifying cancer site from electronic pathology reports to achieve a desired precision. From the mathematical formalism, we propose a heuristic estimate for triage rule based on post-processing the soft-max output from standard machine learning algorithms. We show, in test cases, that the triage rule significantly improve the classification accuracy.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2010

Technical Report: Toward a Scalable Algorithm to Compute High-Dimensional Integrals of Arbitrary Functions

Neutron experiments at the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) frequently generate large amounts of data (on the order of 106-1012 data points). Hence, traditional data analysis tools run on a single CPU take too long to be practical and scientists are unable to efficiently analyze all data generated by experiments. Our goal is to develop a scalable algorithm to efficiently compute high-dimensional integrals of arbitrary functions. This algorithm can then be used to integrate the four-dimensional integrals that arise as part of modeling intensity from the experiments at the SNS. Here, three different one-dimensional numerical integration solvers from the GNU Scientific Library were modified and implemented to solve four-dimensional integrals. The results of these solvers on a final integrand provided by scientists at the SNS can be compared to the results of other methods, such as quasi-Monte Carlo methods, computing the same integral. A parallelized version of the most efficient method can allow scientists the opportunity to more effectively analyze all experimental data.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2010

Efficient pattern matching on GPUs for intrusion detection systems

Pattern matching is at the core of many security applications, like Network Intrusion Detection Systems (NIDS), spam filters and virus scanner. The always growing traffic on networks requires the ability to recognize potentially malicious signatures effectively, fastly and possibly in real time, without afftecting the performance and the latencies of the connections. Unfortunately, pattern matching is a computationally intensive procedure which poses significant challenges on current software and hardware implementations. Graphic Processing Units (GPU) have become an interesting target for such high-througput applications, but the algorithms and the data structures need to be redesigned to be parallelized and adapted to the underlining hardware, coping with the limitations imposed by these architectures. In this paper we present an efficient implementation of the Aho-Corasick pattern matching algorithm on GPU, showing how we progressively redesigned the algorithm and the data structures to fit on the architecture and comparing it with equivalent implementations on the CPU and with previous work. We show that with realistic TCP-IP workloads and signatures, our implementation obtains a speedup of 6.5 with respect to CPU implementations and of two times when compared to previous GPU solutions.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2012

Interfractional Dose Variations in Intensity-Modulated Radiotherapy With Breath-Hold for Pancreatic Cancer

Purpose: To investigate the interfractional dose variations for intensity-modulated radiotherapy (RT) combined with breath-hold (BH) at end-exhalation (EE) for pancreatic cancer. Methods and Materials: A total of 10 consecutive patients with pancreatic cancer were enrolled. Each patient was fixed in the supine position on an individualized vacuum pillow with both arms raised. Computed tomography (CT) scans were performed before RT, and three additional scans were performed during the course of chemoradiotherapy using a conventional RT technique. The CT data were acquired under EE-BH conditions (BH-CT) using a visual feedback technique. The intensity-modulated RT plan, which used five 15-MV coplanar ports, was designed on the initial BH-CT set with a prescription dose of 39 Gy at 2.6 Gy/fraction. After rigid image registration between the initial and subsequent BH-CT scans, the dose distributions were recalculated on the subsequent BH-CT images under the same conditions as in planning. Changes in the dose-volume metrics of the gross tumor volume (GTV), clinical target volume (CTV = GTV + 5 mm), stomach, and duodenum were evaluated. Results: For the GTV and clinical target volume (CTV), the 95th percentile of the interfractional variations in the maximal dose, mean dose, dose covering 95% volume of the region of structure, and percentage of the volume covered by the 90% isodose line were within {+-}3%. Although the volume covered by the 39 Gy isodose line for the stomach and duodenum did not exceed 0.1 mL at planning, the volume covered by the 39 Gy isodose line for these structures was up to 11.4 cm{sup 3} and 1.8 cm{sup 3}, respectively. Conclusions: Despite variations in the gastrointestinal state and abdominal wall position at EE, the GTV and CTV were mostly ensured at the planned dose, with the exception of 1 patient. Compared with the duodenum, large variations in the stomach volume receiving high-dose radiation were observed, which might be beyond the negligible range in achieving dose escalation with intensity-modulated RT combined with BH at EE.

62 RADIOLOGY AND NUCLEAR MEDICINE↗
DOE OSTI2014

Petascale Application of a Coupled CPU-GPU Algorithm for Simulation and Analysis of Multiphase Flow Solutions in Porous Medium Systems, In: 2014 IEEE 28th International Parallel and Distributed Processing Symposium

Large-scale simulation can provide a wide range of information needed to develop and validate theoretical models for multiphase flow in porous medium systems. In this paper, we consider a coupled solution in which a multiphase flow simulator is coupled to an analysis approach used to extract the interfacial geometries as the flow evolves. This has been implemented using MPI to target heterogeneous nodes equipped with GPUs. The GPUs evolve the multiphase flow solution using the lattice Boltzmann method while the CPUs compute up scaled measures of the morphology and topology of the phase distributions and their rate of evolution. Our approach is demonstrated to scale to 4,096 GPUs and 65,536 CPU cores to achieve a maximum performance of 244,754 million-lattice-node updates per second (MLUPS) in double precision execution on Titan. In turn, this approach increases the size of systems that can be considered by an order of magnitude compared with previous work and enables detailed in situ tracking of averaged flow quantities at temporal resolutions that were previously impossible. Furthermore, it virtually eliminates the need for post-processing and intensive I/O and mitigates the potential loss of data associated with node failures.

Computer Science↗
DOE OSTI2016

Technical Note: SCUDA: A software platform for cumulative dose assessment

Purpose: Accurate tracking of anatomical changes and computation of actually delivered dose to the patient are critical for successful adaptive radiation therapy (ART). Additionally, efficient data management and fast processing are practically important for the adoption in clinic as ART involves a large amount of image and treatment data. The purpose of this study was to develop an accurate and efficient Software platform for CUmulative Dose Assessment (SCUDA) that can be seamlessly integrated into the clinical workflow. Methods: SCUDA consists of deformable image registration (DIR), segmentation, dose computation modules, and a graphical user interface. It is connected to our image PACS and radiotherapy informatics databases from which it automatically queries/retrieves patient images, radiotherapy plan, beam data, and daily treatment information, thus providing an efficient and unified workflow. For accurate registration of the planning CT and daily CBCTs, the authors iteratively correct CBCT intensities by matching local intensity histograms during the DIR process. Contours of the target tumor and critical structures are then propagated from the planning CT to daily CBCTs using the computed deformations. The actual delivered daily dose is computed using the registered CT and patient setup information by a superposition/convolution algorithm, and accumulated using the computed deformation fields. Both DIR and dose computation modules are accelerated by a graphics processing unit. Results: The cumulative dose computation process has been validated on 30 head and neck (HN) cancer cases, showing 3.5 ± 5.0 Gy (mean±STD) absolute mean dose differences between the planned and the actually delivered doses in the parotid glands. On average, DIR, dose computation, and segmentation take 20 s/fraction and 17 min for a 35-fraction treatment including additional computation for dose accumulation. Conclusions: The authors developed a unified software platform that provides accurate and efficient monitoring of anatomical changes and computation of actually delivered dose to the patient, thus realizing an efficient cumulative dose computation workflow. Evaluation on HN cases demonstrated the utility of our platform for monitoring the treatment quality and detecting significant dosimetric variations that are keys to successful ART.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2019

Optimizing wind farm siting to reduce power system impacts of wind variability

We introduce and solve two variants of a biobjective optimization model to reduce the negative impact of wind variability on the power system by strategically locating wind farms. The first model variant considers average changes in wind power over time; the second captures extreme fluctuations in wind power. A complementary set of wind sites is selected with the aim of minimizing both residual demand and the variability in residual demand. Because exact optimization is computationally intensive, we develop two heuristics—forward and backward greedy algorithms—to find approximate solutions. The results are compared with the exact optimization results for a well-selected subset of the data as well as to the results from selecting sites based on average wind alone. The two models are solved using demand data and potential wind sites for the Southwest Power Pool. Though both objectives can be improved by adding more sites, for a fixed number of sites, minimizing residual demand and variability in residual demand are competing objectives. Here, we find an approximate efficient frontier to compare trade-offs between the two objectives. We also vary the parameter in the heuristic that controls how the two objectives are prioritized. For the case study, the backward greedy algorithm is more effective at reducing the wind power variability than the forward greedy algorithm. Furthermore, using the backward algorithm for the full dataset is more effective than solving the exact optimization on a subset of the data when the results are evaluated using the full dataset.

17 WIND ENERGY↗
DOE OSTI2017

Automatic translation of MPI source into a latency-tolerant, data-driven form

Hiding communication behind useful computation is an important performance programming technique but remains an inscrutable programming exercise even for the expert. We present Bamboo, a code transformation framework that can realize communication overlap in applications written in MPI without the need to intrusively modify the source code. Bamboo reformulates MPI source into the form of a task dependency graph that expresses a partial ordering among tasks, enabling the program to execute in a data-driven fashion under the control of an external runtime system. Experimental results demonstrate that Bamboo significantly reduces communication delays while requiring only modest amounts of programmer annotation for a variety of applications and platforms, including those employing co-processors and accelerators. Moreover, Bamboo's performance meets or exceeds that of labor-intensive hand coding. The translator is more than a means of hiding communication costs automatically; it demonstrates the utility of semantic level optimization against a wellknown library.

Communication-computation↗