Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “cache”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

PFHydro: A New Watershed-Scale Model for Post-Fire Runoff Simulation

Runoff increases after wildfires that burn vegetation and create a condition of soil-water repellence (SWR). A new post-fire watershed hydrological model, PFHydro, was created to explicitly simulate vegetation interception and SWR effects for four burn severity categories: high, medium, low severity and unburned. The model was applied to simulate post-fire runoff from the Upper Cache Creek Watershed in California, USA. Nash–Sutcliffe modeling efficiency (NSE) was used to assess model performance. The NSE was 0.80 and 0.88 for pre-fire water years (WY) 2000 and 2015, respectively. NSE was 0.88 and 0.93 for WYs 2016 (first year post-fire) and 2017 respectively. The simulated percentage of surface runoff in total runoff of WY 2016 was about six times that of pre-fire WY 2000 and three times that of WY 2015. The modeling results suggest that SWR is an important factor for post-fire runoff generation. In conclusion, the model was successful at simulating SWR behavior.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2016

Holocene cold storage practices on the eastern Snake River Plain: A risk-mitigation strategy for lean times

Previous archaeological research in southern Idaho has suggested that climate change over the past 8000 years was not dramatic enough to alter long-term subsistence practices in the region. However, recent isotopic analyses of bison remains from cold storage caves on the Snake River Plain contest this hypothesis. Our results, when examined against an archaeoclimate model, suggest that cold storage episodes coincided with drier, warmer phases that likely reduced forage and water, and thus limited the availability of bison on the open steppe. Within this context we build a risk model to illustrate how environment might have motivated cold storage behaviors. Caching bison in cold lava tubes would have mitigated both intra-annual and inter-annual food shortages under these conditions. This analysis also suggests that skeletal fat, more than meat, may have influenced the selection, transport and storage of bison carcass parts. We deciphered when and how cold storage caves which was used to provide a more comprehensive understanding of foraging behaviors in a broad range of hunting-gathering economies.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Predicting content consumption from content-to-content relationships

As the majority of Internet traffic today is attributed to content-centric applications, there has been ever-increasing demand for highly scalable and efficient content delivery. An accurate prediction on future content consumption is essential for such demand. To address such an issue, this paper introduces a new computational approach, Content Network (CN) that can capture the relations among contents, and its potential applications. We conduct a measurement study to investigate how contents are inter-related from the viewpoint of content spreading on one of the popular BitTorrent portals: The Pirate Bay. Based on the large-scale dataset that contains 18 K torrents and 9 M users, we construct the CN and investigate its structural properties. Our key finding is that contents in the same community in the CN (i) belong to the same content category with 94% probability, (ii) are uploaded by the same content publisher with 76% probability, and (iii) have the similar titles with 51% probability, which implies that contents in the same community collectively contain common (shared) interests of users. Our trace-driven study demonstrates that the proposed CN model is useful in (i) content recommendation for increasing sales and (ii) content caching for networking efficiency. We believe our work can provide an important insight for content stakeholders, e.g., content providers for efficient publishing strategies, network engineers for networking efficiency, or content marketers for accurate recommendation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Two-level main memory co-design: Multi-threaded algorithmic primitives, analysis, and simulation

A challenge in computer architecture is that processors often cannot be fed data from DRAM as fast as CPUs can consume it. Therefore, many applications are memory-bandwidth bound. With this motivation and the realization that traditional architectures (with all DRAM reachable only via bus) are insufficient to feed groups of modern processing units, vendors have introduced a variety of non-DDR 3D memory technologies (Hybrid Memory Cube (HMC),Wide I/O 2, High Bandwidth Memory (HBM)). These offer higher bandwidth and lower power by stacking DRAM chips on the processor or nearby on a silicon interposer. We will call these solutions “near-memory,” and if user-addressable, “scratchpad.” High-performance systems on the market now offer two levels of main memory: near-memory on package and traditional DRAM further away. In the near term we expect the latencies near-memory and DRAM to be similar. Here, it is natural to think of near-memory as another module on the DRAM level of the memory hierarchy. Vendors are expected to offer modes in which the near memory is used as cache, but we believe that this will be inefficient.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Data preservation at the Fermilab Tevatron

The Fermilab Tevatron collider's data-taking run ended in September 2011, yielding a dataset with rich scientific potential. The CDF and D0 experiments each have approximately 9 PB of collider and simulated data stored on tape. A large computing infrastructure consisting of tape storage, disk cache, and distributed grid computing for physics analysis with the Tevatron data is present at Fermilab. The Fermilab Run II data preservation project intends to keep this analysis capability sustained through the year 2020 and beyond. To achieve this goal, we have implemented a system that utilizes virtualization, automated validation, and migration to new standards in both software and data storage technology and leverages resources available from currently-running experiments at Fermilab. Lastly, these efforts have also provided useful lessons in ensuring long-term data access for numerous experiments, and enable high-quality scientific output for years to come.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2016

An asynchronous traversal engine for graph-based rich metadata management

Rich metadata in high-performance computing (HPC) systems contains extended information about users, jobs, data files, and their relationships. Property graphs are a promising data model to represent heterogeneous rich metadata flexibly. Specifically, a property graph can use vertices to represent different entities and edges to record the relationships between vertices with unique annotations. The high-volume HPC use case, with millions of entities and relationships, naturally requires an out-of-core distributed property graph database, which must support live updates (to ingest production information in real time), low-latency point queries (for frequent metadata operations such as permission checking), and large-scale traversals (for provenance data mining). Among these needs, large-scale property graph traversals are particularly challenging for distributed graph storage systems. Most existing graph systems implement a "level synchronous" breadth-first search algorithm that relies on global synchronization in each traversal step. This performs well in many problem domains; but a rich metadata management system is characterized by imbalanced graphs, long traversal lengths, and concurrent workloads, each of which has the potential to introduce or exacerbate stragglers (i.e., abnormally slow steps or servers in a graph traversal) that lead to low overall throughput for synchronous traversal algorithms. Previous research indicated that the straggler problem can be mitigated by using asynchronous traversal algorithms, and many graph-processing frameworks have successfully demonstrated this approach. Such systems require the graph to be loaded into a separate batch-processing framework instead of being iteratively accessed, however. In this work, we investigate a general asynchronous graph traversal engine that can operate atop a rich metadata graph in its native format. We outline a traversal-aware query language and key optimizations (traversal-affiliate caching and execution merging) necessary for efficient performance. We further explore the effect of different graph partitioning strategies on the traversal performance for both synchronous and asynchronous traversal engines. Our experiments show that the asynchronous graph traversal engine is more efficient than its synchronous counterpart in the case of HPC rich metadata processing, where more servers are involved and larger traversals are needed. Furthermore, the asynchronous traversal engine is more adaptive to different graph partitioning strategies.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2017

HPC formulations of optimization algorithms for tensor completion

Tensor completion is a powerful tool used to estimate or recover missing values in multi-way data. It has seen great success in domains such as product recommendation and healthcare. Tensor completion is most often accomplished via low-rank sparse tensor factorization, a computationally expensive non-convex optimization problem which has only recently been studied in the context of parallel computing. In this work, we study three optimization algorithms that have been successfully applied to tensor completion: alternating least squares (ALS), stochastic gradient descent (SGD), and coordinate descent (CCD++). We explore opportunities for parallelism on shared- and distributed-memory systems and address challenges such as memory- and operation-efficiency, load balance, cache locality, and communication. Among our advancements are a communication-efficient CCD++ algorithm, an ALS algorithm rich in level-3 BLAS routines, and an SGD algorithm which combines stratification with asynchronous communication. Furthermore, we show that introducing randomization during ALS and CCD++ can accelerate convergence. We evaluate our parallel formulations on a variety of real datasets on a modern supercomputer and demonstrate speedups through 16384 cores. Further, these improvements reduce time-to-solution from hours to seconds on real-world datasets. We show that after our optimizations, ALS is advantageous on parallel systems of small-to-moderate scale, while both ALS and CCD++ provide the lowest time-to-solution on large-scale distributed systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Memory access optimization for particle operations in computational fluid dynamics-discrete element method simulations

Computational Fluid Dynamics - Discrete Element Method is used to model gas-solid systems in several applications in energy, pharmaceutical and petrochemical industries. Computational performance bottlenecks often limit the problem sizes that can be simulated at industrial scale. The data structures used to store several millions of particles in such large-scale simulations have a large memory footprint that does not fit into the processor cache hierarchies on current high-performance-computing platforms, leading to reduced computational performance. This paper specifically addresses this aspect of memory access bottlenecks in industrial scale simulations. The use of space-filling curves to improve memory access patterns is described and their impact on computational performance is quantified in both shared and distributed memory parallelization paradigms. The Morton space filling curve applied to uniform grids and k-dimensional tree partitions are used to reorder the particle data-structure thus improving spatial and temporal locality in memory. The performance impact of these techniques when applied to two benchmark problems, namely the homogeneous-cooling-system and a fluidized-bed, are presented. We report these optimization techniques lead to approximately two-fold performance improvement in particle focused operations such as neighbor-list creation and data-exchange, with ~ 1.5 times overall improvement in a fluidization simulation with 1.27 million particles.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

A spherical harmonic transform approach to the indexing of electron back-scattered diffraction patterns

A new approach is proposed for the indexing of electron back-scattered diffraction (EBSD) patterns. The algorithm employs a spherical master EBSD pattern and computes its cross-correlation with a back-projected experimental pattern using the spherical harmonic transform (SHT). This approach is significantly faster than the recent dictionary indexing algorithm, but shares the latter’s robustness against noise. The underlying theory is presented, followed by example applications, one on a series of Ni EBSD data sets recorded with decreasing signal-to-noise ratio, the other on a large shot-peened Al data set. The dependence of indexing speed and memory usage on the SHT bandwidth is explored. The speed gains of the new algorithm are achieved by executing real-valued Fast Fourier Transforms, explicitly incorporating crystallographic symmetry in the cross-correlation computation, and using efficient loop ordering to improve the caching behavior. The algorithm produces a cross-correlation array in the zyz Euler space; an orientation refinement procedure is proposed based on analytical derivatives of the Wigner d functions. As a result, the new approach can be applied to any diffraction modality for which the scattered intensity can be represented on a spherical surface.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Unnatural biosynthesis by an engineered microorganism with heterologously expressed natural enzymes and an artificial metalloenzyme

Synthetic biology enables microbial hosts to produce complex molecules from organisms that are rare or difficult to cultivate, but the structures of these molecules are limited to those formed by reactions of natural enzymes. The integration of artificial metalloenzymes (ArMs) that catalyse unnatural reactions into metabolic networks could broaden the cache of molecules produced biosynthetically. Here, in this work, we report an engineered microbial cell expressing a heterologous biosynthetic pathway, containing both natural enzymes and ArMs, that produces an unnatural product with high diastereoselectivity. We engineered Escherichia coli with a heterologous terpene biosynthetic pathway and an ArM containing an iridium–porphyrin complex that was transported into the cell with a heterologous transport system. We improved the diastereoselectivity and product titre of the unnatural product by evolving the ArM and selecting the appropriate gene induction and cultivation conditions. This work shows that synthetic biology and synthetic chemistry can produce, by combining natural and artificial enzymes in whole cells, molecules that were previously inaccessible to nature.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Bonding similarities and differences between Y–Sb–Te and Sc–Sb–Te phase-change memory materials

The scandium (Sc) – alloyed Sb 2 Te 3 phase-change alloy has recently been found to enable ultrafast crystal nucleation due to the formation of Sc-stabilized octahedral motifs in the amorphous phase, rendering cache-type phase-change memory feasible. When yttrium (Y) is added, however, non-octahedral bonding patterns form in the amorphous Sb 2 Te 3 -based network even though Y has a valence electron configuration similar to that of Sc and also forms perfect octahedral bonding environments with tellurium in the YTe crystal. Here we elucidate the origin of this difference between Sc and Y, by carrying out thorough ab initio simulations and orbital-based bonding analyses on amorphous Y–Sb–Te and Sc–Sb–Te compounds. We also demonstrate how the smooth overlap of atomic positions (SOAP) similarity kernel can be used to quantify the structural similarity of local motifs in the amorphous phase with respect to various crystalline yttrium and scandium tellurides, both in the nearest-neighbor shell and beyond. We discover that the bonding contrast of Y- and Sc-centered structural motifs in amorphous Sb 2 Te 3 stems from their parent crystals at high Te concentrations. The larger atomic radius of Y and the weaker charge transfer when bonded with Te is found to allow more Te neighbors and cause a more open bonding environment, leading to higher coordination numbers and non-octahedral environments. We discuss the implications of the different local environments for practical applications in memory devices.

36 MATERIALS SCIENCE↗
DOE OSTI2018

Structure of the sensory domain of McpX from Sinorhizobium meliloti, the first known bacterial chemotactic sensor for quaternary ammonium compounds

The α-proteobacterium Sinorhizobium meliloti can live freely in the soil or engage in a symbiosis with its legume host. S. meliloti facilitates nitrogen fixation in root nodules, thus providing pivotal, utilizable nitrogen to the host. The organism has eight chemoreceptors, namely McpT to McpZ and IcpA that facilitate chemotaxis. McpX is the first known bacterial sensor of quaternary ammonium compounds (QACs) such as choline and betaines. Because QACs are exuded at chemotaxis-relevant concentrations by germinating alfalfa seeds, McpX has been proposed to contribute to host-specific chemotaxis. We have determined the crystal structure of the McpX periplasmic region (McpX PR ) in complex with the proline betaine at 2.7 Å resolution. In the crystal, the protein forms a symmetric dimer with one proline betaine molecule bound to each monomer of McpX PR within membrane-distal CACHE module. The ligand is bound through cation–π interactions with four aromatic amino acid residues. Mutational analysis in conjunction with binding studies revealed that a conserved aspartate residue is pivotal for ligand binding. We discovered that, in a striking example of convergent evolution, the ligand-binding site of McpX PR resembles that of a group of structurally unrelated betaine-binding proteins including ProX and OpuAC. Through this comparison and docking studies, we rationalized the specificity of McpX PR for this specific group of ligands. Collectively, our structural, biochemical, and molecular docking data have revealed the molecular determinants in McpX that are crucial for its rare ligand specificity for QACs.

Shrestha, Manisha↗
DOE OSTI2023

Fast correlation function calculator: A high-performance pair-counting toolkit

A novel high-performance exact pair-counting toolkit called fast correlation function calculator (FCFC) is presented. With the rapid growth of modern cosmological datasets, the evaluation of correlation functions with observational and simulation catalogues has become a challenge. High-efficiency pair-counting codes are thus in great demand. We introduce different data structures and algorithms that can be used for pair-counting problems, and perform comprehensive benchmarks to identify the most efficient algorithms for real-world cosmological applications. We then describe the three levels of parallelisms used by FCFC, SIMD, OpenMP, and MPI, and run extensive tests to investigate the scalabilities. Finally, we compare the efficiency of FCFC with alternative pair-counting codes. The data structures and histogram update algorithms implemented in FCFC are shown to outperform alternative methods. FCFC does not benefit greatly from SIMD because the bottleneck of our histogram update algorithm is mainly cache latency. Nevertheless, the efficiency of FCFC scales well with the numbers of OpenMP threads and MPI processes, even though speedups may be degraded with over a few thousand threads in total. FCFC is found to be faster than most (if not all) other public pair-counting codes for modern cosmological pair-counting applications.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2019

Understanding the evolution of conditions data access through Frontier for the ATLAS Experiment

The ATLAS Distributed Computing system uses the Frontier system to access the Conditions, Trigger, and Geometry database data stored in the Oracle Offline Database at CERN by means of the HTTP protocol. All ATLAS computing sites use Squid web proxies to cache the data, greatly reducing the load on the Frontier servers and the databases. One feature of the Frontier client is that in the event of failure, it retries with different services. While this allows transient errors and scheduled maintenance to happen transparently, it does open the system up to cascading failures if the load is high enough. Throughout LHC Run 2 there has been an ever increasing demand on the Frontier service. There have been multiple incidents where parts of the service failed due to high load. A significant improvement in the monitoring of the Frontier service was required. The monitoring was needed to identify both problematic tasks, which could then be killed or throttled, and to identify failing site services as the consequence of a cascading failure is much higher. This presentation describes the implementation and features of the monitoring system.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2019

Storage events: distributed users, federation and beyond

For federated storage to work well, some knowledge from each storage system must exist outside that system, regardless of the use case. This is needed to allow coordinated activity; e.g., executing analysis jobs on worker nodes with good accessibility to the data.Currently, this is achieved by clients notifying central services of activity; e.g., a client notifies a replica catalogue after an upload. Unfortunately, this forces end users to use bespoke clients. It also forces clients to wait for asynchronous activities to finish.dCache provides an alternative approach: storage events. In this approach the storage systems (rather than the clients) become the coordinating service, notifying interested parties of key events.At DESY, we are investigating storage events along with Apache OpenWhisk and Kubernetes to build a "serverless" cloud, similar to AWS Lambda or Google Cloud Functions, for photon science use cases.Storage events are more generally useful: catalogues are notified whenever data is uploaded or delete, tape becomes more efficient because analysis can start immediately after the data is on disk, caches can be "smart" fetching new datasets preemptively.In this paper we will present work within dCache to support a new event-based interface, with which these and other use cases become more efficient.

Millar, A.Paul↗
DOE OSTI2018

Performance of the Belle II Conditions Database

The Belle II experiment at KEK observed its first collisions in the summer of 2018. Processing the large amounts of data that will be produced requires conditions data to be readily available to systems worldwide in a fast and efficient manner that is straightforward for both the user and maintainer. This was accomplished by relying on industry-standard tools and methods: the conditions database is built as an HTTP REST service using tools such as Swagger for the API interface development, Payara for the Java EE application server, and Squid for the caching proxy. This article presents the design of the Belle II conditions database production environment as well as details about the capabilities and performance during both Monte Carlo campaigns and data reprocessing.

Wood, Lynn S.↗
DOE OSTI2019

ATLAS Data Carousel

The ATLAS experiment at CERN's LHC stores detector and simulation data in raw and derived data formats across more than 150 Grid sites world-wide, currently in total about 200PB on disk and 250PB on tape. Data have different access characteristics due to various computational workflows, and can be accessed from different media, such as remote I/O, disk cache on hard disk drives or SSDs. Also, larger data centers provide the majority of offline storage capability via tape systems. For the High-Luminosity LHC (HL-LHC), the estimated data storage requirements are several factors bigger than the present forecast of available resources, based on a flat budget assumption. On the computing side, ATLAS Distributed Computing was very successful in the last years with high performance and high throughput computing integration and in using opportunistic computing resources for the Monte Carlo simulation. On the other hand, equivalent opportunistic storage does not exist. ATLAS started the "Data Carousel" project to increase the usage of less expensive storage, i.e. tapes or even commercial storage, so it is not limited to tape technologies exclusively. Data Carousel orchestrates data processing between workload management, data management, and storage services with the bulk data resident on offline storage. The processing is executed by staging and promptly processing a sliding window of inputs onto faster buffer storage, such that only a small percentage of input data are available at any one time. With this project, we aim to demonstrate that this is the natural way to dramatically reduce our storage cost. The first phase of the project was started in the fall of 2018 and was related to I/O tests of the sites archiving systems. Phase II now requires a tight integration of the workload and data management systems. Additionally, the Data Carousel studies the feasibility to run multiple computing workflows from tape. The project is progressing very well and the results presented in this document will be used before the LHC Run 3.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Striped Data Analysis Framework

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Gutsche, Oliver↗