Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “log structured”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Small File Aggregation with PLFS [Slides]

Today’s computational science demands have resulted in ever larger parallel computers, and storage systems have grown to match these demands. Parallel file systems used in this environment are increasingly specialized to extract the highest possible performance for large I/O operations, at the expense of other potential workloads. While some applications have adapted to I/O best practices and can obtain good performance on these systems, the natural I/O patterns of many applications result in the generation of a huge number of small files, the creation of which is poorly served by current parallel file systems at very large scale. This paper describes a new technique for optimizing small file access in parallel file systems for these very large scale systems. The idea is to use a virtual parallel log-structure file system on the compute nodes in order to aggregate large numbers of small files in compute node memory and then stream their data sequentially to a much smaller number of physical files on an underlying parallel file system. The technique is implemented and evaluated using PLFS as the aggregating middleware. We evaluate our system with micro-benchmarks on a local OSX filesystem and with an MPI extension of the standard Postmark to provide results at scale on both Lustre and PanFS parallel filesystems. We observe as much as a 33x improvement in small file create rates on a single host, and 30x improvement in small file write rates, compared to a baseline Lustre configuration on a leadership computing platform using 16,384 cores and achieve an unprecedented create rate of 200 million files per second.

97 MATHEMATICS AND COMPUTING↗

Additively Manufactured Tamper Evident Container

Los Alamos National Laboratory has developed a revolutionary and reliable new tamper evident container (TEC) to monitor and protect the health, security, and distribution of high-value assets, technological advancements, products, materials and more that are critical to the operations of businesses and government organizations. Unlike traditional tamper evident seals, the entire TEC serves as a seal, with encrypted on-board electronics able to continuously monitor and log structural health data including but not limited to force, temperature, pressure, humidity, orientation, radiation, acceleration, or vibration as needed. Analog and encrypted digital boards safely stored inside the TEC permanently record the complete integrity history of the protected items and have the ability to alert the TEC owners or take immediate defensive actions should a disruptive or tamper event take place.

36 MATERIALS SCIENCE↗

Robo-line storage: Low latency, high capacity storage systems over geographically distributed networks

Rapid advances in high performance computing are making possible more complete and accurate computer-based modeling of complex physical phenomena, such as weather front interactions, dynamics of chemical reactions, numerical aerodynamic analysis of airframes, and ocean-land-atmosphere interactions. Many of these 'grand challenge' applications are as demanding of the underlying storage system, in terms of their capacity and bandwidth requirements, as they are on the computational power of the processor. A global view of the Earth's ocean chlorophyll and land vegetation requires over 2 terabytes of raw satellite image data. In this paper, we describe our planned research program in high capacity, high bandwidth storage systems. The project has four overall goals. First, we will examine new methods for high capacity storage systems, made possible by low cost, small form factor magnetic and optical tape systems. Second, access to the storage system will be low latency and high bandwidth. To achieve this, we must interleave data transfer at all levels of the storage system, including devices, controllers, servers, and communications links. Latency will be reduced by extensive caching throughout the storage hierarchy. Third, we will provide effective management of a storage hierarchy, extending the techniques already developed for the Log Structured File System. Finally, we will construct a protototype high capacity file server, suitable for use on the National Research and Education Network (NREN). Such research must be a Cornerstone of any coherent program in high performance computing and communications.

Katz, Randy H.↗

Zebra: A striped network file system

The design of Zebra, a striped network file system, is presented. Zebra applies ideas from log-structured file system (LFS) and RAID research to network file systems, resulting in a network file system that has scalable performance, uses its servers efficiently even when its applications are using small files, and provides high availability. Zebra stripes file data across multiple servers, so that the file transfer rate is not limited by the performance of a single server. High availability is achieved by maintaining parity information for the file system. If a server fails its contents can be reconstructed using the contents of the remaining servers and the parity information. Zebra differs from existing striped file systems in the way it stripes file data: Zebra does not stripe on a per-file basis; instead it stripes the stream of bytes written by each client. Clients write to the servers in units called stripe fragments, which are analogous to segments in an LFS. Stripe fragments contain file blocks that were written recently, without regard to which file they belong. This method of striping has numerous advantages over per-file striping, including increased server efficiency, efficient parity computation, and elimination of parity update.

Hartman, John H.↗

Reducing Concurrency Bottlenecks in Parallel I/O Workloads

To enable high performance parallel checkpointing we introduced the Parallel Log Structured File System (PLFS). PLFS is middleware interposed on the file system stack to transform concurrent writing of one application file into many non-concurrently written component files. The promising effectiveness of PLFS makes it important to examine its performance for workloads other than checkpoint capture, notably the different ways that state snapshots may be later read, to make the case for using PLFS in the Exascale I/O stack. Reading a PLFS file involved reading each of its component files. In this paper we identify performance limitations on broader workloads in an early version of PLFS, specifically the need to build and distribute an index for the overall file, and the pressure on the underlying parallel file system's metadata server, and show how PLFS's decomposed components architecture can be exploited to alleviate bottlenecks in the underlying parallel file system.

COMPUTER ARCHITECTURE↗

Scientific Data Services -- A High-Performance I/O System with Array Semantics

As high-performance computing approaches exascale, the existing I/O system design is having trouble keeping pace in both performance and scalability. We propose to address this challenge by adopting database principles and techniques in parallel I/O systems. First, we propose to adopt an array data model because many scientific applications represent their data in arrays. This strategy follows a cardinal principle from database research, which separates the logical view from the physical layout of data. This high-level data model gives the underlying implementation more freedom to optimize the physical layout and to choose the most effective way of accessing the data. For example, knowing that a set of write operations is working on a single multi-dimensional array makes it possible to keep the subarrays in a log structure during the write operations and reassemble them later into another physical layout as resources permit. While maintaining the high-level view, the storage system could compress the user data to reduce the physical storage requirement, collocate data records that are frequently used together, or replicate data to increase availability and fault-tolerance. Additionally, the system could generate secondary data structures such as database indexes and summary statistics. We expect the proposed Scientific Data Services approach to create a “live” storage system that dynamically adjusts to user demands and evolves with the massively parallel storage hardware.

97 MATHEMATICS AND COMPUTING↗

An analysis of image storage systems for scalable training of deep neural networks

This study presents a principled empirical evaluation of image storage systems for training deep neural networks. We employ the Caffe deep learning framework to train neural network models for three different data sets, MNIST, CIFAR-10, and ImageNet. While training the models, we evaluate five different options to retrieve training image data: (1) PNG-formatted image files on local file system; (2) pushing pixel arrays from image files into a single HDF5 file on local file system; (3) in-memory arrays to hold the pixel arrays in Python and C++; (4) loading the training data into LevelDB, a log-structured merge tree based key-value storage; and (5) loading the training data into LMDB, a B+tree based key-value storage. The experimental results quantitatively highlight the disadvantage of using normal image files on local file systems to train deep neural networks and demonstrate reliable performance with key-value storage based storage systems. When training a model on the ImageNet dataset, the image file option was more than 17 times slower than the key-value storage option. Along with measurements on training time, this study provides in-depth analysis on the cause of performance advantages/disadvantages of each back-end to train deep neural networks. We envision the provided measurements and analysis will shed light on the optimal way to architect systems for training neural networks in a scalable manner.

Lim, Seung-Hwan↗

Parallel file system with metadata distributed across partitioned key-value store c

Improved techniques are provided for storing metadata associated with a plurality of sub-files associated with a single shared file in a parallel file system. The shared file is generated by a plurality of applications executing on a plurality of compute nodes. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file generated by an application executing on the compute node and metadata for the at least one portion of the shared file on one or more object storage servers. The compute node is also configured to implement a partitioned data store for storing a partition of the metadata for the shared file, wherein the partitioned data store communicates with partitioned data stores on other compute nodes using a message passing interface. The partitioned data store can be implemented, for example, using Multidimensional Data Hashing Indexing Middleware (MDHIM).

97 MATHEMATICS AND COMPUTING↗

DENOVA: Deduplication Extended NOVA File System

This paper shows mathematically and experimentally that inline deduplication is not suitable for file systems on ultra-low latency Intel Optane DC PM devices in terms of performance, and proposes DeNova, an offline deduplication specially designed for log-structured NVM file systems such as NOVA. DeNova offers high-performance and low-latency I/O processing and executes deduplication in the background without interfering with foreground I/Os. DeNova employs DRAM-free persistent deduplication metadata, favoring CPU cache line, and ensures failure consistency on any system failure. We implement DeNova in the NOVA file system. Evaluation with DeNova confirms a negligible performance drop of baseline NOVA of less than 1%, while gaining high storage space savings. Extensive experiments show DeNova is failure consistent in all failure scenario cases.

Khan, Awais↗

IMPROVED V II log(gf) VALUES, HYPERFINE STRUCTURE CONSTANTS, AND ABUNDANCE DETERMINATIONS IN THE PHOTOSPHERES OF THE SUN AND METAL-POOR STAR HD 84937

New experimental absolute atomic transition probabilities are reported for 203 lines of V II. Branching fractions are measured from spectra recorded using a Fourier transform spectrometer and an echelle spectrometer. The branching fractions are normalized with radiative lifetime measurements to determine the new transition probabilities. Generally good agreement is found between this work and previously reported V II transition probabilities. Two spectrometers, independent radiometric calibration methods, and independent data analysis routines enable a reduction in systematic uncertainties, in particular those due to optical depth errors. In addition, new hyperfine structure constants are measured for selected levels by least squares fitting line profiles in the FTS spectra. The new V II data are applied to high resolution visible and UV spectra of the Sun and metal-poor star HD 84937 to determine new, more accurate V abundances. Lines covering a range of wavelength and excitation potential are used to search for non-LTE effects. Very good agreement is found between our new solar photospheric V abundance, log ε(V) = 3.95 from 15 V II lines, and the solar-system meteoritic value. In HD 84937, we derive [V/H] = –2.08 from 68 lines, leading to a value of [V/Fe] = 0.24.

79 ASTRONOMY AND ASTROPHYSICS↗

Contrasting Patterns of Damage and Recovery in Logged Amazon Forests From Small Footprint LiDAR Data

Tropical forests ecosystems respond dynamically to climate variability and disturbances on time scales of minutes to millennia. To date, our knowledge of disturbance and recovery processes in tropical forests is derived almost exclusively from networks of forest inventory plots. These plots typically sample small areas (less than or equal to 1 ha) in conservation units that are protected from logging and fire. Amazon forests with frequent disturbances from human activity remain under-studied. Ongoing negotiations on REDD+ (Reducing Emissions from Deforestation and Forest Degradation plus enhancing forest carbon stocks) have placed additional emphasis on identifying degraded forests and quantifying changing carbon stocks in both degraded and intact tropical forests. We evaluated patterns of forest disturbance and recovery at four -1000 ha sites in the Brazilian Amazon using small footprint LiDAR data and coincident field measurements. Large area coverage with airborne LiDAR data in 2011-2012 included logged and unmanaged areas in Cotriguacu (Mato Grosso), Fiona do Jamari (Rondonia), and Floresta Estadual do Antimary (Acre), and unmanaged forest within Reserva Ducke (Amazonas). Logging infrastructure (skid trails, log decks, and roads) was identified using LiDAR returns from understory vegetation and validated based on field data. At each logged site, canopy gaps from logging activity and LiDAR metrics of canopy heights were used to quantify differences in forest structure between logged and unlogged areas. Contrasting patterns of harvesting operations and canopy damages at the three logged sites reflect different levels of pre-harvest planning (i.e., informal logging compared to state or national logging concessions), harvest intensity, and site conditions. Finally, we used multi-temporal LiDAR data from two sites, Reserva Ducke (2009, 2012) and Antimary (2010, 2011), to evaluate gap phase dynamics in unmanaged forest areas. The rates and patterns of canopy gap formation at these sites illustrate potential issues for separating logging damages from natural forest disturbances over longer time scales. Multi-temporal airborne LiDAR data and coincident field measurements provide complementary perspectives on disturbance and recovery processes in intact and degraded Amazon forests. Compared to forest inventory plots, the large size of each individual site permitted analyses of landscape-scale processes that would require extremely high investments to study using traditional forest inventory methods.

Morton, D. C.↗

Using advanced data structures to enable responsive security monitoring

Write-optimized data structures (WODS), offer the potential to keep up with cyberstream event rates and give sub-second query response for key items like IP addresses. These data structures organize logs as the events are observed. To work in a real-world environment and not fill up the disk, WODS must efficiently expire older events. As the basis for our research into organizing security monitoring data, we implemented a tool, called Diventi, to index IP addresses in connection logs using RocksDB (a write-optimized LSM tree). In this work, we extended Diventi to automatically expire data as part of the data structures’ normal operations. We guarantee that Diventi always tracks the N most recent events and tracks no more than N + k events for a parameter k < N, while ensuring the index is opportunistically pruned. To test Diventi at scale in a controlled environment, we used anonymized traces of IP communications collected at SuperComputing 2019. We synthetically extended the 2.4 billion connection events to 100 billion events. We tested Diventi vs. Elasticsearch, a common log indexing tool. In our test environment, Elasticsearch saw an ingestion rate of at best 37,000 events/s while Diventi sustained ingestion rates greater than 171,000 events/s. Our query response times were as much as 100 times faster, typically answering queries in under 80 ms. Furthermore, we saw no noticeable degradation in Diventi from expiration. We have deployed Diventi for many months where it has performed well and supported new security analysis capabilities.

97 MATHEMATICS AND COMPUTING↗

Coupling fracture facies with in-situ permeability measurements to generate stochastic simulations of tight carbonate aquifer properties: Example from the Lower Cretaceous aquifer, Northern Provence, SE France

The relationships between fracture facies and permeability distribution have been explored using a well-constrained 3D implicit structural model of hemipelagic low porosity/low permeability carbonate from the Northern Provence region, SE France. Fracture and permeability facies were determined using an extensive dataset of 99 hydrogeological wells. Data processing was undertaken using a step-by-step approach, involving: (i) identification of the fracture facies based on well logs detailing structure and stratigraphy; (ii) determination of permeability facies from the a priori correlation between the dimension of the hydraulic radius of influence (deduced from slug test analyses) and the type of reservoir heterogeneity (fissure, fracture, fault zone, etc.); (iii) three dimensional plot of fracture and permeability facies in the geological model using a variographic analysis of data. Thirty-three sequential indicator simulations (SIS) based on geostatistic analyses were realised on both fracture and permeability facies. Finally, a connectivity algorithm was developed to compute the probability of connection between selected infiltration areas and the major aquifer springs via moderate- to high-permeability geological bodies. Key results are: (i) fault zones have a greater role in controlling permeability facies distribution than on fracture facies repartition; (ii) there is little correlation between permeability and fracture facies distributions; (iii) connectivity results highlight the compartmentalisation of aquifers in the Cadarache area, the extensions of permeable geological bodies being limited by the N130 faults.

58 GEOSCIENCES↗

Deformation Behavior of Foam Laser Targets Fabricated by Two-Photon Polymerization

Two-photon polymerization (2PP), which is a three-dimensional micro/nano-scale additive manufacturing process, is used to fabricate component for small custom experimental packages (“targets”) to support laser-driven, high-energy-density physics research. Of particular interest is the use of 2PP to deterministically print millimeter-scale, low-density, and low atomic number (CHO) polymer matrices (“foams”). Deformation during development and drying of the foam structures remains a challenge when using certain commercial acrylic photo-resins. Acrylic resins were chosen in order to meet the low atomic number requirement for the foam; that requirement precludes the use of low-shrinkage organic/inorganic hybrid resins. Here, we compare the use of acrylic resins IP-S and IP-Dip. Infrared and Raman spectroscopy are used to quantify the extent of the polymerization during 2PP vs. UV curing. The mechanical strength of beam and foam structures is examined, particularly the degree of deformation that occurs during the development and drying processes. The magnitude of the shrinkage is quantified, and finite element analysis is used in order to simulate the resulting deformation. Capillary drying forces during development are shown to be small and are likely below the elastic limit of the foam log-pile structures. In contrast, the substantial shrinkage in IP-Dip (~5–10%) causes large shear stresses and associated plastic deformation, particularly near constrained boundaries and locations with sharp density transitions. Use of IP-S with an improved writing procedure results in a marked reduction in deformation with a minor loss of resolution.

Liu, Ying↗

A Scientific Data Provenance Harvester for Distributed Applications

Data provenance provides a way for scientists to observe how experimental data originates, conveys process history, and explains influential factors such as experimental rationale and associated environmental factors from system metrics measured at runtime. The US Department of Energy Office of Science Integrated end-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) project has developed a provenance harvester that is capable of collecting observations from file based evidence typically produced by distributed applications. To achieve this, file based evidence is extracted and transformed into an intermediate data format inspired in part by W3C CSV on the Web recommendations, called the Harvester Provenance Application Interface (HAPI) syntax. This syntax provides a general means to pre-stage provenance into messages that are both human readable and capable of being written to a provenance store, Provenance Environment (ProvEn). HAPI is being applied to harvest provenance from climate ensemble runs for Accelerated Climate Modeling for Energy (ACME) project funded under the U.S. Department of Energy’s Office of Biological and Environmental Research (BER) Earth System Modeling (ESM) program. ACME informally provides provenance in a native form through configuration files, directory structures, and log files that contain success/failure indicators, code traces, and performance measurements. Because of its generic format, HAPI is also being applied to harvest tabular job management provenance from Belle II DIRAC scheduler relational database tables as well as other scientific applications that log provenance related information.

Stephan, Eric G.↗

Effect of micron-scale manufacturing flaws on the tensile response of centimeter sized two-photon polymerization microlattices

Two-photon polymerization (TPP) is a unique fabrication technique that enables sub-micron scale resolution. Herein, we report on uniaxial tensile tests on millimeter-sized TPP log-pile structures that were fabricated using a recently developed interdigitated stitching strategy. As expected, the Young’s modulus increased with laser intensity, however elastic modulus, ultimate strength and strain-at-failure varied widely, even for nominally identical samples. Post-mortem analysis revealed a series of print defects resulting from various sources including misalignments, resin shrinkage, laser beam shadowing, and local depletion of the oxygen inhibitor. While some of the defects can be eliminated, such as misalignments due to insufficient precision of the mechanical stage, others are more intrinsic to TPP and thus more difficult to address.

36 MATERIALS SCIENCE↗

TESTING THE PROPAGATING FLUCTUATIONS MODEL WITH A LONG, GLOBAL ACCRETION DISK SIMULATION

The broadband variability of many accreting systems displays characteristic structures; log-normal flux distributions, root-mean square (rms)-flux relations, and long inter-band lags. These characteristics are usually interpreted as inward propagating fluctuations of the mass accretion rate in an accretion disk driven by stochasticity of the angular momentum transport mechanism. We present the first analysis of propagating fluctuations in a long-duration, high-resolution, global three-dimensional magnetohydrodynamic (MHD) simulation of a geometrically thin ( h / r ≈ 0.1) accretion disk around a black hole. While the dynamical-timescale turbulent fluctuations in the Maxwell stresses are too rapid to drive radially coherent fluctuations in the accretion rate, we find that the low-frequency quasi-periodic dynamo action introduces low-frequency fluctuations in the Maxwell stresses, which then drive the propagating fluctuations. Examining both the mass accretion rate and emission proxies, we recover log-normality, linear rms-flux relations, and radial coherence that would produce inter-band lags. Hence, we successfully relate and connect the phenomenology of propagating fluctuations to modern MHD accretion disk theory.

79 ASTRONOMY AND ASTROPHYSICS↗

Combined Structural Analysis of Core and Image Log of TGH 76-31 South East of Mt Baker, Washington State

Despite active volcanism, few geothermal energy resources have been developed in the Cascades Range. Temperature Gradient Hole 76-31 was drilled to ~440 m measured depth to probe for zones where fractures provide fluid conduits that transport deep volcanic heat to shallow depths that could support baseload, carbon neutral electrical generation. These zones were predicted by a Play Fairway assessment (PFA) of resource potential along a zone 11 km west-southwest of the summit of Mount Baker Volcano. Rock core, temperature logs, and an acoustic image log were obtained. By comparison to outcrops, the core has been initially interpreted as the Chilliwack group comprised of partially metamorphosed basaltic to andesitic volcanics. Core mapping reveals complex, steeply dipping networks of fractures and brecciation along slickensided strike slip faults; clay alteration is common in many of these structures. The majority of fractures are thoroughly healed by layers of chlorite and calcite; preservation of pyrite indicates anoxic conditions. The majority of fracture porosity resides in very dense fractures a few centimeters or less in length. The image log provides good insights into attitude of fractures that fully transect the core, but generally underestimates fracture density. The combination of complex, non-planar fracture zones containing many short fractures and healing promote misinterpretation of natural fracture attitude and density in the image log. The healing and anoxic minerals are consistent with the conductive temperature gradient measured in the well below a shallow isothermal zone, although, several fractures are open or only partially healed and resulted in fluid entries into the well. Here, the equilibrated measured temperature gradient of 64ºC/km and calculated heat flow of 145 mW/m2 is more than twice the regional average, indicating local influence of the Mt Baker magmatic system at the Little Park Creek TGH site.

Cascades↗