Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “log structured”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Virtual Log-Structured Storage for High-Performance Streaming

Over the past decade, given the higher number of data sources (e.g., Cloud applications, Internet of things) and critical business demands, Big Data transitioned from batch-oriented to real-time analytics. Stream storage systems, such as Apache Kafka, are well known for their increasing role in real-time Big Data analytics. For scalable stream data ingestion and processing, they logically split a data stream topic into multiple partitions. Stream storage systems keep multiple data stream copies to protect against data loss while implementing a stream partition as a replicated log. This architectural choice enables simplified development while trading cluster size with performance and the number of streams optimally managed. This paper introduces a shared virtual log-structured storage approach for improving the cluster throughput when multiple producers and consumers write and consume in parallel data streams. Stream partitions are associated with shared replicated virtual logs transparently to the user, effectively separating the implementation of stream partitioning (and data ordering) from data replication (and durability). We implement the virtual log technique in the KerA stream storage system. When comparing with Apache Kafka, KerA improves the cluster ingestion throughput by up to 4x when multiple producers write over hundreds of data streams.

consistent stream ordering↗
DOE OSTI2021

Pattern-aware prefetching using parallel log-structured file system

Techniques are provided for pattern-aware prefetching using a parallel log-structured file system. At least a portion of one or more files is accessed by detecting at least one pattern in a non-sequential access of the one or more files; and obtaining at least a portion of the one or more files based on the detected at least one pattern. The obtaining step comprises, for example, a prefetching or pre-allocation of the at least the portion of the one or more files. A prefetch cache can store the portion of the one or more obtained files. The cached portion of the one or more files can be provided from the prefetch cache to an application requesting the at least a portion of the one or more files.

Bent, John M.↗
NASA NTRS1963

A THEORETICAL AND EXPERIMENTAL INVESTIGATION OF FLUSH MOUNTING CONFIGURATIONS FOR LOG PERIODIC STRUCTURES

Many types of log periodic structures have been described in the literature; but, as far as is known, flush-mounted log periodic antennas have not been studied heretofore. The purpose of this study is to investigate, primarily by experimental methods, the input impedance and radiation characteristics of a particular type of log periodic antenna when flush-mounted over a suitable cavity or recess in a conducting surface. The particular type of log periodic antenna chosen for use in the investigation is referred to as a monopole-tooth antenna and is described in detail in Part II of the report. A suitable cavity over which the monopole-tooth antenna may be mounted is described in Part III, and experimentally determined characteristics of the antenna-cavity assembly are presented in Part IV. The first objective of the study was to obtain an optimum or nearoptimum design of the antenna-cavity assembly. This objective has been reached within the limits of the experimental approach, and the results obtained are summarized in Part V of the report. A second objective, which will be reported upon at a future date, is to establish optimum design criteria and performance characteristics of the antenna-cavity assembly when the cavity is filled with low-loss dielectric material.

RADIATION FUNCTION↗
NASA NTRS1993

Using Transparent Informed Prefetching (TIP) to reduce file read latency

As processor performance gains continue to outstrip Input/Output gains, I/O performance is becoming critical to overall system performance. File read latency is the most significant bottleneck for high performance I/O. Other aspects of I/O performance benefit from recent advances in disk bandwidth and throughput resulting from disk arrays, and in write performance derived from buffered write behind and the Log-structured File System. The access gap problem limiting improvements in read latency is exacerbated by distributed file systems operating over networks with diverse bandwidth. Focus is on extending the power of caching and prefetching to reduce file read latencies by exploiting hints from high-levels of a system. Such Transparent Informed Prefetching, TIP, and its benefits are described. It is argued that hints that disclose high level knowledge are a means for transferring optimization information across, without violating, module boundaries. How TIP can be used to convert the high throughput of new technologies such as disk arrays and log-structured file systems into low latency for applications is discussed. Our preliminary experiments show reductions in wall - clock execution time of 13 percent and 20 percent for a multiple module compilation tool (make) accessing data on a local disk and remote Coda file server, respectively, and a reduction of 30 percent for a text search (grep) remotely accessing many small files.

Patterson, R. H.↗
DOE OSTI2025

DEDUPKV: A Space-Efficient and High-Performance Key-Value Store via Fine-Grained Deduplication

Log-Structured Merge Tree (LSM-tree) based key-value stores excel in write-intensive environments but suffer from data duplication, consuming up to 49% of storage space in LSM-tree-based key-value store deployments. Traditional solutions like compression and coarse-grained file system-level deduplication introduce overhead or have limited effectiveness. In this study, we propose DedupKV, a fine-grained deduplication framework tailored for LSM-tree, maximizing data reduction efficiency while minimizing write stalls and read overheads. DedupKV features three key innovations: (1) FLUSH-integrated inline deduplication, which removes duplicates during memory-to-storage writes; (2) WAL file-based offline deduplication, repurposing write-ahead logs to avoid double writes; and (3) elastic execution, dynamically balancing inline and offline deduplication based on memory pressure and workload intensity. Additionally, dynamic granularity management reduces deduplication metadata overhead. We implemented these four ideas in RocksDB for the first time and conducted experiments in a Linux environment. Our evaluation shows that WAL file-based offline deduplication and DedupKV outperform BlobDB by 33% and 23%, respectively, in write-heavy workloads, while reducing write amplification by 1.2 ×, 2 ×, and 1.6 × for real KV datasets.

Jamil, Safdar [Sogang University]↗
DOE OSTI2022

Additively Manufactured Tamper Evident Container

Los Alamos National Laboratory has developed a revolutionary and reliable new tamper evident container (TEC) to monitor and protect the health, security, and distribution of high-value assets, technological advancements, products, materials and more that are critical to the operations of businesses and government organizations. Unlike traditional tamper evident seals, the entire TEC serves as a seal, with encrypted on-board electronics able to continuously monitor and log structural health data including but not limited to force, temperature, pressure, humidity, orientation, radiation, acceleration, or vibration as needed. Analog and encrypted digital boards safely stored inside the TEC permanently record the complete integrity history of the protected items and have the ability to alert the TEC owners or take immediate defensive actions should a disruptive or tamper event take place.

36 MATERIALS SCIENCE↗
NASA NTRS1991

Robo-line storage: Low latency, high capacity storage systems over geographically distributed networks

Rapid advances in high performance computing are making possible more complete and accurate computer-based modeling of complex physical phenomena, such as weather front interactions, dynamics of chemical reactions, numerical aerodynamic analysis of airframes, and ocean-land-atmosphere interactions. Many of these 'grand challenge' applications are as demanding of the underlying storage system, in terms of their capacity and bandwidth requirements, as they are on the computational power of the processor. A global view of the Earth's ocean chlorophyll and land vegetation requires over 2 terabytes of raw satellite image data. In this paper, we describe our planned research program in high capacity, high bandwidth storage systems. The project has four overall goals. First, we will examine new methods for high capacity storage systems, made possible by low cost, small form factor magnetic and optical tape systems. Second, access to the storage system will be low latency and high bandwidth. To achieve this, we must interleave data transfer at all levels of the storage system, including devices, controllers, servers, and communications links. Latency will be reduced by extensive caching throughout the storage hierarchy. Third, we will provide effective management of a storage hierarchy, extending the techniques already developed for the Log Structured File System. Finally, we will construct a protototype high capacity file server, suitable for use on the National Research and Education Network (NREN). Such research must be a Cornerstone of any coherent program in high performance computing and communications.

Katz, Randy H.↗
NASA NTRS1992

Zebra: A striped network file system

The design of Zebra, a striped network file system, is presented. Zebra applies ideas from log-structured file system (LFS) and RAID research to network file systems, resulting in a network file system that has scalable performance, uses its servers efficiently even when its applications are using small files, and provides high availability. Zebra stripes file data across multiple servers, so that the file transfer rate is not limited by the performance of a single server. High availability is achieved by maintaining parity information for the file system. If a server fails its contents can be reconstructed using the contents of the remaining servers and the parity information. Zebra differs from existing striped file systems in the way it stripes file data: Zebra does not stripe on a per-file basis; instead it stripes the stream of bytes written by each client. Clients write to the servers in units called stripe fragments, which are analogous to segments in an LFS. Stripe fragments contain file blocks that were written recently, without regard to which file they belong. This method of striping has numerous advantages over per-file striping, including increased server efficiency, efficient parity computation, and elimination of parity update.

Hartman, John H.↗
DOE OSTI2022

DENOVA: Deduplication Extended NOVA File System

This paper shows mathematically and experimentally that inline deduplication is not suitable for file systems on ultra-low latency Intel Optane DC PM devices in terms of performance, and proposes DeNova, an offline deduplication specially designed for log-structured NVM file systems such as NOVA. DeNova offers high-performance and low-latency I/O processing and executes deduplication in the background without interfering with foreground I/Os. DeNova employs DRAM-free persistent deduplication metadata, favoring CPU cache line, and ensures failure consistency on any system failure. We implement DeNova in the NOVA file system. Evaluation with DeNova confirms a negligible performance drop of baseline NOVA of less than 1%, while gaining high storage space savings. Extensive experiments show DeNova is failure consistent in all failure scenario cases.

Khan, Awais↗
NASA NTRS2012

Contrasting Patterns of Damage and Recovery in Logged Amazon Forests From Small Footprint LiDAR Data

Tropical forests ecosystems respond dynamically to climate variability and disturbances on time scales of minutes to millennia. To date, our knowledge of disturbance and recovery processes in tropical forests is derived almost exclusively from networks of forest inventory plots. These plots typically sample small areas (less than or equal to 1 ha) in conservation units that are protected from logging and fire. Amazon forests with frequent disturbances from human activity remain under-studied. Ongoing negotiations on REDD+ (Reducing Emissions from Deforestation and Forest Degradation plus enhancing forest carbon stocks) have placed additional emphasis on identifying degraded forests and quantifying changing carbon stocks in both degraded and intact tropical forests. We evaluated patterns of forest disturbance and recovery at four -1000 ha sites in the Brazilian Amazon using small footprint LiDAR data and coincident field measurements. Large area coverage with airborne LiDAR data in 2011-2012 included logged and unmanaged areas in Cotriguacu (Mato Grosso), Fiona do Jamari (Rondonia), and Floresta Estadual do Antimary (Acre), and unmanaged forest within Reserva Ducke (Amazonas). Logging infrastructure (skid trails, log decks, and roads) was identified using LiDAR returns from understory vegetation and validated based on field data. At each logged site, canopy gaps from logging activity and LiDAR metrics of canopy heights were used to quantify differences in forest structure between logged and unlogged areas. Contrasting patterns of harvesting operations and canopy damages at the three logged sites reflect different levels of pre-harvest planning (i.e., informal logging compared to state or national logging concessions), harvest intensity, and site conditions. Finally, we used multi-temporal LiDAR data from two sites, Reserva Ducke (2009, 2012) and Antimary (2010, 2011), to evaluate gap phase dynamics in unmanaged forest areas. The rates and patterns of canopy gap formation at these sites illustrate potential issues for separating logging damages from natural forest disturbances over longer time scales. Multi-temporal airborne LiDAR data and coincident field measurements provide complementary perspectives on disturbance and recovery processes in intact and degraded Amazon forests. Compared to forest inventory plots, the large size of each individual site permitted analyses of landscape-scale processes that would require extremely high investments to study using traditional forest inventory methods.

Morton, D. C.↗
DOE OSTI2022

Using advanced data structures to enable responsive security monitoring

Write-optimized data structures (WODS), offer the potential to keep up with cyberstream event rates and give sub-second query response for key items like IP addresses. These data structures organize logs as the events are observed. To work in a real-world environment and not fill up the disk, WODS must efficiently expire older events. As the basis for our research into organizing security monitoring data, we implemented a tool, called Diventi, to index IP addresses in connection logs using RocksDB (a write-optimized LSM tree). In this work, we extended Diventi to automatically expire data as part of the data structures’ normal operations. We guarantee that Diventi always tracks the N most recent events and tracks no more than N + k events for a parameter k < N, while ensuring the index is opportunistically pruned. To test Diventi at scale in a controlled environment, we used anonymized traces of IP communications collected at SuperComputing 2019. We synthetically extended the 2.4 billion connection events to 100 billion events. We tested Diventi vs. Elasticsearch, a common log indexing tool. In our test environment, Elasticsearch saw an ingestion rate of at best 37,000 events/s while Diventi sustained ingestion rates greater than 171,000 events/s. Our query response times were as much as 100 times faster, typically answering queries in under 80 ms. Furthermore, we saw no noticeable degradation in Diventi from expiration. We have deployed Diventi for many months where it has performed well and supported new security analysis capabilities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Effect of micron-scale manufacturing flaws on the tensile response of centimeter sized two-photon polymerization microlattices

Two-photon polymerization (TPP) is a unique fabrication technique that enables sub-micron scale resolution. Herein, we report on uniaxial tensile tests on millimeter-sized TPP log-pile structures that were fabricated using a recently developed interdigitated stitching strategy. As expected, the Young’s modulus increased with laser intensity, however elastic modulus, ultimate strength and strain-at-failure varied widely, even for nominally identical samples. Post-mortem analysis revealed a series of print defects resulting from various sources including misalignments, resin shrinkage, laser beam shadowing, and local depletion of the oxygen inhibitor. While some of the defects can be eliminated, such as misalignments due to insufficient precision of the mechanical stage, others are more intrinsic to TPP and thus more difficult to address.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Combined Structural Analysis of Core and Image Log of TGH 76-31 South East of Mt Baker, Washington State

Despite active volcanism, few geothermal energy resources have been developed in the Cascades Range. Temperature Gradient Hole 76-31 was drilled to ~440 m measured depth to probe for zones where fractures provide fluid conduits that transport deep volcanic heat to shallow depths that could support baseload, carbon neutral electrical generation. These zones were predicted by a Play Fairway assessment (PFA) of resource potential along a zone 11 km west-southwest of the summit of Mount Baker Volcano. Rock core, temperature logs, and an acoustic image log were obtained. By comparison to outcrops, the core has been initially interpreted as the Chilliwack group comprised of partially metamorphosed basaltic to andesitic volcanics. Core mapping reveals complex, steeply dipping networks of fractures and brecciation along slickensided strike slip faults; clay alteration is common in many of these structures. The majority of fractures are thoroughly healed by layers of chlorite and calcite; preservation of pyrite indicates anoxic conditions. The majority of fracture porosity resides in very dense fractures a few centimeters or less in length. The image log provides good insights into attitude of fractures that fully transect the core, but generally underestimates fracture density. The combination of complex, non-planar fracture zones containing many short fractures and healing promote misinterpretation of natural fracture attitude and density in the image log. The healing and anoxic minerals are consistent with the conductive temperature gradient measured in the well below a shallow isothermal zone, although, several fractures are open or only partially healed and resulted in fluid entries into the well. Here, the equilibrated measured temperature gradient of 64ºC/km and calculated heat flow of 145 mW/m2 is more than twice the regional average, indicating local influence of the Mt Baker magmatic system at the Little Park Creek TGH site.

Cascades↗
NASA NTRS2022

AdvoCATE User Guide

This tutorial gives an overview of AdvoCATE - the assurance case automation toolset - which provides a suite of features for safety risk management, including hazard analysis, the linking of hazards to requirement logs, risk modeling using bow tie diagrams, structured arguments, and evidence logs. In this tutorial we show how these related assurance artifacts can be combined with each other in an integrated assurance case.

Safety assurance↗
NASA NTRS2002

Low-Cost Evaluation of EO-1 Hyperion and ALI for Detection and Biophysical Characterization of Forest Logging in Amazonia (NCC5-481)

Major uncertainties exist regarding the rate and intensity of logging in tropical forests worldwide: these uncertainties severely limit economic, ecological, and biogeochemical analyses of these regions. Recent sawmill surveys in the Amazon region of Brazil show that the area logged is nearly equal to total area deforested annually, but conversion of survey data to forest area, forest structural damage, and biomass estimates requires multiple assumptions about logging practices. Remote sensing could provide an independent means to monitor logging activity and to estimate the biophysical consequences of this land use. Previous studies have demonstrated that the detection of logging in Amazon forests is difficult and no studies have developed either the quantitative physical basis or remote sensing approaches needed to estimate the effects of various logging regimes on forest structure. A major reason for these limitations has been a lack of sufficient, well-calibrated optical satellite data, which in turn, has impeded the development and use of physically-based, quantitative approaches for detection and structural characterization of forest logging regimes. We propose to use data from the EO-1 Hyperion imaging spectrometer to greatly increase our ability to estimate the presence and structural attributes of selective logging in the Amazon Basin. Our approach is based on four "biogeophysical indicators" not yet derived simultaneously from any satellite sensor: 1) green canopy leaf area index; 2) degree of shadowing; 3) presence of exposed soil and; 4) non-photosynthetic vegetation material. Airborne, field and modeling studies have shown that the optical reflectance continuum (400-2500 nm) contains sufficient information to derive estimates of each of these indicators. Our ongoing studies in the eastern Amazon basin also suggest that these four indicators are sensitive to logging intensity. Satellite-based estimates of these indicators should provide a means to quantify both the presence and degree of structural disturbance caused by various logging regimes. Our quantitative assessment of Hyperion hyperspectral and ALI multi-spectral data for the detection and structural characterization of selective logging in Amazonia will benefit from data collected through an ongoing project run by the Tropical Forest Foundation, within which we have developed a study of the canopy and landscape biophysics of conventional and reduced-impact logging. We will add to our base of forest structural information in concert with an EO-1 overpass. Using a photon transport model inversion technique that accounts for non-linear mixing of the four biogeophysical indicators, we will estimate these parameters across a gradient of selective logging intensity provided by conventional and reduced impact logging sites. We will also compare our physical ly-based approach to both conventional (e.g., NDVI) and novel (e.g., SWIR-channel) vegetation indices as well as to linear mixture modeling methods. We will cross-compare these approaches using Hyperion and ALI imagers to determine the strengths and limitations of these two sensors for applications of forest biophysics. This effort will yield the first physical ly-based, quantitative analysis of the detection and intensity of selective logging in Amazonia, comparing hyperspectral and improved multi-spectral approaches as well as inverse modeling, linear mixture modeling, and vegetation index techniques.

Asner, Gregory P.↗
DOE OSTI2022

Forest structure and solar-induced fluorescence across intact and degraded forests in the Amazon

Tropical forest degradation (e.g., anthropogenic disturbances such as selective logging and fires) alters forest structure and function and influences the forest's carbon sink. In this study, we explored structure-function relationships across a variety of degradation levels in the southern Brazilian Amazon by 1) investigating how forest structural properties vary as a function of degradation history using airborne lidar data; 2) assessing the effects of degradation on solar-induced chlorophyll fluorescence (SIF) seasonality using TROPOMI data; and 3) quantifying the contribution of structural variables to SIF using multiple regression models with stepwise selection of lidar metrics. Forest degradation history was obtained through Landsat time-series classification. We found that fire, logging, and time since disturbance were major determinants of forest structure, and that forests affected by fires experienced larger variability in leaf area index (LAI), canopy height and vertical structure relative to logged and intact forests. Moreover, only recently burned forests showed significantly depressed SIF during the dry season compared to intact forests. Canopy height and the vertical distribution of foliage were the best predictors of SIF. Unexpectedly, we found that wet-season SIF was higher in active regenerating forests (~ 4 years after fires or logging) compared with intact forests, despite lower LAI. Furthermore, our findings help to elucidate the mechanisms of carbon accumulation in anthropogenically disturbed tropical forests and indicate that they can capture large amounts of carbon while recovering.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1986

Propagation of electromagnetic waves in a turbulent medium

Theoretical modeling of the wealth of experimental data on propagation of electromagnetic radiation through turbulent media has centered on the use of the Heisenberg-Kolmogorov (HK) model, which is, however, valid only for medium to small sized eddies. Ad hoc modifications of the HK model to encompass the large-scale region of the eddy spectrum have been widely used, but a sound physical basis has been lacking. A model for large-scale turbulence that was recently proposed is applied to the above problem. The spectral density of the temperature field is derived and used to calculate the structure function of the index of refraction N. The result is compared with available data, yielding a reasonably good fit. The variance of N is also in accord with the data. The model is also applied to propagation effects. The phase structure function, covariance of the log amplitude, and variance of the log intensity are calculated. The calculated phase structure function is in excellent agreement with available data.

Canuto, V. M.↗
NASA NTRS1991

The Kinematics of Turbulent Boundary Layer Structure

The long history of research into the internal structure of turbulent boundary layers has not provided a unified picture of the physics responsible for turbulence production and dissipation. The goals of the present research are to: (1) define the current state of boundary layer structure knowledge; and (2) utilize direct numerical simulation results to help close the unresolved issues identified in part A and to unify the fragmented knowledge of various coherent motions into a consistent kinematic model of boundary layer structure. The results of the current study show that all classes of coherent motion in the low Reynolds number turbulent boundary layer may be related to vortical structures, but that no single form of vortex is representative of the wide variety of vortical structures observed. In particular, ejection and sweep motions, as well as entrainment from the free-streem are shown to have strong spatial and temporal relationships with vortical structures. Disturbances of vortex size, location, and intensity show that quasi-streamwise vortices dominate the buffer region, while transverse vortices and vortical arches dominate the wake region. Both types of vortical structure are common in the log region. The interrelationships between the various structures and the population distributions of vortices are combined into a conceptual kinematic model for the boundary layer. Aspects of vortical structure dynamics are also postulated, based on time-sequence animations of the numerically simulated flow.

Robinson, Stephen Kern↗