Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “distributed data structures”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,585 records · Page 88

Striped Data Analysis Framework

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Gutsche, Oliver↗

Towards an Introspective Dynamic Model of Globally Distributed Computing Infrastructures

Large-scale scientific collaborations like ATLAS, Belle II, CMS, DUNE, and others involve hundreds of research institutes and thousands of researchers spread across the globe. These experiments generate petabytes of data, with volumes soon expected to reach exabytes. Consequently, there is a growing need for computation, including structured data processing from raw data to consumer-ready derived data, extensive Monte Carlo simulation campaigns, and a wide range of end-user analysis. To manage these computational and storage demands, centralized workflow and data management systems are implemented. However, decisions regarding data placement and payload allocation are often made disjointly and via heuristic means. A significant obstacle in adopting more effective heuristic or AI-driven solutions is the absence of a quick and reliable introspective dynamic model to evaluate and refine alternative approaches. In this study, we aim to develop such an interactive system using real-world data. By examining job execution records from the PanDA workflow management system, we have pinpointed key performance indicators such as queuing time, error rate, and the extent of remote data access. The dataset includes five months of activity. Additionally, we are creating a generative AI model to simulate time series of payloads, which incorporate visible features like category, event count, and submitting group, as well as hidden features like the total computational load—derived from existing PanDA records and computing site capabilities. These hidden features, which are not visible to job allocators, whether heuristic or AI-driven, influence factors such as queuing times and data movement.

kilic, Ozgur Ozan [Brookhaven National Laboratory ↗

Striped Data Server for Scalable Parallel Data Analysis

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. As a result, we have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

HydraGNN_Predictive_GFM_2024 - Ensemble of predictive graph foundation models for ground state atomistic materials modeling

We provide the ensemble of fifteen pre-trained graph foundation models (GFMs) for atomistic materials modeling applications. Each one of the fifteen GFMs has been trained on five open-source datasets that (once aggregated) amount to over 154 million atomistic structures, which cover over two-thirds of the natural elements of the periodic table and that comprises a broad set of organic and inorganic compounds. This vast set of atomistic structures comprises ground state configurations that are dynamically stable (i.e., equilibrated structures with atomic forces approximately close to zero values) as well as dynamically unstable structures (i.e., non-equilibrium structures with non-negligible non-zero values of atomic forces). The ensemble of datasets aggregated does NOT include excited states. The datasets have been curated to remove atomistic structures with spectral norm of the force tensor above 100 eV/angstrom. Moreover, a linear term of the energy was computed for each dataset using a linear regression model that uses the chemical concentration of each natural element as regressor. The linear term predicted by the linear regression model has been subtracted from each original energy value to perform a re-alignment of the energy values across different electronic structures approximation theories performed to generate the diverse multi-source, multi-fidelity datasets. The folder "ADIOS_files" contains the set of pre-processed datasets in Adaptable I/O System (ADIOS) format (https://www.exascaleproject.org/research-project/adios/) that have been used for the development and training of GFMs in this work. The "ADIOS_files" directory contains 6 sub-directories named as follows: - ANI1x-v3.bp - MPTrj-v3.bp - OC2020-20M-v3.bp - OC2020-v3.bp - OC2022-v3.bp - qm7x-v3.bp Each sub-directory contains the pre-processed datasets converted in Adaptable I/O System (ADIOS) format (https://www.exascaleproject.org/research-project/adios/) that have been used to the development, training, and performance testing of the ensemble go predictive graph foundation models. Each GFM was developed using HydraGNN (https://github.com/ORNL/HydraGNN) as underlying graph neural network (GNN) architecture. The multi-task learning (MTL) capability of HydraGNN was used to simultaneously train the GFMs on labeled values for direct predictions of energy (a total system property of an atomistic structure that measures the chemical stability) and atomic forces (an atomic level property of an atomistic structure that measures the dynamical stability). The hyper parameters of the GFM have been tuned using scalable hyperparameter optimization (HPO) algorithms implemented in the software DeepHyper (https://github.com/deephyper/deephyper). The pre-training of each HPO trial was performed using distributed data parallelism (DDP) to scale the training across 128 compute nodes of the exascale OLCF supercomputer Frontier. Each HPO trial was trained only for 10 epochs and an early stopping was performed to avoid wasting significant computational resources on GNN architectures that were clearly underperforming. For each HPO trial, the 'omnistat' tool developed by (AMD Research - Advanced Micro Device) was used to measure the total energy consumption in kWh. The ensemble of GFMs was obtained by selecting the fifteen best performing HPO trials. Four models have been selected for their clear advantage in accuracy, and these are the GFMs with IDs 229, 156, 147, 260. Additional eleven models have been selected based on judicious balance between accuracy and energy consumption needed for training, and these are the GFMs with IDs 165, 78, 137, 1, 175, 171, 181, 67, 179, 167, 351. Each selected GFM of the ensemble was continued to cumulate a total of at most 30 epochs. In some cases, the total number of epochs actually performed was les than 30 due to two combined factors: (1) the size of the GFM (i.e., the number of model parameters to train) and (2) the total wall-clock time for which the computational resources could be allocated on OLCF-Frontier. The "Ensemble_of_models" directory contains 15 sub-directories named as follows: - gfm_0.229 - gfm_0.156 - gfm_0.147 - gfm_0.260 - gfm_0.165 - gfm_0.78 - gfm_0.137 - gfm_0.1 - gfm_0.175 - gfm_0.171 - gfm_0.181 - gfm_0.67 - gfm_0.179 - gfm_0.167 - gfm_0.351 Each one of these sub-directories refers to one of the fifteen HPO trials that have been selected to continue the pre-training with at most 30 epochs. With each sub-directory associated with a specific HPO trial, the following files can be found: - config.json: file for argument parsing to develop and train an HydraGNN architecture - gfm_0.ID_epoch_N.pk: file with model parameters for HPO ID trial after N epochs of training The ensemble of fifteen GFM architectures was used for (1) ensemble averaging to stabilize the predictions of energy and atomic forces after pre-training for post-processing analysis and (2) ensemble uncertainty quantification (UQ). The code used to develop, pre-train, and load the pre-trained models for post-processing analysis is available on the ORNL-GitHub at the following link: https://github.com/ORNL/HydraGNN/tree/Predictive_GFM_2024

36 MATERIALS SCIENCE↗

Washington Geothermal Play Fairway Analysis Data From Potential Field Studies

A recent study which adapts play fairway analysis (PFA) methodology to assess geothermal potential was conducted at three locations (Mount Baker, Mount St. Helens seismic zone, and Wind River valley) along the Washington Cascade Range (Forson et al. 2017). Potential field (gravity and magnetic) methods which can detect subsurface contrasts in physical properties, provides a means for mapping and modeling subsurface geology and structure. As part of the WA-Cascade PFA project, we performed potential field studies by collecting high-resolution gravity and ground-magnetic data, and rock property measurements to (1) identify and constrain fault geometries (2) constrain subsurface lithologic distribution (3) study fault interactions (4) identify areas favorable to hydrothermal flow, and ultimately (5) guide future geothermal exploration at each location.

15 GEOTHERMAL ENERGY↗

Leveraging 35 years of Pinus taeda research in the southeastern US to constrain forest carbon cycle predictions: regional data assimilation using ecosystem experiments

Predicting how forest carbon cycling will change in response to climate change and management depends on the collective knowledge from measurements across environmental gradients, ecosystem manipulations of global change factors, and mathematical models. Formally integrating these sources of knowledge through data assimilation, or model–data fusion, allows the use of past observations to constrain model parameters and estimate prediction uncertainty. Data assimilation (DA) focused on the regional scale has the opportunity to integrate data from both environmental gradients and experimental studies to constrain model parameters. Here, we introduce a hierarchical Bayesian DA approach (Data Assimilation to Predict Productivity for Ecosystems and Regions, DAPPER) that uses observations of carbon stocks, carbon fluxes, water fluxes, and vegetation dynamics from loblolly pine plantation ecosystems across the southeastern US to constrain parameters in a modified version of the Physiological Principles Predicting Growth (3-PG) forest growth model. The observations included major experiments that manipulated atmospheric carbon dioxide (CO 2 ) concentration, water, and nutrients, along with nonexperimental surveys that spanned environmental gradients across an 8.6 × 10 5 km 2 region. We optimized regionally representative posterior distributions for model parameters, which dependably predicted data from plots withheld from the data assimilation. While the mean bias in predictions of nutrient fertilization experiments, irrigation experiments, and CO 2 enrichment experiments was low, future work needs to focus modifications to model structures that decrease the bias in predictions of drought experiments. Predictions of how growth responded to elevated CO 2 strongly depended on whether ecosystem experiments were assimilated and whether the assimilated field plots in the CO 2 study were allowed to have different mortality parameters than the other field plots in the region. We present predictions of stem biomass productivity under elevated CO 2 , decreased precipitation, and increased nutrient availability that include estimates of uncertainty for the southeastern US. Overall, we (1) demonstrated how three decades of research in southeastern US planted pine forests can be used to develop DA techniques that use multiple locations, multiple data streams, and multiple ecosystem experiment types to optimize parameters and (2) developed a tool for the development of future predictions of forest productivity for natural resource managers that leverage a rich dataset of integrated ecosystem observations across a region.

54 ENVIRONMENTAL SCIENCES↗

Investigation of the flow field surrounding circular-arc boattail nozzles at subsonic speeds

The effects of jet exhaust on the subsonic flow field surrounding boattail nozzles with attached and separated boundary layers were investigated. Measurements of local Mach numbers and flow angles were made at free-stream Mach numbers of 0.60 and 0.80 at an angle of attack of 0 deg. Jet exhaust flow was simulated with a solid cylindrical sting and with high pressure air at jet-nozzle total pressure ratios of 2.9 and 5.0. Results show strong effects of the jet-wave structure on the external flow field. The predicted local Mach numbers and flow angles for attached-flow nozzles with solid jet simulators obtained by using subsonic inviscid/viscous-flow theory are in good agreement with experimental data. Prediction of nozzle surface pressure distributions which include jet-entrainment effects also agree with experimental data for attached-flow nozzles with high pressure air jets.

Abeyounis, W. K.↗

Anisotropy and shear-velocity heterogeneities in the upper mantle

Long-period surface waves are used to map lateral heterogeneities of velocity and anisotropy in the upper mantle. The dispersion curves are expanded in spherical harmonics up to degree 6 and inverted to find the depth structure. The data are corrected for the effect of surface layers and both Love and Rayleigh waves are used. Shear wave velocity and shear polarization anisotropy can be resolved down to a depth of about 450 km. The shear wave velocity distribution to 200 km depth correlates with surface tectonics, except in a few anomalous regions. Below that depth the correlation vanishes. Cold subducted material shows up weakly at 350 km as fast S-wave anomalies. In the transition region a large scale pattern appears with fast mantle in the South-Atlantic. S-anisotropy at 200 km can resolve uprising or downwelling currents under some ridges and subduction zones. The Pacific shows a NW-SE fabric.

Nataf, H.-C.↗

Sound generated aerodynamically revisited - Large-scale structures in a turbulent jet as a source of sound

The theoretical capability of identifying the source of turbulent jet noise is assessed in comparison with experimental data. Account is taken of axisymmetric and spiral turbulence modes in Lighthill's (1952) formulation of turbulent noise. Coherent structures interacting with the mean flow and the fine-grained turbulence are the primary noise sources, modeled as an oscillating streamwise distribution. Low-frequencies arise farther downstream while high-frequencies congregate close to the nozzle lip. Previous measurements at various exit velocities, angles with respect to the nozzle axis, the Strouhal number and downstream distance are discussed. The model successfully predicted the angular distribution of noise frequency due to coherent structures. Further work is indicated on compressibility effects.

Mankbadi, R.↗

Weight estimation of unconventional structures by structural optimization

Automated techniques are presented that are used in structural optimization technology, with emphasis on modifications of finite element models to obtain an optimal material distribution for minimum weight while satisfying the prescribed design requirements. It is anticipated that the future development of computer aided engineering (CAE) system will provide environments where structural analysis, a design optimization, and weight evaluation modules are integrated, sharing a common data base. Structural optimization capabilities obtained by integrating a finite element structural analysis program and a numerical optimization code are developed and applied to two illustrative examples: marine gear housing structural weight minimization and joined wing structures.

Miura, Hirokazu↗

A comparison of experimental and calculated thin-shell leading-edge buckling due to thermal stresses

High-temperature thin-shell leading-edge buckling test data are analyzed using NASA structural analysis (NASTRAN) as a finite element tool for predicting thermal buckling characteristics. Buckling points are predicted for several combinations of edge boundary conditions. The problem of relating the appropriate plate area to the edge stress distribution and the stress gradient is addressed in terms of analysis assumptions. Local plasticity was found to occur on the specimen analyzed, and this tended to simplify the basic problem since it effectively equalized the stress gradient from loaded edge to loaded edge. The initial loading was found to be difficult to select for the buckling analysis because of the transient nature of thermal stress. Multiple initial model loadings are likely required for complicated thermal stress time histories before a pertinent finite element buckling analysis can be achieved. The basic mode shapes determined from experimentation were correctly identified from computation.

Jenkins, Jerald M.↗

On the structure, interaction, and breakdown characteristics of slender wing vortices at subsonic, transonic, and supersonic speeds

Slender wing vortex flows at subsonic, transonic, and supersonic speeds were investigated in a 6 x 6 ft wind tunnel. Test data obtained include off-body and surface flow visualizations, wing upper surface static pressure distributions, and six-component forces and moments. The results reveal the transition from the low-speed classical vortex regime to the transonic regime, beginning at a freestream Mach number of 0.60, where vortices coexist with shock waves. It is shown that the onset of core breakdown and the progression of core breakdown with the angle of attack were sensitive to the Mach number, and that the shock effects at transonic speeds were reduced by the interaction of the wing and the lead-edge extension (LEX) vortices. The vortex strengths and direct interaction of the wing and LEX cores (cores wrapping around each other) were found to diminish at transonic and supersonic speeds.

Erickson, Gary E.↗

CINDOC, CSIC, and Spanish R and D

The organizational structure and functional activities of the Spanish Center for Scientific Information and Documentation (CINDOC) are discussed. The library holds 8,500 journals, including all the Spanish scientific journals; 16,000 books and 20 CD-ROM data bases. CINDOC creates and distributes its own data bases that include all the articles published in Spanish scientific journals.

Delaviesca, Rosa↗

Modeling Approaches in Planetary Seismology

Of the many geophysical means that can be used to probe a planet's interior, seismology remains the most direct. Given that the seismic data gathered on the Moon over 40 years ago revolutionized our understanding of the Moon and are still being used today to produce new insight into the state of the lunar interior, it is no wonder that many future missions, both real and conceptual, plan to take seismometers to other planets. To best facilitate the return of high-quality data from these instruments, as well as to further our understanding of the dynamic processes that modify a planet's interior, various modeling approaches are used to quantify parameters such as the amount and distribution of seismicity, tidal deformation, and seismic structure on and of the terrestrial planets. In addition, recent advances in wavefield modeling have permitted a renewed look at seismic energy transmission and the effects of attenuation and scattering, as well as the presence and effect of a core, on recorded seismograms. In this chapter, we will review these approaches.

Weber, Renee↗

Planetary Seismology

Of the many geophysical means that can be used to probe a planet's interior, seismology remains the most direct. In addition to Earth, seismometers have been installed on Venus, Mars, and the Moon. Given that the seismic data gathered on the Moon (now over 40 years ago) revolutionized our understanding of the Moon and are still being used today to produce new insight into the state of the lunar interior, it is no wonder that many future missions, both real and conceptual, plan to take seismometers to other planets. To best facilitate the return of high-quality data from these instruments, as well as to further our understanding of the dynamic processes that modify a planet's interior, various modeling approaches are used to quantify parameters such as the amount and distribution of seismicity, tidal deformation, and seismic structure of the terrestrial planets. In addition, recent advances in wavefield modeling have permitted a renewed look at seismic energy transmission and the effects of attenuation and scattering, as well as the presence and effect of a core, on recorded seismograms. In this talk I will discuss some of these methods and review the history of planetary seismology.

planetary seismology↗

Electrodynamic Dust Shield (EDS) Preparation for MISSE-11 Launch

The dusty surfaces of the Moon, Mars, and various asteroids present a significant challenge to NASA’s manned and unmanned space exploration efforts. The fine, electrostatically charged dust is difficult to remove and has resulted in vision obscuration, false instrument readings, contamination, elevated temperatures, performance reduction, and equipment failure. To alleviate these problems, NASA KSC’s Electrostatics and Surface Physics Laboratory (ESPL) is developing active dust mitigation systems for solar system exploration and in situ resource utilization (ISRU). Part of KSC’s Swamp Works system, the lab emphasizes fast-paced, hands-on, cost-effective, and collaborative innovation. The Electrodynamic Dust Shield (EDS) is an ESPL technology being developed to prevent dust accumulation on space components. It consists of a dielectric substrate embedded with parallel electrodes which, when applied with high voltage out-of-phase waveforms, produce a traveling electric field wave that scatters dust particles from its surface via dielectrophoretic and electrostatic forces. The EDS is currently fully functional and has been tested for scalability and endurance in reduced gravity/pressure flight. Next, it will be part of the May 2019 Materials International Space Station Experiment-11 (MISSE-11) payload, which will test small shield samples for performance durability with exposure to the harsh space environment. The summer objective was to test EDS functionality before active and passive dust shield samples launch on MISSE-11, in order to screen for good flight/control samples as well as to compare pre-flight and post-flight clearing efficiencies. I imaged glass and Kapton shields of 2-phase and 3-phase electrode configurations with the Keyence VHX-5000 digital microscope to check for damage before and after all testing for a qualitative pre-flight baseline; helped with vacuum testing of high-voltage shield breakdown; assembled shields and wires using Electrobond 004 silver epoxy; and helped with vacuum and cryogenic dust clearance testing of assembled EDS systems. I learned CAD (computer-aided design) and 3D printing skills to create parts for a shaker table/motor assembly, which will allow for repeatable experiments by evening out EDS dust distribution in a consistent manner prior to dust scattering testing. I will also be improving the interfacing with Alpha Space’s prototype Data Communications Unit (DCU), a payload component to control power to the EDS and store data, by mapping the DCU structure upon power-up and writing shell scripts to improve various processes. Finally, I also CADed and 3D-printed plastic supports for three different geometries of the Electrostatic Precipitator (ESP), a project that aims to use high voltage electrodes to charge and filter out dust from the Martian atmosphere drawn for ISRU devices. This work on dust mitigation technologies will help to support future robotic and human space exploration missions, including NASA’s Journey to Mars; the EDS has especially significant applications to spacecraft solar panels, thermal radiators, viewports, and astronaut visors. This opportunity has also allowed me to become more familiar with various lab equipment and hardware testing, develop key modeling skills in SolidWorks, explore past and present components critical to America’s space program, work with inspiring peers and mentors, and ascertain my dream of working permanently for NASA.

space↗

The Anatomy of Software Changes and Bugs in Autonomous Operating System

Cyberphysical systems with autonomous functions are complex pieces of software, consisting of many components, some of which implement autonomous functionality and some may use AI or machine learning algorithms. Software bugs in an autonomous system are of particular concern, as they can have catastrophic consequences. However, detailed studies based on empirical data are rare and therefore these bugs are not well understood. This paper aims to contribute towards filling that gap by investigating the software changes and bugs in Autonomy Operating System (AOS) for Unmanned Aircraft Systems (UAS), which consist of 26 components containing about 103,000 lines of code and having a total of 772 bugfixes. Based on the data extracted from the code repository and semi-structured interviews with the developers of AOS, we explore the differences among autonomous software components, components developed using Model-based Software Engineering, and reuse with respect to change proneness, fault proneness, distribution of bugfixes among AOS components and files of these components, and characteristics of bugs of different AOS components. Our results show that the autonomous components were significantly more change prone (measured in number of commits and code churn) and fault prone (measured in bugfixes per KLoC) than non-autonomous components. The distribution of the locations of bugfixes was skewed, both at component and file level (i.e., a small number of components / files contained the majority of bugs). These evidence-based findings provide important insights to researchers and practitioners alike and can be used to efficiently improve the quality and reliability of autonomous systems.

Katerina Goseva-Popstojanova↗

Striped Data Analysis Framework

Traditionally, High Energy data analysis is based on the model where data are stored in files and analyzed by running multiple analysis processes, each reading one or more of the data files. This process involves repeated data reduction step, which produces smaller files, which is time consuming and leads to data duplication. We propose an alternative approach to data storage and analysis, based on the Big Data technologies. The idea is to store each element of data once and only once in a distributed scalable database and analyze data by reading only "interesting" pieces of data from the database. To make this approach possible, we developed columnar Striped Data Representation Format as the basis of the framework. Traditional columnar approach allows for efficient analysis of complex data structures. While keeping all the benefits of columnar data representation, striped mechanism goes further by enabling efficient parallelization of computations and flexible dist ribution of data analysis. The framework includes scalable and elastic data storage, compute and user analysis backend components. The framework uses off-the shelf web services and data caching technologies as the compute/data co-location mechanism. Flexible architecture allows the framework run in the cloud using container technologies. The framework offers Python/Jupyter as the user analysis backend platform, but can also run in command line or batch mode. In the article, we will present the results of the FNAL-LDRD-2016-032 FNAL LDRD project, the design, implementation, features and performance characteristics of the Striped Data Analysis Framework.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗