SEARCH · Engineering Papers

Results for “Distributed memory”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,099 records · Page 61

All sources NASA NTRS DOE OSTI

DOE OSTI2022

FORCE-DISPATCHES Integration - Initial Demonstration

Integrated energy systems (IES) combine, in mutually beneficial ways, power from variable renewable energy sources and nuclear power plants (NPP) to improve economic viability under uncertain market and weather conditions. The open-source Framework for Optimization of Resources and Economics (FORCE) tool suite, developed at Idaho National Laboratory (INL), has enabled comprehensive modeling and simulation of IES. The capabilities within FORCE include grid portfolio optimization through the Holistic Energy Resource Optimization Network (HERON) and the transient process model analysis library HYBRID, among others. Continuous efforts and investments from the IES programs have been made to expand and improve the versatility of the FORCE toolset in fiscal year 2022. Code-coupling and cross-tool communication have been important methods for improving this versatility. This report focuses on an additional workflow in the HERON tool for capacity and dispatch stochastic optimization through integration with the external tool Design Integration and Synthesis Platform to Advance Tightly Coupled Hybrid Energy Systems (DISPATCHES). DISPATCHES was primarily developed by the National Energy Technology Laboratory, in collaboration with other national laboratories, which included INL, universities, and industry partners. It is coupled to a library of algebraic models for specific plant components, and to a framework for stochastic optimization different from that provided in the current Risk Analysis Virtual Environment (RAVEN)-running-RAVEN algorithm in HERON. HERON currently conducts stochastic optimization via an outer-inner loop: it optimizes over variable capacity on the outer loop, and at each step within the capacity parameter space, conducts an inner optimization over scenarios (of market signals, demand, and/or weather patterns) and hourly dispatch throughout a user-specified number of years. On the other hand, DISPATCHES conducts stochastic optimization via an “all-at-once” strategy in which capacity variables are optimized at the same level as dispatch variables, as all scenarios are considered at once. The latter method works especially well for projects of limited size and project length, as the necessary computational power and memory increases with the number of variables and scenarios. The new capability to use the DISPATCHES workflow in HERON enhances standalone simulations by leveraging FORCE tools—namely, the economic metrics from the Tool for Economic Analysis (TEAL) and reduced-order model (ROM) sampling from RAVEN. The initial demonstration of the DISPATCHES workflow simulates an existing nuclear-case flowsheet within the DISPATCHES repository—this models a NPP with a secondary revenue stream for hydrogen production. Electrical output from the plant is converted to hydrogen via a proton-exchange membrane (PEM) electrolyzer, hydrogen tanks are used for storage, and an additional turbine is added for hydrogen combustion. Continued work regarding this FORCE-DISPATCHES integration will include automatic generation of DISPATCHES models from HERON inputs, offering analysts the option of using either the RAVEN-runsRAVEN or DISPATCHES workflow to solve technoeconomic optimization problems.

24 POWER TRANSMISSION AND DISTRIBUTION↗

NASA NTRS2012

NASA Tech Briefs, August 2012

Topics covered include: Mars Science Laboratory Drill; Ultra-Compact Motor Controller; A Reversible Thermally Driven Pump for Use in a Sub-Kelvin Magnetic Refrigerator; Shape Memory Composite Hybrid Hinge; Binding Causes of Printed Wiring Assemblies with Card-Loks; Coring Sample Acquisition Tool; Joining and Assembly of Bulk Metallic Glass Composites Through Capacitive Discharge; 670-GHz Schottky Diode-Based Subharmonic Mixer with CPW Circuits and 70-GHz IF; Self-Nulling Lock-in Detection Electronics for Capacitance Probe Electrometer; Discontinuous Mode Power Supply; Optimal Dynamic Sub-Threshold Technique for Extreme Low Power Consumption for VLSI; Hardware for Accelerating N-Modular Redundant Systems for High-Reliability Computing; Blocking Filters with Enhanced Throughput for X-Ray Microcalorimetry; High-Thermal-Conductivity Fabrics; Imidazolium-Based Polymeric Materials as Alkaline Anion-Exchange Fuel Cell Membranes; Electrospun Nanofiber Coating of Fiber Materials: A Composite Toughening Approach; Experimental Modeling of Sterilization Effects for Atmospheric Entry Heating on Microorganisms; Saliva Preservative for Diagnostic Purposes; Hands-Free Transcranial Color Doppler Probe; Aerosol and Surface Parameter Retrievals for a Multi-Angle, Multiband Spectrometer LogScope; TraceContract; AIRS Maps from Space Processing Software; POSTMAN: Point of Sail Tacking for Maritime Autonomous Navigation; Space Operations Learning Center; OVERSMART Reporting Tool for Flow Computations Over Large Grid Systems; Large Eddy Simulation (LES) of Particle-Laden Temporal Mixing Layers; Projection of Stabilized Aerial Imagery Onto Digital Elevation Maps for Geo-Rectified and Jitter-Free Viewing; Iterative Transform Phase Diversity: An Image-Based Object and Wavefront Recovery; 3D Drop Size Distribution Extrapolation Algorithm Using a Single Disdrometer; Social Networking Adapted for Distributed Scientific Collaboration; General Methodology for Designing Spacecraft Trajectories; Hemispherical Field-of-View Above-Water Surface Imager for Submarines; and Quantum-Well Infrared Photodetector (QWIP) Focal Plane Assembly.

Source record↗

DOE OSTI2022

Proximity Portability and in Transit , M-to-N Data Partitioning and Movement in SENSEI [Book Chapter]

In high-performance parallel in situ processing, the term in transit processing refers to those configurations where data must move from a producer to a consumer that runs on separate resources. In the context of parallel and distributed computing on an HPC platform one of the central challenges is to determine a mapping of data from producer ranks to consumer ranks. This problem is complicated by the heterogeneity that arises in producer-consumer pairs, such as when producer and consumer codes have different levels of concurrency, different scaling characteristics, or different data models. The resulting mapping and movement of data from M producer to N consumer ranks can have a significant impact on aggregate application performance, particularly when the data consumer requires only a subset of the overall data for its task. This chapter focuses on the design considerations that underlie SENSEI’s implementation to this challenging problem. These design considerations extend the core SENSEI architecture and include ideas like the need to accommodate flexibility in the choice of different partitioning methods, the ability for a data consumer to request and receive only the subset of data needed for its particular operation, and the ability to leverage any of several different data transport tools. The idea of proximity portability, being able to use different data transport methods as part of an in transit workflow, is illustrated through the use of three different transport layers where switching from one transport tool to another is accomplished with only a configuration file change. Here, the chapter also includes a performance analysis summary showing the performance gains that are possible in terms of multiple metrics, such as memory footprint, time to solution, and amount of data moved, when using optimized partitioners in an in transit setting, gains that are made possible by the implementation shaped by specific design considerations.

Bethel, E. Wes↗

NASA NTRS1992

The VLBA correlator: Real-time in the distributed era

The correlator is the signal processing engine of the Very Long Baseline Array (VLBA). Radio signals are recorded on special wideband (128 Mb/s) digital recorders at the 10 telescopes, with sampling times controlled by hydrogen maser clocks. The magnetic tapes are shipped to the Array Operations Center in Socorro, New Mexico, where they are played back simultaneously into the correlator. Real-time software and firmware controls the playback drives to achieve synchronization, compute models of the wavefront delay, control the numerous modules of the correlator, and record FITS files of the fringe visibilities at the back-end of the correlator. In addition to the more than 3000 custom VLSI chips which handle the massive data flow of the signal processing, the correlator contains a total of more than 100 programmable computers, 8-, 16- and 32-bit CPUs. Code is downloaded into front-end CPU's dependent on operating mode. Low-level code is assembly language, high-level code is C running under a RT OS. We use VxWorks on Motorola MVME147 CPU's. Code development is on a complex of SPARC workstations connected to the RT CPU's by Ethernet. The overall management of the correlation process is dependent on a database management system. We use Ingres running on a Sparcstation-2. We transfer logging information from the database of the VLBA Monitor and Control System to our database using Ingres/NET. Job scripts are computed and are transferred to the real-time computers using NFS, and correlation job execution logs and status flow back by the route. Operator status and control displays use windows on workstations, interfaced to the real-time processes by network protocols. The extensive network protocol support provided by VxWorks is invaluable. The VLBA Correlator's dependence on network protocols is an example of the radical transformation of the real-time world over the past five years. Real-time is becoming more like conventional computing. Paradoxically, 'conventional' computing is also adopting practices from the real-time world: semaphores, shared memory, light-weight threads, and concurrency. This appears to be a convergence of thinking.

Wells, D. C.↗

NASA NTRS2024

Molecular Dynamics Simulations of Microstructural Effects on Austenite-Martensite Interfaces in NiTi

Formation and migration of austenite-martensite interfaces plays the key role in reversible martensitic transformations of shape memory alloys (SMAs). How these interfaces interact with the SMA microstructure is a primary determining factor in important functional properties such as hysteresis and transformation span. As such, successful microstructural engineering of SMAs requires in-depth knowledge of interface behavior. The rapid nature of martensitic transformations makes experimental observations of moving austenite-martensite interfaces challenging. Molecular dynamics (MD) simulation is a unique tool which can probe the atomic-scale details of austenite-martensite interfaces as they migrate through different microstructures. However, in focusing on the entire transformation process, including the nucleation of new phases, MD studies are usually performed so far from equilibrium that their relevance to experiment is questionable. Here, we demonstrate new MD techniques to generate energetically preferred austenite-martensite interfaces in NiTi under near-equilibrium conditions. The interfaces are semi-coherent, exhibiting a series of structural disconnections, and they can migrate rapidly through single crystals under only small thermodynamic driving forces. In contrast, when interfaces migrate in polycrystals, their motion is impeded by thermoelastic effects as well as changes in orientation relationships at grain boundaries. Microstructures which accumulate large amounts of elastic energy tend to release some fraction through irreversible, hysteresis-inducing mechanisms. We demonstrate that engineering microstructures with less constraints is a viable strategy to produce SMAs with reduced hysteresis and transformation span. Similar thermoelastic and hysteresis-inducing mechanisms also arise when austenite-martensite interfaces encounter precipitates and can be controlled by tuning characteristics of the precipitate distribution.

Gabriel Plummer↗

NASA NTRSDate not supplied

Molecular Dynamics Simulations of Microstructural Effects on Austenite-Martensite Interfaces in NiTi

The formation and migration of austenite-martensite interfaces plays the key role in reversible martensitic transformations of shape memory alloys (SMAs). How these interfaces interact with the SMA microstructure is a primary determining factor in important functional properties such as hysteresis and transformation span. As such, successful microstructural engineering of SMAs requires in-depth knowledge of interface behavior. The rapid nature of martensitic transformations makes experimental observations of moving austenite-martensite interfaces challenging. Molecular dynamics (MD) simulation is a unique tool which can probe the atomic-scale details of austenite-martensite interfaces as they migrate through different microstructures. However, in focusing on the entire transformation process, including the nucleation of new phases, MD studies are usually performed so far from equilibrium that their relevance to experiment is questionable. Here, we demonstrate new MD techniques to generate energetically preferred austenite-martensite interfaces in NiTi under near-equilibrium conditions. The interfaces are semi-coherent, exhibiting a series of structural disconnections, and they can migrate rapidly through single crystals under only small thermodynamic driving forces. In contrast, when interfaces migrate in polycrystals, their motion is impeded by thermoelastic effects as well as changes in orientation relationships at grain boundaries. Microstructures which accumulate large amounts of elastic energy tend to release some fraction through irreversible, hysteresis-inducing mechanisms. We demonstrate that engineering microstructures with less constraints is a viable strategy to produce SMAs with reduced hysteresis and transformation span. Similar thermoelastic and hysteresis-inducing mechanisms also arise when austenite-martensite interfaces encounter precipitates and can be controlled by tuning characteristics of the precipitate distribution.

Gabriel Plummer↗

NASA NTRSDate not supplied

Molecular Dynamics Simulations of Microstructural Effects on Austenite-Martensite Interfaces in NiTi

Gabriel Plummer↗

NASA NTRS1999

Parallelization of the Physical-Space Statistical Analysis System (PSAS)

Atmospheric data assimilation is a method of combining observations with model forecasts to produce a more accurate description of the atmosphere than the observations or forecast alone can provide. Data assimilation plays an increasingly important role in the study of climate and atmospheric chemistry. The NASA Data Assimilation Office (DAO) has developed the Goddard Earth Observing System Data Assimilation System (GEOS DAS) to create assimilated datasets. The core computational components of the GEOS DAS include the GEOS General Circulation Model (GCM) and the Physical-space Statistical Analysis System (PSAS). The need for timely validation of scientific enhancements to the data assimilation system poses computational demands that are best met by distributed parallel software. PSAS is implemented in Fortran 90 using object-based design principles. The analysis portions of the code solve two equations. The first of these is the "innovation" equation, which is solved on the unstructured observation grid using a preconditioned conjugate gradient (CG) method. The "analysis" equation is a transformation from the observation grid back to a structured grid, and is solved by a direct matrix-vector multiplication. Use of a factored-operator formulation reduces the computational complexity of both the CG solver and the matrix-vector multiplication, rendering the matrix-vector multiplications as a successive product of operators on a vector. Sparsity is introduced to these operators by partitioning the observations using an icosahedral decomposition scheme. PSAS builds a large (approx. 128MB) run-time database of parameters used in the calculation of these operators. Implementing a message passing parallel computing paradigm into an existing yet developing computational system as complex as PSAS is nontrivial. One of the technical challenges is balancing the requirements for computational reproducibility with the need for high performance. The problem of computational reproducibility is well known in the parallel computing community. It is a requirement that the parallel code perform calculations in a fashion that will yield identical results on different configurations of processing elements on the same platform. In some cases this problem can be solved by sacrificing performance. Meeting this requirement and still achieving high performance is very difficult. Topics to be discussed include: current PSAS design and parallelization strategy; reproducibility issues; load balance vs. database memory demands, possible solutions to these problems.

Larson, J. W.↗

NASA NTRS2000

First-Generation Jet Propulsion Laboratory "Hockey-Puck" Free-Flying Magnetometers for Distributed In-Situ Multiprobe Measurement of Current Density Filamentation in the Northern Auroral Zone: Enstrophy Mission

The sub-orbital rocket mission was a collaborative project between the University of New Hampshire, Cornell University, and the Jet Propulsion Laboratory (JPL) to study filamentation phenomena in the northern Auroral zone. The Enstrophy mission test flies the JPL Free-Flying Magnetometer (FFM) concept. The FFM technology development task has been funded by NASA develop miniaturized, low-power, integrated "sensorcrafts". JPL's role was to design, integrate, test, and deliver four FFMs for deployment from the sounding rocket, allowing a unique determination of curl-B. This provides a direct measurement of magnetic-field-aligned current density along the rocket trajectory. A miniaturized three-axis fluxgate magnetometer was integrated with a 4-channel 22-bit sigma-delta Analog to Digital Converter (ADC), four temperature sensors, digital control electronics, seven (Li-SOCl2) batteries, two (4 deg x 170 deg field of view) sun-sensors, a fan-shaped-beam laser diode beacon, a (16 MHz) stable Temperature Compensated Crystal Oscillator (TCXO) clock, Radio Frequency (RF) communication subsystem, and an antenna for approximately 15 minutes of operation where data was collected continuously and transmitted in three (3) bursts (approximately 26 seconds each) to ground station antennas at Poker Flat, Alaska. FFMs were stowed within two trays onboard the rocket during the rocket launch and were released simultaneously using the spinning action of the rocket at approximately 300 km altitude (approximately 100 sec. into the flight). FFMs were deployed with spin rate of approximately 17 Hz and approximately 3 m/sec linear velocity with respect to the rocket. For testing purposes while the rocket was in the launch pad and during flight prior to release of FFMs from the rocket, commands (such as "power on", "test", "flight", "power off', and clock "Reset" signal) were transmitted via a infrared Light Emitting Diode to an infrared detector in the FFM. Special attention was paid to low magnetic signature electronic design and choice of materials in packaging. The miniaturized fluxgate magnetometers had a range of 1-60000 nT with 0.1% full-scale linearity. The frequency range of interest for magnetic measurement was 10 mHz - 50 Hz. Digital data from the magnetometer's three axes were placed in a 4MB Static Random Access Memory (SRAM) in data packages (frames) formatted together with time tags and frame ID. After a specified time was elapsed, the data were Viterbi encoded and transmitted at a rate of 100 kbps (BPSK). Each of the four FFMs transmitted at different frequency. These carrier frequencies were in the range of 2200-2300 MHz. The antenna was a single patch on a high dielectric constant substrate covering one end-plate of the hockey-puck-sized unit. The local clocks aboard the FFMs were reset at the start of the mission and stayed synchronized within 3 msec during the mission. Position of each FFM with respect to the rocket is calculated by the knowledge of its release velocity (measured at exit point of the FFM launcher tract) providing an accuracy of 1 m over the maximum range of 3 km. Spatial and temporal nature of observants can be separated to within 3 m in space or 3 msec time interval.

Javadi, H.↗

DOE OSTI2026

Estimating Flexibility Envelopes for Residential Customers From Utility Smart Meter Data: Preprint

Demand response from residential customers has significant potential to support power system operations, but accurate flexibility estimation is challenging due to the limited resolution of advanced metering infrastructure (AMI) data. Most utility AMI measurements are recorded at hourly intervals, with only a small portion at higher resolutions, and even fewer households have appliance-level energy usage data. To address this issue, this paper proposes a two-stage long short-term memory (LSTM) framework for estimating household flexibility envelopes from low-resolution AMI data. In the first stage, the heating, ventilating, and air-conditioning (HVAC) load and non-HVAC loads are estimated by using a model trained on a small set of households with appliance-level profiles. These estimated data are then used to compute the upper- and lower-flexibility bounds, which are subsequently down-sampled to lower-resolution data. In the second stage, these flexibility bounds serve as training inputs for another LSTM model, enabling direct prediction of flexibility envelopes for households with only hourly AMI data. This method is validated using Pecan Street data from two different areas, and the results demonstrate its applicability and effectiveness.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2021

Short-term apartment-level load forecasting using a modified neural network with selected auto-regressive features

Residential electricity load profiles and their diversity have become increasingly important to realize the benefits of Smart or Transactive Energy Networks (TENs). An important element of TENs will be practical, accurate, and implementable residential load forecasting techniques. While there have been many approaches to short-term load forecasting, few have included forecasting for individual households, partly because the high volatility and idiosyncrasies present in individual household load data can pose significant challenges. In this study, we develop a Convolutional Long Short-Term Memory-based neural network with Selected Autoregressive Features (termed a CLSAF model) to improve short-term household electricity load forecasting accuracy by employing three strategies: autoregressive features selection, exogenous features selection, and a “default” state to avoid overfitting at times of high load volatility. We include aggregations of apartments to floor and building level, because utilities may favor transactive approaches that rely on aggregator models, e.g., a cluster of consumers as opposed to an individual. We demonstrate that the CLSAF model, by virtue of its enhanced feature representation and modest computational resources, can accomplish load forecasting in a multi-family residential building across three spatial granularities (individual apartment/household, floor, and building levels), with an accuracy improvement of up to 25% compared to a persistence model. We propose a data screening technique to characterize time-series electricity-load data. This technique is suitable for integration into a TEN ecosystem and allows one to estimate confidence levels of the load forecasts to optimize computational resources and the risks associated with uncertain forecasts.

24 POWER TRANSMISSION AND DISTRIBUTION↗

NASA NTRS2015

The Effects of Long Duration Bed Rest as a Spaceflight Analogue on Resting State Sensorimotor Network Functional Connectivity and Neurocognitive Performance

Long duration spaceflight has been associated with detrimental alterations in human sensorimotor systems and neurocognitive performance. Prolonged exposure to a head-down tilt position during long duration bed rest can resemble several effects of the microgravity environment such as reduced sensory inputs, body unloading and increased cephalic fluid distribution. The question of whether microgravity affects other central nervous system functions such as brain functional connectivity and its relationship with neurocognitive performance is largely unknown, but of potential importance to the health and performance of astronauts both during and post-flight. The aims of the present study are 1) to identify changes in sensorimotor resting state functional connectivity that occur with extended bed rest exposure, and to characterize their recovery time course; 2) to evaluate how these neural changes correlate with neurocognitive performance. Resting-state functional magnetic resonance imaging (rsfMRI) data were collected from 17 male participants. The data were acquired through the NASA bed rest facility, located at the University of Texas Medical Branch (Galveston, TX). Participants remained in bed with their heads tilted down six degrees below their feet for 70 consecutive days. RsfMRI data were obtained at seven time points: 7 and 12 days before bed rest; 7, 50, and 65 days during bed rest; and 7 and 12 days after bed rest. Functional connectivity magnetic resonance imaging (fcMRI) analysis was performed to measure the connectivity of sensorimotor networks in the brain before, during, and post-bed rest. We found a decrease in left putamen connectivity with the pre- and post-central gyri from pre bed rest to the last day in bed rest. In addition, vestibular cortex connectivity with the posterior cingulate cortex decreased from pre to post bed rest. Furthermore, connectivity between cerebellar right superior posterior fissure and other cerebellar regions decreased from pre bed rest to the last day in bed rest. In contrast, connectivity within the default mode network remained stable over the course of bed rest. We also utilized a battery of behavioral measures including spatial working memory tasks and measures of functional mobility and balance. These behavioral measurements were collected before, during, and after bed rest. We will report the preliminary findings of correlations observed between brain functional connectivity and behavioral performance changes. Our results suggest that sensorimotor brain networks exhibit decoupling with extended periods of reduced usage. The findings from this study could aid in the understanding and future design of targeted countermeasures to alleviate the detrimental health and neurocognitive effects of long-duration spaceflight.

Cassady, K.↗

DOE OSTI2026

VRN3P: Variational Recurrent Neural Network Based Net-Load Prediction under High Solar Penetration

This is the final technical report for the SETO-funded VRN3P project (PNNL# 76914). The goal of this project, led by Pacific Northwest National Laboratory (PNNL), in collaboration with Lawrence Livermore National Laboratory (LLNL) and Portland General Electric (PGE), was to develop and validate a deep variational recurrent neural network-based net-load prediction (VRN3P) framework for probabilistic time-series forecasting of day-ahead net-load under high solar penetration scenarios. The project team reports successful design of a novel probabilistic net-load forecasting architecture, comprising of a variational autoencoder and a recurrent neural network, which demonstrates 30% improvement in forecast performance, 60% improvement in training time, and consumes 44% less memory, when compared with conventional baseline models. The team tested the VRN3P model performance on GridLAB-D test-cases representing varying BTM solar penetration levels of 20%, 30%, and 50%, with integrated time-series net-load profiles provided by the utility partner (PGE). The VRN3P model demonstrate <2% hourly MAPE (averaged over the year) for day- ahead net-load forecast on the test scenario with 20% BTM solar. Transfer learning extension of the VRN3P model has demonstrated 8.33× speed-up in training, while still achieving acceptable forecast performance of 2.24% hourly MAPE on the 30% BTM solar penetration test-scenario. A preliminary version of the VRN3P GridAPPS-D™has been developed, along with a web-based interactive user-interface (named ‘Forte’) which has made available on GitHub for public use.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2024

Color symmetry and confinement as an underlying superconformal structure in holographic QCD

Dedicated to the memory of our colleague, Harald Fritzsch, who, together with Murray Gell-Mann, introduced the color quantum number as the exact symmetry responsible for the strong interaction, thus establishing quantum chromodynamics (QCD) as a fundamental non-Abelian gauge theory. A basic understanding of hadron properties, however, such as confinement and the emergence of a mass scale, from first principles QCD has remained elusive: Hadronic characteristics are not explicit properties of the QCD Lagrangian and perturbative QCD, so successful in the large transverse momentum domain, is not applicable at large distances. In this article, we shall examine how this daunting obstacle is overcome in holographic QCD with the introduction of a superconformal symmetry in anti de Sitter (AdS) space which is responsible for confinement and the introduction of a mass scale within the superconformal group. When mapped to light-front coordinates in physical spacetime, this approach incorporates supersymmetric relations between the Regge trajectories of meson, baryon and tetraquark states which can be visualized in terms of specific SU(3) C color representations of quarks. Finally, we will also briefly discuss here the implications of holographic models for QCD color transparency in view of the present experimental interest.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2023

Design and implementation of I/O performance prediction scheme on HPC systems through large-scale log analysis

Abstract Large-scale high performance computing (HPC) systems typically consist of many thousands of CPUs and storage units used by hundreds to thousands of users simultaneously. Applications from large numbers of users have diverse characteristics, such as varying computation, communication, memory, and I/O intensity. A good understanding of the performance characteristics of each user application is important for job scheduling and resource provisioning. Among these performance characteristics, I/O performance is becoming increasingly important as data sizes rapidly increase and large-scale applications, such as simulation and model training, are widely adopted. However, predicting I/O performance is difficult because I/O systems are shared among all users and involve many layers of software and hardware stack, including the application, network interconnect, operating system, file system, and storage devices. Furthermore, updates to these layers and changes in system management policy can significantly alter the I/O behavior of applications and the entire system. To improve the prediction of the I/O performance on HPC systems, we propose integrating information from several different system logs and developing a regression-based approach to predict the I/O performance. Our proposed scheme can dynamically select the most relevant features from the log entries using various feature selection algorithms and scoring functions, and can automatically select the regression algorithm with the best accuracy for the prediction task. The evaluation results show that our proposed scheme can predict the write performance with up to 90% prediction accuracy and the read performance with up to 99% prediction accuracy using the real logs from the Cori supercomputer system at NERSC.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2013

NASA Tech Briefs, September 2013

Topics include: ISS Ammonia Leak Detection Through X-Ray Fluorescence; A System for Measuring the Sway of the Vehicle Assembly Building; Fast, High-Precision Readout Circuit for Detector Arrays; Victim Simulator for Victim Detection Radar; Hydrometeor Size Distribution Measurements by Imaging the Attenuation of a Laser Spot; Quasi-Linear Circuit; High-Speed, High-Resolution Time-to-Digital Conversion; Li-Ion Battery and Supercapacitor Hybrid Design for Long Extravehicular Activities; Ultrasonic Low-Friction Containment Plate for Thermal and Ultrasonic Stir Weld Processes; High-Powered, Ultrasonically Assisted Thermal Stir Welding; Next-Generation MKIII Lightweight HUT/Hatch Assembly; Centrifugal Sieve for Gravity-Level-Independent Size; Segregation of Granular Materials; Ion Exchange Technology Development in Support of the Urine Processor Assembly; Nickel-Graphite Composite Compliant Interface and/or Hot Shoe Material; UltraSail CubeSat Solar Sail Flight Experiment; Mechanism for Deploying a Long, Thin-Film Antenna From a Rover; Counterflow Regolith Heat Exchanger; Acquisition and Retaining Granular Samples via a Rotating Coring Bit; Very-Low-Cost, Rugged Vacuum System; Medicine Delivery Device With Integrated Sterilization and Detection; FRET-Aptamer Assays for Bone Marker Assessment, C-Telopeptide, Creatinine, and Vitamin D; Multimode Directional Coupler for Utilization of Harmonic Frequencies from TWTAs; Dual-Polarization, Multi-Frequency Antenna Array for use with Hurricane Imaging Radiometer; Complementary Barrier Infrared Detector (CBIRD) Contact Methods; Autonomous Control of Space Nuclear Reactors; High-Power, High-Speed Electro-Optic Pockels Cell Modulator; Covariance Analysis Tool (G-CAT) for Computing Ascent, Descent, and Landing Errors; Enigma Version 12; Micrometeoroid and Orbital Debris (MMOD) Shield Ballistic Limit Analysis Program; Spitzer Telemetry Processing System; Planetary Protection Bioburden Analysis Program; Wing Leading Edge RCC Rapid Response Damage Prediction Tool (IMPACT2); ISSM: Ice Sheet System Model; Automated Loads Analysis System (ATLAS); Integrated Main Propulsion System Performance Reconstruction Process/Models. Phoenix Telemetry Processor; Contact Graph Routing Enhancements Developed in ION for DTN; GFEChutes Lo-Fi; Advanced Strategic and Tactical Relay Request Management for the Mars Relay Operations Service; Software for Generating Troposphere Corrections for InSAR Using GPS and Weather Model Data; Ionospheric Specifications for SAR Interferometry (ISSI); Implementation of a Wavefront-Sensing Algorithm; Sally Ride EarthKAM - Automated Image Geo-Referencing Using Google Earth Web Plug-In; Trade Space Specification Tool (TSST) for Rapid Mission Architecture (Version 1.2); Acoustic Emission Analysis Applet (AEAA) Software; Memory-Efficient Onboard Rock Segmentation; Advanced Multimission Operations System (ATMO); Robot Sequencing and Visualization Program (RSVP); Automating Hyperspectral Data for Rapid Response in Volcanic Emergencies; Raster-Based Approach to Solar Pressure Modeling; Space Images for NASA JPL Android Version; Kinect Engineering with Learning (KEWL); Spacecraft 3D Augmented Reality Mobile App; MPST Software: grl_pef_check; Real-Time Multimission Event Notification System for Mars Relay; SIM_EXPLORE: Software for Directed Exploration of Complex Systems; Mobile Timekeeping Application Built on Reverse-Engineered JPL Infrastructure; Advanced Query and Data Mining Capabilities for MaROS; Jettison Engineering Trajectory Tool; MPST Software: grl_suppdoc; PredGuid+A: Orion Entry Guidance Modified for Aerocapture; Planning Coverage Campaigns for Mission Design and Analysis: CLASP for DESDynl; and Space Place Prime.

Source record↗

NASA NTRS2007

Efficient Implementation of an Optimal Interpolator for Large Spatial Data Sets

Scattered data interpolation is a problem of interest in numerous areas such as electronic imaging, smooth surface modeling, and computational geometry. Our motivation arises from applications in geology and mining, which often involve large scattered data sets and a demand for high accuracy. The method of choice is ordinary kriging. This is because it is a best unbiased estimator. Unfortunately, this interpolant is computationally very expensive to compute exactly. For n scattered data points, computing the value of a single interpolant involves solving a dense linear system of size roughly n x n. This is infeasible for large n. In practice, kriging is solved approximately by local approaches that are based on considering only a relatively small'number of points that lie close to the query point. There are many problems with this local approach, however. The first is that determining the proper neighborhood size is tricky, and is usually solved by ad hoc methods such as selecting a fixed number of nearest neighbors or all the points lying within a fixed radius. Such fixed neighborhood sizes may not work well for all query points, depending on local density of the point distribution. Local methods also suffer from the problem that the resulting interpolant is not continuous. Meyer showed that while kriging produces smooth continues surfaces, it has zero order continuity along its borders. Thus, at interface boundaries where the neighborhood changes, the interpolant behaves discontinuously. Therefore, it is important to consider and solve the global system for each interpolant. However, solving such large dense systems for each query point is impractical. Recently a more principled approach to approximating kriging has been proposed based on a technique called covariance tapering. The problems arise from the fact that the covariance functions that are used in kriging have global support. Our implementations combine, utilize, and enhance a number of different approaches that have been introduced in literature for solving large linear systems for interpolation of scattered data points. For very large systems, exact methods such as Gaussian elimination are impractical since they require 0(n(exp 3)) time and 0(n(exp 2)) storage. As Billings et al. suggested, we use an iterative approach. In particular, we use the SYMMLQ method, for solving the large but sparse ordinary kriging systems that result from tapering. The main technical issue that need to be overcome in our algorithmic solution is that the points' covariance matrix for kriging should be symmetric positive definite. The goal of tapering is to obtain a sparse approximate representation of the covariance matrix while maintaining its positive definiteness. Furrer et al. used tapering to obtain a sparse linear system of the form Ax = b, where A is the tapered symmetric positive definite covariance matrix. Thus, Cholesky factorization could be used to solve their linear systems. They implemented an efficient sparse Cholesky decomposition method. They also showed if these tapers are used for a limited class of covariance models, the solution of the system converges to the solution of the original system. Matrix A in the ordinary kriging system, while symmetric, is not positive definite. Thus, their approach is not applicable to the ordinary kriging system. Therefore, we use tapering only to obtain a sparse linear system. Then, we use SYMMLQ to solve the ordinary kriging system. We show that solving large kriging systems becomes practical via tapering and iterative methods, and results in lower estimation errors compared to traditional local approaches, and significant memory savings compared to the original global system. We also developed a more efficient variant of the sparse SYMMLQ method for large ordinary kriging systems. This approach adaptively finds the correct local neighborhood for each query point in the interpolation process.

Memarsadeghi, Nargess↗

DOE OSTI2020

Ad Hoc File Systems for High-Performance Computing

Storage backends of parallel compute clusters are still based mostly on magnetic disks, while newer and faster storage technologies such as flash-based SSDs or non-volatile random access memory (NVRAM) are deployed within compute nodes. Including these new storage technologies into scientific workflows is unfortunately today a mostly manual task, and most scientists therefore do not take advantage of the faster storage media. One approach to systematically include nodelocal SSDs or NVRAMs into scientific workflows is to deploy ad hoc file systems over a set of compute nodes, which serve as temporary storage systems for single applications or longer-running campaigns. This paper presents results from the Dagstuhl Seminar 17202 “Challenges and Opportunities of User-Level File Systems for HPC” and discusses application scenarios as well as design strategies for ad hoc file systems using node-local storage media. The discussion includes open research questions, such as how to couple ad hoc file systems with the batch scheduling environment and how to schedule stage-in and stage-out processes of data between the storage backend and the ad hoc file systems. Also presented are strategies to build ad hoc file systems by using reusable components for networking and how to improve storage device compatibility. Various interfaces and semantics are presented, for example those used by the three ad hoc file systems BeeOND, GekkoFS, and BurstFS. Their presentation covers a range from file systems running in production to cutting-edge research focusing on reaching the performance limits of the underlying devices.

97 MATHEMATICS AND COMPUTING↗