The National Microbiome Data Collaborative Data Portal: an integrated multi-omics microbiome data resource
Explore the source record for details and available documents.
Eloe-Fadrosh, Emiley A↗
SEARCH · Engineering Papers
Results for “Data Distribution”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Ring Buffered Network Bus
This report describes the research effort to demonstrate the integration of a data sharing technology, Ring Buffered Network Bus, in development by Dryden Flight Research Center, with an engine simulation application, the Java Gas Turbine Simulator, in development at the University of Toledo under a grant from the Glenn Research Center. The objective of this task was to examine the application of the RBNB technologies as a key component in the data sharing, health monitoring and system wide modeling elements of the NASA Aviation Safety Program (AVSP) [Golding, 1997]. System-wide monitoring and modeling of aircraft and air safety systems will require access to all data sources which are relative factors when monitoring or modeling the national airspace such as radar, weather, aircraft performance, engine performance, schedule and planning, airport configuration, flight operations, etc. The data sharing portion of the overall AVSP program is responsible for providing the hardware and software architecture to access and distribute data, including real-time flight operations data, among all of the AVSP elements. The integration of an engine code capable of numerically "flying" through recorded flight paths and weather data using a software tool that allows for distributed access of data to this engine code demonstrates initial steps toward building a system capable of monitoring and modeling the National Airspace.
Source record↗
Distributed-Memory Sparse Deep Neural Network Inference Using Global Arrays
Partitioned Global Address Space (PGAS) models exhibit tremendous promise in developing efficient and productive distributed-memory parallel applications. They have been used extensively in scientific computations due to conveniently offering a ``shared-memory''-like model and convenient interfaces that separate communication with synchronization. Traditionally, PGAS communication models have been applied to dense/contiguously distributed data, but most modern applications depict varied levels of sparsity. Existing PGAS models require certain adaptations to support distributed sparse computations, since associated computations often require matrix arithmetic, in addition to data movement. The Global Arrays toolkit from Pacific Northwest National Laboratory (PNNL) is one of the earliest PGAS models to combine one-sided data communication and distributed matrix operations and is still used in the popular NWChem quantum chemistry suite. Recently, we have expanded the Global Arrays toolkit to support common sparse operations, like sparse matrix-dense matrix multiplies (SpMM), sparse matrix-sparse matrix multiplication (SpGEMM) and Sampled Dense-Dense Matrix Multiplication (SDDMM). As it turns out, these operations are the bedrock of sparse Deep Learning (DL); sparse deep neural networks and Graph Neural Networks (GNNs) have gained increasing attention recently in achieving speedups on training and inference with reduced memory footprints. Unlike scientific applications in High Performance Computing (HPC), modern (distributed-memory capable) DL toolkits often rely on non-standardized and closed-source vendor software optimizations, creating challenges in software-hardware co-design at scale. Our goal is to support a variety of distributed-memory sparse matrix operations and helper functions in the newly created Sparse Global Arrays (SGA), such that it is possible to build portable and productive Machine Learning scenarios for algorithm/software and hardware codesign purposes. Contemporary data-parallel schemes for training/inference are undergoing a major overhaul since model replication limits scalability and causes resource inefficiencies. As such, we have adopted tensor parallelism in decomposing the model and inputs, to mitigate memory issues. Current implementation is built on top of MPI and uses CPUs to maximize the portability across the platforms.
Distributed computing, machine learning↗
Inference for Nonparanormal Partial Correlation via Regularized Rank-Based Nodewise Regression
Abstract Partial correlation is a common tool in studying conditional dependence for Gaussian distributed data. However, partial correlation being zero may not be equivalent to conditional independence under non-Gaussian distributions. In this paper, we propose a statistical inference procedure for partial correlations under the high-dimensional nonparanormal (NPN) model where the observed data are normally distributed after certain monotone transformations. The NPN partial correlation is the partial correlation of the normal transformed data under the NPN model, which is a more general measure of conditional dependence. We estimate the NPN partial correlations by regularized nodewise regression based on the empirical ranks of the original data. A multiple testing procedure is proposed to identify the nonzero NPN partial correlations. The proposed method can be carried out by a simple coordinate descent algorithm for lasso optimization. It is easy-to-implement and computationally more efficient compared to the existing methods for estimating NPN graphical models. Theoretical results are developed to show the asymptotic normality of the proposed estimator and to justify the proposed multiple testing procedure. Numerical simulations and a case study on brain imaging data demonstrate the utility of the proposed procedure and evaluate its performance compared to the existing methods. Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database.
97 MATHEMATICS AND COMPUTING↗
Automation of Data Traffic Control on DSM Architecture
The design of distributed shared memory (DSM) computers liberates users from the duty to distribute data across processors and allows for the incremental development of parallel programs using, for example, OpenMP or Java threads. DSM architecture greatly simplifies the development of parallel programs having good performance on a few processors. However, to achieve a good program scalability on DSM computers requires that the user understand data flow in the application and use various techniques to avoid data traffic congestions. In this paper we discuss a number of such techniques, including data blocking, data placement, data transposition and page size control and evaluate their efficiency on the NAS (NASA Advanced Supercomputing) Parallel Benchmarks. We also present a tool which automates the detection of constructs causing data congestions in Fortran array oriented codes and advises the user on code transformations for improving data traffic in the application.
Frumkin, Michael↗
(abstract) Satellite Physical Oceanography Data Available From an EOSDIS Archive
The Physical Oceanography Distributed Active Archive Center (PO.DAAC) at the Jet Propulsion Laboratory archives and distributes data as part of the Earth Observing System Data and Information System (EOSDIS). Products available from JPL are largely satellite derived and include sea-surface height, surface-wind speed and vectors, integrated water vapor, atmospheric liquid water, sea-surface temperature, heat flux, and in-situ data as it pertains to satellite data. Much of the data is global and spans fourteen years.There is email access, a WWW site, product catalogs, and FTP capabilities. Data is free of charge.
oceanography Earth Observing System oceans data cl↗
MDLoader: A Hybrid Model-Driven Data Loader for Distributed Graph Neural Network Training
Scalable data management is essential for processing large scientific dataset on HPC platforms for distributed deep learning. In-memory distributed storage is preferred for its speed, enabling rapid, random, and frequent data access required by stochastic optimizers. Processes use one-sided or collective communication to fetch remote data, with optimal performance depending on (i) dataset characteristics, (ii) training scale, and (iii) interconnection network. Empirical analysis shows collective communication excels with larger mini-batch sizes and/or fewer processes, whereas one-sided communication outperforms at larger scales. We propose MDLoader, a hybrid in-memory data loader for distributed graph neural network training. MDLoader features a model-driven performance estimator that dynamically selects between one-sided and collective communication at the beginning of training using Tree of Parzen Estimators (TPE). Evaluations on NERSC Perlmutter and OLCF Summit show MDLoader outperforms single-backend loaders by up to 2.83 × and predicts the suitable communication method with 96.3% (Perlmutter) and 94.3% (Summit) success rate.
Bae, Jonghyun↗
Retrieving the Vertical Structure of the Effective Aerosol Complex Index of Refraction from a Combination of Aerosol in Situ and Remote Sensing Measurements During TARFOX
The largest uncertainty in estimates of the effects of atmospheric aerosols on climate stems from uncertainties in the determination of their microphysical properties, including the aerosol complex index of refraction, which in turn determines their optical properties. A novel technique is used to estimate the aerosol complex index of refraction in distinct vertical layers from a combination of aerosol in situ size distribution and remote sensing measurements during the Tropospheric Aerosol Radiative Forcing Observational Experiment (TARFOX). In particular, aerosol backscatter measurements using the NASA Langley LASE (Lidar Atmospheric Sensing Experiment) instrument and in situ aerosol size distribution data are utilized to derive vertical profiles of the "effective" aerosol complex index of refraction at 815 nm (i.e., the refractive index that would provide the same backscatter signal in a forward calculation on the basis of the measured in situ particle size distributions for homogeneous, spherical aerosols). A sensitivity study shows that this method yields small errors in the retrieved aerosol refractive indices, provided the errors in the lidar-derived aerosol backscatter are less than 30% and random in nature. Absolute errors in the estimated aerosol refractive indices are generally less than 0.04 for the real part and can be as much as 0.042 for the imaginary part in the case of a 30% error in the lidar-derived aerosol backscatter. The measurements of aerosol optical depth from the NASA Ames Airborne Tracking Sunphotometer (AATS-6) are successfully incorporated into the new technique and help constrain the retrieved aerosol refractive indices. An application of the technique to two TARFOX case studies yields the occurrence of vertical layers of distinct aerosol refractive indices. Values of the estimated complex aerosol refractive index range from 1.33 to 1.45 for the real part and 0.001 to 0.008 for the imaginary part. The methodology devised in this study provides, for the first time, a complete set of vertically resolved aerosol size distribution and refractive index data. yielding the vertical distribution of aerosol optical properties required for the determination of aerosol-induced radiative flux changes.
Redemann, J.↗
Retrieving the Vertical Structure of the Effective Aerosol Complex Index of Refraction from a Combination of Aerosol in Situ and Remote Sensing Measurements During TARFOX
The largest uncertainty in estimates of the effects of atmospheric aerosols on climate stems from uncertainties in the determination of their microphysical properties, including the aerosol complex index of refraction, which in turn determines their optical properties. A novel technique is used to estimate the aerosol complex index of refraction in distinct vertical layers from a combination of aerosol in situ size distribution and remote sensing measurements during the Tropospheric Aerosol Radiative Forcing Observational Experiment (TARFOX). In particular, aerosol backscatter measurements using the NASA Langley LASE (Lidar Atmospheric Sensing Experiment) instrument and in situ aerosol size distribution data are utilized to derive vertical profiles of the 'effective' aerosol complex index of refraction at 815 nm (i.e., the refractive index that would provide the same backscatter signal in a forward calculation on the basis of the measured in situ particle size distributions for homogeneous, spherical aerosols). A sensitivity study shows that this method yields small errors in the retrieved aerosol refractive indices, provided the errors in the lidar derived aerosol backscatter are less than 30% and random in nature. Absolute errors in the estimated aerosol refractive indices are generally less than 0.04 for the real part and can be as much as 0.042 for the imaginary part in the case of a 30% error in the lidar-derived aerosol backscatter. The measurements of aerosol optical depth from the NASA Ames Airborne Tracking Sunphotometer (AATS-6) are successfully incorporated into the new technique and help constrain the retrieved aerosol refractive indices. An application of the technique to two TARFOX case studies yields the occurrence of vertical layers of distinct aerosol refractive indices. Values of the estimated complex aerosol refractive index range from 1.33 to 1.45 for the real part and 0.001 to 0.008 for the imaginary part. The methodology devised in this study provides, for the first time a complete set of vertically resolved aerosol size distribution and refractive index data, yielding the vertical distribution of aerosol optical properties required for the determination of aersol-induced radiative flux changes
Redemann, J.↗
Low-energy electron transport with the method of discrete ordinates
The one-dimensional discrete ordinates code ANISN was adapted to transport low energy (a few MeV) electrons. Calculated results obtained with ANISN were compared with experimental data for transmitted electron energy and angular distribution data for electrons normally incident on aluminum slabs of various thicknesses. The calculated and experimental results are in good agreement for a thin slab (0.2 of the electron range), but not for the thicker slabs (0.6 of the electron range). Calculated results obtained with ANISN were also compared with results obtained using Monte Carlo methods.
Bartine, D. E.↗
Shuttle ascent and shock impingement aerodynamic heating studies
The collection and analysis of aerodynamic heating data obtained from shock impingement experimental investigation were completed. The data were categorized into four interference areas; fin leading edge, wing/fuselage fin/plate corners, and space shuttle configurations. The effects of shock impingement were found to increase the heating rates 10 to 40 times the undisturbed values. A test program was completed at NASA/Langley Research Center to investigate the magnitudes and surface patterns of the mated shock interference flowfield. A 0.0065 scale thin-skin model of the MDAC 256-20 space shuttle booster mated with a Stycast model of the MDAC Internal tank orbiter was tested in the 20-inch M=6 tunnel, the 31-inch M=10 tunnel, and the 48-inch Unitary Plan Tunnel. The gap region of the ascent configuration was the principal area of interest where both thermocouple and phase-change paint data were obtained. Pressure and heat transfer distributions data on the leeward surface of a 75-degree sweep slab delta wing are presented. The effects of surface roughness on boundary layer transition and aerodynamic heating were investigated.
Lanning, W. D.↗
The Seasat commercial demonstration program
The background and development of the Seasat commercial demonstration program are reviewed and the Seasat spacecraft and its sensors (altimeter, wind field scatterometer, synthetic aperture radar, and scanning multichannel microwave radiometer) are described. The satellite data distribution system allows for selected sets of data, reformatted or tailored to specific needs and geographical regions, to be available to commercial users. Products include sea level and upper atmospheric pressure, sea surface temperature, marine winds, significant wave heights, primary wave direction and period, and spectral wave data. The results of a set of retrospective case studies performed for the commercial demonstration program are described. These are in areas of application such as marine weather and ocean condition forecasting, offshore resource exploration and development, commercial fishing, and marine transportation.
Mccandless, S. W.↗
Generic functional requirements for a NASA general-purpose data base management system
Generic functional requirements for a general-purpose, multi-mission data base management system (DBMS) for application to remotely sensed scientific data bases are detailed. The motivation for utilizing DBMS technology in this environment is explained. The major requirements include: (1) a DBMS for scientific observational data; (2) a multi-mission capability; (3) user-friendly; (4) extensive and integrated information about data; (5) robust languages for defining data structures and formats; (6) scientific data types and structures; (7) flexible physical access mechanisms; (8) ways of representing spatial relationships; (9) a high level nonprocedural interactive query and data manipulation language; (10) data base maintenance utilities; (11) high rate input/output and large data volume storage; and adaptability to a distributed data base and/or data base machine configuration. Detailed functions are specified in a top-down hierarchic fashion. Implementation, performance, and support requirements are also given.
Lohman, G. M.↗
The 13-inch magnetic suspension and balance system wind tunnel
NASA Langley has a small, subsonic wind tunnel in use with the 13-inch Magnetic Suspension and Balance System (MSBS). The tunnel is capable of speeds up to Mach 0.5. This report presents tunnel design and construction details. It includes flow uniformity, angularity, and velocity fluctuation data. It also compares experimental Mach number distribution data with computed results for the General Electric Streamtube Curvature Program.
Johnson, William G., Jr.↗
Anomaly detection of turbopump vibration in Space Shuttle Main Engine using statistics and neural networks
The statistical and neural networks methods have been applied to investigate the feasibility in detecting anomalies in turbopump vibration of SSME. The anomalies are detected based on the amplitude of peaks of fundamental and harmonic frequencies in the power spectral density. These data are reduced to the proper format from sensor data measured by strain gauges and accelerometers. Both methods are feasible to detect the vibration anomalies. The statistical method requires sufficient data points to establish a reasonable statistical distribution data bank. This method is applicable for on-line operation. The neural networks method also needs to have enough data basis to train the neural networks. The testing procedure can be utilized at any time so long as the characteristics of components remain unchanged.
Lo, C. F.↗
A land-surface testbed for EOSDIS
We propose to develop an on-line data distribution and interactive display system for the collection, archival, distribution and analysis of operational weather satellite data for applications in land surface studies. A 1,000 square km scene of the western U.S. (centered on the Colorado Rockies) will be extracted from Advanced Very High Resolution Radiometer (AVHRR) imagery collected from morning and afternoon passes of the NOAA polar-orbiters at the direct readout stations operated by CU/CCAR. All five channels of these AVHRR data will be navigated and map registered at CU/CCAR and then be transferred to NCAR for storage in an on-line data system. Software will also be available at NCAR to process and navigate the raw AVHRR data as needed. A display workstation software, based on a Macintosh II computer, will be developed that will display and further process the AVHRR data for studies of vegetation monitoring and snowpack assessment. Various options of presently used techniques for both vegetation and snowpack monitoring will be implemented in the workstation software to provide the individual investigator with the freedom to interact with the satellite image data. The display software will be freely distrributed online to interested investigators, and the AVHRR data will be made available on-line to anyone interested. In addition, potential users will be sought out and connected to the on-line data archive. This experiment with an active on-line archive and interactive analysis systems will provide experience with a small scale EOSDIS.
Emery, William↗
Benchmark of FDNS CFD Code For Direct Connect RBCC Test Data
Computational Fluid Dynamics (CFD) analysis results are compared with experimental data from the Pennsylvania State University's (PSU) Propulsion Engineering Research Center (PERC) rocket based combined cycle (RBCC) rocket-ejector experiments. The PERC RBCC experimental hardware was in a direct-connect configuration in diffusion and afterburning (DAB) operation. The objective of the present work was to validate the Finite Difference Navier Stokes (FDNS) CFD code for the rocket-ejector mode internal fluid mechanics and combustion phenomena. A second objective was determine the best application procedures to use FDNS as a predictive/engineering tool. Three-dimensional CFD analysis was performed. Solution methodology and grid requirements are discussed. CFD results are compared to experimental data for static pressure, Raman Spectroscopy species distribution data and RBCC net thrust and specified impulse.
Ruf, J. H.↗
Dynamic Cloud-Based Data Collection System
A schema-less cloud-based distributed data collection system with dynamic validation and comprehensive system status and alerting provided a reconfigurable data collection system to solve evolving incoming data schemas. Successful adaptation of the dynamic validation was useful in conforming to the evolution of incoming data.
dynamic data collection↗