Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data parallelism”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 721 records · Page 40
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Building a FAIR data ecosystem for incorporating single-cell transcriptomics data into agricultural genome to phenome research

Introduction The agriculture genomics community has numerous data submission standards available, but the standards for describing and storing single-cell (SC, e.g., scRNA- seq) data are comparatively underdeveloped. Methods To bridge this gap, we leveraged recent advancements in human genomics infrastructure, such as the integration of the Human Cell Atlas Data Portal with Terra, a secure, scalable, open-source platform for biomedical researchers to access data, run analysis tools, and collaborate. In parallel, the Single Cell Expression Atlas at EMBL-EBI offers a comprehensive data ingestion portal for high-throughput sequencing datasets, including plants, protists, and animals (including humans). Developing data tools connecting these resources would offer significant advantages to the agricultural genomics community. The FAANG data portal at EMBL-EBI emphasizes delivering rich metadata and highly accurate and reliable annotation of farmed animals but is not computationally linked to either of these resources. Results Herein, we describe a pilot-scale project that determines whether the current FAANG metadata standards for livestock can be used to ingest scRNA-seq datasets into Terra in a manner consistent with HCA Data Portal standards. Importantly, rich scRNA-seq metadata can now be brokered through the FAANG data portal using a semi-automated process, thereby avoiding the need for substantial expert curation. We have further extended the functionality of this tool so that validated and ingested SC files within the HCA Data Portal are transferred to Terra for further analysis. In addition, we verified data ingestion into Terra, hosted on Azure, and demonstrated the use of a workflow to analyze the first ingested porcine scRNA-seq dataset. Additionally, we have also developed prototype tools to visualize the output of scRNA-seq analyses on genome browsers to compare gene expression patterns across tissues and cell populations. This JBrowse tool now features distinct tracks, showcasing PBMC scRNA-seq alongside two bulk RNA-seq experiments. Discussion We intend to further build upon these existing tools to construct a scientist-friendly data resource and analytical ecosystem based on Findable, Accessible, Interoperable, and Reusable (FAIR) SC principles to facilitate SC-level genomic analysis through data ingestion, storage, retrieval, re-use, visualization, and comparative annotation across agricultural species.

Genetics & Heredity↗
NASA NTRS1989

Hardware impacts to software development strategies - The history of the development of the Mars Observer Payload Data Subsystem embedded real-time software

Ways in which parallel hardware development and high level requirements changes have influenced Mars Observer Payload Data Subsystem (PDS) flight software development are discussed. Particular attention is given to ways in which the evolving hardware product and changing requirements have led to repeated modification to software requirements, design, code, and test tools and a delay in the closure of corresponding phases of the software development life cycle. Design and implementation problems which were encountered during the PDS software development effort are described.

Elson, Anne B.↗
NASA NTRS1991

System Decommutes And Displays Telemetry Data

TDPlus computer program software system for decommutation of pulse-code-modulation (PCM) telemetry signals. Provides synchronization, conversion into engineering units, and display of serial bit streams. Transforms IBM PC-compatible computer into PCM-telemetry-decommutation system. Synchronizes telemetric signals data and enables conversion back into such meaningful forms as voltage, current, pressure, and the like. Also controls operation of digital-to-analog converters to ship data to paper strip charts or to parallel digital ports for offloading to other computers. Software used to process actual data only when telemetry-data-processing computer modified in accordance with specifications contained in "TDPlus TM Data Processor" (GSC-13291). Written in Turbo C and 8088 Assembly language.

Massey, D. E.↗
DOE OSTI2021

Machine Learning Driven Contouring of High-Frequency Four-Dimensional Cardiac Ultrasound Data

Automatic boundary detection of 4D ultrasound (4DUS) cardiac data is a promising yet challenging application at the intersection of machine learning and medicine. Using recently developed murine 4DUS cardiac imaging data, we demonstrate here a set of three machine learning models that predict left ventricular wall kinematics along both the endo- and epi-cardial boundaries. Each model is fundamentally built on three key features: (1) the projection of raw US data to a lower dimensional subspace, (2) a smoothing spline basis across time, and (3) a strategic parameterization of the left ventricular boundaries. Model 1 is constructed such that boundary predictions are based on individual short-axis images, regardless of their relative position in the ventricle. Model 2 simultaneously incorporates parallel short-axis image data into their predictions. Model 3 builds on the multi-slice approach of model 2, but assists predictions with a single ground-truth position at end-diastole. To assess the performance of each model, Monte Carlo cross validation was used to assess the performance of each model on unseen data. For predicting the radial distance of the endocardium, models 1, 2, and 3 yielded average R2 values of 0.41, 0.49, and 0.71, respectively. Monte Carlo simulations of the endocardial wall showed significantly closer predictions when using model 2 versus model 1 at a rate of 48.67%, and using model 3 versus model 2 at a rate of 83.50%. These finding suggest that a machine learning approach where multi-slice data are simultaneously used as input and predictions are aided by a single user input yields the most robust performance. Subsequently, we explore the how metrics of cardiac kinematics compare between ground-truth contours and predicted boundaries. We observed negligible deviations from ground-truth when using predicted boundaries alone, except in the case of early diastolic strain rate, providing confidence for the use of such machine learning models for rapid and reliable assessments of murine cardiac function. To our knowledge, this is the first application of machine learning to murine left ventricular 4DUS data. Future work will be needed to strengthen both model performance and applicability to different cardiac disease models.

4D ultrasound↗
DOE OSTI2022

Proximity Portability and in Transit , M-to-N Data Partitioning and Movement in SENSEI [Book Chapter]

In high-performance parallel in situ processing, the term in transit processing refers to those configurations where data must move from a producer to a consumer that runs on separate resources. In the context of parallel and distributed computing on an HPC platform one of the central challenges is to determine a mapping of data from producer ranks to consumer ranks. This problem is complicated by the heterogeneity that arises in producer-consumer pairs, such as when producer and consumer codes have different levels of concurrency, different scaling characteristics, or different data models. The resulting mapping and movement of data from M producer to N consumer ranks can have a significant impact on aggregate application performance, particularly when the data consumer requires only a subset of the overall data for its task. This chapter focuses on the design considerations that underlie SENSEI’s implementation to this challenging problem. These design considerations extend the core SENSEI architecture and include ideas like the need to accommodate flexibility in the choice of different partitioning methods, the ability for a data consumer to request and receive only the subset of data needed for its particular operation, and the ability to leverage any of several different data transport tools. The idea of proximity portability, being able to use different data transport methods as part of an in transit workflow, is illustrated through the use of three different transport layers where switching from one transport tool to another is accomplished with only a configuration file change. Here, the chapter also includes a performance analysis summary showing the performance gains that are possible in terms of multiple metrics, such as memory footprint, time to solution, and amount of data moved, when using optimized partitioners in an in transit setting, gains that are made possible by the implementation shaped by specific design considerations.

Bethel, E. Wes↗
NASA NTRS2001

An OFDM System Using Polyphase Filter and DFT Architecture for Very High Data Rate Applications

This paper presents a conceptual architectural design of a four-channel Orthogonal Frequency Division Multiplexing (OFDM) system with an aggregate information throughput of 622 megabits per second (Mbps). Primary emphasis is placed on the generation and detection of the composite waveform using polyphase filter and Discrete Fourier Transform (DFT) approaches to digitally stack and bandlimit the individual carriers. The four-channel approach enables the implementation of a system that can be both power and bandwidth efficient, yet enough parallelism exists to meet higher data rate goals. It also enables a DC power efficient transmitter that is suitable for on-board satellite systems, and a moderately complex receiver that is suitable for low-cost ground terminals. The major advantage of the system as compared to a single channel system is lower complexity and DC power consumption. This is because the highest sample rate is half that of the single channel system and synchronization can occur at most, depending on the synchronization technique, a quarter of the rate of a single channel system. The major disadvantage is the increased peak-to-average power ratio over the single channel system. Simulation results in a form of bit-error-rate (BER) curves are presented in this paper.

Kifle, Muli↗
DOE OSTI2020

Pass-efficient methods for compression of high-dimensional turbulent flow data

The future of high-performance computing, specifically on future Exascale computers, will presumably see memory capacity and bandwidth fail to keep pace with data generated, for instance, from massively parallel partial differential equation (PDE) systems. Current strategies proposed to address this bottleneck entail the omission of large fractions of data, as well as the incorporation of in situ compression algorithms to avoid overuse of memory. To ensure that post-processing operations are successful, this must be done in a way that a sufficiently accurate representation of the solution is stored. Moreover, in situations where the input/output system becomes a bottleneck in analysis, visualization, etc., or the execution of the PDE solver is expensive, the number of passes made over the data must be minimized. In the interest of addressing this problem, this work focuses on the utility of pass-efficient, parallelizable, low-rank, matrix decomposition methods in compressing high-dimensional simulation data from turbulent flows. Additionally, a particular emphasis is placed on using coarse representation of the data – compatible with the PDE discretization grid – to accelerate the construction of the low-rank factorization. This includes the presentation of a novel single-pass matrix decomposition algorithm for computing the so-called interpolative decomposition. The methods are described extensively and numerical experiments on two turbulent channel flow data are performed. In the first (unladen) channel flow case, compression factors exceeding 400 are achieved while maintaining accuracy with respect to first- and second-order flow statistics. In the particle-laden case, compression factors of 100 are achieved and the compressed data is used to recover particle velocities. These results show that these compression methods can enable efficient computation of various quantities of interest in both the carrier and disperse phases.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
NASA NTRS1997

High Performance Morphological Filtering of Cirrus Emission from Infrared Images

The project was designed to explore new morphological filtering, techniques for the removal of foreground Galactic 'Cirrus' emission from NASA Infrared Astronomical Satellite (IRAS) data, especially at 100 microns, using parallel processors as the main engine for achieving this result. The ultimate aim was to provide NASA with completely filtered data by the end of the grant period for the entire IRAS 100 and possibly 60 micron database. If successful, the filtered data would reveal many new sources of IR emission, especially at low galactic latitudes, which had previously been heavily confused with diffuse Galactic 'cirrus'.

Appleton, P. N.↗
NASA NTRSDate not supplied

Nanopore Sequencing-Based Microbial Air Profiling Method for Crewed Spacecraft

Microbial monitoring of the International Space Station (ISS) atmosphere is vital to maintaining the health of the spacecraft and crew. Key to NASA’s microbial risk assessment is the identity of contaminating microorganisms in the environment. Historically, this has been achieved through impaction-based air sampling followed by culture. Identification of the microorganisms present requires sample return to Earth and lab-based analyses. While this culture-based approach has served to provide alerts to anomalies and overall confidence in the controls in place, it is not suitable for exploration missions with no sample return. Recently, significant advancements in molecular-based microbial monitoring via nanopore sequencing have been implemented onboard the ISS. Building on this work, multiple commercially available air samplers, compatible with downstream molecular analysis, were evaluated for use in the spaceflight environment. Through this assessment, the Coriolis Compact (Bertin Technologies), which uses cyclonic technology to collect bioaerosols onto the surface of a sterile cone, was selected for a larger-scale comparison to the current culture-based monitoring method. Using the Coriolis Compact, 1000 L of air was collected from the breakroom of an office building and a fitness center. The buffer used to dissociate the microbial cells from the surface of the cone was split between the NASA Microbiology Laboratory’s standard culture and Sanger sequencing-based method and the culture-independent nanopore sequencing method. The bacteria identified through culture were present in the nanopore data, with Micrococcus, Staphylococcus, and Moraxella being the most common cultured isolates, which is expected based on the media and growth conditions. Not surprisingly, the nanopore data yielded much higher diversity and paralleled that of previous atmospheric microbiome studies of human-occupied built environments. As compared to the culture-based data where the breakroom and fitness center data sets cluster in proximity, the nanopore data depicts the contrast of these atmospheric microbiomes. Moreover, the nanopore data were sufficient to meet NASA’s risk assessment needs and noted the culturable isolates routinely observed. This nanopore-based atmospheric microbial profiling method will enable near real-time environmental monitoring of crewed spacecraft as future missions extend beyond low-Earth orbit.

Brandon Dunbar↗
NASA NTRS1977

Analysis of low altitude atmospheric turbulence data measured in flight

All three components of turbulence were measured simultaneously in flight at each wing tip of a Beech D-18 aircraft. The flights were conducted at low altitude, 30.5 - 61.0 meters (100-200 ft.), over water in the presence of wind driven turbulence. Statistical properties of flight measured turbulence were compared with Gaussian and non-Gaussian turbulence models. Spatial characteristics of the turbulence were analyzed using the data from flight perpendicular and parallel to the wind. The probability density distributions of the vertical gusts show distinctly non-Gaussian characteristics. The distributions of the longitudinal and lateral gusts are generally Gaussian. The power spectra compare in the inertial subrange at some points better with the Dryden spectrum, while at other points the von Karman spectrum is a better approximation. In the low frequency range the data show peaks or dips in the power spectral density. The cross between vertical gusts in the direction of the mean wind were compared with a matched non-Gaussian model. The real component of the cross spectrum is in general close to the non-Gaussian model. The imaginary component, however, indicated a larger phase shift between these two gust components than was found in previous research.

Ganzer, V. M.↗
NASA NTRS1972

Post-analysis report on Chesapeake Bay data processing

The additional processing performed on data collected over the Rhode River Test Site and Forestry Site in November 1970 is reported. The techniques and procedures used to obtain the processed results are described. Thermal data collected over three approximately parallel lines of the site were contoured, and the results color coded, for the purpose of delineating important scene constituents and to identify trees attacked by pine bark beetles. Contouring work and histogram preparation are reviewed and the important conclusions from the spectral analysis and recognition computer (SPARC) signature extension work are summarized. The SPARC setup and processing records are presented and recommendations are made for future data collection over the site.

Thomson, F.↗
NASA NTRS1997

High Performance Input/Output Systems for High Performance Computing and Four-Dimensional Data Assimilation

The approach of this task was to apply leading parallel computing research to a number of existing techniques for assimilation, and extract parameters indicating where and how input/output limits computational performance. The following was used for detailed knowledge of the application problems: 1. Developing a parallel input/output system specifically for this application 2. Extracting the important input/output characteristics of data assimilation problems; and 3. Building these characteristics s parameters into our runtime library (Fortran D/High Performance Fortran) for parallel input/output support.

Fox, Geoffrey C.↗
NASA NTRS1999

Parallelization of the Physical-Space Statistical Analysis System (PSAS)

Atmospheric data assimilation is a method of combining observations with model forecasts to produce a more accurate description of the atmosphere than the observations or forecast alone can provide. Data assimilation plays an increasingly important role in the study of climate and atmospheric chemistry. The NASA Data Assimilation Office (DAO) has developed the Goddard Earth Observing System Data Assimilation System (GEOS DAS) to create assimilated datasets. The core computational components of the GEOS DAS include the GEOS General Circulation Model (GCM) and the Physical-space Statistical Analysis System (PSAS). The need for timely validation of scientific enhancements to the data assimilation system poses computational demands that are best met by distributed parallel software. PSAS is implemented in Fortran 90 using object-based design principles. The analysis portions of the code solve two equations. The first of these is the "innovation" equation, which is solved on the unstructured observation grid using a preconditioned conjugate gradient (CG) method. The "analysis" equation is a transformation from the observation grid back to a structured grid, and is solved by a direct matrix-vector multiplication. Use of a factored-operator formulation reduces the computational complexity of both the CG solver and the matrix-vector multiplication, rendering the matrix-vector multiplications as a successive product of operators on a vector. Sparsity is introduced to these operators by partitioning the observations using an icosahedral decomposition scheme. PSAS builds a large (approx. 128MB) run-time database of parameters used in the calculation of these operators. Implementing a message passing parallel computing paradigm into an existing yet developing computational system as complex as PSAS is nontrivial. One of the technical challenges is balancing the requirements for computational reproducibility with the need for high performance. The problem of computational reproducibility is well known in the parallel computing community. It is a requirement that the parallel code perform calculations in a fashion that will yield identical results on different configurations of processing elements on the same platform. In some cases this problem can be solved by sacrificing performance. Meeting this requirement and still achieving high performance is very difficult. Topics to be discussed include: current PSAS design and parallelization strategy; reproducibility issues; load balance vs. database memory demands, possible solutions to these problems.

Larson, J. W.↗
DOE OSTI2023

In-depth analysis on parallel processing patterns for high-performance Dataframes

The Data Science domain has expanded monumentally in both research and industry communities during the past decade, predominantly owing to the Big Data revolution. Artificial Intelligence (AI) and Machine Learning (ML) are bringing more complexities to data engineering applications, which are now integrated into data processing pipelines to process terabytes of data. Typically, a significant amount of time is spent on data preprocessing in these pipelines, and hence improving its efficiency directly impacts the overall pipeline performance. The community has recently embraced the concept of Dataframes as the de-facto data structure for data representation and manipulation. However, the most widely used serial Dataframes today (R, pandas) experience performance limitations while working on even moderately large data sets. We believe that there is plenty of room for improvement by taking a look at this problem from a high-performance computing point of view. In a prior publication, we presented a set of parallel processing patterns for distributed dataframe operators and the reference runtime implementation, Cylon. In this paper, we are expanding on the initial concept by introducing a cost model for evaluating the said patterns. Furthermore, we evaluate the performance of Cylon on the ORNL Summit supercomputer.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1975

Evaluation of ERTS multispectral signatures in relation to ground control signatures using a nested-sampling approach

The author has identified the following significant results. Ground measured spectral signatures of wavelength bands matching ERTS MSS were collected using a radiometer at several Californian and Nevadan sites, and directly compared with similar data from ERTS CCTs. The comparison was tested at the highest possible spatial resolution for ERTS, using deconvoluted MSS data, and contrasted with that of ground measured spectra, originally from 1 meter squares. In the mobile traverses of the grassland sites, these one meter fields of view were integrated into eighty meter transects along the five km track across four major rock/soil types. Suitable software was developed to read the MSS CCT tapes, to shadeprint individual bands with user-determined greyscale stretching. Four new algorithms for unsupervised and supervised, normalized and unnormalized clustering were developed, into a program termed STANSORT. Parallel software allowed the field data to be calibrated, and by using concurrently continuously collected, upward- and downward-viewing, 4 band radiometers, bidirectional reflectances could be calculated.

Lyon, R. J. P.↗
DOE OSTI2024

High-performance data management for whole slide image analysis in digital pathology

When dealing with giga-pixel digital pathology in whole-slide imaging, a notable proportion of data records holds relevance during each analysis operation. For instance, when deploying an image analysis algorithm on whole-slide images (WSI), the computational bottleneck often lies in the input-output (I/O) system. This is particularly notable as patch-level processing introduces a considerable I/O load onto the computer system. However, this data management process could be further paralleled, given the typical independence of patch-level image processes across different patches. This paper details our endeavors in tackling this data access challenge by implementing the Adaptable IO System version 2 (ADIOS2). Our focus has been constructing and releasing a digital pathology-centric pipeline using ADIOS2, which facilitates streamlined data management across WSIs. Additionally, we’ve developed strategies aimed at curtailing data retrieval times. The performance evaluation encompasses two key scenarios: (1) a pure CPU-based image analysis scenario (“CPU scenario”), and (2) a GPU-based deep learning framework scenario (“GPU scenario”). Our findings reveal noteworthy outcomes. Under the CPU scenario, ADIOS2 showcases an impressive two-fold speed-up compared to the brute-force approach. In the GPU scenario, its performance stands on par with the cutting-edge GPU I/O acceleration framework, NVIDIA Magnum IO GPU Direct Storage (GDS). From what we know, this appears to be among the initial instances, if any, of utilizing ADIOS2 within the field of digital pathology. The source code has been made publicly available at https://github.com/hrlblab/adios.

Wang, Xiao↗
NASA NTRS1986

Mountain building processes in the Central Andes

False color composite images of the Thematic Mapper (TM) bands 5, 4, and 2 were examined to make visual interpretations of geological features. The use of the roam mode of image display with the International Imaging Systems (IIS) System 600 image processing package running on the IIS Model 75 was very useful. Several areas in which good comparisons with ground data existed, were examined in detail. Parallel to the visual approach, image processing methods are being developed which allow the complete use of the seven TM bands. The data was organized into easily accessible files and a visual cataloging of the quads (quarter TM scenes) with preliminary registration with the best available charts for the region. The catalog has proved to be a valuable tool for the rapid scanning of quads for a specific investigation. Integration of the data into a complete approach to the problems of uplift, deformation, and magnetism in relation to the Nazca-South American plate interaction is at an initial stage.

Bloom, A. L.↗
NASA NTRS1994

Analysis and Optimization of Parallel Software Pipeline Performance

Pipelining is a common strategy for extracting parallelism from a collection of independent computational tasks, each of which is spread among a number of processors and has an implied data dependence. When implemented on MIMD parallel computers with finite process interrupt times, pipeline algorithms suffer from slowdown--in addition to the expected pipeline fill time--due to a wave-like propagation of delays. This phenomenon, which has been observed experimentally using the performance monitoring system AIMS, is investigated analytically, and an optimal correction is derived to eliminate the wave. Efficiency increase through the correction is verified experimentally.

VanderWijngaart, Rob F.↗