Results for “Data Intensive Computing” — Page 16

DOE OSTI2015

GOTTCHA, Version2

One major challenge in the field of shotgun metagenomics is the accurate identification of the organisms present within the community, based on classification of short sequence reads. Though microbial community profiling methods have emerged to attempt to rapidly classify the millions of reads output from contemporary sequencers, the combination of incomplete databases, similarity among otherwise divergent genomes, and the large volumes of sequencing data required for metagenome sequencing has led to unacceptably high false discovery rates (FDR). Here we present the application of a novel, gene-independent and signature-based metagenomic taxonomic profiling tool with significantly smaller FDR, which is also capable of classifying never-before seen genomes into the appropriate parent taxa.The algorithm is based upon three primary computational phases: (I) genomic decomposition into bit vectors, (II) bit vector intersections to identify shared regions, and (III) bit vector subtractions to remove shared regions and reveal unique, signature regions. In the Decomposition phase, genomic data is first masked to highlight only the valid (non-ambiguous) regions and then decomposed into overlapping 24-mers. The k-mers are sorted along with their start positions, de-replicated, and then prefixed, to minimize data duplication. The prefixes are indexed and an identical data structure is created for the start positions to mimic that of the k-mer data structure. During the Intersection phase -- which is the most computationally intensive phase -- as an all-vs-all comparison is made, the number of comparisons is first reduced by four methods: (a) Prefix restriction, (b) Overlap detection, (c) Overlap restriction, and (d) Result recording. In Prefix restriction, only k-mers of the same prefix are compared. Within that group, potential overlap of k-mer suffixes that would result in a non-empty set intersection are screened for. If such an overlap exists, the region which intersects is first reduced by performing a binary search of the boundary suffixes of the smaller set into the larger set, which defines the limits of the zipper-based intersection process. Rather than recording the actual k-mers of the intersection, another data structure of identical "shape" is created which consists of only bit vectors so that only a 1 or 0 will be stored in the location of the k-mer suffix that was found in the intersection. This reduces the amount of data generated and stored considerably. During the Subtraction phase, relevant intersection bitmasks are first unionized together to form a single bitmask which is then applied over the original genome to reveal only those regions of the genome that are unique. These regions are then exported to disk in FASTA format and used in the application of determining the constituents of an unknown metagenomic community.

Freitas, Tracey↗

DOE OSTI2015

GOTTCHA, Version 1

One major challenge in the field of shotgun metagenomics is the accurate identification of the organisms present within the community, based on classification of short sequence reads. Though microbial community profiling methods have emerged to attempt to rapidly classify the millions of reads output from contemporary sequencers, the combination of incomplete databases, similarity among otherwise divergent genomes, and the large volumes of sequencing data required for metagenome sequencing has led to unacceptably high false discovery rates (FDR). Here we present the application of a novel, gene-independent and signature-based metagenomic taxonomic profiling tool with significantly smaller FDR, which is also capable of classifying never-before seen genomes into the appropriate parent taxa.The algorithm is based upon three primary computational phases: (I) genomic decomposition into bit vectors, (II) bit vector intersections to identify shared regions, and (III) bit vector subtractions to remove shared regions and reveal unique, signature regions. In the Decomposition phase, genomic data is first masked to highlight only the valid (non-ambiguous) regions and then decomposed into overlapping 24-mers. The k-mers are sorted along with their start positions, de-replicated, and then prefixed, to minimize data duplication. The prefixes are indexed and an identical data structure is created for the start positions to mimic that of the k-mer data structure. During the Intersection phase -- which is the most computationally intensive phase -- as an all-vs-all comparison is made, the number of comparisons is first reduced by four methods: (a) Prefix restriction, (b) Overlap detection, (c) Overlap restriction, and (d) Result recording. In Prefix restriction, only k-mers of the same prefix are compared. Within that group, potential overlap of k-mer suffixes that would result in a non-empty set intersection are screened for. If such an overlap exists, the region which intersects is first reduced by performing a binary search of the boundary suffixes of the smaller set into the larger set, which defines the limits of the zipper-based intersection process. Rather than recording the actual k-mers of the intersection, another data structure of identical "shape" is created which consists of only bit vectors so that only a 1 or 0 will be stored in the location of the k-mer suffix that was found in the intersection. This reduces the amount of data generated and stored considerably. During the Subtraction phase, relevant intersection bitmasks are first unionized together to form a single bitmask which is then applied over the original genome to reveal only those regions of the genome that are unique. These regions are then exported to disk in FASTA format and used in the application of determining the constituents of an unknown metagenomic community.

Freitas, Tracey↗

DOE OSTI2025

Conservative projection-based data-driven model order reduction of a fluid-kinetic spectral solver

Kinetic simulations are computationally intensive due to six-dimensional phase space discretization. Many kinetic spectral solvers use the asymmetrically weighted Hermite expansion due to its conservation and fluid-kinetic coupling properties, i.e., the lower-order Hermite moments capture and describe the macroscopic fluid dynamics, and higher-order Hermite moments describe the microscopic kinetic dynamics. We leverage this structure by developing a parametric data-driven reduced-order model based on the proper orthogonal decomposition, which projects the higher-order kinetic moments while retaining the fluid moments intact. We demonstrate analytically and numerically that the method ensures local and global mass, momentum, and energy conservation. The numerical results show that the proposed method effectively replicates the high-dimensional spectral simulations at a fraction of the computational cost and memory, as validated on the weak Landau damping and two-stream instability benchmark problems.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

NASA NTRS2014

High-Speed On-Board Data Processing for Science Instruments

A new development of on-board data processing platform has been in progress at NASA Langley Research Center since April, 2012, and the overall review of such work is presented in this paper. The project is called High-Speed On-Board Data Processing for Science Instruments (HOPS) and focuses on a high-speed scalable data processing platform for three particular National Research Council's Decadal Survey missions such as Active Sensing of CO2 Emissions over Nights, Days, and Seasons (ASCENDS), Aerosol-Cloud-Ecosystems (ACE), and Doppler Aerosol Wind Lidar (DAWN) 3-D Winds. HOPS utilizes advanced general purpose computing with Field Programmable Gate Array (FPGA) based algorithm implementation techniques. The significance of HOPS is to enable high speed on-board data processing for current and future science missions with its reconfigurable and scalable data processing platform. A single HOPS processing board is expected to provide approximately 66 times faster data processing speed for ASCENDS, more than 70% reduction in both power and weight, and about two orders of cost reduction compared to the state-of-the-art (SOA) on-board data processing system. Such benchmark predictions are based on the data when HOPS was originally proposed in August, 2011. The details of these improvement measures are also presented. The two facets of HOPS development are identifying the most computationally intensive algorithm segments of each mission and implementing them in a FPGA-based data processing board. A general introduction of such facets is also the purpose of this paper.

Beyon, Jeffrey Y.↗

DOE OSTI2015

Electronic spectra from TDDFT and machine learning in chemical space

Due to its favorable computational efficiency, time-dependent (TD) density functional theory (DFT) enables the prediction of electronic spectra in a high-throughput manner across chemical space. Its predictions, however, can be quite inaccurate. We resolve this issue with machine learning models trained on deviations of reference second-order approximate coupled-cluster (CC2) singles and doubles spectra from TDDFT counterparts, or even from DFT gap. We applied this approach to low-lying singlet-singlet vertical electronic spectra of over 20 000 synthetically feasible small organic molecules with up to eight CONF atoms. The prediction errors decay monotonously as a function of training set size. For a training set of 10 000 molecules, CC2 excitation energies can be reproduced to within ±0.1 eV for the remaining molecules. Analysis of our spectral database via chromophore counting suggests that even higher accuracies can be achieved. Based on the evidence collected, we discuss open challenges associated with data-driven modeling of high-lying spectra and transition intensities.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

NASA NTRS1990

Free-field propagation of high intensity noise

Observed spectral data from supersonic jet aircraft are known to contain much more high frequency energy than can be explained by linear acoustic propagation theory. It is believed that the high frequency energy is an effect of nonlinear distortion due to the extremely high acoustic levels generated by the jet engines. The objective, to measure acoustic waveform distortion for spherically diverging high intensity noise, was reached by using an electropneumatic acoustic source capable of generating sound pressure levels in the range of 140 to 160 decibels (re 20 micro Pa). The noise spectrum was shaped to represent the spectra generated by jet engines. Two microphones were used to capture the acoustic pressure waveform at different points along the propagation path in order to provide a direct measure of the waveform distortion as well as spectral distortion. A secondary objective was to determine that the observed distortion is an acoustic effect. To do this an existing computer prediction code that deals with nonlinear acoustic propagation was used on data representative of the measured data. The results clearly demonstrate that high intensity jet noise does shift the energy in the spectrum to the higher frequencies along the propagation path. In addition, the data from the computer model are in good agreement with the measurements, thus demonstrating that the waveform distortion can be accounted for with nonlinear acoustic theory.

Welz, Joseph P.↗

NASA NTRS2007

Representation-Independent Iteration of Sparse Data Arrays

An approach is defined that describes a method of iterating over massively large arrays containing sparse data using an approach that is implementation independent of how the contents of the sparse arrays are laid out in memory. What is unique and important here is the decoupling of the iteration over the sparse set of array elements from how they are internally represented in memory. This enables this approach to be backward compatible with existing schemes for representing sparse arrays as well as new approaches. What is novel here is a new approach for efficiently iterating over sparse arrays that is independent of the underlying memory layout representation of the array. A functional interface is defined for implementing sparse arrays in any modern programming language with a particular focus for the Chapel programming language. Examples are provided that show the translation of a loop that computes a matrix vector product into this representation for both the distributed and not-distributed cases. This work is directly applicable to NASA and its High Productivity Computing Systems (HPCS) program that JPL and our current program are engaged in. The goal of this program is to create powerful, scalable, and economically viable high-powered computer systems suitable for use in national security and industry by 2010. This is important to NASA for its computationally intensive requirements for analyzing and understanding the volumes of science data from our returned missions.

James, Mark↗

NASA NTRS2004

A Numerical Simulation and Statistical Modeling of High Intensity Radiated Fields Experiment Data

Tests are conducted on a quad-redundant fault tolerant flight control computer to establish upset characteristics of an avionics system in an electromagnetic field. A numerical simulation and statistical model are described in this work to analyze the open loop experiment data collected in the reverberation chamber at NASA LaRC as a part of an effort to examine the effects of electromagnetic interference on fly-by-wire aircraft control systems. By comparing thousands of simulation and model outputs, the models that best describe the data are first identified and then a systematic statistical analysis is performed on the data. All of these efforts are combined which culminate in an extrapolation of values that are in turn used to support previous efforts used in evaluating the data.

Smith, Laura J.↗

NASA NTRS1995

The neutral oxygen spectrum. 1: Collisionally excited level populations and line intensities under optically thin conditions

This is the first paper in a projected program to produce quantitative information on the spectrum of the neutral oxygen atom under a variety of excitation conditions. Radiative rates and effective collision strengths are assembled from the recent literature where available, or are calculated for as yet untreated transitions using the University College superstructure/distorted-wave computer package, to produce a complete set of atomic data for a 13 hybrid level model of neutral oxygen. Level populations and relative intensities for 28 allowed, inter-combination, and forbidden oxygen lines are computed, under optically thin conditions, for the electron density range 4.0 less than log N(sub e) less than 12.0 and the electron temperature values T(sub e) = 5000, 10,000, 20,000, 50,000, and 100,000 K. Preliminary applications to observed intercombination/allowed and forbidden/allowed line ratios are discussed.

Bhatia, A. K.↗

DOE OSTI2018

GPU acceleration of a petascale application for turbulent mixing at high Schmidt number using OpenMP 4.5

Not Available

Clay, M. P.↗

NASA NTRS1993

Thin film ferroelectric electro-optic memory

An electrically programmable, optically readable data or memory cell is configured from a thin film of ferroelectric material, such as PZT, sandwiched between a transparent top electrode and a bottom electrode. The output photoresponse, which may be a photocurrent or photo-emf, is a function of the product of the remanent polarization from a previously applied polarization voltage and the incident light intensity. The cell is useful for analog and digital data storage as well as opto-electric computing. The optical read operation is non-destructive of the remanent polarization. The cell provides a method for computing the product of stored data and incident optical data by applying an electrical signal to store data by polarizing the thin film ferroelectric material, and then applying an intensity modulated optical signal incident onto the thin film material to generate a photoresponse therein related to the product of the electrical and optical signals.

Thakoor, Sarita↗

NASA NTRS1990

Status of the Delco Systems Operations forward looking windshear detection program

Delco Systems Operations, a division of General Motors Hughes Electronics Corporation, is developing a Forward Looking Windshear Detection System based on the integration of infrared remote sensing and accelerometer reactive sensing technologies. The infrared sensor is a multi-spectral, scanning radiometer operating in the 8 to 14 micron region. A 2 x 5 detector array with parallel-serial scanning produces 60 degrees horizontal and 10 degrees vertical-fields of view. Using multiple wavelength signals, azimuth temperature gradients are analyzed for characteristic signatures of thermally induced windshear phenomena. Elevation temperature gradients are processed through an atmosphere model to continuously compute a stability index for arming microburst detection criteria. The atmosphere model and proprietary computer processing algorithms combine to generate coarse estimates of disturbance ranges based on multiple wavelength radiance data with different extinction coefficients. Computer outputs of atmospheric stability, disturbance intensity, and azimuth and range information provide a situation display capability. A ground operated, experimental radiometer has been developed and is being used to verify the detection and discrimination concepts at an atmospheric and simulated rain test facility in Milwaukee. A prototype airborne radiometer is being developed for flight test evaluation during the summer of 1989.

Gallagher, Brian J.↗

NASA NTRS1997

Sulfur in Cometary Dust

The computer-intensive project consisted of the analysis and synthesis of existing data on composition of comet Halley dust particles. The main objective was to obtain a complete inventory of sulfur containing compounds in the comet Halley dust by building upon the existing classification of organic and inorganic compounds and applying a variety of statistical techniques for cluster and cross-correlational analyses. A student hired for this project wrote and tested the software to perform cluster analysis. The following tasks were carried out: (1) selecting the data from existing database for the proposed project; (2) finding access to a standard library of statistical routines for cluster analysis; (3) reformatting the data as necessary for input into the library routines; (4) performing cluster analysis and constructing hierarchical cluster trees using three methods to define the proximity of clusters; (5) presenting the output results in different formats to facilitate the interpretation of the obtained cluster trees; (6) selecting groups of data points common for all three trees as stable clusters. We have also considered the chemistry of sulfur in inorganic compounds.

Fomenkova, M. N.↗

NASA NTRS2010

Kepler Science Operations Center Pipeline Framework

The Kepler mission is designed to continuously monitor up to 170,000 stars at a 30 minute cadence for 3.5 years searching for Earth-size planets. The data are processed at the Science Operations Center (SOC) at NASA Ames Research Center. Because of the large volume of data and the memory and CPU-intensive nature of the analysis, significant computing hardware is required. We have developed generic pipeline framework software that is used to distribute and synchronize the processing across a cluster of CPUs and to manage the resulting products. The framework is written in Java and is therefore platform-independent, and scales from a single, standalone workstation (for development and research on small data sets) to a full cluster of homogeneous or heterogeneous hardware with minimal configuration changes. A plug-in architecture provides customized control of the unit of work without the need to modify the framework itself. Distributed transaction services provide for atomic storage of pipeline products for a unit of work across a relational database and the custom Kepler DB. Generic parameter management and data accountability services are provided to record the parameter values, software versions, and other meta-data used for each pipeline execution. A graphical console allows for the configuration, execution, and monitoring of pipelines. An alert and metrics subsystem is used to monitor the health and performance of the pipeline. The framework was developed for the Kepler project based on Kepler requirements, but the framework itself is generic and could be used for a variety of applications where these features are needed.

Klaus, Todd C.↗

NASA NTRS1977

The estimation of pointing angle and normalized surface scattering cross section from GEOS-3 radar altimeter measurements

The statistical error of the pointing angle estimation technique is determined as a function of the effective receiver signal to noise ratio. Other sources of error are addressed and evaluated with inadequate calibration being of major concern. The impact of pointing error on the computation of normalized surface scattering cross section (sigma) from radar and the waveform attitude induced altitude bias is considered and quantitative results are presented. Pointing angle and sigma processing algorithms are presented along with some initial data. The intensive mode clean vs. clutter AGC calibration problem is analytically resolved. The use clutter AGC data in the intensive mode is confirmed as the correct calibration set for the sigma computations.

Brown, G. S.↗

DOE OSTI2014

DIRAQ: scalable in situ data- and resource-aware indexing for optimized query performance

Scientific data analytics in high-performance computing environments has been evolving along with the advancement of computing capabilities. With the onset of exascale computing, the increasing gap between compute performance and I/O bandwidth has rendered the traditional post-simulation processing a tedious process. Despite the challenges due to increased data production, there exists an opportunity to benefit from "cheap" computing power to perform query-driven exploration and visualization during simulation time. To accelerate such analyses, applications traditionally augment, post-simulation, raw data with large indexes, which are then repeatedly utilized for data exploration. However, the generation of current state-of-the-art indexes involves a compute- and memory-intensive processing, thus rendering them inapplicable in an in situ context. In this paper we propose DIRAQ, a parallel in situ, in network data encoding and reorganization technique that enables the transformation of simulation output into a query-efficient form, with negligible runtime overhead to the simulation run. DIRAQ's effective core-local, precision-based encoding approach incorporates an embedded compressed index that is 3---6 $$\times $$ ? smaller than current state-of-the-art indexing schemes. Its data-aware index adjustmentation improves performance of group-level index layout creation by up to 35 % and reduces the size of the generated index by up to 27 %. Moreover, DIRAQ's in network index merging strategy enables the creation of aggregated indexes that speed up spatial-context query responses by up to $$10\times $$ 10 ? versus alternative techniques. DIRAQ's topology-, data-, and memory-aware aggregation strategy results in efficient I/O and yields overall end-to-end encoding and I/O time that is less than that required to write the raw data with MPI collective I/O.

Computer Science↗

DOE OSTI2020

Coordinative Entities: Forms of Organizing in Data Intensive Science

Scientific collaboration is a long-standing subject of CSCW scholarship that typically focuses on the development and use of computing systems to facilitate research. The research presented in this article investigates the sociality of science by identifying and describing particular, common forms of organizing that researchers in four different scientific realms employ to conduct work in both local contexts and as part of distributed, global projects. This paper introduces five prototypical forms of organizing we categorize as coordinative entities: the Principal Group, Intermittent Exchange, Sustained Aggregation, Federation, and Facility Organization. Coordinative entities as a categorization help specify, articulate, compare, and trace overlapping and evolving arrangements scientists use to facilitate data intensive research. We use this typology to unpack complexities of data intensive scientific collaboration in four cases, showing how scientists invoke different coordinative entities across three types of research activities: data collection, processing, and analysis. Finally, our contribution scrutinizes the sociality of scientific work to illustrate how these actors engage in relational work within and among diverse, dispersed forms of organizing across project, funding, and disciplinary boundaries.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2026

Toward memory-efficient melt pool monitoring: a classification framework using event-based imaging and sparse sensing technique

Vision sensors like CMOS and CCD cameras are often used for in-process monitoring of melt pools in laser-based additive and welding processes, but they require transferring large amounts of data and computational processing resources. Event-based neuromorphic imagery, on the other hand, detects only the change in pixel intensity, thus potentially reducing the data amount and latency. With an event imager, this study develops a framework for melt pool condition classification, including image construction, time scale selection, optimal pixel selection, and sparse classification, to achieve a highly memory-efficient scheme. These are based on sparse sensing techniques with singular value decomposition (SVD) and QR pivoting, the two fundamental matrix transformations for linear dimensionality reduction. The framework is then validated by classifying a controlled experiment by exciting various mode shapes of liquid gallium pools of varying depths (3, 6, and 8 mm). At 200 pixels, the classifier can reach overall accuracy of 75%, while at 2000 pixels (0.013% of the total possible pixels), the accuracy is nearly 90% (89.86%). At the same number of pixels, random selection can only achieve 46% and 67%, respectively. The memory savings of the sparsely sampled event data compared to a conventional imager is about 500 times. In addition to performance, implementation and limitations of the framework are also discussed.

42 ENGINEERING↗