Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11

On the Uncertainty in Single Molecule Fluorescent Lifetime and Energy Emission Measurements

Time-correlated single photon counting has recently been combined with mode-locked picosecond pulsed excitation to measure the fluorescent lifetimes and energy emissions of single molecules in a flow stream. Maximum likelihood (ML) and least squares methods agree and are optimal when the number of detected photons is large, however, in single molecule fluorescence experiments the number of detected photons can be less than 20, 67 percent of those can be noise, and the detection time is restricted to 10 nanoseconds. Under the assumption that the photon signal and background noise are two independent inhomogeneous Poisson processes, we derive the exact joint arrival time probability density of the photons collected in a single counting experiment performed in the presence of background noise. The model obviates the need to bin experimental data for analysis, and makes it possible to analyze formally the effect of background noise on the photon detection experiment using both ML or Bayesian methods. For both methods we derive the joint and marginal probability densities of the fluorescent lifetime and fluorescent emission. The ML and Bayesian methods are compared in an analysis of simulated single molecule fluorescence experiments of Rhodamine 110 using different combinations of expected background noise and expected fluorescence emission. While both the ML or Bayesian procedures perform well for analyzing fluorescence emissions, the Bayesian methods provide more realistic measures of uncertainty in the fluorescent lifetimes. The Bayesian methods would be especially useful for measuring uncertainty in fluorescent lifetime estimates in current single molecule flow stream experiments where the expected fluorescence emission is low. Both the ML and Bayesian algorithms can be automated for applications in molecular biology.

Brown, Emery N.↗

Level 1 Processing of MODIS Direct Broadcast Data at the GSFC DAAC

The GSFC DAAC is working to test and package the MODIS Level 1 Processing software for Aqua Direct Broadcast data. This entails the same code base, but different lookup tables for Aqua and Terra. However, the most significant change is the use of ancillary attitude and ephemeris files instead of orbit/attitude information within the science data stream (as with Terra). In addition, we are working on Linux: ports of the algorithms, which could eventually enable processing on PC clusters. Finally, the GSFC DAAC is also working with the GSFC Direct Readout laboratory to ingest Level 0 data from the GSFC DB antenna into the main DAAC, enabling level 1 production in near real time in support of applications users, such as the Synergy project. The mechanism developed for this could conceivably be extended to other participating stations.

Lynnes, Christopher↗

Web-based Visualization and Analytics of Petascale Data: Equity as a Tide that Lifts All Boats

Scientists generate petabytes of data daily to help uncover environmental trends or behaviors that are hard to predict. For example, understanding climate simulations based on the long-term average of temperature, precipitation, and other environmental variables is essential to predicting and establishing root causes of future undesirable scenarios and assessing possible mitigation strategies. While supercomputer centers provide a powerful infrastructure for generating petabytes of simulation output, accessing and analyzing these datasets interactively remains challenging on multiple fronts. This paper presents an approach to managing, visualizing, and analyzing petabytes of data within a browser on equipment ranging from the top NASA supercomputer to commodity hardware like a laptop. Our novel data fabric abstraction layer allows user-friendly querying of scientific information while hid-ing the complexities of dealing with file systems or cloud services.We also optimize network utilization while streaming from petas-cale repositories through state-of-the-art progressive compression algorithms. Based on this abstraction, we provide customizable dashboards that can be accessed from any device with any inter-net connection, enabling interactive visual analysis of vast amounts of data to a wide range of users - from top scientists with access to leadership-class computing environments to undergraduate students of disadvantaged backgrounds from minority-serving institutions. We focus on NASA’s use of petascale climate datasets as an example of particular societal impact and, therefore, a case where achieving equity in science participation is critical. We validate our approach by improving the ability of climate scientists to visually explore their data via two fully interactive dashboards. We further validate our approach by deploying the dashboards and simplified training materials in the classroom at a minority-serving institution.These dashboards, released in simplified form to the general public, contribute significantly to a broader push to democratize the access and use of climate data.

Data visualization↗

The Collection 6 'dark-target' MODIS Aerosol Products

Aerosol retrieval algorithms are applied to Moderate resolution Imaging Spectroradiometer (MODIS) sensors on both Terra and Aqua, creating two streams of decade-plus aerosol information. Products of aerosol optical depth (AOD) and aerosol size are used for many applications, but the primary concern is that these global products are comprehensive and consistent enough for use in climate studies. One of our major customers is the international modeling comparison study known as AEROCOM, which relies on the MODIS data as a benchmark. In order to keep up with the needs of AEROCOM and other MODIS data users, while utilizing new science and tools, we have improved the algorithms and products. The code, and the associated products, will be known as Collection 6 (C6). While not a major overhaul from the previous Collection 5 (C5) version, there are enough changes that there are significant impacts to the products and their interpretation. In its entirety, the C6 algorithm is comprised of three sub-algorithms for retrieving aerosol properties over different surfaces: These include the dark-target DT algorithms to retrieve over (1) ocean and (2) vegetated-dark-soiled land, plus the (3) Deep Blue (DB) algorithm, originally developed to retrieve over desert-arid land. Focusing on the two DT algorithms, we have updated assumptions for central wavelengths, Rayleigh optical depths and gas (H2O, O3, CO2, etc.) absorption corrections, while relaxing the solar zenith angle limit (up to 84) to increase pole-ward coverage. For DT-land, we have updated the cloud mask to allow heavy smoke retrievals, fine-tuned the assignments for aerosol type as function of season location, corrected bugs in the Quality Assurance (QA) logic, and added diagnostic parameters such as topographic altitude. For DT-ocean, improvements include a revised cloud mask for thin-cirrus detection, inclusion of wind speed dependence in the retrieval, updates to logic of QA Confidence flag (QAC) assignment, and additions of important diagnostic information. At the same time as we have introduced algorithm changes, we have also accounted for upstream changes including: new instrument calibration, revised land-sea masking, and changed cloud masking. Upstream changes also impact the coverage and global statistics of the retrieved AOD. Although our responsibility is to the DT code and products, we have also added a product that merges DT and DB product over semi-arid land surfaces to provide a more gap-free dataset, primarily for visualization purposes. Preliminary validation shows that compared to surface-based sunphotometer data, the C6, Level 2 (along swath) DT-products compare at least as well as those from C5. C6 will include new diagnostic information about clouds in the aerosol field, including an aerosol cloud mask at 500 m resolution, and calculations of the distance to the nearest cloud from clear pixels. Finally, we have revised the strategy for aggregating and averaging the Level 2 (swath) data to become Level 3 (gridded) data. All together, the changes to the DT algorithms will result in reduced global AOD (by 0.02) over ocean and increased AOD (by 0.02) over land, along with changes in spatial coverage. Changes in calibration will have more impact to Terras time series, especially over land. This will result in a significant reduction in artificial differences in the Terra and Aqua datasets, and will stabilize the MODIS data as a target for AEROCOM studie

Aerosol retrieval algorithms↗

Theory of an Automatic Seepage Meter and Ramifications for Applications

A new approach for measuring fluxes across surface water—groundwater interfaces was recently proposed. The Automatic Seepage Meter (ASM) is equipped with a precise water level sensor and digital memory that analyzes water level time series in a vertical tube inserted into a streambed. The ability to infer flux values with high temporal resolution relies on an accurate interpretation of water level dynamics inside the tube. Here, we reduce the three-dimensional hydrodynamic problem that describes the ASM water level in a variety of field conditions to a single ordinary differential equation. This novel general analytical solution for estimating ASM responses is more comprehensive and flexible than previous approaches and is applicable to the entire range of field conditions, including steady or transient stream stages, evaporation, rainfall, and noise. For example, our analysis determines the timing of the nonmonotonic ASM response to a monotonic linear stream stage variation and explains previously used empirical parabolic approximation for estimating fluxes. We present algorithms for simultaneous inference of vertical interface flux and hydraulic conductivity values together with an example code. We quantify how the accuracy of parameter estimation depends on test duration and noise amplitude and propose how our analysis can be used to optimize field test protocols. On this basis, changing the ASM geometry by increasing the radius and decreasing tube insertion depth may enable ASM field test protocols that estimate interface flux and hydraulic conductivity faster while maintaining desired accuracy. Potential applications of joint parameter estimation are suggested.

47 OTHER INSTRUMENTATION↗

A machine learning decision criterion for reducing scan time for hyperspectral neutron computed tomography systems

We present the first machine learning-based autonomous hyperspectral neutron computed tomography experiment performed at the Spallation Neutron Source. Hyperspectral neutron computed tomography allows the characterization of samples by enabling the reconstruction of crystallographic information and elemental/isotopic composition of objects relevant to materials science. High quality reconstructions using traditional algorithms such as the filtered back projection require a high signal-to-noise ratio across a wide wavelength range combined with a large number of projections. This results in scan times of several days to acquire hundreds of hyperspectral projections, during which end users have minimal feedback. To address these challenges, a golden ratio scanning protocol combined with model-based image reconstruction algorithms have been proposed. This novel approach enables high quality real-time reconstructions from streaming experimental data, thus providing feedback to users, while requiring fewer yet a fixed number of projections compared to the filtered back projection method. In this paper, we propose a novel machine learning criterion that can terminate a streaming neutron tomography scan once sufficient information is obtained based on the current set of measurements. Our decision criterion uses a quality score which combines a reference-free image quality metric computed using a pre-trained deep neural network with a metric that measures differences between consecutive reconstructions. The results show that our method can reduce the measurement time by approximately a factor of five compared to a baseline method based on filtered back projection for the samples we studied while automatically terminating the scans.

97 MATHEMATICS AND COMPUTING↗

Alert Classification for the ALeRCE Broker System: The Light Curve Classifier

We present the first version of the Automatic Learning for the Rapid Classification of Events (ALeRCE) broker light curve classifier. ALeRCE is currently processing the Zwicky Transient Facility (ZTF) alert stream, in preparation for the Vera C. Rubin Observatory. The ALeRCE light curve classifier uses variability features computed from the ZTF alert stream and colors obtained from AllWISE and ZTF photometry. We apply a balanced random forest algorithm with a two-level scheme where the top level classifies each source as periodic, stochastic, or transient, and the bottom level further resolves each of these hierarchical classes among 15 total classes. This classifier corresponds to the first attempt to classify multiple classes of stochastic variables (including core- and host-dominated active galactic nuclei, blazars, young stellar objects, and cataclysmic variables) in addition to different classes of periodic and transient sources, using real data. We created a labeled set using various public catalogs (such as the Catalina Surveys and Gaia DR2 variable stars catalogs, and the Million Quasars catalog), and we classify all objects with ≥6 g-band or ≥6 r-band detections in ZTF (868,371 sources as of 2020 June 9), providing updated classifications for sources with new alerts every day. For the top level we obtain macro-averaged precision and recall scores of 0.96 and 0.99, respectively, and for the bottom level we obtain macro-averaged precision and recall scores of 0.57 and 0.76, respectively. Updated classifications from the light curve classifier can be found at the ALeRCE Explorer website (http://alerce.online).

47 OTHER INSTRUMENTATION↗

Viscous flow simulations in VTOL aerodynamics

The critical issues in viscous flow simulations, such as boundary-layer separation, entrainment, turbulence modeling, and compressibility, are discussed with regard to the ground effects problem for vertical-takeoff-and-landing (VTOL) aircraft. A simulation of the two-dimensional incompressible lift jet in ground proximity is based on solution of the Reynolds-averaged Navier-Stokes equations and a turbulence-model equation which are written in stream function-vorticity form and are solved using Hoffman's augmented-central-difference algorithm. The resulting equations and their shortcomings are discussed when the technique is extended to two-dimensional compressible and three-dimensional incompressible flows.

Bower, W. W.↗

Systolic VLSI Reed-Solomon Decoder

Decoder for digital communications provides high-speed, pipelined ReedSolomon (RS) error-correction decoding of data streams. Principal new feature of proposed decoder is modification of Euclid greatest-common-divisor algorithm to avoid need for time-consuming computations of inverse of certain Galois-field quantities. Decoder architecture suitable for implementation on very-large-scale integrated (VLSI) chips with negative-channel metaloxide/silicon circuitry.

Shao, H. M.↗

Multiprocessing on supercomputers for computational aerodynamics

Very little use is made of multiple processors available on current supercomputers (computers with a theoretical peak performance capability equal to 100 MFLOPs or more) in computational aerodynamics to significantly improve turnaround time. The productivity of a computer user is directly related to this turnaround time. In a time-sharing environment, the improvement in this speed is achieved when multiple processors are used efficiently to execute an algorithm. The concept of multiple instructions and multiple data (MIMD) through multi-tasking is applied via a strategy which requires relatively minor modifications to an existing code for a single processor. Essentially, this approach maps the available memory to multiple processors, exploiting the C-FORTRAN-Unix interface. The existing single processor code is mapped without the need for developing a new algorithm. The procedure for building a code utilizing this approach is automated with the Unix stream editor. As a demonstration of this approach, a Multiple Processor Multiple Grid (MPMG) code is developed. It is capable of using nine processors, and can be easily extended to a larger number of processors. This code solves the three-dimensional, Reynolds averaged, thin-layer and slender-layer Navier-Stokes equations with an implicit, approximately factored and diagonalized method. The solver is applied to generic oblique-wing aircraft problem on a four processor Cray-2 computer. A tricubic interpolation scheme is developed to increase the accuracy of coupling of overlapped grids. For the oblique-wing aircraft problem, a speedup of two in elapsed (turnaround) time is observed in a saturated time-sharing environment.

Yarrow, Maurice↗

Multi-processing on supercomputers for computational aerodynamics

The MIMD concept is applied, through multitasking, with relatively minor modifications to an existing code for a single processor. This approach maps the available memory to multiple processors, exploiting the C-FORTRAN-Unix interface. An existing single processor algorithm is mapped without the need for developing a new algorithm. The procedure of designing a code utilizing this approach is automated with the Unix stream editor. A Multiple Processor Multiple Grid (MPMG) code is developed as a demonstration of this approach. This code solves the three-dimensional, Reynolds-averaged, thin-layer and slender-layer Navier-Stokes equations with an implicit, approximately factored and diagonalized method. This solver is applied to a generic, oblique-wing aircraft problem on a four-processor computer using one process for data management and nonparallel computations and three processes for pseudotime advance on three different grid systems.

Yarrow, Maurice↗

Multiprocessing on supercomputers for computational aerodynamics

Little use is made of multiple processors available on current supercomputers (computers with a theoretical peak performance capability equal to 100 MFLOPS or more) to improve turnaround time in computational aerodynamics. The productivity of a computer user is directly related to this turnaround time. In a time-sharing environment, such improvement in this speed is achieved when multiple processors are used efficiently to execute an algorithm. The concept of multiple instructions and multiple data (MIMD) is applied through multitasking via a strategy that requires relatively minor modifications to an existing code for a single processor. This approach maps the available memory to multiple processors, exploiting the C-Fortran-Unix interface. The existing code is mapped without the need for developing a new algorithm. The procedure for building a code utilizing this approach is automated with the Unix stream editor.

Yarrow, Maurice↗

The computation of inviscid rotational gasdynamic flows using an alternate velocity decompositon

The present investigation has the objective to present a method which can be considered to be a modification of existing potential flow algorithms. The method can be employed for the accurate and efficient solution of rotational gasdynamic flows. The general approach to computing inviscid rotational flows considered makes use of a velocity decomposition discussed by Clebsch (1859). Recent applications of the Clebsch representation in inviscid flows have involved the convection of an externally generated vorticity. Attention is given to the case of a steady, inviscid flow of a perfect gas with a uniform incoming stream. The developed procedure makes it possible to obtain a rapid solution to inviscid, compressible, rotational flows with solution algorithms which are a modification of existing potential flow codes. An application of the procedure to supersonic rotational conical flow is discussed.

Grossman, B.↗

A Reproducible Validation of Algorithms for Estimating Array Tilt and Azimuth from Photovoltaic Power Time Series

In this research, we assess the viability of four different, publicly available algorithms for estimating the azimuth and tilt parameters of solar photovoltaic systems using only the associated AC power time series data and site latitude-longitude coordinates. In this work, we curated a benchmarking data set of 44 fixed-tilt systems, comprising 275 measured AC power inverter data streams, with known azimuth and tilt parameters. Additionally, we isolated test cases in the data set with real-world issues, including shading and clipping, to determine how algorithm performance varies based on the presence of these phenomena. Using this data set for benchmarking, we evaluated the estimated vs. actual system characteristics for each algorithm, as well as the associated algorithm execution time using a standardized benchmarking process. The two highest performing algorithms were the Solar Data Tools and the PVWatts 5-based methods, which both achieved a median absolute error of approximately 5 and 1 degrees for azimuth and tilt, respectively. During run time analysis, the SDT method was approximately 5 times faster than the PVWatts 5-based method, with the median execution time for a stream varying between 6 and 8 seconds vs. a median run time of 31 seconds for the PVWatts 5-based method.

algorithm validation↗

Contemplating Synergistic Algorithms for the NASA ACE Mission

ACE is a proposed Tier 2 NASA Decadal Survey mission that will focus on clouds, aerosols, and precipitation as well as ocean ecosystems. The primary objective of the clouds component of this mission is to advance our ability to predict changes to the Earth's hydrological cycle and energy balance in response to climate forcings by generating observational constraints on future science questions, especially those associated with the effects of aerosol on clouds and precipitation. ACE will continue and extend the measurement heritage that began with the A-Train and that will continue through Earthcare. ACE planning efforts have identified several data streams that can contribute significantly to characterizing the properties of clouds and precipitation and the physical processes that force these properties. These include dual frequency Doppler radar, high spectral resolution lidar, polarimetric visible imagers, passive microwave and submillimeter wave radiometry. While all these data streams are technologically feasible, their total cost is substantial and likely prohibitive. It is, therefore, necessary to critically evaluate their contributions to the ACE science goals. We have begun developing algorithms to explore this trade space. Specifically, we will describe our early exploratory algorithms that take as input the set of potential ACE-like data streams and evaluate critically to what extent each data stream influences the error in a specific cloud quantity retrieval.

precipitation↗

On the Advantages of Using Harmonized Landsat Sentinel-2 Data for Monitoring Environmental Change

NASA coordinates the Satellite Needs Working Group, dedicated to identifying, communicating, and addressing Earth observation needs of federal agencies. In 2016, the Harmonized Landsat Sentinel-2 (HLS) dataset was formulated and implemented to fulfill multiple needs. The combination of acquisitions from the Landsat and Sentinel-2 platforms results in a global dataset of surface reflectance with a temporal resolution of two days, while retaining the geometry and 30-meter spatial resolution of Landsat data. This harmonization allows for seamless integration with the 40-year archive of Landsat data. The HLS dataset is now available on the Google Earth Engine, enabling HLS utilization in various algorithms and frameworks essential for monitoring environmental change worldwide. During this presentation, we will demonstrate and discuss the advantages of using HLS data in comparison to using separate streams of Landsat and Sentinel-2 data in existing time series-based frameworks for change monitoring. Specifically, we will explore the application of HLS for continuous monitoring of deforestation using time series-based algorithms traditionally run with Landsat data. Additionally, we will showcase the benefits of HLS data for near real-time monitoring of forest disturbance in tropical regions. These examples underscore the value and utility of the HLS dataset for environmental monitoring and analysis.

Pontus Olofsson↗

Parallel Algorithms for Computing the Tensor-Train Decomposition

The tensor-train (TT) decomposition expresses a tensor in a data-sparse format used in molecular simulations, high-order correlation functions, and optimization. In this paper, we propose four parallelizable algorithms that compute the TT format from various tensor inputs: (1) Parallel-TTSVD for traditional format, (2) PSTT and its variants for streaming data, (3) Tucker2TT for Tucker format, and (4) TT-fADI for solutions of Sylvester tensor equations. We provide theoretical guarantees of accuracy, parallelization methods, scaling analysis, and numerical results. For example, for a d-dimension tensor in $\mathbb{R}$ $n\times∙∙∙$$\times$$n$ a two-sided sketching algorithm PSTT2 is shown to have a memory complexity of $O(n^{[d/2]})$, improving upon $O(n^{d—1})$ from previous algorithms.

97 MATHEMATICS AND COMPUTING↗

Prototype Data Acquisition and Slow Control Systems for the Mu2e Experiment

The Mu2e experiment at the Fermilab Muon Campus will search for the coherent neutrinoless conversion of a muon into an electron in the field of an aluminum nucleus with a sensitivity improvement by a factor of 10 000 over existing limits. Such a charged lepton flavor-violating reaction probes new physics at a scale unavailable with direct searches at either present or planned high-energy colliders. The Mu2e Trigger and Data Acquisition (TDAQ) system exploits otsdaq as its online Data Acquisition System (DAQ) solution. Furthermore, developed at Fermilab, otsdaq integrates both the artdaq DAQ and the art analysis frameworks for event transfer, filtering, and processing. otsdaq is an online DAQ software suite with a focus on flexibility and scalability and provides a multi-user, web-based, interface accessible through a web browser. The read out controllers (ROCs) stream out zero-suppressed data continuously from the detector subsystems to the data transfer controllers (DTCs). The data stream is then read over the peripheral component interconnect express (PCIe) bus to a software filter algorithm that selects events which are combined with the data flux coming from a cosmic-ray veto (CRV) system. The detector control system (DCS) has been developed using the experimental physics and industrial control system (EPICS) open source platform for monitoring, controlling, alarming, and archiving. The DCS has been integrated into otsdaq. A prototype of the TDAQ system and the DCS has been built at Fermilab's Feynman Computing Center. In this article, we report on the progress of the integration of this prototype in the online otsdaq software.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗