Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,369 records · Page 76

Ocean observations with EOS/MODIS: Algorithm development and post launch studies

Several significant accomplishments were made during the present reporting period. We have completed our basic study of using the 1.38 micron MODIS band for removal of the effects of thin cirrus clouds and stratospheric aerosol. The results suggest that it should be possible to correct imagery for thin cirrus clouds with optical thicknesses as large as 0.5 to 1.0. We have also acquired reflectance data for oceanic whitecaps during a cruise on the RV Malcolm Baldrige in the Gulf of Mexico. The reflectance spectrum of whitecaps was found to be similar to that for breaking waves in the surf zone measured by Frouin, Schwindling and Deschamps. We installed a CIMEL sun photometer at Fort Jefferson on the Dry Tortugas off Key West in the Gulf of Mexico. The instrument has yielded a continuous stream of data since February. It shows that the aerosol optical thickness at 669 nm is often less than 0.1 in winter. This suggests that the Southern Gulf of Mexico will be an excellent winter site for vicarious calibration. In addition, we completed a study of the effect of vicarious calibration, i.e., the accuracy with which the radiance at the top of the atmosphere (TOA) can be predicted from measurement of the sky radiance at the bottom of the atmosphere (BOA). The results suggest that the neglect of polarization in the aerosol optical property inversion algorithm and in the prediction code for the TOA radiances is the largest error associated with the radiative transfer process. Overall, the study showed that the accuracy of the TOA radiance prediction is now limited by the racliometric calibration error in the sky radiometer. Finally, considerable coccolith light scattering data were obtained in the Gulf of Maine with a flow-through instrument, along with data relating to calcite concentration and the rate of calcite production.

Gordan, Howard R.↗

Low-Cutoff, High-Pass Digital Filtering of Neural Signals

The figure depicts the major functional blocks of a system, now undergoing development, for conditioning neural signals acquired by electrodes implanted in a brain. The overall functions to be performed by this system can be summarized as preamplification, multiplexing, digitization, and high-pass filtering. Other systems under development for recording neural signals typically contain resistor-capacitor analog low-pass filters characterized by cutoff frequencies in the vicinity of 100 Hz. In the application for which this system is being developed, there is a requirement for a cutoff frequency of 5 Hz. Because the resistors needed to obtain such a low cutoff frequency would be impractically large, it was decided to perform low-pass filtering by use of digital rather than analog circuitry. In addition, it was decided to timemultiplex the digitized signals from the multiple input channels into a single stream of data in a single output channel. The signal in each input channel is first processed by a preamplifier having a voltage gain of approximately 50. Embedded in each preamplifier is a low-pass anti-aliasing filter having a cutoff frequency of approximately 10 kHz. The anti-aliasing filters make it possible to couple the outputs of the preamplifiers to the input ports of a multiplexer. The output of the multiplexer is a single stream of time-multiplexed samples of analog signals. This stream is processed by a main differential amplifier, the output of which is sent to an analog-to-digital converter (ADC). The output of the ADC is sent to a digital signal processor (DSP).

Mojarradi,Mohammad↗

Model Data Archive Associated with Manuscript "Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon"

This data package supports the publication “Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon” by Li et al. (2026). The package contains processed model inputs, configuration files, restart files, simulation outputs, scripts, and visualization products used to evaluate post-fire dissolved organic carbon (DOC) dynamics in the Naches River Watershed, Washington, USA, following the 2021 Schneider Springs Fire. The modeling workflow couples ELM-BGC, the biogeochemistry-enabled Energy Exascale Earth System Model Land Model; ATS, the Advanced Terrestrial Simulator for integrated surface-subsurface hydrology; and PFLOTRAN, a reactive transport model for multicomponent aqueous geochemistry. Together, these models simulate how wildfire-induced changes in vegetation, litter, coarse woody debris, and soil organic matter influence DOC production, transport, and reaction from burned hillslopes to stream networks. The archive includes preprocessed meteorological, geospatial, hydrologic, and biogeochemical forcing data; ELM-BGC-derived DOC source terms; ATS mesh files; PFLOTRAN reactive-transport inputs; model configuration files; spin-up and transient restart files; watershed-scale diagnostic outputs; stream concentration time series; and figures or visualization files used to inspect and reproduce key results. File types include Hierarchical Data Format 5 (HDF5) files for gridded forcing and model-coupling data, model input and configuration files for ELM-BGC, ATS, and PFLOTRAN, restart and simulation-output files generated by the modeling workflow, tabular or time-series diagnostic outputs, scripts for post-processing and figure generation, and image or visualization products associated with the manuscript. Use of the package depends on the intended task. Re-running the simulations requires the relevant modeling software, including ELM-BGC, ATS, and PFLOTRAN as ATS's geochemical engine. Inspecting outputs and reproducing figures requires Python with scientific plotting libraries such as Matplotlib, and three-dimensional model outputs may be viewed with ParaView. Geographic information system files or maps may be inspected with ArcGIS Pro or comparable GIS software. The data package is intended to enable traceability, reuse, and partial reproduction of the coupled land-to-watershed hydro-biogeochemical modeling workflow used to test how wildfire disturbance affects terrestrial carbon pools and downstream DOC dynamics.

ATS↗

Fault-Tolerant and Elastic Streaming MapReduce with Decentralized Coordination

The MapReduce programming model, due to its simplicity and scalability, has become an essential tool for processing large data volumes in distributed environments. Recent Stream Processing Systems (SPS) extend this model to provide low-latency analysis of high-velocity continuous data streams. However, integrating MapReduce with streaming poses challenges: first, the runtime variations in data characteristics such as data-rates and key-distribution cause resource overload, that inturn leads to fluctuations in the Quality of the Service (QoS); and second, the stateful reducers, whose state depends on the complete tuple history, necessitates efficient fault-recovery mechanisms to maintain the desired QoS in the presence of resource failures. We propose an integrated streaming MapReduce architecture leveraging the concept of consistent hashing to support runtime elasticity along with locality-aware data and state replication to provide efficient load-balancing with low-overhead fault-tolerance and parallel fault-recovery from multiple simultaneous failures. Our evaluation on a private cloud shows up to 2:8 improvement in peak throughput compared to Apache Storm SPS, and a low recovery latency of 700 -1500 ms from multiple failures.

97 MATHEMATICS AND COMPUTING↗

Streaming Readout and Remote Compute

Streaming Readout Data Acquisition systems coupled with distributed resources spread over vast geographic distances present new challenges to the next generation of experiments. High bandwidth modern network connectivity opens the possibility to utilize large, general-use, HTC systems that are not necessarily located close to the experiment. Near real-time response rates and workflow colocation can provide high reliability solutions to ensure efficient use of beam time. This talk will focus on a few technologies currently being developed at Jefferson Lab and in collaboration with ESnet to support fully streaming DAQ systems.

Lawrence, David↗

A high-speed lossless data compression system for space applications

This paper reports on the integration of a lossless data compression/decompression chipset into a space data system architecture. For its compression engine, the data system incorporates the Universal Source Encoder (USE) designed for the NASA/Goddard Space Flight Center. Currently, the data compression testbed generates video frames consisting of 512 lines of 512 pixels having 8-bit resolution. Each image is passed through the USE where the lines are internally partitioned into 16-word blocks. These blocks are adaptively encoded across widely varying entropy levels using a Rice 12-option set coding algorithm. The current system operates at an Input/Output rate of 10 Msamples/s or 80 Mbits/s for each buffered input line. Frame and line synchronization for each image are maintained through the use of uniquely decodable command words. Length information of each variable length compressed image line is also included in the output stream. The data and command information are passed to the next stage of the system architecture through a serial fiber-optic transmitter. The initial segment of this stage consists of packetizer hardware which adds an appropriate CCSDS header to the received source data. An uncompressed mode is optionally available to pass image lines directly to the packetizer hardware. A data decompression testbed has also been developed to confirm the data compression operation.

Miko, Joe↗

Model data for numerical evaluation of photosensitive tracers as a strategy for separating surface and subsurface transient storage in streams

This model-data archive pertains to a study aimed at the numerical evaluation of the photosensitive tracers as a potential strategy for separating the effects of surface and hyporheic storage zones (SSZs and HSZs, respectively). Separating effects of SSZ and HSZ are important for accurately representing stream function as HSZs and SSZs expose solutes to significantly different biogeochemical conditions. We perform numerical experiments using a multiscale reactive transport model for stream corridors implemented in ATS code, which allows for representing multiple storage zones with their respective travel time distributions and biogeochemistry. For each of the numerical experiment, we provide in this model-data archive python wrapper to drive ATS (forward_model.py), synthetic observation (synthetic_btc.py, BTC_observed.csv), forward model files including ATS input (multiscale_transport.tpl), PFLOTRAN inputs (reactions_channel.tpl, reactions_hz.tpl, reactions_sz.tpl), MCMC (mcmc_run.py), predictive uncertainty (pred_uncert.py, BTCs_simulated_day.csv, BTCs_simulated_night.csv), MCMC outputs (tracer_test_GR.npy, tracer_test_logps.npy, tracer_test_parameters.npy) and Jupyter notebook for post-processing and visualization (post-processing.ipynb). For the denitrification application, ATS input file (denitrification_multisubgrid.tpl), PFLOTRAN input files (denitrification_channel.in, denitrification_hz.in), predictive uncertainty (pred_uncert_denitrification.py, BTCs_simulated_DO.csv, BTCs_simulated_DOC.csv, BTCs_simulated_Nitrate.csv).

54 ENVIRONMENTAL SCIENCES↗

Accelerating DNA analysis applications on GPU clusters

DNA analysis is an emerging application of high performance bioinformatic. Modern sequencing machinery are able to provide, in few hours, large input streams of data which needs to be matched against exponentially growing databases known fragments. The ability to recognize these patterns effectively and fastly may allow extending the scale and the reach of the investigations performed by biology scientists. Aho-Corasick is an exact, multiple pattern matching algorithm often at the base of this application. High performance systems are a promising platform to accelerate this algorithm, which is computationally intensive but also inherently parallel. Nowadays, high performance systems also include heterogeneous processing elements, such as Graphic Processing Units (GPUs), to further accelerate parallel algorithms. Unfortunately, the Aho-Corasick algorithm exhibits large performance variabilities, depending on the size of the input streams, on the number of patterns to search and on the number of matches, and poses significant challenges on current high performance software and hardware implementations. An adequate mapping of the algorithm on the target architecture, coping with the limit of the underlining hardware, is required to reach the desired high throughputs. Load balancing also plays a crucial role when considering the limited bandwidth among the nodes of these systems. In this paper we present an efficient implementation of the Aho-Corasick algorithm for high performance clusters accelerated with GPUs. We discuss how we partitioned and adapted the algorithm to fit the Tesla C1060 GPU and then present a MPI based implementation for a heterogeneous high performance cluster. We compare this implementation to MPI and MPI with pthreads based implementations for a homogeneous cluster of x86 processors, discussing the stability vs. the performance and the scaling of the solutions, taking into consideration aspects such as the bandwidth among the different nodes.

59 BASIC BIOLOGICAL SCIENCES↗

REACH: Real-Time Data Awareness in Multi-Spacecraft Missions

Missions have been proposed that will use multiple spacecraft to perform scientific or commercial tasks. Indeed, in the commercial world, some spacecraft constellations already exist. Aside from the technical challenges of constructing and flying these missions, there is also the financial challenge presented by the tradition model of the flight operations team (FOT) when it is applied to a constellation mission. Proposed constellation missions range in size from three spacecraft to more than 50. If the current ratio of three-to-five FOT personnel per spacecraft is maintained, the size of the FOT becomes cost prohibitive. The Advanced Architectures and Automation Branch at the Goddard Space Flight Center (GSFC Code 588) saw the potential to reduce the cost of these missions by creating new user interfaces to the ground system health-and-safety data. The goal is to enable a smaller FOT to remain aware and responsive to the increased amount of ground system information in a multi-spacecraft environment. Rather than abandon the tried and true, these interfaces were developed to run alongside existing ground system software to provide additional support to the FOT. These new user interfaces have been combined in a tool called REACH. REACH-the Real-time Evaluation and Analysis of Consolidated Health-is a software product that uses advanced visualization techniques to make spacecraft anomalies easy to spot, no matter how many spacecraft are in the constellation. REACH reads a real-time stream of data from the ground system and displays it to the FOT such that anomalies are easy to pick out and investigate. Data visualization has been used in ground system operations for many years. To provide a unique visualization tool, we developed a unique source of data to visualize: the REACH Health Model Engine. The Health Model Engine is rule-based software that receives real-time telemetry information and outputs "health" information related to the subsystems and spacecraft that the telemetry belong to. The Health Engine can run out-of-the-box or can be tailored with a scripting language. Out of the box, it uses limit violations to determine the health of subsystems and spacecraft; when tailored, it determines health using equations combining the values and limits of any telemetry in the spacecraft. The REACH visualizations then "roll up" the information from the Health Engine into high level, summary displays. These summary visualizations can be "zoomed" into for increasing levels of detail. Currently REACH is installed in the Small Explorer (SMEX) lab at GSFC, and is monitoring three of their five spacecraft. We are scheduled to install REACH in the Mid-sized Explorer (MIDEX) lab, which will allow us to monitor up to six more spacecraft. The process of installing and using our "research" software in an operational environment has provided many insights into which parts of REACH are a step forward and which of our ideas are missteps. Our paper explores both the new concepts in spacecraft health-and-safety visualization, the difficulties of such systems in the operational environment, and the cost and safety issues of multi-spacecraft missions.

Maks, Lori↗

REACH: Real-Time Data Awareness in Multi-Spacecraft Missions

NASA's Advanced Architectures and Automation Branch at the Goddard Space Flight Center (Code 588) saw the potential to reduce the cost of constellation missions by creating new user interfaces to the ground system health-and-safety data. The goal is to enable a small Flight Operations Team (FOT) to remain aware and responsive to the increased amount of ground system information in a multi-spacecraft environment. Rather than abandon the tried and true, these interfaces were developed to run alongside existing ground system software to provide additional support to the FOT. These new user interfaces have been combined in a tool called REACH. REACH-the Real-time Evaluation and Analysis of Consolidated Health-is a software product that uses advanced visualization techniques to make spacecraft anomalies easy to spot, no matter how many spacecraft are in the constellation. REACH reads numerous real-time streams of data from the ground system(s) and displays synthesized information to the FOT such that anomalies are easy to pick out and investigate.

Maks, Lori↗

Heterogeneous data-processing optimization with CLARA’s adaptive workflow orchestrator

The hardware landscape used in HEP and NP is changing from homogeneous multi-core systems towards heterogeneous systems with many different computing units, each with their own characteristics. To achieve maximum performance with data processing, the main challenge is to place the right computing on the right hardware. In this paper, we discuss CLAS12 charge particle tracking workflow orchestration that allows us to utilize both CPU and GPU to improve the performance. The tracking application algorithm was decomposed into micro-services that are deployed on CPU and GPU processing units, where the best features of both are intelligently combined to achieve maximum performance. In this heterogeneous environment, CLARA aims to match the requirements of each micro-service to the strength of a CPU or a GPU architecture. A predefined execution of a micro-service on a CPU or a GPU may not be the most optimal solution due to the streaming data-quantum size and the data-quantum transfer latency between CPU and GPU. So, the CLARA workflow orchestrator is designed to dynamically assign micro-service execution to a CPU or a GPU, based on the online benchmark results analyzed for a period of real-time data-processing.

Gyurjyan, Vardan↗

A Measurement-Based Analysis of the Hydrokinetic Energy in the Gulf Stream

This study uses ocean current measurements made off the East Coast of the United States of America to evaluate the ocean current-based electricity production potential in the Gulf Stream. Drifter data from the World Ocean Circulation Experiment (WOCE) are used to map the energy density from South Florida to North Carolina, showing that the most energy dense areas exist off the east coast of Central Florida. Bottom- and vessel-mounted Acoustic Doppler Current Profiler (ADCP) measurements are used to conduct a more detailed study near 26°N and 27°N, respectively. Furthermore, these results quantify the energy profiles in these regions, suggesting that 54% more energy dense currents are available at 27°N than at 26°N.

16 TIDAL AND WAVE POWER↗

Metering Plan: Intelligent Operational Strategies Through Enhanced Metering Systems

The Sustainability Program at Pacific Northwest National Laboratory (PNNL) has adopted a “triple-bottom-line” approach of environmental stewardship, social responsibility, and economic prosperity to its operations. Metering at PNNL works in support of all three, specifically to measure and inform building energy use and greenhouse gas emissions and minimize water use. The foundation for metering at PNNL is a core goal set, which consists of four objectives: providing accurate data without interruption, analyzing data while it is still new, providing actionable recommendations to operations management, and ensuring PNNL’s compliance with contract metering requirements. These core objectives guide the decisions that we make during annual planning and as we operate throughout the year. This 2016 edition of the Metering Plan conveys the metering practices for and vision of the Sustainability Program. Changes in this plan from the 2015 edition include updated tables and an enhanced discussion on energy tracking systems used at PNNL. This plan also discusses updated benchmarking strategies using PNNL’s graphics and analytics tool, BuildingOS by Lucid Design Group. This plan presents our progress toward the metering goals shared by all federal agencies and highlights our successful completion of metering requirements. Currently, PNNL is fully compliant with the applicable legislative and Executive Order metering requirements. PNNL’s approach to the installation of new meters will be discussed. Perhaps most importantly, this plan details the analysis techniques utilized at PNNL that rely on the endless streams of data newly available as a result of increased meter deployment over the last several years. Previous Metering Plans have documented specific meter connection schemes as PNNL focused on deploying meters in a first step toward managing energy and water use. This plan serves not only to highlight PNNL’s successful completion of agency metering goals, but also can be used as a guideline for meter installation and data analysis.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Jet stream winds - Comparisons of analyses with independent aircraft data over Southwest Asia

Cruise-level wind data from commercial aircraft are obtained, and these data are compared with operational jet stream analyses over southwest Asia, an area of limited conventional data. Results from an ensemble of 11 cases during January 1989 and individual cases during December 1988-March 1989 are presented. The key results are: (1) European Centre for Medium-Range Weather Forecasts (ECMWF), National Meteorological Center, and United Kingdom Meteorological Office analyses of the subtropical jet in southwest Asia are 11 percent, 17 percent, and 17 percent weaker, respectively, than aircraft observations; (2) analyzed poleward shears range up to 1 f (0.00007/s) compared with up to 3 f (0.00021/s) in the aircraft observations where f is the local Coriolis parameters; (3) the ECMWF errors are largest at the base of the jet; (4) the mean ECMWF core location is latitudinally correct but has an rms latitude variance of 1.5 deg; (5) isolated erroneous radiosondes produce unmeteorological structures in the analyzed subtropical jet stream; and (6) the increased utilization of automated aircraft reports is likely to produce a spurious secular increase in the apparent strength of the jets. The magnitude and spatial extent of the errors seen are near limits of current GCM resolution (100 km) but should be resolvable. The results imply that studies of GCM systematic jet stream wind errors in weather and climate forecasts must be interpreted with caution in this region.

Tenenbaum, J.↗

A Comprehensive View of Global Potential for Hydro-generated Electricity

In this study, we assess global hydropower potential using runoff and stream flow data, along with turbine technology performance, cost assumptions, and environmental considerations. The results provide the first comprehensive quantification of global hydropower potential including gross, technical, economic, and exploitable estimates. Total global potential of gross, technical, economic, and exploitable hydropower are estimated to be approximately 128, 39, 32, and 27 petawatt hours per year, respectively. The economic and exploitable potential of hydropower are calculated at less than 9 cents/kWh. We find that hydropower has the potential to supply a significant portion of the world energy needs, although this potential varies substantially by region. Globally, hydropower can potentially supply about 1.5 times the total electricity demand in 2005. Estimated hydropower resources in a number of countries are sufficient to accommodate their demand for electricity in 2005, e.g., Brazil (5.6 times), Russia (4.6 times), and Canada (3.5 times). A sensitivity analysis indicates that hydropower estimates are not highly sensitive to five key parameters: design flow (varying by -2% to +1% at less than 9 cents/kWh), cost and financing options (by -7% to +6%), turbine efficiency (by -10% to +10%), stream flow (by -10% to +10%), and fixed charge rate (by -6% to 5%). This sensitivity analysis emphasizes the reliable role of hydropower for future energy systems, when compared to other renewable energy resources with larger uncertainty in their future potentials.

Zhou, Yuyu↗

Stream discharge and water quality data for East Fork Poplar Creek beginning 2012

This data note describes 15-min discharge and in situ water quality data at two locations along East Fork Poplar Creek in east Tennessee, USA. Data records include temperature, gauge height, water surface elevation above mean sea level, and volumetric discharge. Water quality measurements include temperature, specific conductance, pH, dissolved oxygen (percent saturation and concentration), turbidity, and less extensive fDOM data at one site. The data records begin in 2012 at one site and 2015 at the second site; monitoring at both sites is ongoing (as of 2021). The goal of this data collection is to improve understanding of watershed functions, hydrologic dynamics, and material flux. The data will contribute to site conceptual and numerical models, exposure and risk evaluation, remediation selection and design, and performance monitoring. The data are publicly available and can be accessed via unique url or DOI.

54 ENVIRONMENTAL SCIENCES↗

How well do terrestrial biosphere models simulate coarse-scale runoff in the contiguous United States?

Significant changes in the water cycle are expected under current global environmental change. Robust assessment of present-day water cycle dynamics at continental to global scales is confounded by shortcomings in the observed record. Modeled assessments also yield conflicting results which are linked to differences in model structure and simulation protocol. Here we compare simulated gridded (1 spatial resolution) runoff from six terrestrial biosphere models (TBMs), seven reanalysis products, and one gridded surface station product in the contiguous United States (CONUS) from 2001 to 2005. We evaluate the consistency of these 14 estimates with stream gauge data, both as depleted flow and corrected for net withdrawals (2005 only), at the CONUS and water resource region scale, as well as examining similarity across TBMs and reanalysis products at the grid cell scale. Mean runoff across all simulated products and regions varies widely (range: 71 to 356 mm yr(-1)) relative to observed continental-scale runoff (209 or 280 mm yr(-1) when corrected for net withdrawals). Across all 14 products 8 exhibit Nash-Sutcliffe efficiency values in excess of 0.8 and three are within 10% of the observed value. Region-level mismatch exhibits a weak pattern of overestimation in western and underestimation in eastern regions although two products are systematically biased across all regions and largely scales with water use. Although gridded composite TBM and reanalysis runoff show some regional similarities, individual product values are highly variable. At the coarse scales used here we find that progress in better constraining simulated runoff requires standardized forcing data and the explicit incorporation of human effects (e.g., water withdrawals by source, fire, and land use change). (C) 2015 Elsevier B.V. All rights reserved.

54 ENVIRONMENTAL SCIENCES↗

Rate-capability of the VMM3a front-end in the RD51 Scalable Readout System

The VMM3a is an Application Specific Integrated Circuit (ASIC), specifically developed for the readout of gaseous detectors. Originally developed within the ATLAS New Small Wheel (NSW) upgrade, it has been successfully integrated into the Scalable Readout System (SRS) of the RD51 collaboration. This allows, to use the VMM3a also in small laboratory set-ups and mid-scale experiments, which make use of Micro-Pattern Gaseous Detectors (MPGDs). As part of the integration of the VMM3a into the SRS, the readout and data transfer scheme was optimised to reach a high rate-capability of the entire readout system and profit from the VMM3a's high single-channel rate-capability of 3.6Mhits/s. The optimisation focused mainly on the handling of the data output stream of the VMM3a, but also on the development of a trigger-logic between the front-end cards and the DAQ computer. In this article, two firmware implementations of the non-ATLAS continuous readout mode are presented, as well as the implementation of the trigger-logic. Afterwards, a short overview on X-ray imaging results is presented, to illustrate the high rate-capability from an application point-of-view.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗