Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,207 records · Page 67

Levoglucosan data from five coastal streams impacted by the 2020 CZU Lightning Complex Fires, California, United States

This dataset includes levoglucosan data for five coastal California (United States) streams impacted by the 2020 CZU Lightning Complex Fires which burned from August 16th through September 22nd. Levoglucosan is a highly soluble and biolabile fraction of pyrogenic carbon. The five watersheds (San Lorenzo River, Pescadero Creek, Majors Creek, Laguna Creek, and Scott Creek) were impacted by the fires with watersheds experiencing a range of burn severity and extents. Grab samples were collected from each stream between October 2020 and May 2021, targeting both baseflow and event flow hydrologic conditions. Additional biogeochemistry data (i.e., organic and black carbon concentrations) can be found in a separate data package (https://doi.org/10.4211/hs.421c0226bb38460c8393d67fe0c4f802). This data package consists of one main data folder that contains (1) readme; (2) file-level metadata; (3) data dictionary; (4) field metadata with international generic sample numbers (IGSN); (5) methods codes; and (6) levoglucosan data. All files are .csv or .pdf.

2020 CZU Lightning Complex Fires↗

Monthly Maps of Sea Surface Height in the North Atlantic and Zonal Indices for the Gulf Stream Using TOPEX/Poseidon Altimeter Data

Monthly Maps of sea surface height are constructed for the North Atlantic Ocean using TOPEX/Poseidon altimeter data. Mean sea surface height is reconstructed using a weighted combination of historical, hydrographic data and a synthetic mean obtained by fitting a Gaussian model of the Gulf Stream jet to altimeter data. The resultant mean shows increased resolution over the hydrographic mean, and incorporates recirculation information that is absent in the synthetic mean. Monthly maps, obtained by adding the mean field to altimeter sea surface height residuals, are used to derive a set of zonal indices that describe the annual cycle of meandering as well as position and strength of the Gulf Stream.

GULF STREAM↗

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Burst-Compression And -Expansion For TDMA Communication

Burst-compression and -expansion technique enables interconnection of users transmitting and receiving data at rates asynchronous with respect to clocks within ground terminals of satellite-switched, time-division-multiple-access (TDMA) communication network. Matrix switch aboard satellite routes bursts of data from source users received on uplink antennas to downlink antennas illuminating ground areas containing destination users. TDMA ground terminal compresses streams of data from source users into rapid bursts for transmission and reexpands bursts of received data into slower streams of data for delivery to destination users. Greater flexibility in interconnecting widely dispersed users achieved by use of hopping beams.

Budinger, James M.↗

Walker Branch Watershed: Temperature Response of Organic-Matter Decomposition in Headwater Stream

This data set reports the results of a field study investigating the effect of temperature on organic-matter decomposition in the West Fork of Walker Branch, a headwater stream on the Oak Ridge Reservation in east Tennessee. The goal of this project was to evaluate the effect of temperature on organic-matter decomposition in a stream ecosystem to better understand how carbon dynamics across terrestrial-aquatic carbon interfaces may change with warming. The West Fork is a spring-fed headwater stream and the spring inputs create a natural longitudinal temperature gradient. In winter, the upstream sites near the springs are warmer due to the spring input. In summer, the temperature gradient reverses and the upstream sites are cooler. One experiment examined the decomposition of senesced leaves from 3 tree species (red maple [Acer rubrum], tulip poplar [Liriodendron tulipifera], and white oak [Quercus alba]) along the temperature gradient in winter (December 2011 - March 2012). The second experiment examined the breakdown of cotton strips, a substrate of consistent quality (95% cellulose), along the temperature gradient with approximately month-long incubations over ~2 years (August 2011 to October 2013). This dataset contains six data files in comma separate (*.csv) format.

54 ENVIRONMENTAL SCIENCES↗

Predictive Interfaces for Long-Distance Tele-Operations

We address the development of predictive tele-operator interfaces for humanoid robots with respect to two basic challenges. Firstly, we address automating the transition from fully tele-operated systems towards degrees of autonomy. Secondly, we develop compensation for the time-delay that exists when sending telemetry data from a remote operation point to robots located at low earth orbit and beyond. Humanoid robots have a great advantage over other robotic platforms for use in space-based construction and maintenance because they can use the same tools as astronauts do. The major disadvantage is that they are difficult to control due to the large number of degrees of freedom, which makes it difficult to synthesize autonomous behaviors using conventional means. We are working with the NASA Johnson Space Center's Robonaut which is an anthropomorphic robot with fully articulated hands, arms, and neck. We have trained hidden Markov models that make use of the command data, sensory streams, and other relevant data sources to predict a tele-operator's intent. This allows us to achieve subgoal level commanding without the use of predefined command dictionaries, and to create sub-goal autonomy via sequence generation from generative models. Our method works as a means to incrementally transition from manual tele-operation to semi-autonomous, supervised operation. The multi-agent laboratory experiments conducted by Ambrose et. al. have shown that it is feasible to directly tele-operate multiple Robonauts with humans to perform complex tasks such as truss assembly. However, once a time-delay is introduced into the system, the rate of tele\ioperation slows down to mimic a bump and wait type of activity. We would like to maintain the same interface to the operator despite time-delays. To this end, we are developing an interface which will allow for us to predict the intentions of the operator while interacting with a 3D virtual representation of the expected state of the robot. The predictive interface anticipates the intention of the operator, and then uses this prediction to initiate appropriate sub-goal autonomy tasks.

Wheeler, Kevin R.↗

Integrated inertial stellar attitude sensor

An integrated inertial stellar attitude sensor for an aerospace vehicle includes a star camera system, a gyroscope system, a controller system for synchronously integrating an output of said star camera system and an output of said gyroscope system into a stream of data, and a flight computer responsive to said stream of data for determining from the star camera system output and the gyroscope system output the attitude of the aerospace vehicle.

Brady, Tye M.↗

Closing the Gap between FAIR Data Repositories and Hierarchical Data Formats

Many in the scientific community, particularly in publicly funded research, are pushing to adhere to more accessible data standards to maximize the findability, accessibility, interoperability, and reusability (FAIR) of scientific data, especially with the growing prevalence of machine learning augmented research. Online FAIR data repositories, such as the Open Science Framework (OSF), help facilitate the adoption of these standards by providing frameworks for storage, access, search, APIs, and other features that create organized hubs of scientific data. However, the wider acceptance of such repositories is hindered by the lack of support of hierarchical data formats, such as Technical Data Management Streaming (TDMS) and Hierarchical Data Format 5 (HDF5), that many researchers rely on to organize their datasets. Various tools and strategies should be used to allow hierarchical data formats, FAIR data repositories, and scientific organizations to work more seamlessly together. A pilot project at Los Alamos National Laboratory (LANL) addresses the disconnect between them by integrating the OSF FAIR data repository with hierarchical data renderers, extending support for additional file types in their framework. The multifaceted interactive renderer displays a tree of metadata alongside a table and plot of the data channels in the file. This allows users to quickly and efficiently load large and complex data files directly in the OSF webapp. Users who are browsing files can quickly and intuitively see the files in the way they or their colleagues structured the hierarchical form and immediately grasp their contents. This solution helps bridge the gap between hierarchical data storage techniques and FAIR data repositories, making both of them more viable options for scientific institutions like LANL which have been put off by the lack of integration between them.

97 MATHEMATICS AND COMPUTING↗

Fast Vector-Quantizing Data Compressor

Proposed encoding digital electronic data-processing system compresses 7-Mb/s input stream of data into 1-Mb/s output stream. Intended to reduce flow of data from orbiting synthetic-aperture-radar (SAR) system to rate low enough for transmission to ground. Features include reliability, light weight, and low power. Proves advantageous in such terrestrial applications as compression of speech and of video signals.

Chang, Chi-Yung↗

Seismology in the Cloud: A New Streaming Workflow

Data-intensive research in seismology is experiencing a recent boom, driven in part by large volumes of available data and advances in the growing field of data science. However, there are significant barriers to processing large data volumes, such as long retrieval times from data repositories, complex data management, and limited computational resources. New tools and platforms have reduced the barriers to entry for scientific cluster computing, including the maturation of the commercial cloud as an accessible instrument for research. Here, we build a customized research cluster in the cloud to test a new workflow for large-scale seismic analysis, in which data are processed as a stream (retrieved on-the-fly and acted upon without storing), with data from the Incorporated Research Institutions for Seismology Data Management Center. We use this workflow to deploy a spectral peak detection algorithm over 5.6 TB of compressed continuous seismic data from 2074 stations of the USArray Transportable Array EarthScope network. Using a 50-node cluster in the cloud, we completed the noise survey in 80 hr, with an average data throughput of 1.7 GB per minute. By varying cluster sizes, we find the scaling of our analysis to be sublinear, due to a combination of algorithmic limitations and data center response times. The cloud-based streaming workflow represents an order-of-magnitude increase in acquisition and processing speed compared to a traditional download-store-process workflow, and offers the additional benefits of employing a flexible, accessible, and widely used computing architecture. It is limited, however, due to its reliance on Internet transfer speeds and data center service capacity, and may not work well for repeated analyses or those for which even higher data throughputs are needed. These research applications will require a new class of cloud-native approaches in which both data and analysis are in the cloud.

58 GEOSCIENCES↗

Walker Branch Watershed: Weekly Stream Water Chemistry

This data set reports weekly stream water chemistry in the West and East Forks of Walker Branch Watershed (WBW). The long-term Walker Branch stream chemistry monitoring is intended to provide data on watershed output of chemicals via streamflow and long-term changes in stream chemical composition. This data set contains two data files of stream water chemistry collected at weekly time intervals from 1989 through 2013. Weekly sampling of stream water started in 1989 in the West Fork and in 1995 in the East Fork. Measured parameters include water level at the weir and stream discharge, water temperature, specific conductivity, pH, alkalinity, and dissolved organic carbon (DOC), soluble reactive phosphorus (SRP), total dissolved phosphorus (TDP), ammonium-N, nitrate+nitrite-N, total dissolved nitrogen (TDN), anion (chloride and sulfate), and cation and trace metal (calcium, magnesium, sodium, potassium, iron, manganese, silicon, aluminum, barium, cadmium, nickel, lead, strontium, zinc, copper, molybdenum) concentrations. The Walker Branch Watershed is located on the U.S. Department of Energy's Oak Ridge Reservation in eastern Tennessee.

54 ENVIRONMENTAL SCIENCES↗

Preliminary estimates of Gulf Stream characteristics from TOPEX data and a precise gravimetric geoid

TOPEX sea surface height data has been used, with a gravimetric geoid, to calculate sea surface topography across the Gulf Stream. This topography was initially computed for nine tracks on cycles 21 to 29. Due to inaccurate geoid undulations on one track, results for eight tracks are reported. The sea surface topography estimates were used to calculate parameters that describe Gulf Stream characteristics from two models of the Gulf Stream. One model was based on a Gaussian representation of the velocity while the other was a hyperbolic representation of velocity or the sea surface topography. The parameters of the Gaussian velocity model fit were a width parameter, a maximum velocity value, and the location of the maximum velocity. The parameters of the hyperbolic sea surface topography model were the width, the height jump, position, and sea surface topography at the center of the stream. Both models were used for the eight tracks and nine cycles studied. Comparisons were made between the width parameters, the maximum velocities, and the height jumps. Some of the parameter estimates were found to be highly (0.9) correlated when the hyperbolic sea surface topography fit was carried out, but such correlations were reduced for either the Gaussian velocity fits or the hyperbolic velocity model fit. A comparison of the parameters derived from 1-year TOPEX data showed good agreement with values derived by Kelly (1991) using 2.5 years of Geosat data near 38 deg N, 66 deg W longitude. Accuracy of the geoid undulations used in the calculations was of order of +/- 16 cm with the accuracy of a geoid undulation difference equal to +/- 15 cm over a 100-km line in areas with good terrestrial data coverage. This paper demonstrates that our knowledge or geoid undulations and undulation differences, in a portion of the Gulf Stream region, is sufficiently accurate to determine characteristics of the jet when used with TOPEX altimeter data. The method used here has not been shown to be more accurate than methods that average altimeter data to form a reference surface used in analysis to obtain the Gulf Stream characteristics. However, the results show the geoid approach may be used in areas where lack of current meandering reduces the accuracy of the average surface procedure.

Rapp, Richard H.↗

Spatial Study 2022: Surface Water Samples, Cotton Strip Degradation, and Hydrologic Sensor Data across the Yakima River Basin, Washington, USA (v3)

This dataset supports a broader study examining the drivers of spatial variability in sediment respiration rates in the Yakima River Basin. The dataset provides data and photos generated from sample collection during the same one-week period at 48 sites within multiple rivers throughout the Yakima River Basin in Washington, USA. The contents include surface water geochemistry data; river substrate grain size photos; stream depth data; manual chamber open channel respiration data; and field metadata (including qualitative information on instream and river corridor characteristics). Grain size photos can be used to improve estimates of channel substrate D50 data. The dataset also includes tensile strength and photos from cotton strip field degradation experiments; five-week sensor time series temperature, dissolved oxygen, pressure, pH, specific conductance, chlorophyll A, and turbidity data; plots of the sensor data; and R scripts used to generate the plots. Samples collected during this study were labeled as “Second Spatial Study” or “SSS.” A subset of data from the SSS samples were published in the contiguous United States (CONUS)-Scale Model-Sample (CM) study data package available at https://data.ess-dive.lbl.gov/view/doi:10.15485/1923689 that presents data from across the CONUS. SSS data published in the CM data package were not included in this data package. They include dissolved organic carbon (DOC, measured as non-purgeable organic carbon, NPOC), total nitrogen (TN), grain size, aerobic sediment respiration, dissolved oxygen (DO), and temperature. Parent IDs and Site IDs are consistent between the SSS and CM data packages, and they can be mapped directly so data across packages can be used together. Additionally, sensor data from a similar 2021 spatial study can be found at https://data.ess-dive.lbl.gov/view/doi:10.15485/1892052 and 2021 sample data can be found at https://data.ess-dive.lbl.gov/view/doi:10.15485/1898914. The 2021 spatial study had some sites in common with this 2022 spatial study. This dataset is comprised of three photo folders and one main data folder with six subfolders. The photo folders contain photographs and videos of cotton strip retrieval and sediment quadrats. The main data folder consists of (1) file-level metadata; (2) data dictionary; (3) field metadata; (4) total suspended solids (TSS) data and cotton strip tensile strength data and averages; (5) field protocol; (6) readme; (7) methods codes; (8) international generic sample number (IGSN) mapping file; (9) sensor installation methods summary; (10) stream depth and averages; and (11) Ultrameter data and averages. The Sonar subfolder consists of Sonar time-series depth data and a processing script. The BarotrollAtm, DepthHOBO, MantaRiver, miniDOT, and miniDOTManualChamber subfolders contain time-series data, plots, and summary files. All files are .csv, .pdf, .txt, .R, .Rmd, .jpg, .jpeg, .AVI, .mp4, or .mov. The data package was originally published in April 2023. It was updated in August 2023 (v2; modified files) and September 2024 (v3; modified files). See the change history section in the readme for details. We acknowledge the Yakama Nation as owners and caretakers of the lands where we collected these data. We thank the Confederated Tribes and Bands of the Yakama Nation Tribal Council and Yakama Nation Fisheries for working with us to facilitate sample collection and optimization of data usage according to their values and worldview.

54 ENVIRONMENTAL SCIENCES↗

The Sagittarius stream in Gaia Early Data Release 3 and the origin of the bifurcations

The Sagittarius dwarf spheroidal (Sgr) is a dissolving galaxy being tidally disrupted by the Milky Way (MW). Its stellar stream still poses serious modelling challenges, which hinders our ability to use it effectively as a prospective probe of the MW gravitational potential at large radii. Our goal is to construct the largest and most stringent sample of stars in the stream with which we can advance our understanding of the Sgr-MW interaction, focusing on the characterization of the bifurcations. We improved on previous methods based on the use of the wavelet transform to systematically search for the kinematic signature of the Sgr stream throughout the whole sky in the Gaia data. We then refined our selection via the use of a clustering algorithm on the statistical properties of the colour-magnitude diagrams. Our final sample contains more than 700 000 candidate stars and is three times larger than previous Gaia samples. With it, we have been able to detect the bifurcation of the stream in both the northern and southern hemispheres, which requires four branches (two bright and two faint) to fully describe the system. We present the detailed proper motion distribution of the trailing arm as a function of the angular coordinate along the stream, showing, for the first time, the presence of a sharp edge (on the side of the small proper motions) beyond which there are no Sgr stars. We also characterize the correlation between kinematics and distance. Finally, the chemical analysis of our sample shows that the faint branch of the bifurcation is more metal poor than the bright. We provide analytical descriptions for the proper motion trends as well as for the sky distribution of the four branches of the stream. Based on our analysis, we interpret the bifurcations as a misaligned overlap of the material stripped at the antepenultimate pericentre (faint branches) with the stars ejected at the penultimate pericentre (bright branch), given that Sgr just went through its perigalacticon. The source of this misalignment is still unknown, but we argue that models with some internal rotation in the progenitor – at least during the time of stripping of the stars that are now in the faint branches – are worth exploring.

79 ASTRONOMY AND ASTROPHYSICS↗

Active workflow system for near real-time extreme-scale science. In: PPAA '14 Proceedings of the first workshop on Parallel programming for analytics applications

In recent years, streaming-based data processing has been gaining substantial traction for dealing with overwhelming data generated by real-time applications, from both enterprise sources and scientific computing. In this work, however, we look at an emerging class of scientific data with Near Real-Time (NRT) requirement, in which data is typically generated in a bursty fashion with the near real-time constraints being applied primarily between bursts, rather than within a stream. A key challenge for this types of data sources is that the processing time per data element is not uniform, and not always feasible to predict. Given the observations on the increasing unpredictability of compute load and system dynamics, this work looks to adapt streaming-based approach to the context of this new class of large experiments and simulations that have complex run-time control and analysis issues. In particular, we deploy a novel two-tier scheme for handling the increasing unpredictability of runtime behaviors: Instead of relying on determining what and where to run the scientific workflows beforehand or partial dynamically, the decision will also be adaptively enhanced online according to system runtime status. This is enabled by embedding workflow along with data streams. Specifically, we break data outputs generated from experiments or simulations into multiple self-describing "chunks", which we call active data objects. As such, if there is a transient hotspot observed, a data object with unfinished workflow pipeline can break its previous schedule and search for a least loaded location to continue the execution. Our preliminary experiment results based on synthetic workloads demonstrate the proposed active workflow system as a very promising solution by outperforming the state-of-the-art semi-dynamic workflow schedulers with an improved workflow completion time, as well as a good scalability.

Zhang, Yanwei↗

Jet stream winds - Enhanced aircraft data acquisition and analysis over Southwest Asia

A project is described for providing the accurate initial and verification analyses for the jet stream in regions where general circulation models are known to have large systematic errors, either due to the extreme sparsity of data or to incorrect physical parameterizations. For this purpose, finely spaced aircraft-based meteorological data for the Southwest Asian regions collected for three 10-day periods in the winter of 1988-1989 will be used, together with corresponding data for the North American regions used as a control, to rerun the assimilation cycles and forecast models of the NMC and the NASA Goddard Laboratory for Atmospheres. Data for Southeast Asia will be collected by three carriers with extensive wide-body routes crossing the total region, while data for the North American region will be obtained from the archives of ACARS and GTS.

Tenenbaum, J.↗

DynPaC: Coarse-Grained, Dynamic, and Partially Reconfigurable Array for Streaming Applications

Coarse-grained reconfigurable arrays (CGRAs) provide higher flexibility than application-specific integrated circuits (ASICs) and higher efficiency than fine-grained reconfigurable devices such as Field Programmable Gate Arrays (FPGAs). However, CGRAs are generally designed to support offloading of a single kernel. While their design, based on communicating functional units, appears to naturally suit streaming applications composed of multiple cooperating kernels, current approaches only statically partition the resources across kernels. However, streaming applications often are data-dependent, leading to variable kernel execution times depending on the input data and impacting the throughput of the entire pipeline if resources are statically allocated. Therefore, in this paper, we discuss the design of DynPaC — a coarse-grained, dynamically, and partially reconfigurable array for data-dependent streaming applications. We discuss the required software and hardware components to manage partial dynamic reconfiguration. We demonstrate that by supporting partial dynamic reconfiguration, we can obtain an average speedup of 1.44X for a representative set of applications w.r.t. static partitioning, with a limited area overhead (6.4% of the entire chip).

Tan, Cheng↗

Continuous Spacecraft Communications via Make-Before-Break Antenna Array Beam Steering

Active phased array (APA) antennas can electronically form multiple beams to track several targets simultaneously. Spacecraft equipped with these antennas can achieve continuous communications through a constellation of relay satellites by forming a second beam to an upcoming relay satellite before the spacecraft moves beyond the current relay’s line of sight. These make-before-break operations have utility for streaming critical data without gaps. Using characterization data from a commercially-available APA-based satellite communications terminal, we simulate performance from low-Earth orbit to a representative geosynchronous relay satellite system at Ka-band. Average data rates of 3 Mbps are achievable with a minimum rate of 0.8 Mbps during worst-case handovers. We prototype the beam-splitting algorithm in the terminal hardware and evaluate its performance in an antenna range. In an over-the-air test emulating a handover between two relays we observe error-free data in the combined telemetry stream.

active phased array antennas↗