Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data streams”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,351 records · Page 75

Ellicott City Disasters II: Enhancing a Statistical Flood Risk Model to Continue Improving Early Warning Systems and Public Safety in Ellicott City, Maryland

As flooding events in the United States grow in frequency and intensity, the use of technological advancements and applied science are increasingly necessary for effective flood monitoring and warning systems. The NASA DEVELOP Ellicott City Disasters II project investigated the use of machine learning for applications in flood risk detection to support the improvement of early warning systems. To strengthen the efforts of the Howard County Office of Emergency Management (OEM) in building a more robust flood monitoring system, the project improved the original statistical flood risk model, FLuME (Flood Learning Model Environment), programmed by the first DEVELOP term The enhancements incorporated an additional six years of precipitation and soil moisture data from the North American Land Data Assimilation System (NLDAS), modeled using Aqua Advanced Microwave Scanning Radiometer for EOS and Tropical Rainfall Measuring Mission TRMM Microwave Imager. These Earth observations were supplemented by stream gauge data from the OEM and the US Geological Survey. The resultant flood risk model FLASH (Flood Learning Environment and Severity Assessment Hub) was trained to evaluate input variables and predict stage height in Ellicott City in real time. The addition of an advanced deep learning framework known as long short-term memory improved the model’s ability to capture relationships between variables. To assess the effectiveness of the new model, FLASH produced a model efficiency metric of 0.99, a significant improvement over the 0.85 value produced by the previous model. The project assisted the OEM in pursuing the integration of open data and NASA Earth observations into a threat matrix capable of informing near real-time decision making.

Disasters↗

Deep near-infrared survey of the Southern Sky (DENIS)

DENIS (Deep Near-Infrared Survey of the Southern Sky) will be the first complete census of astronomical sources in the near-infrared spectral range. The challenges of this novel survey are both scientific and technical. Phenomena radiating in the near-infrared range from brown dwarfs to galaxies in the early stages of cosmological evolution, the scientific exploitation of data relevant over such a wide range requires pooling expertise from several of the leading European astronomical centers. The technical challenges of a project which will provide an order of magnitude more sources than given by the IRAS space mission, and which will involve advanced data-handling and image-processing techniques, likewise require pooling of hardware and software resources, as well as of human expertise. The DENIS project team is composed of some 40 scientists, computer specialists, and engineers located in 5 European Community countries (France, Germany, Italy, The Netherlands, and Spain), with important contributions from specialists in Australia, Brazil, Chile, and Hungary. DENIS will survey the entire southern sky in 3 colors, namely in the I band at a wavelength of 0.8 micron, in the 1.25 micron J band, and in the 2.15 micron K' band. The sensitivity limits will be 18th magnitude in the I band, 16th in the J band, and 14.5th in the K' band. The angular resolution achieved will be 1 arcsecond in the I band, and 3.0 arcseconds in the J and K' bands. The European Southern Observatory 1 m telescope on La Silla will be dedicated to survey use during operations expected to last four years, commencing in late 1993. DENIS aims to provide the astronomical community with complete digitized infrared images of the full southern sky and a catalogue of extracted objects, both of the best quality and in readily accessible form. This will be achieved through dedicated software packages and specialized catalogues, and with assistance from the Leiden and Paris Data Analysis Centers. The data will be mailed on DAT tapes from La Silla to the two Data Analysis Centers for further processing. Two centers are necessary because of the shear quantity of data and because of the complementary roles the Centers will develop, each exploiting its own particular expertise. The Leiden Data Analysis Center (LDAC) will extract objects, establish their parameters, and archive them into a source catalogue. The LDAC will collaborate with the Groningen Space Research group that has gained experience in infrared image handling from the IRAS satellite. The Paris Data Analysis Center (PDAC) will be responsible for archiving and preprocessing the raw data to provide a homogeneous set of data suitable for further reduction in both the Leiden and Paris data analysis streams. The PDAC will also extract and archive images for the sources flagged by the LDAC as extended, and create a catalogue of galaxies. In exploiting the DENIS data we foresee the collaboration with other data analysis centers, such as the Observatoire de Lyon where the relevant DENIS catalogue of galaxies can be incorporated into their extragalactic database. The Point Sources and the Small Extended Sources catalogues could be incorporated in the Late Type Star database at Montpellier, and in the SIMBAD database as CDS. At Groningen the IRAS Point Source catalogue and/or image data can be merged with the DENIS catalogues. At Meudon algorithms and software will be developed with main goal assessing the limits reachable for the homogeneity and intrinsic consistency between the ensemble of the images in the data base (flat-fielding, relative positioning of the fields, bootstrapped flux calibration) but also for the data analysis.

Deul, E.↗

An empirical determination of the polytropic index for the free-streaming solar wind using Helios 1 data

Observations of solar wind proton temperatures indicate that the solar wind is heated as it moves outward toward the orbit of Earth. This heating, which may be the results of electron heat conduction and perhaps MHD waves, has proven difficult to quantify and hence is often neglected in MHD models of the solar wind. An alternate approach to finding explicit heating terms for the MHD energy equation is to use a polytropic approximation. This paper discusses the properties of the polytropic approximation and its application to the solar wind plasma. By using data from the Helios 1 spacecraft, an empirical value for the polytropic index of the free-streaming solar wind is determined. Various corrections to the data are made to account for velocity, nonuniformity in radial sampling, and stream interaction regions. The polytropic index, as derived from proton data, is found to indepedent of speed state, within statistical error, and has an average value of 1.46. If magnetic pressure is included, the polytropic index has an average value of 1.58.

Totten, T. L.↗

The Practical Obstacles of Data Transfer: Why researchers still love scp

The importance of computing facilities is heralded every six months with the announcement of the new Top500 list, showcasing the world s fastest supercomputers. Unfortu- nately, with great computing capability does not come great long-term data storage capacity, which often means users must move their data to their local site archive, to remote sites where they may be doing future computation or anal- ysis, or back to their home institution, else face the dreaded data purge that most HPC centers employ to keep utiliza- tion of large parallel filesystems low to manage performance and capacity. At HPC centers, data transfer is crucial to the scientific workflow and will increase in importance as computing systems grow in size. The Energy Sciences Net- work (ESnet) recently launched its fifth generation network, a 100 Gbps high-performance, unclassified national network connecting more than 40 DOE research sites to support scientific research and collaboration. Despite the tenfold increase in bandwidth to DOE research sites amenable to multiple data transfer streams and high throughput, in prac- tice, researchers often under-utilize the network and resort to painfully-slow single stream transfer methods such as scp to avoid the complexity of using multiple stream tools such as GridFTP and bbcp, and contend with frustration from the lack of consistency of available tools between sites. In this study we survey and assess the data transfer methods pro- vided at several DOE supported computing facilities, includ- ing both leadership-computing facilities, connected through ESnet. We present observed transfer rates, suggested opti- mizations, and discuss the obstacles the tools must overcome to receive wide-spread adoption over scp.

Nam, Hai Ah↗

STREAM: A Scalable Federated HPC Telemetry Platform

Obtaining and analyzing high performance computing (HPC) telemetry in real time is a complex task that can impact algo- rithmic performance, operating costs, and ultimately scientific outcomes. If your organization operates multiple HPC systems, filesystems, and clusters, telemetry streams can be synthesized in order to ease operational and analytics burden. In order to collect this telemetry, the Oak Ridge Leadership Computing Facility (OLCF) has deployed STREAM (Streaming Telemetry for Resource Events, Analytics, and Monitoring), which is a distributed and high-performance message bus based on Apache Kafka. STREAM collects center-wide performance information and must interface with many sources, including five HPE deployed supercomputers, each with their own Kafka cluster which is managed by HPCM. OLCF Supercomputers and their attached scratch filesystems currently send more than 300 million messages to over 200 topics producing around 1.3 Terabytes per day of telemetry data to STREAM. This paper describes the architectural principles that enable STREAM to be both resilient and highly performant while supporting multiple upstream Kafka clusters and other data sources. It also discusses the design challenges and decisions faced in adapting our existing system- monitoring infrastructure to support the first Exascale computing platform.

Adamson, Ryan↗

One SQL to Rule Them All - an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables

Real-time data analysis and management are increasingly critical for today's businesses. SQL is the de facto lingua franca for these endeavors, yet support for robust streaming analysis and management with SQL remains limited. Many approaches restrict semantics to a reduced subset of features and/or require a suite of non-standard constructs. Additionally, use of event timestamps to provide native support for analyzing events according to when they actually occurred is not pervasive, and often comes with important limitations. We present a three-part proposal for integrating robust streaming into SQL, namely: (1) time-varying relations as a foundation for classical tables as well as streaming data, (2) event time semantics, (3) a limited set of optional keyword extensions to control the materialization of time-varying query results. We show how with these minimal additions it is possible to utilize the complete suite of standard SQL semantics to perform robust stream processing. We motivate and illustrate these concepts using examples and describe lessons learned from implementations in Apache Calcite, Apache Flink, and Apache Beam. We conclude with syntax and semantics of a concrete proposal for extensions of the SQL standard and note further areas of exploration.

Begoli, Edmon↗

Exploiting analytics techniques in CMS computing monitoring

The CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been done, but are of crucial importance for a better understanding of how CMS did successful operations, and to reach an adequate and adaptive modelling of the CMS operations, in order to allow detailed optimizations and eventually a prediction of system behaviours. These data are now streamed into the CERN Hadoop data cluster for further analysis. Specific sets of information (e.g. data on how many replicas of datasets CMS wrote on disks at WLCG Tiers, data on which datasets were primarily requested for analysis, etc) were collected on Hadoop and processed with MapReduce applications profiting of the parallelization on the Hadoop cluster. We present the implementation of new monitoring applications on Hadoop, and discuss the new possibilities in CMS computing monitoring introduced with the ability to quickly process big data sets from mulltiple sources, looking forward to a predictive modeling of the system.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Coding for Efficient Image Transmission

NASA publication second in series on data-coding techniques for noiseless channels. Techniques used even in noisy channels, provided data further processed with Reed-Solomon or other error-correcting code. Techniques discussed in context of transmission of monochrome imagery from Voyager II spacecraft but applicable to other streams of data. Objective of this type coding to "compress" data; that is, to transmit using as few bits as possible by omitting as much as possible of portion of information repeated in subsequent samples (or picture elements).

Rice, R. F.↗

Performance and usability enhancements for continuous subgraph matching queries on graph-structured data

A query graph, which includes vertices and edges, represents a query on graph-structured data. The query graph is decomposed into query subgraphs. A network analysis tool performs continuous subgraph matching queries to facilitate analysis of computer network traffic, social media events, or other streams of data represented as a dynamic data graph (graph-structured data). This can help identify emerging trends in the data. Some features of the network analysis tool enhance performance by effectively utilizing distributed computing resources (including processing cores and memory at different nodes of a cluster) to speed up the process of updating the dynamic data graph and detecting matches of query subgraphs. Features of a query graph building tool enhance usability by providing intuitive ways to specify query graphs and their subgraphs. Features of a results visualization tool enhance usability by providing an intuitive way to present the results of continuous subgraph matching queries.

Choudhury, Sutanay↗

via machinae : Searching for stellar streams using unsupervised machine learning

ABSTRACT We develop a new machine learning algorithm, via machinae, to identify cold stellar streams in data from the Gaia telescope. via machinae is based on ANODE, a general method that uses conditional density estimation and sideband interpolation to detect local overdensities in the data in a model agnostic way. By applying ANODE to the positions, proper motions, and photometry of stars observed by Gaia, via machinae obtains a collection of those stars deemed most likely to belong to a stellar stream. We further apply an automated line-finding method based on the Hough transform to search for line-like features in patches of the sky. In this paper, we describe the via machinae algorithm in detail and demonstrate our approach on the prominent stream GD-1. Though some parts of the algorithm are tuned to increase sensitivity to cold streams, the via machinae technique itself does not rely on astrophysical assumptions, such as the potential of the Milky Way or stellar isochrones. This flexibility suggests that it may have further applications in identifying other anomalous structures within the Gaia data set, for example debris flow and globular clusters.

79 ASTRONOMY AND ASTROPHYSICS↗

Infrared Imaging System for Studying Brain Function

A proposed special-purpose infrared imaging system would be a compact, portable, less-expensive alternative to functional magnetic resonance imaging (fMRI) systems heretofore used to study brain function. Whereas a typical fMRI system fills a large room, and must be magnetically isolated, this system would fit into a bicycle helmet. The system would include an assembly that would be mounted inside the padding in a modified bicycle helmet or other suitable headgear. The assembly would include newly designed infrared photodetectors and data-acquisition circuits on integrated-circuit chips on low-thermal-conductivity supports in evacuated housings (see figure) arranged in multiple rows and columns that would define image coordinates. Each housing would be spring-loaded against the wearer s head. The chips would be cooled by a small Stirling Engine mounted contiguous to, but thermally isolated from, the portions of the assembly in thermal contact with the wearer s head. Flexible wires or cables for transmitting data from the aforementioned chips would be routed to an integrated, multichannel transmitter and thence through the top of the assembly to a patch antenna on the outside of the helmet. The multiple streams of data from the infrared-detector chips would be sent to a remote site, where they would be processed, by software, into a three-dimensional display of evoked potentials that would represent firing neuronal bundles and thereby indicate locations of neuronal activity associated with mental or physical activity. The 3D images will be analogous to current fMRI images. The data would also be made available, in real-time, for comparison with data in local or internationally accessible relational databases that already exist in universities and research centers. Hence, this system could be used in research on, and for the diagnosis of response from the wearer s brain to physiological, psychological, and environmental changes in real time. The images would also be stored in a relational database for comparison with corresponding responses previously observed in other subjects.

Mintz, Frederick↗

LITE data processing

The Lidar In-space Technology Experiment (LITE) was a payload onboard the Space Shuttle Discovery as part of the September 1994 STS-64 mission. LITE successfully made both atmospheric and surface measurements at the 355 nm, 532 nm, and 1064 nm wavelengths [Winker et al., 1996]. At mission completion, nearly 45 Gbytes of data had been obtained. As the data were collected they were organized into three data sets. The largest of these data sets contained the single-shot full resolution digitized lidar signals for all three wavelengths. The other two data sets contained the instrument status data block (ISDB) and the quick look science data (QLSD). ISDB information was generated once per second and contained timing parameters and engineering data related to the health, status, and configuration of the instrument. The QLSD are 100shot averages of the 355 nm and 532 nm lidar signals and were generated at ten second intervals. As the LITE data were generated, they were output to low-rate and high-rate telemetry streams. The low-rate data contained the ISDB and QLSD. The lowrate data were transmitted to the ground by the Shuttle S-band system through the Tracking and Data Relay Satellite System (TDRSS). The high-rate data contained the full resolution digitized lidar signals for all three wavelengths, the ISDB, and QLSD. The high-rate data were transmitted in real time by the Ku-band system through TDRSS downlink to the LITE operations center at the Johnson Space Center (JSC). High-rate data were only obtained when the orbiter and TDRSS telemetry link coincided with the real-time operations of LITE. The low-rate data were backed up by a Shuttle recorder during periods when the telemetry link was unavailable, and the entire lowrate data stream was transferred to the ground over the course of the mission. A total of 53 hours of lowrate data and 45.5 hours of high-rate data were acquired. All of the LITE raw telemetry data have been archived at the NASA Langley Research Center (LaRC), and the LITE high-rate data have been processed to the level 1 data product. This paper describes the data processing steps required to convert the LITE raw, high-rate telemetry data to the LITE level 1 data product.

Powell, Kathleen A.↗

A machine learning decision criterion for reducing scan time for hyperspectral neutron computed tomography systems

We present the first machine learning-based autonomous hyperspectral neutron computed tomography experiment performed at the Spallation Neutron Source. Hyperspectral neutron computed tomography allows the characterization of samples by enabling the reconstruction of crystallographic information and elemental/isotopic composition of objects relevant to materials science. High quality reconstructions using traditional algorithms such as the filtered back projection require a high signal-to-noise ratio across a wide wavelength range combined with a large number of projections. This results in scan times of several days to acquire hundreds of hyperspectral projections, during which end users have minimal feedback. To address these challenges, a golden ratio scanning protocol combined with model-based image reconstruction algorithms have been proposed. This novel approach enables high quality real-time reconstructions from streaming experimental data, thus providing feedback to users, while requiring fewer yet a fixed number of projections compared to the filtered back projection method. In this paper, we propose a novel machine learning criterion that can terminate a streaming neutron tomography scan once sufficient information is obtained based on the current set of measurements. Our decision criterion uses a quality score which combines a reference-free image quality metric computed using a pre-trained deep neural network with a metric that measures differences between consecutive reconstructions. The results show that our method can reduce the measurement time by approximately a factor of five compared to a baseline method based on filtered back projection for the samples we studied while automatically terminating the scans.

97 MATHEMATICS AND COMPUTING↗

THE M31 VELOCITY VECTOR. I. HUBBLE SPACE TELESCOPE PROPER-MOTION MEASUREMENTS

We present the first proper-motion (PM) measurements for the galaxy M31. We obtained new V-band imaging data with the Hubble Space Telescope ACS/WFC and the WFC3/UVIS instruments of three fields: a spheroid field near the minor axis, an outer disk field along the major axis, and a field on the Giant Southern Stream. The data provide five to seven year time baselines with respect to pre-existing deep first-epoch observations of the same fields. We measure the positions of thousands of M31 stars and hundreds of compact background galaxies in each field. High accuracy and robustness is achieved by building and fitting a unique template for each individual object. The average PM for each field is obtained from the average motion of the M31 stars between the epochs with respect to the background galaxies. For the three fields, the observed PMs ({mu}{sub W}, {mu}{sub N}) are, in units of mas yr{sup -1}, (- 0.0458, -0.0376) {+-} (0.0165, 0.0154), (- 0.0533, -0.0104) {+-} (0.0246, 0.0244), and (- 0.0179, -0.0357) {+-} (0.0278, 0.0272), respectively. The ability to average over large numbers of objects and over the three fields yields a final displacement accuracy of a few thousandths of a pixel, corresponding to only 12 {mu}as yr{sup -1}. This is comparable to what has been achieved for other Local Group galaxies using Very Long Baseline Array observations of water masers. Potential systematic errors are controlled by an analysis strategy that corrects for detector charge transfer inefficiency, spatially and time-dependent geometric distortion, and point-spread function variations. The robustness of the PM measurements and uncertainties are supported by the fact that data from different instruments, taken at different times and with different telescope orientations, as well as measurements of different fields, all yield statistically consistent results. Papers II and III of this series explore the implications of the new measurements for our understanding of the history, future, and mass of the Local Group.

79 ASTRONOMY AND ASTROPHYSICS↗

Quantifying the Role of Snowmelt in Stream Discharge in an Alaskan Watershed: An Analysis Using a Spatially Distributed Surface Hydrology Model

Here, this study uses a spatially distributed surface hydrology model to investigate the role of snowmelt in stream discharge for the Tanana Flats Basin in interior Alaska. The Parameter ESTimation code is used to calibrate the model with observed stream discharge data. The model was further evaluated using remote sensing-based snow cover product and in situ snowpack water equivalent (SWE) observations. A 36 year (1980–2015) U.S. Geological Survey Precipitation-Runoff Modeling System simulation shows (1) the monthly stream discharge from the Tanana Flats Basin in April decreased by 44%; (2) snow cover area at high altitudes (above 2000 m) decreased in summer, both SWE and snowmelt also decreased significantly, especially in spring; (3) the timings of snowmelt onset and ending shifted by 2 (earlier) and 5 (later) days per decade, respectively; and (4) snowmelt accounts for 40% of the annual stream discharge. This study provides a quantitative tool to investigating hydrological systems considering the impacts of snow dynamics in cold regions. Additionally, this study suggests that future warming will further decrease snow coverage, advance snow melting time, and hereafter change the stream discharge dynamics in the Arctic.

58 GEOSCIENCES↗

LC-IM-TOF Instrument Control & Data Visualization Software

Liquid Chromatography-Ion Mobility-time of Flight Instrument Control and Data Visualization software is designed to control instrument voltages for the Ion Mobility drift tube. It collects and stores information collected from the Agilent TOF instrument and analyses/displays the ion intensity information acquired. The software interface can be split into 3 categories -- Instrument Settings/Controls, Data Acquisition, and Viewer. The Instrument Settings/Controls prepares the instrument for Data Acquisition. The Viewer contains common objects that are used by Instrument Settings/Controls and Data Acquisition. Intensity information is collected in 1 nanosec bins and separated by TOF pulses called scans. A collection of scans are stored side by side making up an accumulation. In order for the computer to keep up with the stream of data, 30-50 accumulations are commonly summed into a single frame. A collection of frames makes up an experiment. The Viewer software then takes the experiment and presents the data in several possible ways, each frame can be viewed in TOF bins or m/z (mass to charge ratio). The experiment can be viewed frame by frame, merging several frames, or by viewing the peak chromatogram. The user can zoom into the data, export data, and/or animate frames. Additional features include calibration of the data and even post-processing multiplexed data.

Danielson III, WilliamF.↗

Large-scale real-time signal processing in physics experiments: the ALICE TPC FPGA pipeline

For LHC Run 3, the ALICE Time Projection Chamber was upgraded to operate in continuous readout mode. Interaction rates of up to 50 kHz in Pb-Pb collisions require real-time processing of more than 3 TB s -1 of raw detector data. This requirement is met by a custom FPGA-based processing pipeline that performs the complete front-end data treatment fully in-stream, including common-mode correction, pedestal subtraction, ion-tail filtering, zero suppression, and dense data packing. A central element of the design is a highly parallel common-mode correction algorithm operating directly on the streaming data. It robustly identifies signal-free readout channels on a time-bin basis and applies pad-dependent scaling to compensate for local variations in capacitive coupling in the GEM readout. In combination with pedestal subtraction and ion-tail filtering, this enables accurate baseline restoration under extreme high-occupancy conditions, preventing signal loss while efficiently suppressing noise prior to zero suppression. The pipeline operates continuously at the full detector bandwidth and reduces the raw input rate of approximately 3 TB s -1 to about 900 GBps for Pb-Pb collisions at the target interaction rate. Overall, it represents a large-scale FPGA-based real-time signal-processing implementation for high-energy physics detector readout.

Digital signal processing (DSP)↗

Predicting and Detecting Emerging Cyberattack Patterns Using StreamWorks

The number and sophistication of cyberattacks on industries and governments have dramatically grown in recent years. To counter this movement, new advanced tools and techniques are needed to detect cyberattacks in their early stages such that defensive actions may be taken to avert or mitigate potential damage. From a cybersecurity analysis perspective, detecting cyberattacks may be cast as a problem of identifying patterns in computer network traffic. Logically and intuitively, these patterns may take on the form of a directed graph that conveys how an attack or intrusion propagates through the computers of a network. Such cyberattack graphs could provide cybersecurity analysts with powerful conceptual representations that are natural to express and analyze. We have been researching and developing graph-centric approaches and algorithms for dynamic cyberattack detection. The advanced dynamic graph algorithms we are developing will be packaged into a streaming network analysis framework known as StreamWorks. With StreamWorks, a scientist or analyst may detect and identify precursor events and patterns as they emerge in complex networks. This analysis framework is intended to be used in a dynamic environment where network data is streamed in and is appended to a large-scale dynamic graph. Specific graphical query patterns are decomposed and collected into a graph query library. The individual decomposed subpatterns in the library are continuously and efficiently matched against the dynamic graph as it evolves to identify and detect early, partial subgraph patterns. The scalable emerging subgraph pattern algorithms will match on both structural and semantic network properties.

Chin, George↗