Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data streams”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 595 records · Page 33

Moving NASA Remote Sensing Data to the GIS Environment for Health Studies

There has been an increasing demand by the health community for improved data on many different environmental factors relevant to the links between the environment and disease occurrence and transmission. These data are important for GIS-based monitoring, risk mapping, and surveillance of epidemiological parameters on a large number of different spatial, temporal, and spectral resolutions. Accordingly, NASA is developing new approaches to data collection and distribution in order to improve access to multiple sources of data streams to increase spatial and temporal coverage. Methods are being developed to incorporate different, scalable capabilities to handle multiple data sources by adding, deleting and replacing components as required as well as associated tools for their management. An approach has been to search for innovative solutions focused on the creation, use and manipulation of data stored in many different archives. These include data transformation and combination as well as data and information tools that can assist the public health and science community to use existing and anticipated products in new and flexible ways. This presentation will provide an inventory of geophysical parameters derived from satellite remote sensing sensors that are useful for GIS-based public health studies. The presentation will also discuss the physical and scientific limitations of access to and use of these data for health applications such as resolution and format differences, lack of software interoperability, data access problems. Finally, there will be a summary of the recent steps the NASA program has taken to bring NASA-generated satellite products to a wider range of users in the GIS community.

Vicente, Gilberto A.↗

Improving Geolocation Accuracy of the Advanced Meteorological Imager on the GEO-KOMPSAT-2A

GeoNEX is a collaborative project led by scientists from NASA and many other international institutes to generate Earth monitoring products using data streams from the latest geostationary (GEO) sensors. Its consistent processing and common gridding systems can produce research-quality data products from GEO sensors and leverage GEO-GEO or GEO-LEO (low earth orbit) synergistic uses. Currently, GeoNEX has produced and disseminated L1G (geometrically corrected Level 1 products) from GOES 16/17 ABIs and Himawari-8 AHI, but a new Korean geostationary sensor (Advanced Meteorological Imager, AMI) onboard Geo-KOMPSAT-2A covering a large proportion of Asia and all of Oceania is in development. Our recent efforts on assessing geolocation accuracy in ABI and AHI suggest a nontrivial residual exists in both level 1B data with varying spatiotemporal patterns. The findings urge us to prioritize identifying and correcting geolocation residuals of AMI to generate accurate and consistent GeoNEX top-of-atmosphere (TOA) reflectance products and following processing chains. Here we implement a phase correlation correction approach to a visible band (0.64 μm, 500 m) using landmarks prepared from finer scale digital terrain models. We characterize spatiotemporal patterns (e.g., diurnal & daily) of geolocation residuals of AMI before and after correction. We then assess stability of datasets and quantify impact of unexpected geolocation errors on terrestrial monitoring. The geolocation corrected AMI data are further compared with GeoNEX AHI L1G products which are able to create unique stereo-type observations with AMI through leveraging the similarities of spectral bands and the sun-target-sensor geometry. Further, we discuss challenges in utilizing the GEO-GEO (e.g., AMI & AHI) satellite data for potential applications.

Geostationary Satellites↗

A Machine Learning Approach for Business Intelligence Analysis using Commercial Shipping Transaction Data

Business intelligence problems are particularly challenging due to the use of large volume and high velocity data in attempts to model and explain complex underlying phenomena. Incremental machine learning based approaches for summarizing trends and identifying anomalous behavior are often desirable in such conditions to assist domain experts in characterizing their data. The overall goal of this research is to develop a machine learning algorithm that enables predictive analysis on streaming data, detects changes and anomalies in the data, and can evolve based on the dynamic behavior of the data. Commercial shipping transaction data for the U.S. is used to develop and test a Naïve Bayes model that classifies several companies into lines of businesses and demonstrates an ability to predict when the behavior of these companies changes by venturing into other lines of businesses.

Bramer, Lisa M.↗

Multiple Laser Tracker Synchronization for the Vibration Analysis

Measuring the mechanical coupling and rigidity of a large apparatus presents unique challenges for 3D metrology. In this case the motion of well separated points attached to a rotor blade must be rapidly and repeatedly measured and the resulting data synchronized. This presentation describes the methodology and results for the mechanical coupling test of the magnetic field mapping system (FMS) for the Mu2e experiment at Fermi National Accelerator Laboratory (Fermilab). Three laser trackers, 2 API Radians and an API TIII, simultaneously measured the position of three cylindrical corner cube reflectors mounted on the FMS while the system was in motion and after coming to rest. The laser trackers took measurements at 83Hz for the T3 and 100Hz for the Radians, with fine spatial resolution, resulting in over 20,000 points measured per instrument in observation cycle. The laser tracker hardware and utilized software are unable to synchronize the three independent data streams directly during the measurement process. Instead a custom software program was developed to perform this synchronization of the data sequences during post-processing using features of the targets motion common to all three fiducial locations.

43 PARTICLE ACCELERATORS↗

Mu2e DAQ and slow control systems

The Mu2e experiment at the Fermilab Muon Campus will search for the coherent neutrinoless conversion of a muon into an electron in the feld of an aluminum nucleus with a sensitivity improvement by a factor of 10,000 over existing limits. The Mu2e Trigger and Data Acquisition System (TDAQ) uses otsdaq as the online Data Acquisition System (DAQ) solution. Developed at Fermilab, otsdaq integrates both the artdaq DAQ and the art analysis frameworks for event transfer, fltering, and processing. otsdaq is an online DAQ software suite with a focus on fexibility and scalability and provides a multiuser, web-based, interface accessible through a web browser. The data stream from the detector subsystems is read by a software flter algorithm that selects events which are combined with the data fux coming from a Cosmic Ray Veto System. The Detector Control System (DCS) has been developed using the Experimental Physics and Industrial Control System (EPICS) open source platform for monitoring, controlling, alarming, and archiving. The DCS System has been integrated into otsdaq. A prototype of the TDAQ and the DCS systems has been built at Fermilab’s Feynman Computing Center. In this paper, we report on the progress of the integration of this prototype in the online otsdaq software.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Data acquisition and slow control interface for the Mu2e experiment

The Mu2e experiment at the Fermilab Muon Campus will search for the coherent neutrinoless conversion of a muon into an electron in the field of an aluminum nucleus with a sensitivity improvement by a factor of 10000 over existing limits. The Mu2e Trigger and Data Acquisition System (TDAQ) uses otsdaq as the online Data Acquisition System (DAQ) solution. Developed at Fermilab, otsdaq integrates both the artdaq DAQ and the art analysis frameworks for event transfer, filtering, and processing. otsdaq is an online DAQ software suite with a focus on flexibility and scalability and provides a multi-user, web-based, interface accessible through a web browser. The data stream from the detector subsystems is read by a software filter algorithm that selects events which are combined with the data flux coming from a cosmic ray veto system. The Detector Control System (DCS) has been developed using the Experimental Physics and Industrial Control System (EPICS) open source platform for monitoring, controlling, alarming, and archiving. The DCS system has been integrated into otsdaq. A prototype of the TDAQ and the DCS systems has been built at Fermilab's Feynman Computing Center. In this study, we report on the progress of the integration of this prototype in the online otsdaq software.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Integrated Multi-Scale Data Analytics and Machine Learning for the Distribution Grid

We consider the field of machine learning and where it is both useful, and not useful, for the distribution grid and buildings interface. While analytics, in general, is a growing field of interest, and often seen as the golden goose in the burgeoning distribution grid industry, its application is often limited by communications infrastructure, or lack of a focused technical application. Overall, the linkage of analytics to purposeful application in the grid space has been limited. In this paper we consider the field of machine learning as a subset of analytical techniques, and discuss its ability and limitations to enable the future distribution grid. To that end, we also consider the potential for mixing distributed and centralized analytics and the pros and cons of these approaches. There is an exponentially expanding volume of measured data being generated on the distribution grid, which, with appropriate application of analytics, may be transformed into intelligible, actionable information that can be provided to the right actors - such as grid and building operators, at the appropriate time to enhance grid or building resilience, efficiency, and operations against various metrics or goals - such as total carbon reduction or other economic benefit to customers. While some basic analysis into these data streams can provide a wealth of information, computational and human boundaries on performing the analysis are becoming significant, with more data and multi-objective concerns. Efficient applications of analysis and the machine learning field are being considered in the loop. This paper describes benefits and limits of present machine-learning applications for use on the grid and presents a series of case studies that illustrate the potential benefits of developing advanced local multi-variate analytics machine-learning-based applications.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Integrated Multi-Scale Data Analytics and Machine Learning for the Distribution Grid and Building-to-Grid Interface

This white paper introduces the application of advanced data analytics to the modernized grid. In particular, we consider the field of machine learning and where it is both useful, and not useful, for the particular field of the distribution grid and buildings interface. While analytics, in general, is a growing field of interest, and often seen as the golden goose in the burgeoning distribution grid industry, its application is often limited by communications infrastructure, or lack of a focused technical application. Overall, the linkage of analytics to purposeful application in the grid space has been limited. In this paper we consider the field of machine learning as a subset of analytical techniques, and discuss its ability and limitations to enable the future distribution grid and the building-to-grid interface. To that end, we also consider the potential for mixing distributed and centralized analytics and the pros and cons of these approaches. Machine learning is a subfield of computer science that studies and constructs algorithms that can learn from data and make predictions and improve forecasts. Incorporation of machine learning in grid monitoring and analysis tools may have the potential to solve data and operational challenges that result from increasing penetration of distributed and behind-the-meter energy resources. There is an exponentially expanding volume of measured data being generated on the distribution grid, which, with appropriate application of analytics, may be transformed into intelligible, actionable information that can be provided to the right actors – such as grid and building operators, at the appropriate time to enhance grid or building resilience, efficiency, and operations against various metrics or goals – such as total carbon reduction or other economic benefit to customers. While some basic analysis into these data streams can provide a wealth of information, computational and human boundaries on performing the analysis are becoming significant, with more data and multi-objective concerns. Efficient applications of analysis and the machine learning field are being considered in the loop.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Advanced Data Structures for Improved Cyber Resilience and Awareness in Untrusted Environments: LDRD Report

This report summarizes the work performed under the project "Advanced Data Structures for Improved Cyber Resilience and Awareness in Untrusted Environments." The goal of the project was to design, analyze, and test new data structures for cybersecurity applications. We had two major thrusts: 1) using new/improved write-optimized data structures and/or algorithms to better man- age and analyze high-speed massive-volume cyberstreams, and 2) adding security features to data structures at minimum cost. Write optimization allows data structures to better use secondary memory to store and search a larger amount of useful information. Secondary memory is large compared to main memory, but data movement to and from secondary memory must be carefully managed to run quickly enough to keep up with fast streams. The first thrust included managing cyberstreams in parallel, both multi-threaded and distributed, and improving the benchmarking infrastructure for testing new streaming data structures. We considered both (near) real-time discovery of particular patterns, and improved logging for improved forensics. We considered two kinds of security-feature problem. The first was high-performance history- independent external-memory data structures. These provide certain protections to data if a disk is stolen. We also prove some trade-offs between speed and security in this setting. The second data-security problem is more secure data look-up in secret-shared data bases. This report summarizes the project's major accomplishments, with the background to under- stand these accomplishments. It gathers the abstracts and references for the six refereed publications that have appeared as part of this work. We summarize several accomplishments that will be submitted for publication. We then archive one piece of partial work that is not likely to be published in the near future: validation of history-independent data structure implementations.

97 MATHEMATICS AND COMPUTING↗

Evaluation of a streaming readout solution for Jefferson Lab experiments

Jefferson Lab (JLab) operates the CEBAF accelerator that provides a high energy (up to 12 GeV) and high current (up to 150 µA) polarized electron beam for fixed target experiments. JLab's mission is to provide substantial progress in understanding the Quantum Chromo Dynamics (QCD), the current theory of the spectrum and the dynamics of hadronic matter. Given the broad scientific program a general-purpose detector, CLAS12, has been designed and constructed. Such a detector requires a sophisticated trigger and current experiments use an on-line FPGA-based system that relies upon custom firmware and electronics both of which are difficult reconfigure from one experiment to the next. To overcome these challenges transitioning to a streaming readout has been proposed since it would allow a more flexible, easier to debug, software trigger to be developed. To facilitate this JLab is gaining experience with streaming readout systems and is actively working to design and test a streaming data acquisition architecture. Significant work has already been done to make existing front-end electronics compatible with operation in streaming mode. Software-wise, much work remains to be done but an existing framework, TriDAS, which was developed for underwater neutrino telescopes can, in principle, be used with the existing JLab hardware. A test stand is already in place in the INDRA DAQ development lab at JLab. This paper will present the development of this TriDAS based test system and it?s operation in-beam using parts of the CLAS12 detector to evaluate streaming readout for a large general purpose detector.

Battaglieri, Marco↗

Definition and fabrication of an airborne scatterometer radar signal processor

A hardware/software system which incorporates a microprocessor design and software for the calculation of normalized radar cross section in real time was developed. Interface is provided to decommutate the NASA ADAS data stream for aircraft parameters used in processing and to provide output in the form of strip chart and pcm compatible data recording.

Source record↗

User data management

The primary objective is to identify, develop, and demonstrate key data management technologies to support user access to Space Station data. To accomplish this objective, there are several technical challenges which must be addressed. The first is how to provide routine customer access to high volume, dynamic and distributed data bases. This access will encompass the functions of mission and payload planning and operations, data processing and analysis, and data archive and distribution. Secondly, there must be some analysis of architectures for handling high volume data streams like those expected from the Space Station. This analysis will examine the use of packetized versus non-packetized data formats, modular expansion capabilities, real-time versus non-real-time data processing, and the interfaces and architecture required to support telescience operations. The task will also determine benchmarks of performance capabilities for technology operations, such as varied data base structures, data access procedures, distributed data base design, and data base machines. Information is provided here in outline form.

Source record↗

Do Better Satellite Precipitation Algorithms Improve Landslide Hazard Assessment?

Satellites make it possible to estimate precipitation in near real time. Given the challenges of achieving global coverage by other means, these data are used widely. However, few systems for landslide hazard assessment rely on satellite precipitation estimates. This could be due in part to perceptions of accuracy, although latency, spatial resolution, and other factors may also be important. We test whether recent changes to data streams from the Global Precipitation Measurement mission (GPM) have improved its potential for use in landslide prediction. Specifically, we examine data produced by the Integrated Multi-satellitERetrievals for the GPM (IMERG) algorithm, which was upgraded to version 7 this year. IMERG relies upon other algorithms, including the Goddard Profiling Algorithm (GPROF) and the GPM Combined Radar-Radiometer Algorithm (CORRA). Many changes have been made during the switch from IMERG version 6 to version 7. These include upgrading CORRA and GPROF to version 7, to improve the accuracy of precipitation in frozen, mountainous, and coastal areas. The measured intensity of some storms has been enhanced with a new algorithm, the Scheme for Histogram Adjustment with Ranked Precipitation Estimates in the Neighborhood. Combined with many others, these changes to IMERG should improve its utility for landslide hazard assessment in a variety of contexts. To test this idea, we retrain the global Landslide Hazard Assessment for Situational Awareness (LHASA) model twice—first with data from IMERG version 6B and second with 7B. Since current daily rainfall is the most important variable in determining outcomes predicted by LHASA, it should reflect changes made to that input. First, we grid the landslides at a daily, thirty-arcsecond resolution. This serves as the response variable. At each of these sites current and antecedent rainfall are extracted, along with antecedent snow mass and soil moisture, slope, and PGA. In addition, one million grid cells are selected at random points to represent conditions under which landslides (probably) do not occur. After merging these data, we hold back 20% of the dataset for validation purposes and train a machine-learning model with the rest. We assess both the model’s overall ability to identify landslides and its ability to predict specific large landslide disasters.

Thomas A Stanley↗

Redis-Based Streaming Architecture for Accelerator Beam Instrumentation DAQ Systems

The Fermilab Acceleraor Division, Beam Instrumentation Department, is always adopting modern and current software methodologies for complex DAQ architectures. This paper highlights the Redis Adapter (RA) as the key software component enabling high performance, modular communication between digitizers and distributed control systems by leveraging Redis and containerization. The RA provides a unified, efficient interface between Redis based data streams and consumer systems. In the legacy architecture, digitized data flowed through the custom, UDP based Distributed Data Communication Protocol in the middle layer. In the current system, DDCP remains the ingestion path, while the RA serves as the decoupling layer. The proposed system replaces old VME digitizers with a SOM-based digitizer that communicates with Redis using the RA. The RA acts as both a performance-critical bridge and a protocol-agnostic adapter, ensuring compatibility with legacy control frameworks while enabling future scalability and modularity. This restructuring of the middle layer also helps the system achieve high throughput, reduce latency, and simplify the data path. Finally, we will demonstrate how RA is utilized in our two core products to deliver both legacy compatibility and future flexibility.

Joshi, S. [Fermilab]↗

Bridging microscopy with molecular dynamics and quantum simulations: an atomAI based pipeline

Recent advances in (scanning) transmission electron microscopy have enabled a routine generation of large volumes of high-veracity structural data on 2D and 3D materials, naturally offering the challenge of using these as starting inputs for atomistic simulations. In this fashion, the theory will address experimentally emerging structures, as opposed to the full range of theoretically possible atomic configurations. However, this challenge is highly nontrivial due to the extreme disparity between intrinsic timescales accessible to modern simulations and microscopy, as well as latencies of microscopy and simulations per se. Addressing this issue requires as a first step bridging the instrumental data flow and physics-based simulation environment, to enable the selection of regions of interest and exploring them using physical simulations. Here we report the development of the machine learning workflow that directly bridges the instrument data stream into Python-based molecular dynamics and density functional theory environments using pre-trained neural networks to convert imaging data to physical descriptors. Additionally, the pathways to ensure structural stability and compensate for the observational biases universally present in the data are identified in the workflow. This approach is used for a graphene system to reconstruct optimized geometry and simulate temperature-dependent dynamics including adsorption of Cr as an ad-atom and graphene healing effects. However, it is universal and can be used for other material systems.

36 MATERIALS SCIENCE↗

The trigger and data acquisition for the NEMO-Phase 2 tower

In the framework of the Phase 2 of the NEMO neutrino telescope project, a tower with 32 optical modules is being operated since march 2013. A new scalable Trigger and Data Acquisition System (TriDAS) has been developed and extensively tested with the data from this tower. Adopting the all-data-to-shore concept, the NEMO TriDAS is optimized to deal with a continuous data-stream from off-shore to on-shore with a large bandwidth. The TriDAS consists of four computing layers: (i) data aggregation of isochronal hits from all optical modules; (ii) data filtering by means of concurrent trigger algorithms; (iii) composition of the filtered events into post-trigger files; (iv) persistent data storage. The TriDAS implementation is reported together with a review of dedicated on-line monitoring tools.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Extended testing of a general contextual classifier using the massively parallel processor - Preliminary results and test plans

Earlier encouraging test results of a contextual classifier that combines spatial and spectral information employing a general statistical approach are expanded. The earlier results were of limited meaning because they were produced from small (50-by-50 pixel) data sets. An implementation of the contextual classifier on NASA Goddard's Massively Parallel Processor (MPP) is presented; for the first time the MPP makes feasible the testing of the classifier on large data sets (a 12-hour test on a VAX-11/780 minicomputer now takes 5 minutes on the MPP). The MPP is a Single-Instruction, Multiple Data Stream computer, consisting of 16,384 bit serial microprocessors connected in a 128-by-128 mesh array with each element having data transfer connections with its four nearest neighbors so that the MPP is capable of billions of operations per second. Preliminary results are given (with more expected for the conference) and plans are mentioned for extended testing of the contextual classifier on Thematic Mapper data sets.

Tilton, J. C.↗