Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26

Optimization of distributed compute resources utilization in the CMS Global Pool

The CMS Submission Infrastructure is the primary system for managing computing resources for CMS workflows, including data processing, simulation, and analysis. It integrates geographically distributed resources from Grid, HPC, and cloud providers into federated pools managed by HTCondor and Glidein- WMS, for a total of around 500k CPU cores. This system dynamically manages workloads based on priorities defined by the collaboration. Additionally, CMS scheduling strategies must be flexible to handle multiple concurrent workloads while considering changing processing demands and resource availability from various providers.Efficient utilization of vast amounts of distributed compute resources is a key element for the success of the scientific programs of the LHC experiments. Optimizing the system is essential to maximize resource efficiency and fully utilize the distributed computing power. The CMS Submission Infrastructure team thus systematically investigates sources of inefficiency in workload scheduling to reduce their impact. In addition, a strategy of pilot overloading has been introduced to compensate for other inefficiency sources, thereby optimizing resource utilization and enhancing computational throughput.

Mascheroni, Marco [UC, San Diego (main)]↗

Network-Capable Application Process and Wireless Intelligent Sensors for ISHM

Intelligent sensor technology and systems are increasingly becoming attractive means to serve as frameworks for intelligent rocket test facilities with embedded intelligent sensor elements, distributed data acquisition elements, and onboard data acquisition elements. Networked intelligent processors enable users and systems integrators to automatically configure their measurement automation systems for analog sensors. NASA and leading sensor vendors are working together to apply the IEEE 1451 standard for adding plug-and-play capabilities for wireless analog transducers through the use of a Transducer Electronic Data Sheet (TEDS) in order to simplify sensor setup, use, and maintenance, to automatically obtain calibration data, and to eliminate manual data entry and error. A TEDS contains the critical information needed by an instrument or measurement system to identify, characterize, interface, and properly use the signal from an analog sensor. A TEDS is deployed for a sensor in one of two ways. First, the TEDS can reside in embedded, nonvolatile memory (typically flash memory) within the intelligent processor. Second, a virtual TEDS can exist as a separate file, downloadable from the Internet. This concept of virtual TEDS extends the benefits of the standardized TEDS to legacy sensors and applications where the embedded memory is not available. An HTML-based user interface provides a visual tool to interface with those distributed sensors that a TEDS is associated with, to automate the sensor management process. Implementing and deploying the IEEE 1451.1-based Network-Capable Application Process (NCAP) can achieve support for intelligent process in Integrated Systems Health Management (ISHM) for the purpose of monitoring, detection of anomalies, diagnosis of causes of anomalies, prediction of future anomalies, mitigation to maintain operability, and integrated awareness of system health by the operator. It can also support local data collection and storage. This invention enables wide-area sensing and employs numerous globally distributed sensing devices that observe the physical world through the existing sensor network. This innovation enables distributed storage, distributed processing, distributed intelligence, and the availability of DiaK (Data, Information, and Knowledge) to any element as needed. It also enables the simultaneous execution of multiple processes, and represents models that contribute to the determination of the condition and health of each element in the system. The NCAP (intelligent process) can configure data-collection and filtering processes in reaction to sensed data, allowing it to decide when and how to adapt collection and processing with regard to sophisticated analysis of data derived from multiple sensors. The user will be able to view the sensing device network as a single unit that supports a high-level query language. Each query would be able to operate over data collected from across the global sensor network just as a search query encompasses millions of Web pages. The sensor web can preserve ubiquitous information access between the querier and the queried data. Pervasive monitoring of the physical world raises significant data and privacy concerns. This innovation enables different authorities to control portions of the sensing infrastructure, and sensor service authors may wish to compose services across authority boundaries.

Figueroa, Fernando↗

WetNet operations

WetNet is an interdisciplinary Earth science data analysis and research project with an emphasis on the study of the global hydrological cycle. The project goals are to facilitate scientific discussion, collaboration, and interaction among a selected group of investigators by providing data access and data analysis software on a personal computer. The WetNet system fulfills some of the functionality of a prototype Product Generation System (PGS), Data Archive and Distribution System (DADS), and Information Management System for the Distributed Active Archive Center. The PGS functionality is satisfied in WetNet by processing the Special Sensor Microwave/Imager (SSM/I) data into a standard format (McIDAS) data sets and generating geophysical parameter Level II browse data sets. The DADS functionality is fulfilled when the data sets are archived on magneto optical cartridges and distributed to the WetNet investigators. The WetNet data sets on the magneto optical cartridges contain the complete WetNet processing, catalogue, and menu software in addition to SSM/I orbit data for the respective two week time period.

Goodman, H. Michael↗

Application of ESE Data and Tools to Air Quality Management: Services for Helping the Air Quality Community use ESE Data (SHAirED)

The goal of this REASoN applications and technology project is to deliver and use Earth Science Enterprise (ESE) data and tools in support of air quality management. Its scope falls within the domain of air quality management and aims to develop a federated air quality information sharing network that includes data from NASA, EPA, US States and others. Project goals were achieved through a access of satellite and ground observation data, web services information technology, interoperability standards, and air quality community collaboration. In contributing to a network of NASA ESE data in support of particulate air quality management, the project will develop access to distributed data, build Web infrastructure, and create tools for data processing and analysis. The key technologies used in the project include emerging web services for developing self describing and modular data access and processing tools, and service oriented architecture for chaining web services together to assemble customized air quality management applications. The technology and tools required for this project were developed within DataFed.net, a shared infrastructure that supports collaborative atmospheric data sharing and processing web services. Much of the collaboration was facilitated through community interactions through the Federation of Earth Science Information Partners (ESIP) Air Quality Workgroup. The main activities during the project that successfully advanced DataFed, enabled air quality applications and established community-oriented infrastructures were: develop access to distributed data (surface and satellite), build Web infrastructure to support data access, processing and analysis create tools for data processing and analysis foster air quality community collaboration and interoperability.

Falke, Stefan↗

DER Cybersecurity Detection and Response Suite

SAND2024-08475O The Distributed Energy Resource (DER) Cybersecurity Detection and Response Suite is a solution for distributed energy resource (DER) systems. The DER Security Orchestration, Automation, and Response (SOAR) solution that uses alerts from signature- and behavior-based Intrusion Detection Systems are intended to be deployed as bump-in-the-wire (BITW) devices in front of DER equipment. The fielded application would use multiple intrusion detection systems that report data to SOAR to respond to cyberattacks. The suite consists of two software components: • The proactive intrusion detection and mitigation system (PIDMS) secures grid-edge photovoltaic smart inverters and other equipment in distributed energy resource systems. It is a distributed BITW solution; cyber and physical data are automatically processed using network inspection tools and custom machine learning algorithms to detect abnormal events and correlate cyber-physical events. • The Security Orchestration, Automation, and Response for Distributed Energy Resources (SOAR4DER) application ingests data from several intrusion detection systems to quickly block attacks and revert DER systems to good states. Using a collection of intrusion detection system technologies on a BITW device, it incorporates physical and cyber data to detect abnormal and potential malicious behaviors. Multiple SOAR playbooks then use the intrusion detection system data streams to automatically defend the system. SOAR4DER system testing showed detection and response times under 30 seconds for all adversary reconnaissance, denial-of-service attacks, malicious Modbus commands, brute-force logins, and machine-in-the-middle attacks. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Johnson, Jay↗

Distributing Power Grid State Estimation on HPC Clusters A System Architecture Prototype

The future power grid is expected to further expand with highly distributed energy sources and smart loads. The increased size and complexity lead to increased burden on existing computational resources in energy control centers. Thus the need to perform real-time assessment on such systems entails efficient means to distribute centralized functions such as state estimation in the power system. In this paper, we present our early prototype of a system architecture that connects distributed state estimators individually running parallel programs to solve non-linear estimation procedure. The prototype consists of a middleware and data processing toolkits that allows data exchange in the distributed state estimation. We build a test case based on the IEEE 118 bus system and partition the state estimation of the whole system model to available HPC clusters. The measurement from the testbed demonstrates the low overhead of our solution.

Distributed architecture↗

An Overview of ARM Program Climate Research Facility Data Quality Assurance

This report documents key aspects of the Atmospheric Radiation Measurement (ARM) Program's data quality assurance program. Processes described include data archival, display, and distribution, data stream reprocessing, and future directions in ARM data quality checking. A comprehensive, end-to-end data quality assurance program, from instrument siting, to calibration and maintenance, through data quality control and well-documented dissemination, is essential for producing a high-quality research data set. The processes developed over the past 15 years by the ARM Program offer a possible framework for use by other instrumentation-diverse networks, and are offered here to highlight the myriad aspects that go into producing such a data set.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

The problem analysis of data processing performance in nuclear power plant computerized information and control system - 288

This paper is based on the data processing performance problems of operating command delay and system display not updating real-time for nuclear power plant computerized information and control system. The plant computerized information and control system is responsible for monitoring and controlling the plant operation, and it is a key part in the plant DCS system. Its data processing performance will affect the operation stability of nuclear power plant. The architecture, hardware configuration and data communication for plant computerized information and control system among the different levels in DCS are described. The main reasons affecting data processing performance in different signal channels are analyzed according to the plant load test results. The CPU load curves for real time server and central processing server are displayed, although the detailed cause for operating command delay and system display not updating realtime are not found, central processing server load is heavier than real time server which remind us to analyze the data processing of software resource scheduling deeply. So the resource distribution problem of software execution task is also analyzed according to CPU load test results. This paper provides the optimization analysis for data processing performance, and it is available and effective according to the latest test results. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Aerial Lidar and Geomorphic Change Detection Software- Tools for Monitoring Canyon Sediments at LANL - 18668

The Los Alamos National Laboratory Environmental Remediation program has conducted extensive geomorphic surveys of sediment deposits in the canyon bottoms within and surrounding the facility. The goal of the geomorphic surveys was to provide basic information needed to characterize chemical inventories of the sediments downstream of historical release sites. Sediment units were mapped based on depositional setting and age. Representative characterization samples were collected for each unit and submitted for chemical analysis. Chemical inventories were estimated by calculating the total volume of each depositional unit and multiplying by average chemical concentration. This method of characterization provides a spatial distribution of chemical data based on fluvial processes. In general, for the canyon bottom sediments, the higher concentrations of industrial chemicals were associated with fine-grained floodplain deposits closest to the source. Since initial characterization, many of the reaches have undergone changes in response to post-wildfire floods and severe rain events. These events impacted the channel and floodplain deposits. Much of the elevated concentrations are confined to the over-bank deposits distal from the primary channel. In order to ensure that the sediment units containing elevated chemical concentrations are stable, repeat aerial light detection and ranging (LiDAR) surveys are conducted and changes are assessed using the Geomorphic Change Detection (GCD) plug-in for ArcGIS. These tools provide high confidence uncertainty estimates and extensive areal coverage. This is a valuable method for long-term monitoring of surface sediments with low-level but widespread contaminant distributions. All of the monitored canyon bottom reaches pass risk assessment and present technically infeasible sediment removal options. Following the 2014 monsoon season, aerial LiDAR data has been collected annually and the resulting digital elevation models (DEMs) processed using the GCD method. Changes are identified using GCD and are verified by field walk downs. If erosion is identified in areas which are vulnerable to mobilizing contaminated sediment, enhanced controls can be installed to stabilize banks or other areas with high erosion potential. Annual sediment monitoring samples are collected where there is evidence for flow and deposition within the past year. Since 2014, no severe floods have been catalogued using the LiDAR surveys and GCD software. The recent rainfall/runoff events have not caused major changes to the canyon bottom channels and floodplains. Geomorphic processes identified in the monitoring results include channel aggradation and incision that over the course of a monsoon season results in minor changes to the system. Other active processes contributing to observed changes are minor slides, flows, slumps, and falls of unconsolidated sediment on steep bedrock or soil surfaces. A comprehensive baseline has been established by geomorphic mapping and characterization sampling. Periodic monitoring consists of LiDAR surveys and GCD analysis which offers an effective and defensible methodology for documenting stability of surface sediments. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Decision-level Information Fusion to Assess Threat Likelihood in Shipped Containers

A significant challenge in the distribution of goods is assessing the potential threat that an individual shipping container poses. Due to the high volume of shipped goods, a primary concern is balancing accuracy and container scan time. The application of information fusion to the problem enables automated threat determination and the presentation of relevant data to an operator, in a decision support capacity, in order to maintain a sufficient level of processing. This paper outlines an approach to container threat assessment that combines data from multiple sources in order to reliably score the likelihood that a given container holds a threat. Fused data is also leveraged as a tool to optimize the routing of containers through a scanning system comprised of multiple data acquisition stations and providing data in multiple modes. Furthermore, we propose methods for the consolidated presentation of fused data to an operator in order to both minimize the time expended in container evaluation and maximize the accuracy of the assessment.

97 MATHEMATICS AND COMPUTING↗

Multi-Fidelity Gaussian Process for Distribution System Voltage Probabilistic Estimation with PVs

The increasing penetration of behind-the-meter PVs causes challenges to maintain voltage security due to the lack of distribution system visibility. This paper proposes a nonlinear autoregressive Gaussian process (NARGP) approach to fuse limited number of SCADA/AMI data together with historical pseudo measurements for distribution node voltage probabilistic estimation. The high-fidelity SCADA data are fused with the low-fidelity AMI and pseudo measurements by the autoregressive algorithm embedded in the Gaussian process. This allows us to use multi-fidelity data to achieve entire distribution system voltage visibility. Numerical results carried out on the IEEE 123node system demonstrate that the NARGP method is able to obtain high accuracy in estimating bus voltage and quantifying estimation uncertainties as compared to other approaches.

distribution system estimation↗

Spatio-Temporal Signal Twice-Whitening Algorithms on the hx3100 Ultra-Low Power Multicore Processor

While modern signal detection theory fully accounts for spatially distributed sensors, exploiting these techniques for real-time sensing using large, underwater acoustic arrays requires advances in the spatio-temporal signal processing algorithms. In particular, the computational complexity of many spatio-temporal processing techniques is so large that conventional computer processors lack sufficient throughput to provide real-time processing of large spatio-temporal data sets. These limits are exacerbated when constraints, such as power consumption or footprint, reduce the available computational resources. In this report, we demonstrate an implementation of a signal twice-whitening algorithm that is better suited for processing spatio-temporal data in real time. We emphasize these advances by implementing data whitening on the Coherent Logix hx3100 processor, a programmable multicore processor intended for low-power and high-throughput signal processing. These results serve as an example of how the novel capabilities available from emerging multicore processor platforms can provide real-time, software-defined processing of large data sets acquired by spatially distributed sensing.

99 GENERAL AND MISCELLANEOUS↗

Neural network for processing both spatial and temporal data with time based back-propagation

Neural networks are computing systems modeled after the paradigm of the biological brain. For years, researchers using various forms of neural networks have attempted to model the brain's information processing and decision-making capabilities. Neural network algorithms have impressively demonstrated the capability of modeling spatial information. On the other hand, the application of parallel distributed models to the processing of temporal data has been severely restricted. The invention introduces a novel technique which adds the dimension of time to the well known back-propagation neural network algorithm. In the space-time neural network disclosed herein, the synaptic weights between two artificial neurons (processing elements) are replaced with an adaptable-adjustable filter. Instead of a single synaptic weight, the invention provides a plurality of weights representing not only association, but also temporal dependencies. In this case, the synaptic weights are the coefficients to the adaptable digital filters. Novelty is believed to lie in the disclosure of a processing element and a network of the processing elements which are capable of processing temporal as well as spacial data.

Villarreal, James A.↗

The Solar Dynamics Observatory: Your Eye on the Sun

The Solar Dynamics Observatory (SDO) was launched on February 11,2010 into partly cloudy skies over Cape Canaveral, Florida. SDO moved into a 28 degree inclined geosynchronous orbit over the longitude of the ground station in New Mexico. SDO is the first Space Weather Mission in NASA's Living With a Star Program. SDO's main goal is to understand and predict those solar variations that influence life on Earth and our technological systems. The SDO science investigations will determine how the Sun's magnetic field is generated and structured, how this stored magnetic energy is released into the heliosphere as the solar wind, energetic particles, and variations in the solar irradiance. The SDO mission consists of three scientific investigations (AIA, EVE, and HMI), a spacecraft bus, and a dedicated Ka-band ground station to handle the 150 Mbps data flow. SDO continues a long tradition of NASA missions providing calibrated solar spectral irradiance data, in this case using multiple measurements of the irradiance and rocket underflights of the spacecraft. The other instruments on SDO will be used to explain and develop predictive models of the solar spectral irradiance in the extreme ultraviolet. Science teams at LMSAL, LASP, and Stanford are responsible for processing, analyzing, distributing, and archiving the science data. We will talk about the building of SDO, its launch, and the data and science it will provide to NASA.

Pesnell, William Dean↗

The Solar Dynamics Observatory: Your On-Orbit Eye on the Sun

The Solar Dynamics Observatory (SDO) was launched on February 11, 2010 into the partly cloudy skies above Cape Canaveral, Florida. Over the next month SDO moved into a 28 degree inclined geosynchronous orbit at the longitude of the ground station in New Mexico. SDO is the first Space Weather Mission in NASA's Living With a Star Program. SDO's main goal is to understand and predict those solar variations that influence life on Earth and our technological systems. The SDO science investigations will determine how the Sun's magnetic field is generated and structured, how this stored magnetic energy is released into the heliosphere as the solar wind, energetic particles, and variations in the solar irradiance. The SDO mission consists of three scientific investigations (AIA, EVE, and HMI), a spacecraft bus, and a dedicated Ka-band ground station to handle the 150 Mbps data flow. SDO continues a long tradition of NASA missions providing calibrated solar spectral irradiance data, in this case using multiple measurements of the irradiance and rocket underflights of the spacecraft. The other instruments on SDO will be used to explain and develop predictive models of the solar spectral irradiance in the extreme ultraviolet. Science teams at LMSAL, LASP, and Stanford are responsible for processing, analyzing, distributing, and archiving the science data. We will talk about the launch of SDO and describe the data and science it is providing to NASA.

Pesnell, W. Dean↗

E4D_RT

Time-lapse ERT imaging for monitoring both natural and engineered subsurface processes has advanced rapidly over the past 15-20 years. However, imaging results generally required a significant amount of manual and computational effort, and therefore were not available while the process was occurring. Although the value of real-time imaging was recognized, several obstacles prevented it's implementation. E4D_RT addresses these obstacles by 1) providing specialized algorithms that negate the need for user intervention, thereby automating the time-lapse data processing steps, 2) linking field data collection systems with parallel supercomputing systems via wireless data transfer link, and 3) addressing the computational burdens by utilizing distributed memory supercomputing resources, thereby enabling rapid data processing and imaging results.

Source record↗

Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

As system scales and application complexity grow, managing and processing simulation data has become a significant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management requirements. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic placement and scheduling of data processing kernels, and (3) dynamic allocation of in-transit resources. We also exploit coordinated approaches that dynamically combine these adaptations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the Intrepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and increasing resource efficiency.

Computer Science↗

Testing the Drop-Size Distribution-Based Separation of Stratiform and Convective Rain Using Radar and Disdrometer Data from a Mid-Latitude Coastal Region

Stratiform and convective rain are associated with different microphysical processes and generally produce drop-size distributions (DSDs) with different characteristics. Previous studies using data from (a) a tropical coastal location, (b) a mid-latitude continental location with semi-arid climate, and (c) a sub-tropical continental location, found that the two rain types could be separated in the NW–Dm space, where Dm is the mass-weighted mean diameter and NW is the normalized intercept parameter. In this paper, we investigate the same separation technique using data and observations from a mid-latitude coastal region. Three-minute DSDs from disdrometer measurements are used for the NW- versus Dm-based classification and are compared with simultaneous observations from an S-band polarimetric radar 38 km away from the disdrometer site. Specifically, RHI (range-height indicator) scans over the disdrometer were used for confirmation. Results show that there was no need to modify the separation criteria from previous studies. Three-minute DSDs from the same location were used as input to scattering calculations to derive retrieval equations for NW and Dm for the S-band radar using an improved technique and applied to the RHI scans to identify convective and stratiform rain regions. Two events are shown as illustrative examples.

Merhala Thurai↗