Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

Keyless Infrastructure Security Solution (KISS): VOLTTRON™ KSI ® Blockchain Design and Specification

This document provides the technical design specifications that are required for Keyless Infrastructure Security Solution (KISS) development, including, definitions and roles of the new KISS-related products—GridAware and LinkLite—an overview of the DNP3 communications the VOLTTRON™ DNP3 agent is using for distribution control, data flow, historian processing, setup, and installation. These technical specifications provide step-by-step instructions for configuration, implementation, and deployment of the KISS subsystem, docket creation, and verification using the GridAware dashboard. Based on this document and the KISS specification requirements and device risk assessment documents, KISS will be developed by integrating keyless signature infrastructure (KSI) blockchain to increase the cybersecurity, integrity, and trustworthiness of critical energy delivery systems (EDSs). The KISS architecture consists of a VOLTTRON™ agent-based distribution control system platform and GridAware (KSI blockchain capability) for secure energy exchanges in a decentralized electrical network.

42 ENGINEERING↗

Preparation of the Multi-Site Data Processing at the Vera C. Rubin Observatory

The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) Camera is scheduled to start taking data in the summer of 2025. The Data Release Production will run the LSST Science Pipe software at data facilities in the US, France and the UK. The LSST Science Pipeline consists of complex directed acyclic graphs (DAGs) of tasks. Rubin will use the Production and Distributed Analysis (PanDA) workflow and workload management system to orchestrate this complex workflow and the distribution of workloads to the data facilities. When run end-to-end by a team of data production staff, this processing (the Science Pipelines, distributed by the workflow and workload management system) is referred to as a 'campaign'. This paper describes the central services and data facility specific services that support this multi-site data process model, including the service deployment infrastructure, the workload and workflow system, the Campaign Management tools, and connection to Rubin Data Management. This paper will also mention the experience of processing the Rubin Commissioning Camera data. All these are part of the effort to scale up the processing capabilities for the expected very large data volume from the LSST Camera.

Yang, Wei [SLAC]↗

Multipartite nonlocality distillation

The stronger nonlocality than that allowed in quantum theory can provide an advantage in information processing and computation. Since quantum entanglement is distillable, can nonlocality be distilled in the nonsignalling condition? The answer is positive in the bipartite case. In this article the distillability of the multipartite nonlocality is investigated. We propose a distillation protocol solely exploiting xor operations on output bits. The probability-distribution vectors and matrix are introduced to tackle the correlators. It is shown that only the correlators with extreme values can survive the distillation process. As the main result, the amplified nonlocality cannot maximally violate any Bell-type inequality. Accordingly, a distillability criterion in the postquantum region is proposed.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Instrumentation complex for Langley Research Center's National Transonic Facility

The instrumentation discussed in the present paper was developed to ensure reliable operation for a 2.5-meter cryogenic high-Reynolds-number fan-driven transonic wind tunnel. It will incorporate four CPU's and associated analog and digital input/output equipment, necessary for acquiring research data, controlling the tunnel parameters, and monitoring the process conditions. Connected in a multipoint distributed network, the CPU's will support data base management and processing; research measurement data acquisition and display; process monitoring; and communication control. The design will allow essential processes to continue, in the case of major hardware failures, by switching input/output equipment to alternate CPU's and by eliminating nonessential functions. It will also permit software modularization by CPU activity and thereby reduce complexity and development time.

Russell, C. H.↗

Extra Dimensions: 3D and Time in PDF Documentation

Experimental science is replete with multi-dimensional information which is often poorly represented by the two dimensions of presentation slides and print media. Past efforts to disseminate such information to a wider audience have failed for a number of reasons, including a lack of standards which are easy to implement and have broad support. Adobe's Portable Document Format (PDF) has in recent years become the de facto standard for secure, dependable electronic information exchange. It has done so by creating an open format, providing support for multiple platforms and being reliable and extensible. By providing support for the ECMA standard Universal 3D (U3D) file format in its free Adobe Reader software, Adobe has made it easy to distribute and interact with 3D content. By providing support for scripting and animation, temporal data can also be easily distributed to a wide, non-technical audience. We discuss how the field of radiation imaging could benefit from incorporating full 3D information about not only the detectors, but also the results of the experimental analyses, in its electronic publications. In this article, we present examples drawn from high-energy physics, mathematics and molecular biology which take advantage of this functionality. We demonstrate how 3D detector elements can be documented, using either CAD drawings or other sources such as GEANT visualizations as input.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Plate motions and deformations from geologic and geodetic data

The very long baseline interferometry (VLBI) measurements made in the western U.S. since 1979 provide discrete samples of the temporal and spatial deformation field. The interpretation of the VLBI derived rates of deformation requires an examination of geologic information and more densely sampled ground based geodetic data. Triangulation and trilateration data measured on two regional networks, one in the central Mojave Desert and one in the Coast Ranges east of the San Andreas fault, were processed. At the spatial scales spanned by these local geodetic networks, auxiliary geologic and geophysical data were utilized to examine the relation between measured incremental strain and the accommodation of strain seen in local geologic structures, strain release in earthquakes, and principal stress directions inferred from in situ measurements. VLBI data was also processed from stations distributed across the Pacific-North America plate boundary zone in the western U.S. The VLBI data were used to constrain the integrated rate of deformation across portions of the continental plate boundary in California and to provide a tectonic framework to interpret regional geodetic and geologic studies.

Jordan, Thomas H.↗

Distributed computing strategies for processing of FT-ICR MS imaging datasets for continuous mode data visualization

High-resolution Fourier transform ion cyclotron resonance (FT-ICR) mass spectrometry imaging enables the spatial mapping and identification of biomolecules from complex surfaces. The need for long time-domain transients, and thus large raw file sizes, results in a large amount of raw data (“big data”) that must be processed efficiently and rapidly. This can be compounded by largearea imaging and/or high spatial resolution imaging. For FT-ICR, data processing and data reduction must not compromise the high mass resolution afforded by the mass spectrometer. The continuous mode “Mosaic Datacube” approach allows high mass resolution visualization (0.001 Da) of mass spectrometry imaging data, but requires additional processing as compared to featurebased processing. We describe the use of distributed computing for processing of FT-ICR MS imaging datasets with generation of continuous mode Mosaic Datacubes for high mass resolution visualization. An eight-fold improvement in processing time is demonstrated using a Dutch nationally available cloud service.

Environmental Molecular Sciences Laboratory↗

Deep kernel methods learn better: from cards to process optimization

Abstract The ability of deep learning methods to perform classification and regression tasks relies heavily on their capacity to uncover manifolds in high-dimensional data spaces and project them into low-dimensional representation spaces. In this study, we investigate the structure and character of the manifolds generated by classical variational autoencoder (VAE) approaches and deep kernel learning (DKL). In the former case, the structure of the latent space is determined by the properties of the input data alone, while in the latter, the latent manifold forms as a result of an active learning process that balances the data distribution and target functionalities. We show that DKL with active learning can produce a more compact and smooth latent space which is more conducive to optimization compared to previously reported methods, such as the VAE. We demonstrate this behavior using a simple cards dataset and extend it to the optimization of domain-generated trajectories in physical systems. Our findings suggest that latent manifolds constructed through active learning have a more beneficial structure for optimization problems, especially in feature-rich target-poor scenarios that are common in domain sciences, such as materials synthesis, energy storage, and molecular discovery. The Jupyter Notebooks that encapsulate the complete analysis accompany the article.

97 MATHEMATICS AND COMPUTING↗

Information adaptive system of NEEDS

The NASA End-to-End Data System (NEEDS) program was initiated by NASA to improve significantly the state of the art in acquisition, processing, and distribution of space-acquired data for the mid-1980s and beyond. The information adaptive system (IAS) is a program element under NEEDS Phase II which addresses sensor specific processing on board the spacecraft. The IAS program is a logical first step toward smart sensors, and IAS developments - particularly the system components and key technology improvements - are applicable to future smart efforts. The paper describes the design goals and functional elements of the IAS. In addition, the schedule for IAS development and demonstration is discussed.

Howle, W. M., Jr.↗

GES DISC Data Recipes in Jupyter Notebooks

The Earth Science Data and Information System (ESDIS) Project manages twelve Distributed Active Archive Centers (DAACs) which are geographically dispersed across the United States. The DAACs are responsible for ingesting, processing, archiving, and distributing Earth science data produced from various sources (satellites, aircraft, field measurements, etc.). In response to projections of an exponential increase in data production, there has been a recent effort to prototype various DAAC activities in the cloud computing environment. This, in turn, led to the creation of an initiative, called the Cloud Analysis Toolkit to Enable Earth Science (CATEES), to develop a Python software package in order to transition Earth science data processing to the cloud. This project, in particular, supports CATEES and has two primary goals. One, to transition data recipes created by the Goddard Earth Science Data and Information Service Center (GES DISC) into an interactive and educational environment using JupyterNotebooks. Two, to acclimate Earth scientists to cloud computing. To accomplish these goals, we create JupyterNotebooks to compartmentalize the different steps of data analysis and help users obtain and parse data from the command line. We also develop a Docker container, comprised of Jupyter Notebooks, Python dependencies, and command line tools, and configure it into an easy-to-deploy package. The end result is an end-to-end product that simulates the use case of end users working in the cloud computing environment.

discoverability↗

Sensing system for detection and control of deposition on pendant tubes in recovery and power boilers

A system for detection and control of deposition on pendant tubes in recovery and power boilers includes one or more deposit monitoring sensors operating in infrared regions of about 4 or 8.7 microns and directly producing images of the interior of the boiler, or producing feeding signals to a data processing system for information to enable a distributed control system by which the boilers are operated to operate said boilers more efficiently. The data processing system includes an image pre-processing circuit in which a 2-D image formed by the video data input is captured, and includes a low pass filter for performing noise filtering of said video input. It also includes an image compensation system for array compensation to correct for pixel variation and dead cells, etc., and for correcting geometric distortion. An image segmentation module receives a cleaned image from the image pre-processing circuit for separating the image of the recovery boiler interior into background, pendant tubes, and deposition. It also accomplishes thresholding/clustering on gray scale/texture and makes morphological transforms to smooth regions, and identifies regions by connected components. An image-understanding unit receives a segmented image sent from the image segmentation module and matches derived regions to a 3-D model of said boiler. It derives a 3-D structure the deposition on pendant tubes in the boiler and provides the information about deposits to the plant distributed control system for more efficient operation of the plant pendant tube cleaning and operating systems.

47 OTHER INSTRUMENTATION↗

Density estimation via measure transport: Outlook for applications in the biological sciences

Abstract One among several advantages of measure transport methods is that they allow or a unified framework for processing and analysis of data distributed according to a wide class of probability measures. Within this context, we present results from computational studies aimed at assessing the potential of measure transport techniques, specifically, the use of triangular transport maps, as part of a workflow intended to support research in the biological sciences. Scenarios characterized by the availability of limited amount of sample data, which are common in domains such as radiation biology, are of particular interest. We find that when estimating a distribution density function given limited amount of sample data, adaptive transport maps are advantageous. In particular, statistics gathered from computing series of adaptive transport maps, trained on a series of randomly chosen subsets of the set of available data samples, leads to uncovering information hidden in the data. As a result, in the radiation biology application considered here, this approach provides a tool for generating hypotheses about gene relationships and their dynamics under radiation exposure.

gene expression data↗

The creation of a comprehensive metocean data set for offshore wind turbine simulations: Comprehensive metocean data set

A database of meteorological and ocean conditions is presented for use in offshore wind energy research and design. The original data are from 23 ocean sites around the USA and were obtained from the National Data Buoy Center run by the National Oceanic and Atmospheric Administration. The data are presented in a processed form that includes the variables of interest for offshore wind energy design: wind speed, significant wave height, wave peak-spectral period, wind direction and wave direction. For each site, a binning process is conducted to create conditional probability functions for each of these variables. The sites are then grouped according to geographic location and combined to create three representative sites, including a West Coast site, an East Coast site and a Gulf of Mexico site. Both the processed data and the probability distribution parameters for the individual and representative sites are being hosted on a publicly available domain by the National Renewable Energy Laboratory, with the intent of providing a standard basis of comparison for meteorological and ocean conditions for offshore wind energy research worldwide.

17 WIND ENERGY↗

Distributed non-negative matrix factorization with determination of the number of latent features

The holistic analysis and understanding of the latent (that is, not directly observable) variables and patterns buried in large datasets is crucial for data-driven science, decision making and emergency response. Such exploratory analyses require devising unsupervised learning methods for data mining and extraction of the latent features, and non-negative matrix factorization (NMF) is one of the prominent such methods. NMF is based on compute-intense non-convex constrained minimization, which, for large datasets requires fast and distributed algorithms. However, current parallel implementations of NMF fail to estimate the number of latent features. In practice, identifying these features is both difficult and significant for pattern recognition and latent feature analysis, especially for large dense matrices. Here, we introduce a distributed NMF algorithm coupled with distributed custom clustering followed by a stability analysis on dense data, which we call DnMFk, to determine the number of latent variables. The results on synthetic data and the classical Swimmer data set demonstrate the accuracy of model determination while scaling nearly linearly across multiple processors for large data. Further, we employ DnMFk to determine the number of hidden features from a terabyte matrix.

97 MATHEMATICS AND COMPUTING↗

Life Cycle Analysis of Natural Gas Supply Chain and End Use Applications in the United States

Natural gas (NG) plays a crucial role in current and future energy systems in the United States due to its abundance and affordability. In this study a life cycle analysis of the NG supply chain in the United States was conducted using Argonne's R&D GREET model, examining stages from recovery to distribution using reported field data processed and documented by National Energy Technology Laboratory. Supply chain emissions were evaluated across multiple spatial scales, including national average, overall regional production, region-to-region, and basin-to-region scenarios. The GHG intensity of the U.S. average NG supply chain was estimated at 10.3 kg CO 2 e/MMBtu (lower heating value), with a range across regions from 7.8 kg CO 2 e/MMBtu (Northeast) to 15.1 kg CO 2 e/MMBtu (Pacific). The analysis further assessed how upstream NG emissions influence the life cycle GHG emissions of key end-use applications, including electricity generation (0.044–0.086 kg CO 2 e/kWh from upstream NG in combined cycle facilities), hydrogen production (1.04–2.20 kg CO 2 e/kg H 2 for steam methane reforming [SMR] and 1.06–2.23 kg CO 2 e/kg for autothermal reforming [ATR]), and transit bus operation utilizing compressed natural gas fuel (0.19–0.37 kg CO 2 e/mile) and hydrogen fuel (0.12–0.25 kg CO 2 e/mile for hydrogen produced in SMR and ATR).

compression↗

Extra dimensions: 3D in PDF documentation

Experimental science is replete with multi-dimensional information which is often poorly represented by the two dimensions of presentation slides and print media. Past efforts to disseminate such information to a wider audience have failed for a number of reasons, including a lack of standards which are easy to implement and have broad support. Adobe's Portable Document Format (PDF) has in recent years become the de facto standard for secure, dependable electronic information exchange. It has done so by creating an open format, providing support for multiple platforms and being reliable and extensible. By providing support for the ECMA standard Universal 3D (U3D) file format in its free Adobe Reader software, Adobe has made it easy to distribute and interact with 3D content. By providing support for scripting and animation, temporal data can also be easily distributed to a wide, non-technical audience. We discuss how the field of radiation imaging could benefit from incorporating full 3D information about not only the detectors, but also the results of the experimental analyses, in its electronic publications. In this article, we present examples drawn from high-energy physics, mathematics and molecular biology which take advantage of this functionality. Furthermore, we demonstrate how 3D detector elements can be documented, using either CAD drawings or other sources such as GEANT visualizations as input.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗