Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20

Density estimation via measure transport: Outlook for applications in the biological sciences

Abstract One among several advantages of measure transport methods is that they allow or a unified framework for processing and analysis of data distributed according to a wide class of probability measures. Within this context, we present results from computational studies aimed at assessing the potential of measure transport techniques, specifically, the use of triangular transport maps, as part of a workflow intended to support research in the biological sciences. Scenarios characterized by the availability of limited amount of sample data, which are common in domains such as radiation biology, are of particular interest. We find that when estimating a distribution density function given limited amount of sample data, adaptive transport maps are advantageous. In particular, statistics gathered from computing series of adaptive transport maps, trained on a series of randomly chosen subsets of the set of available data samples, leads to uncovering information hidden in the data. As a result, in the radiation biology application considered here, this approach provides a tool for generating hypotheses about gene relationships and their dynamics under radiation exposure.

gene expression data↗

The creation of a comprehensive metocean data set for offshore wind turbine simulations: Comprehensive metocean data set

A database of meteorological and ocean conditions is presented for use in offshore wind energy research and design. The original data are from 23 ocean sites around the USA and were obtained from the National Data Buoy Center run by the National Oceanic and Atmospheric Administration. The data are presented in a processed form that includes the variables of interest for offshore wind energy design: wind speed, significant wave height, wave peak-spectral period, wind direction and wave direction. For each site, a binning process is conducted to create conditional probability functions for each of these variables. The sites are then grouped according to geographic location and combined to create three representative sites, including a West Coast site, an East Coast site and a Gulf of Mexico site. Both the processed data and the probability distribution parameters for the individual and representative sites are being hosted on a publicly available domain by the National Renewable Energy Laboratory, with the intent of providing a standard basis of comparison for meteorological and ocean conditions for offshore wind energy research worldwide.

17 WIND ENERGY↗

Distributed non-negative matrix factorization with determination of the number of latent features

The holistic analysis and understanding of the latent (that is, not directly observable) variables and patterns buried in large datasets is crucial for data-driven science, decision making and emergency response. Such exploratory analyses require devising unsupervised learning methods for data mining and extraction of the latent features, and non-negative matrix factorization (NMF) is one of the prominent such methods. NMF is based on compute-intense non-convex constrained minimization, which, for large datasets requires fast and distributed algorithms. However, current parallel implementations of NMF fail to estimate the number of latent features. In practice, identifying these features is both difficult and significant for pattern recognition and latent feature analysis, especially for large dense matrices. Here, we introduce a distributed NMF algorithm coupled with distributed custom clustering followed by a stability analysis on dense data, which we call DnMFk, to determine the number of latent variables. The results on synthetic data and the classical Swimmer data set demonstrate the accuracy of model determination while scaling nearly linearly across multiple processors for large data. Further, we employ DnMFk to determine the number of hidden features from a terabyte matrix.

97 MATHEMATICS AND COMPUTING↗

Life Cycle Analysis of Natural Gas Supply Chain and End Use Applications in the United States

Natural gas (NG) plays a crucial role in current and future energy systems in the United States due to its abundance and affordability. In this study a life cycle analysis of the NG supply chain in the United States was conducted using Argonne's R&D GREET model, examining stages from recovery to distribution using reported field data processed and documented by National Energy Technology Laboratory. Supply chain emissions were evaluated across multiple spatial scales, including national average, overall regional production, region-to-region, and basin-to-region scenarios. The GHG intensity of the U.S. average NG supply chain was estimated at 10.3 kg CO 2 e/MMBtu (lower heating value), with a range across regions from 7.8 kg CO 2 e/MMBtu (Northeast) to 15.1 kg CO 2 e/MMBtu (Pacific). The analysis further assessed how upstream NG emissions influence the life cycle GHG emissions of key end-use applications, including electricity generation (0.044–0.086 kg CO 2 e/kWh from upstream NG in combined cycle facilities), hydrogen production (1.04–2.20 kg CO 2 e/kg H 2 for steam methane reforming [SMR] and 1.06–2.23 kg CO 2 e/kg for autothermal reforming [ATR]), and transit bus operation utilizing compressed natural gas fuel (0.19–0.37 kg CO 2 e/mile) and hydrogen fuel (0.12–0.25 kg CO 2 e/mile for hydrogen produced in SMR and ATR).

compression↗

Extra dimensions: 3D in PDF documentation

Experimental science is replete with multi-dimensional information which is often poorly represented by the two dimensions of presentation slides and print media. Past efforts to disseminate such information to a wider audience have failed for a number of reasons, including a lack of standards which are easy to implement and have broad support. Adobe's Portable Document Format (PDF) has in recent years become the de facto standard for secure, dependable electronic information exchange. It has done so by creating an open format, providing support for multiple platforms and being reliable and extensible. By providing support for the ECMA standard Universal 3D (U3D) file format in its free Adobe Reader software, Adobe has made it easy to distribute and interact with 3D content. By providing support for scripting and animation, temporal data can also be easily distributed to a wide, non-technical audience. We discuss how the field of radiation imaging could benefit from incorporating full 3D information about not only the detectors, but also the results of the experimental analyses, in its electronic publications. In this article, we present examples drawn from high-energy physics, mathematics and molecular biology which take advantage of this functionality. Furthermore, we demonstrate how 3D detector elements can be documented, using either CAD drawings or other sources such as GEANT visualizations as input.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

PNNL-Comp-Mass-Spec/LIQUID

The OMICS group at PNNL has developed a wide range of software utilities over the last 15+ years. These tools are used to manage data files, transform data, visualize data, and automate data processing. The software is distributed on GitHub as open source, typically under the Apache 2.0 license agreement. See https://github.com/PNNL-Comp-Mass-Spec Lipid analysis tool developed by PNNL

Monroe, Matthew↗

Final Technical Report

During the 1-year duration of this project a new Shortwave Spectrometer (SWS) was designed and developed for deployment at the Southern Great Plains Central Facility to measure zenith solar spectral radiance. The SWS is comprised of two Zeiss miniature monolithic spectrometers (MMS-1 and MMS-NIR) for visible and near-infrared detection in the wavelength range between 350 and 2250 nm. Spectral resolution is 8 nm for the MMS-1 and 12 nm for the MMS-NIR. The light collector is a narrow field of view (±1.5 º) collimator at the front end of a high-grade custom-made fiber optic bundle. The data acquisition and control system is a 933 MHz Pentium based PC in a PC104 format with a USB interface between the computer and the spectrometers. Spectral sampling rate is approximately 1 Hz. A prototype SWS was deployed at SGP in November and December 2004 and it collected zenith-sky solar spectra at 1 Hz continuously over a 29 day period. Prior to deployment it was calibrated and characterized at the NASA Ames Airborne Sensor Facility (ASF) using a 30 inch Integrating Sphere. The SWS was also calibrated using a portable 12 inch integrating sphere at the Central Facility. The testing and calibration procedures were developed during this implementation. The planning and scheduling for permanent installation of the new SWS as well as data processing, calibration, archiving, and distribution was conducted.

47 OTHER INSTRUMENTATION↗

A Dynamic Landsat Derived Normalized Difference Vegetation Index (NDVI) Product for the Conterminous United States

Satellite derived vegetation indices (VIs) are broadly used in ecological research, ecosystem modeling, and land surface monitoring. The Normalized Difference Vegetation Index (NDVI), perhaps the most utilized VI, has countless applications across ecology, forestry, agriculture, wildlife, biodiversity, and other disciplines. Calculating satellite derived NDVI is not always straight-forward, however, as satellite remote sensing datasets are inherently noisy due to cloud and atmospheric contamination, data processing failures, and instrument malfunction. Readily available NDVI products that account for these complexities are generally at coarse resolution; high resolution NDVI datasets are not conveniently accessible and developing them often presents numerous technical and methodological challenges. Here, we address this deficiency by producing a Landsat derived, high resolution (30 m), long-term (30+ years) NDVI dataset for the conterminous United States. We use Google Earth Engine, a planetary-scale cloud-based geospatial analysis platform, for processing the Landsat data and distributing the final dataset. We use a climatology driven approach to fill missing data and validate the dataset with established remote sensing products at multiple scales. We provide access to the composites through a simple web application, allowing users to customize key parameters appropriate for their application, question, and region of interest.

42 ENGINEERING↗

International Ultraviolet Explorer Observatory operations

This volume contains the final report for the International Ultraviolet Explorer IUE Observatory Operations contract. The fundamental operational objective of the International Ultraviolet Explorer (IUE) program is to translate competitively selected observing programs into IUE observations, to reduce these observations into meaningful scientific data, and then to present these data to the Guest Observer in a form amenable to the pursuit of scientific research. The IUE Observatory is the key to this objective since it is the central control and support facility for all science operations functions within the IUE Project. In carrying out the operation of this facility, a number of complex functions were provided beginning with telescope scheduling and operation, proceeding to data processing, and ending with data distribution and scientific data analysis. In support of these critical-path functions, a number of other significant activities were also provided, including scientific instrument calibration, systems analysis, and software support. Routine activities have been summarized briefly whenever possible.

Source record↗

Development and analysis of SCR requirements tables for system scenarios

We describe the use of scenarios to develop and refine requirement tables for parts of the Earth Observing System Data and Information System (EOSDIS). The National Aeronautics and Space Administration (NASA) is developing EOSDIS as part of its Mission-To-Planet-Earth (MTPE) project to accept instrument/platform observation requests from end-user scientists, schedule and perform requested observations of the Earth from space, collect and process the observed data, and distribute data to scientists and archives. Current requirements for the system are managed with tools that allow developers to trace the relationships between requirements and other development artifacts, including other requirements. In addition, the user community (e.g., earth and atmospheric scientists), in conjunction with NASA, has generated scenarios describing the actions of EOSDIS subsystems in response to user requests and other system activities. As part of a research effort in verification and validation techniques, this paper describes our efforts to develop requirements tables from these scenarios for the EOSDIS Core System (ECS). The tables specify event-driven mode transitions based on techniques developed by the Naval Research Lab's (NRL) Software Cost Reduction (SCR) project. The SCR approach has proven effective in specifying requirements for large systems in an unambiguous, terse format that enhance identification of incomplete and inconsistent requirements. We describe development of SCR tables from user scenarios and identify the strengths and weaknesses of our approach in contrast to the requirements tracing approach. We also evaluate the capabilities of both approach to respond to the volatility of requirements in large, complex systems.

Callahan, John R.↗

Research and Development in Very Long Baseline Interferometry (VLBI)

Contents include the following: 1.Observation coordination. 2. Data acquisition system control software. 3. Station support. 4. Correlation, data processing, and analysis. 5. Data distribution and archiving. 6. Technique improvement and research. 7. Computer support.

Himwich, William E.↗

ASF RADARSAT Processor System Performance Summary

The Alaska SAR Facility (ASF) located at the University of Alaska Fairbanks (UAF) has been in operations since 1991 serving as a key data acquisition, processing, archive and distribution center for a number of polar orbiting SAR (synthetic aperture radar) satelittes including ERS 1/2 and JERS-1.

synthetic aperture radar RADARSAT Mapping↗

Small Satellites for NASA Earth Science

NASA’s Earth Science Division (ESD) seeks to develop a scientific understanding of Earth and its response to natural and human-induced changes. Earth is a system comprised of diverse components interacting in complex ways. Understanding Earth’s atmosphere, surface and interior, oceans and surface water, ice and snow, and life as a single connected system is necessary in order to improve our predictions of climate, weather, and natural hazards. The ESD’s Flight Program consists of a coordinated series of satellite and airborne systems providing long and short-term, global and regional observations. In addition, the Flight Program provides infrastructure for operating these missions, processing their scientific data, and distributing them on a free and open basis to researchers, operational users, and the public. The Flight Program currently has 24 operating Earth observing space missions and instruments. There are 18 more missions and instruments planned for launch over the next five years. These comprise missions recommended by the National Academies 2017 Earth Science Decadal Survey, missions and selected instruments to ensure availability of key climate data sets, operational missions to sustain the land imaging provided by the Landsat system, and small-sized competitively selected orbital and instrument missions of opportunity belonging to the Earth Venture (EV) program. The Earth Science Decadal Survey, released in early 2018, recommended four new Flight Program elements in addition to the above activities that comprise the Program of Record (POR). Small satellites (~500 kg or less) are essential components of these activities. Presently, there is an increasing use of micro and nanosatellites (or CubeSats) in constellations to support NASA ESD’s scientific objectives. These include the Cyclone Global Navigation Satellite System (CYGNSS) for observing tropical cyclone intensification and genesis factors, the Timed-Resolved Observations of Precipitation structure and storm Intensity with a Constellation of Smallsats (TROPICS) mission, and the Polar Radiant Energy in the Far InfraRed Experiment (PREFIRE) CubeSat mission. ESD small satellite initiatives like the Small Satellite Constellation Data Buy and Venture Class Launch Services (VCLS) are also underway. The Earth Science Technology Office’s (ESTO) In-Space Validation of Earth Science Technologies (InVEST) and the Venture Technology program elements have launched seven 3U and 6U CubeSat missions to validate advanced instruments and related technologies. An equivalent number of InVEST and other technology demonstration CubeSats are being prepared for launch in the next year. An overview of plans and current status including topics related to small satellite enabling activities will be presented.

Neeck, Steven P.↗

Event recognition using signal spectrograms in long pulse experiments

As discharge duration increases, real-time complex analysis of the signal becomes more important. In this context, data acquisition and processing systems must provide models for designing experiments which use event oriented plasma control. One example of advanced data analysis is signal classification. The off-line statistical analysis of a large number of discharges provides information to develop algorithms for the determination of the plasma parameters from measurements of magnetohydrodinamic waves, for example, to detect density fluctuations induced by the Alfven cascades using morphological patterns. The need to apply different algorithms to the signals and to address different processing algorithms using the previous results necessitates the use of an event-based experiment. The Intelligent Test and Measurement System platform is an example of architecture designed to implement distributed data acquisition and real-time processing systems. The processing algorithm sequence is modeled using an event-based paradigm. The adaptive capacity of this model is based on the logic defined by the use of state machines in SCXML. The Intelligent Test and Measurement System platform mixes a local multiprocessing model with a distributed deployment of services based on Jini.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

SWARM: Reimagining scientific workflow management systems in a distributed world

Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. In conclusion, this paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.

Swarm intelligence↗