Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16

Mission operations update for the restructured Earth Observing System (EOS) mission

The National Aeronautics and Space Administration's (NASA) Earth Observing System (EOS) will provide a comprehensive long term set of observations of the Earth to the Earth science research community. The data will aid in determining global changes caused both naturally and through human interaction. Understanding man's impact on the global environment will allow sound policy decisions to be made to protect our future. EOS is a major component of the Mission to Planet Earth program, which is NASA's contribution to the U.S. Global Change Research Program. EOS consists of numerous instruments on multiple spacecraft and a distributed ground system. The EOS Data and Information System (EOSDIS) is the major ground system developed to support EOS. The EOSDIS will provide EOS spacecraft command and control, data processing, product generation, and data archival and distribution services for EOS spacecraft. Data from EOS instruments on other Earth science missions (e.g., Tropical Rainfall Measuring Mission (TRMM)) will also be processed, distributed, and archived in EOSDIS. The U.S. and various International Partners (IP) (e.g., the European Space Agency (ESA), the Ministry of International Trade and Industry (MITI) of Japan, and the Canadian Space Agency (CSA)) participate in and contribute to the international EOS program. The EOSDIS will also archive processed data from other designated NASA Earth science missions (e.g., UARS) that are under the broad umbrella of Mission to Planet Earth.

Kelly, Angelita Castro↗

CMS Tier-0 data processing during the detector commissioning in Run 3

The CMS Tier-0 system is responsible for the prompt processing and distribution of the data collected by the CMS Experiment. A number of upgrades were implemented during the long shutdown 2 of the Large Hadron Collider, which improved the performance and reliability of the system. In this report, these upgrades are discussed and a more detailed description of the Tier-0 system is given. The experience of the data taking during Run 3 detector commissioning as well as the performance of the system are highlighted.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Ocean color - Availability of the global data set

The use of satellite observations of ocean color to provide reliable estimates of marine phytoplankton biomass on synoptic scales is examined. An overview is given of the Coastal Zone Color Scanner data processing system. The archiving and distribution of ocean color data are discussed, and NASA-sponsored archive sites are listed.

Feldman, Gene↗

Characterizing the LANDSAT Global Long-Term Data Record

The effects of global climate change are fast becoming politically, sociologically, and personally important: increasing storm frequency and intensity, lengthening cycles of drought and flood, expanding desertification and soil salinization. A vital asset in the analysis of climate change on a global basis is the 34-year record of Landsat imagery. In recognition of its increasing importance, a detailed analysis of the Landsat observation coverage within the US archive was commissioned. Results to date indicate some unexpected gaps in the US-held archive. Fortunately, throughout the Landsat program, data have been downlinked routinely to International Cooperator (IC) ground stations for archival, processing, and distribution. These IC data could be combined with the current US holdings to build a nearly global, annual observation record over this 34-year period. Today, we have inadequate information as to which scenes are available from which IC archives. Our best estimate is that there are over four million digital scenes in the IC archives, compared with the nearly two million scenes held in the US archive. This vast pool of Landsat observations needs to be accurately documented, via metadata, to determine the existence of complementary scenes and to characterize the potential scope of the global Landsat observation record. Of course, knowing the extent and completeness of the data record is but the first step. It will be necessary to assure that the data record is easy to use, internally consistent in terms of calibration and data format, and fully accessible in order to fully realize its potential.

Arvidson, T.↗

The development of a post-test diagnostic system for rocket engines

An effort was undertaken by NASA to develop an automated post-test, post-flight diagnostic system for rocket engines. The automated system is designed to be generic and to automate the rocket engine data review process. A modular, distributed architecture with a generic software core was chosen to meet the design requirements. The diagnostic system is initially being applied to the Space Shuttle Main Engine data review process. The system modules currently under development are the session/message manager, and portions of the applications section, the component analysis section, and the intelligent knowledge server. An overview is presented of a rocket engine data review process, the design requirements and guidelines, the architecture and modules, and the projected benefits of the automated diagnostic system.

Zakrajsek, June F.↗

The development of a post-test diagnostic system for rocket engines

An effort was undertaken by NASA to develop an automated post-test, post-flight diagnostic system for rocket engines. The automated system is designed to be generic and to automate the rocket engine data review process. A modular, distributed architecture with a generic software core was chosen to meet the design requirements. The diagnostic system is initially being applied to the Space Shuttle Main Engine data review process. The system modules currently under development are the session/message manager, and portions of the applications section, the component analysis section, and the intelligent knowledge server. An overview is presented of a rocket engine data review process, the design requirements and guidelines, the architecture and modules, and the projected benefits of the automated diagnostic system.

Zakrajsek, June F.↗

MDLoader: A Hybrid Model-Driven Data Loader for Distributed Graph Neural Network Training

Scalable data management is essential for processing large scientific dataset on HPC platforms for distributed deep learning. In-memory distributed storage is preferred for its speed, enabling rapid, random, and frequent data access required by stochastic optimizers. Processes use one-sided or collective communication to fetch remote data, with optimal performance depending on (i) dataset characteristics, (ii) training scale, and (iii) interconnection network. Empirical analysis shows collective communication excels with larger mini-batch sizes and/or fewer processes, whereas one-sided communication outperforms at larger scales. We propose MDLoader, a hybrid in-memory data loader for distributed graph neural network training. MDLoader features a model-driven performance estimator that dynamically selects between one-sided and collective communication at the beginning of training using Tree of Parzen Estimators (TPE). Evaluations on NERSC Perlmutter and OLCF Summit show MDLoader outperforms single-backend loaders by up to 2.83 × and predicts the suitable communication method with 96.3% (Perlmutter) and 94.3% (Summit) success rate.

Bae, Jonghyun↗

Passive microwave data for snow and ice research - Planned products from the DMSP SSM/I system

Recommendations which have been made for processing and distributing passive microwave data for snow and ice research obtained with the Defense Meteorological Satellite Program (DMSP) Special Sensor Microwave Imager (SSM/I) are discussed. The general objectives for SSM/I data are reviewed, and the sensor and data flow are described. The SSM/I sea ice products are discussed, and algorithm/product validation is addressed. Proposed services and implementation after SSM/I launch are summarized.

Weawer, Ronald↗

Software-Based Challenges of Developing the Future Distribution Grid

The software that the utility industry currently uses may be insufficient to analyze the distribution grid as it rapidly modernizes to include active resources such as distributed generation, switch and voltage control, automation, and increasingly complex loads. Although planners and operators have traditionally viewed the distribution grid as a passive load, utilities and consultants increasingly need enhanced analysis that incorporates active distribution grid loads in order to ensure grid reliability. Numerous commercial and open-source tools are available for analyzing distribution grid systems. These tools vary in complexity from providing basic load-flow and capacity analysis under steady-state conditions to time-series analysis and even geographical representations of dynamic and transient events. The need for each type of analysis is not well understood in the industry, nor are the reasons that distribution analysis requires different techniques and tools both from those now available and from those used for transmission analysis. In addition, there is limited understanding of basic capability of the tools and how they should be practically applied to the evolving distribution system. The study reviews the features and state of the art capability of current tools, including usability and visualization, basic analysis functionality, advanced analysis including inverters, and renewable generation and load modeling. We also discuss the need for each type of distribution grid system analysis. In addition to reviewing basic functionality current models, we discuss dynamics and transient simulation in detail and draw conclusions about existing software?s ability to address the needs of the future distribution grid as well as the barriers to modernization of the distribution grid that are posed by the current state of software and model development. Among our conclusions are that accuracy, data transfer, and data processing abilities are key to future distribution grid modeling, and measured data sources are a key missing element . Modeling tools need to be calibrated based on measured grid data to validate their output in varied conditions such as high renewables penetration and rapidly changing topology. In addition, establishing a standardized data modeling format would enable users to transfer data among tools to take advantage of different analysis features. ?

99 GENERAL AND MISCELLANEOUS↗

NASA's Earth Science Flight Program Meets the Challenges of Today and Tomorrow

NASA's Earth science flight program is a dynamic undertaking that consists of a large fleet of operating satellites, an array of satellite and instrument projects in various stages of development, a robust airborne science program, and a massive data archiving and distribution system. Each element of the flight program is complex and present unique challenges. NASA builds upon its successes and learns from its setbacks to manage this evolving portfolio to meet NASA's Earth science objectives. NASA fleet of 16 operating missions provide a wide range of scientific measurements made from dedicated Earth science satellites and from instruments mounted to the International Space Station. For operational missions, the program must address issues such as an aging satellites operating well beyond their prime mission, constellation flying, and collision avoidance with other spacecraft and orbital debris. Projects in development are divided into two broad categories: systematic missions and pathfinders. The Earth Systematic Missions (ESM) include a broad range of multi-disciplinary Earth-observing research satellite missions aimed at understanding the Earth system and its response to natural and human-induced forces and changes. Understanding these forces will help determine how to predict future changes, and how to mitigate or adapt to these changes. The Earth System Science Pathfinder (ESSP) program provides frequent, regular, competitively selected Earth science research opportunities that accommodate new and emerging scientific priorities and measurement capabilities. This results in a series of relatively low-cost, small-sized investigations and missions. Principal investigators whose scientific objectives support a variety of studies lead these missions, including studies of the atmosphere, oceans, land surface, polar ice regions, or solid Earth. This portfolio of missions and investigations provides opportunity for investment in innovative Earth science that enhances NASA's capability for better understanding the current state of the Earth system. ESM and ESSP projects often involve partnerships with other US agencies and/or international organizations. This adds to the complexity of mission development, but allows for a greater scientific return on NASA's investments. The Earth Science Airborne Science Program provides manned and unmanned aircraft systems that further science and advance the use of satellite data. NASA uses these assets worldwide in campaigns to investigate extreme weather events, observe Earth system processes, obtain data for Earth science modeling activities, and calibrate instruments flying aboard Earth science spacecraft. The Airborne Science Program has six dedicated aircraft and access to many other platforms. The Earth Science Multi-Mission Operations program acquires, preserves, and distributes observational data from operating spacecraft to support Earth Science research focus areas. The Earth Observing System Data and Information System (EOSDIS), which has been in operations since 1994, primarily accomplishes this. EOSDIS acquires, processes, archives, and distributes Earth Science data and information products. The archiving of NASA Earth Science information happens at eight Distributed Active Archive Centers (DAACs) and four disciplinary data centers located across the United States. The DAACs specialize by topic area, and make their data available to researchers around the world. The DAACs currently house over 9 petabytes of data, growing at a rate of 6.4 terabytes per day. NASA's current Earth Science portfolio is responsive to the National Research Council (NRC) 2007 Earth Science Decadal Survey and well as the 2010 NASA Response to President Obama's Climate Plan. As the program evolves into the future it will leverage the lessons learned from the current missions in operations and development, and plan for adjustments to future objectives in response to the anticipated 2017 NRC Decadal Survey.

Flight Program↗

NASA’s Unsteady Pressure-Sensitive Paint Research and Operational Capability Developments

To address challenges in the field of unsteady aerodynamics, NASA has developed a new state-of-the-art capability called Unsteady Pressure-Sensitive Paint (uPSP). It has been developed as an operational surface-pressure measurement capability for deployment in NASA’s AETC ground test facilities. Offering unprecedented spatiotemporal resolution, uPSP is an extremely powerful tool for investigating unsteady separated flows. To accelerate scientific discovery, uPSP data is to be processed and distributed as rapidly as it is acquired. Several demonstrations of the technology collected valuable data that has been used to develop data products, a robust processing pipeline, and other computational capabilities. A collection of papers documents the most recent research and development work on the uPSP technology and capability. This paper summarizes the current state of this effort at NASA.

Pressure-Sensitive Paint↗

NASA’s Unsteady Pressure-Sensitive Paint Research and Operational Capability Developments

To address challenges in the field of unsteady aerodynamics, NASA has developed a new stateof-the-art capability called Unsteady Pressure-Sensitive Paint (uPSP). It has been developed as an operational surface-pressure measurement capability for deployment in NASA’s AETC ground test facilities. Offering unprecedented spatiotemporal resolution, uPSP is an extremely powerful tool for investigating unsteady separated flows. To accelerate scientific discovery, uPSP data is to be processed and distributed as rapidly as it is acquired. Several demonstrations of the technology collected valuable data that has been used to develop data products, a robust processing pipeline, and other computational capabilities. A collection of papers documents the most recent research and development work on the uPSP technology and capability. This paper summarizes the current state of this effort at NASA.

Pressure-Sensitive Paint↗

Performance analysis of data intensive cloud systems based on data management and replication: a survey

As we delve deeper into the ‘Digital Age’, we witness an explosive growth in the volume, velocity, and variety of the data available on the Internet. For example, in 2012 about 2.5 quintillion bytes of data was created on a daily basis that originated from myriad of sources and applications including mobiledevices, sensors, individual archives, social networks, Internet of Things, enterprises, cameras, software logs, etc. Such ‘Data Explosions’ has led to one of the most challenging research issues of the current Information and Communication Technology era: how to optimally manage (e.g., store, replicated, filter, and the like) such large amount of data and identify new ways to analyze large amounts of data for unlocking information. It is clear that such large data streams cannot be managed by setting up on-premises enterprise database systems as it leads to a large up-front cost in buying and administering the hardware and software systems. Therefore, next generation data management systems must be deployed on cloud. The cloud computing paradigm provides scalable and elastic resources, such as data and services accessible over the Internet Every Cloud Service Provider must assure that data is efficiently processed and distributed in a way that does not compromise end-users’ Quality of Service (QoS) in terms of data availability, data search delay, data analysis delay, and the like. In the aforementioned perspective, data replication is used in the cloud for improving the performance (e.g., read and write delay) of applications that access data. Through replication a data intensive application or system can achieve high availability, better fault tolerance, and data recovery. In this paper, we survey data management and replication approaches (from 2007 to 2011) that are developed by both industrial and research communities. The focus of the survey is to discuss and characterize the existing approaches of data replication and management that tackle the resource usage and QoS provisioning with different levels of efficiencies. Moreover, the breakdown of both influential expressions (data replication and management) to provide different QoS attributes is deliberated. Furthermore, the performance advantages and disadvantages of data replication and management approaches in the cloud computing environments are analyzed. Open issues and future challenges related to data consistency, scalability, load balancing, processing and placement are also reported.

Malik, Saif Ur Rehman↗

CMS Data transfer operations after the first years of LHC collisions

CMS experiment utilizes distributed computing infrastructure and its performance heavily depends on the fast and smooth distribution of data between different CMS sites. Data must be transferred from the Tier-0 (CERN) to the Tier-1s for processing, storing and archiving, and time and good quality are vital to avoid overflowing CERN storage buffers. At the same time, processed data has to be distributed from Tier-1 sites to all Tier-2 sites for physics analysis while Monte Carlo simulations sent back to Tier-1 sites for further archival. At the core of all transferring machinery is PhEDEx (Physics Experiment Data Export) data transfer system. It is very important to ensure reliable operation of the system, and the operational tasks comprise monitoring and debugging all transfer issues. Based on transfer quality information Site Readiness tool is used to create plans for resources utilization in the future. We review the operational procedures created to enforce reliable data delivery to CMS distributed sites all over the world. Additionally, we need to keep data and meta-data consistent at all sites and both on disk and on tape. In this presentation, we describe the principles and actions taken to keep data consistent on sites storage systems and central CMS Data Replication Database (TMDB/DBS) while ensuring fast and reliable data samples delivery of hundreds of terabytes to the entire CMS physics community.

Kaselis, Rapolas↗

NJOY2016

NJOY2016 is used to convert evaluations in the Evaluated Nuclear Data Files (ENDF) format into forms useful for practical applications such as fission and fusion reactor analysis, criticality safety, radiation shielding, nuclear waste management, nuclear medicine procedures, and more. It begins with the generation of pointwise libraries, including reaction and resonance reconstruction, Doppler broadening, radiation heating and damage, thermal scattering data, unresolved resonance data, and gas production. It reviews the production of libraries for the continuous-energy Monte Carlo code MCNP, multigroup neutron, photon, and particle cross sections and matrices, and photon interaction data. Uncertainty information for ENDF data is processed, including the capability for calculating covariances of resonance data, angular distributions, energy distributions, and radioactive nuclide production. NJOY can prepare and process thermal scattering data evaluations into application libraries. It can also plot cross sections, angular distributions and secondary emission spectra.

Kahler III, A. C. (Skip)↗

NJOY2016

NJOY2016 is used to convert evaluations in the Evaluated Nuclear Data Files (ENDF) format into forms useful for practical applications such as fission and fusion reactor analysis, criticality safety, radiation shielding, nuclear waste management, nuclear medicine procedures, and more. It begins with the generation of pointwise libraries, including reaction and resonance reconstruction, Doppler broadening, radiation heating and damage, thermal scattering data, unresolved resonance data, and gas production. It reviews the production of libraries for the continuous-energy Monte Carlo code MCNP, multigroup neutron, photon, and particle cross sections and matrices, and photon interaction data. Uncertainty information for ENDF data is processed, including the capability for calculating covariances of resonance data, angular distributions, energy distributions, and radioactive nuclide production. NJOY can prepare and process thermal scattering data evaluations into application libraries. It can also plot cross sections, angular distributions and secondary emission spectra.

Kahler III, A. C. (Skip)↗

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly.

Lynnes, Christopher↗