Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15

An Efficient Storage-Driven Machine Learning Model for Performance in the Era of Multimodal Scientific Data

Scientific workflows are increasingly relying on machine learning (ML), simulation, and hybrid techniques to predict, understand, and optimize the behavior of complex experiments. High-performance computing has greatly improved researchers’ ability to acquire diverse data modalities in these workflows. Recent studies suggest that the performance of machine learning models can be improved by integrating data from various sources. Unfortunately, these workloads pose unprecedent pressure on the network storage to meet the demands associated with accessing these multimodal data. To mitigate the impact of intensive IO, we propose a solution that utilizes a multi-tier High-Performance Computing (HPC) distributed storage and data processing framework, placing computation where the data resides for better performance. By adopting this project, the scientific community will gain new opportunities to explore multimodal storage-driven possibilities, integrating multiple scientific data sources with advanced streaming frameworks. Additionally, our framework effectively utilizes computing resources and bridges the gaps identified by HPC experts. Our proposed approach tackles scalability and persistence challenges by leveraging native persistency, which has posed difficulties in traditional approaches. Furthermore, we seek to enhance fault-tolerance and load-balance of computations by leveraging real-time streaming in diverse scientific computing environments, thereby propelling advanced scientific computing research into the next generation.

97 MATHEMATICS AND COMPUTING↗

Data collection system: Earth Resources Technology Satellite-1

Subjects covered at the meeting concerned results on the overall data collection system including sensors, interface hardware, power supplies, environmental enclosures, data transmission, processing and distribution, maintenance and integration in resources management systems.

Cooper, S.↗

The McDonald Observatory lunar laser ranging project

A summary of the activities of the McDonald lunar laser ranging station at Fort Davis for the FY 77-78 fiscal year is presented. The lunar laser experiment uses the observatory 2.7m reflecting telescope on a thrice-per-day, 21-day-per-lunation schedule. Data are recorded on magnetic tapes and sent to the University of Texas at Austin where the data is processed. After processing, the data is distributed to interested analysis centers and later to the National Space Science Data Center where it is available for routine distribution. Detailed reports are published on the McDonald operations after every fourth lunation or approximately once every 115 days. These reports contain a day-by-day documentation of the ranging activity, detailed discussions of the equipment development efforts, and an abundance of other information as is needed to document and archive this important data type.

Silverberg, E. C.↗

A concept for a transparent data acquisition and distribution system for spaceflight applications

The paper describes the Modular Data System concept of the NASA End-to-End Data System program. Data from a single instrument along with the necessary ancillary data will be assembled into self-contained packets and subsequently transmitted over various communications links (space telemetry channel, ground communications circuits, etc.) to the experimenter's facility in real time; reliable error control coding will be included in each link transmission to protect the integrity of the data packets. A major objective is to make the entire data acquisition and distribution process transparent to the experimenter in the sense that the output terminal of the distribution system will be physically, logically and electrically identical to that of the experiment output channel.

Greene, E. P.↗

CNPQ/INPE LANDSAT system

The current status of the Brazilian LANDSAT facilities is described and main accomplishments are outlined. Receiving, recording, and processing substations and data distribution centers are discussed. Examples of the preliminary TM product produced by the Brazilian station are given.

Parada, N. D. J.↗

B-2 and other current flight test control center system architectures

The topics covered include the following: (1) telemetry processing independent of source; (2) generic system/software; (3) time homogeneity; (4) low latency networks; (5) distributed processing; (6) all data available to all workstations; and (7) recall of data in real time.

Hill, Jerry↗

CCD data acquisition systems at Lick and Keck Observatories

This paper will describe and compare two distinct but related CCD data acquisition systems (DAS) currently under development at Lick and Keck Observatories. Although these two systems have a number of major architectural differences, they share a considerable amount of common hardware and software. Both of these new systems build on a large body of proven software that is the foundation of the existing CCD DAS currently in use at Lick Observatory. Both will provide support for reading up to four on-chip amplifiers per CCD and/or reading out mosaics of CCD chips. In addition, they will provide the capability for interactive, real-time adjustment of CCD waveforms for engineering purposes. Each of these two systems is composed of three major subsystems: (1) an instrument computer and its software; (2) a data capture computer and its software; and (3) a CCD/dewar controller and its software. The instrument computer is a Unix workstation, and the functions it provides include user interfaces, the interactive real-time display of CCD images, and the recording of image and FITS header data to disk and/or tape. The data capture computer is responsible for the packaging and high-speed transfer of the CCD pixel data stream into a bulk RAM, and the subsequent transfer of this data to the instrument computer. The CCD/dewar controller generates the waveforms for clocking the CCD, digitizes the pixel data, and transmits it via high-speed link to the data capture computer. It is also responsible for monitoring and controlling the dewar temperature and cryogen levels. Given the number of different types of processors and high-speed data links employed in both systems, a major emphasis of this paper will be on the various forms of interprocessor communications utilized for data transfer and distributed process synchronization.

Kibrick, R. I.↗

Radio-science performance analysis software

The Radio Science Systems Group (RSSG) provides various support functions for several flight project radio-science teams. Among these support functions are uplink and sequence planning, real-time operations monitoring and support, data validation, archiving and distribution functions, and data processing and analysis. This article describes the support functions that encompass radio-science data performance analysis. The primary tool used by the RSSG to fulfill this support function is the STBLTY program set. STBLTY is used to reconstruct observable frequencies and calculate model frequencies, frequency residuals, frequency stability in terms of Allan deviation, reconstructed phase, frequency and phase power spectral density, and frequency drift rates. In the case of one-way data, using an ultrastable oscillator (USO) as a frequency reference, the program set computes the spacecraft transmitted frequency and maintains a database containing the in-flight history of the USO measurements. The program set also produces graphical displays. Some examples and discussions on operating the program set on Galileo and Ulysses data will be presented.

Morabito, D. D.↗

ASF archive issues: Current status, past history, and questions for the future

The Alaska SAR Facility (ASF) collects, processes, archives, and distributes data from synthetic aperture radar (SAR) satellites in support of scientific research. ASF has been in operation since 1991 and presently has an archive of over 100 terabytes of data. ASF is performing an analysis of its magnetic tape storage system to ensure long-term preservation of this archive. Future satellite missions have the possibility of doubling to tripling the amounts of data that ASF acquires. ASF is examining the current data systems and the high volume storage, and exploring future concerns and solutions.

Goula, Crystal A.↗

Cloud Surprises in Moving NASA EOSDIS Applications into Amazon Web Services

NASA ESDIS has been moving a variety of data ingest, distribution, and science data processing applications into a cloud environment over the last 2 years. As expected, there have been a number of challenges in migrating primarily on-premises applications into a cloud-based environment, related to architecture and taking advantage of cloud-based services. What was not expected is a number of issues that were beyond purely technical application re-architectures. We ran into surprising network policy limitations, billing challenges in a government-based cost model, and difficulty in obtaining certificates in an NASA security-compliant manner. On the other hand, this approach has allowed us to move a number of applications from local hosting to the cloud in a matter of hours (yes, hours!!), and our CMR application now services 95% of granule searches and an astonishing 99% of all collection searches in under a second. And most surprising of all, well, you'll just have to wait and see the realization that caught our entire team off guard!

Cloud↗

PBL Height from AIRS, GPS RO, and MERRA-2 Products in NASA GES DISC and Their 10 Year Seasonal Mean Intercomparison

Within the planetary boundary layer (PBL), surface forcing response, drag, turbulence, and vertical mixing are important processes and play a more critical role here than in the overlying “free atmosphere”. The PBL Height (PBLH) is an important parameter in climate models, weather forecasts, and air quality prediction. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) provides data processing, archiving, and distribution services for numerous Earth science products. PBLH is a parameter in three products served by GES DISC, which are from the Atmospheric Infrared Sounder (AIRS), the Global Positioning System (GPS) radio occultation (RO) experiment, and the NASA reanalysis product Modern-Era Retrospective analysis for Research and Applications – 2 (MERRA-2). These products have different spatial and temporal resolutions and coverages, and their PBLH definitions are also different. To better serve the PBL research community, we have summarized the specifications of these products. A ten-year seasonal mean intercomparison is also conducted to provide further guidance to users. The intercomparison results show that MERRA-2 has a much shallower PBL than AIRS and GPS RO. An experimental study indicates the different PBLH definition in MERRA-2 caused smaller values of PBLH. The improvement of the water vapor retrieval in AIRS version 7 over version 6 results in the version 7 PBLH agreeing better with GPS RO and MERRA-2 than version 6, especially near the equator and low latitudes.

Feng Ding↗

SPHEREx: NASA’s Near-Infrared Spectrophotmetric All-Sky Survey

SPHEREx, the Spectro-Photometer for the History of the Universe, Epoch of Reionization, and ices Explorer, isa NASA MIDEX mission planned for launch in 2024. SPHEREx will carry out the first all-sky spectral survey at wavelengths between 0.75μm and 5μm with spectral resolving power ~40 between 0.75 and 3.8μm and ~120 between 3.8 and 5μm At the end of its two-year mission, SPHEREx will provide 14 billion 0.75-to-5μm spectra of each 6."2 x 6."2 pixel on the sky. This paper updates an earlier description of SPHEREx presenting changes made during the mission's Preliminary Design Phase, including a discussion of instrument integration and test flow and a summary of the data processing, analysis, and distribution plans.

Zemcov, Michael↗

Characterizing Wildfires in Western US.: A Cloud-based Case Study for Interdisciplinary Research using NASA Resources

This presentation will demonstrate a case study of interdisciplinary research done in the Amazon Web Services (AWS) cloud platform, in addition to in the local machine. We conduct data analysis next to data by leveraging various cloud-based data in NASA Earthdata Cloud, which are distributed by different missions/NASA Distributed Active Archive Centers (DAACs), and cloud computing resources at NASA. For instance, we directly access multiple datasets stored in the AWS Simple Storage Service (S3) buckets using a Python Jupyter notebook through a JupyterHub interface hosted in AWS (without having to download data), and conduct data analysis next to data in the cloud. We will also show how to share the research results following Open Source policy. This case study characterizes the change in wildfire events in the western United States during the past 20 years. In particular, we focus on the wildfires in California in 2021, one of the most severe wildfire years occurring in the most recent 20 years in California. We will analyze the possible causes of wildfires, such as drought conditions and climate variability, and examine the impacts of wildfires on air quality and atmospheric composition, and on land cover. We will examine the data distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), including aerosols and meteorological data from the NASA Modern-Era Retrospective analysis for Research and Applications version 2 (MERRA-2), precipitation from the Global Precipitation Measurement (GPM) and Global Precipitation Climate Project (GPCP), and aerosol index from Ozone Monitoring Instrument (OMI). We also utilize the data distributed by the Physical Oceanography (PO) DAAC, such as Sea Surface Temperature (SST) data from the Group for High Resolution Sea Surface Temperature (GHRSST), and the data distributed by Land Processes (LP) DAAC, such as Normalized Difference Vegetation Index (NDVI).

Xiaohua Pan↗

Overview of the Astrophysics Data System

The Astrophysics Division of NASA has built a geographically- and logically-distributed heterogeneous information system for the dissemination and coordinated multispectral analysis of data from astrophysics missions. The Astrophysics Data System (ADS) is a truly distributed system in which the data and the required processing are physically distributed. To accommodate the anticipated growth and changes in both requirements and technology, the ADS employs a server/client architecture which allows services and data to be added or replaced without having to change the basic architecture or interfaces. Current datasets accessible through the system include all the tabular astronomical data available at each of six existing astrophysics data centers. Additional data nodes, at both NASA data centers and academic institutions, will be added shortly. The future evolution of the system will be driven in large part by user services mounted both by the ADS project itself and by members of the astrophysics community.

Good, John C.↗

Overview of the Astrophysics Data System

The Astrophysics Division of NASA has built a geographically and logically distributed heterogeneous information system for the dissemination and coordinated multispectral analysis of data from astrophysics missions. The Astrophysics Data System (ADS) is a truly distributed system in which the data and the required processing are physically distributed. To accommodate the anticipated growth and changes in both requirements and technology, the ADS employs a server/client architecture which allows services and data to be added or replaced without having to change the basic architecture or interfaces. Current datasets accessible through the system include all the tabular astronomical data available at each of six existing astrophysics data centers. Additional data nodes, at both NASA data centers and academic institutions, will be added shortly. The future evolution of the system will be driven in large part by user services mounted both by the ADS project itself and by members of the astrophysics community.

Good, John C.↗

Monitoring Volcanic Eruptions by OMI/OMPS, ASTER, and MERRA-2

Volcanic lava flows and/or the gas eruptions are the most common characteristics that can be remotely monitored with satellite technology in the global perspective and on different timescales. Atmospheric Sulfur Dioxide (SO2), one of the most abundant gases from volcanic eruptions apart from atmospheric common gases Carbon Dioxide and water vapor, can be directly detected by space-based sensors on satellites. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is one of the 12 Distributed Active Archive Centers (DAACs) within NASA's Earth Observing System Data and Information System (EOSDIS), which archives SO2 data sets from the Nimbus-7 Total Ozone Mapping Spectrometer (TOMS) in 1978, till the ongoing Ozone Monitoring Instrument (OMI) on NASA's EOS-Aura satellite, the Ozone Mapping Profiler Suite (OMPS) Nadir Mapper (NM) on both the Suomi National Polar-Orbiting Partnership (Suomi-NPP or SNPP) and the Joint Polar-orbiting Satellite System-1 (JPSS-1) satellites, into the future JPSS missions. In addition to the standard OMI/Aura and OMPS/S-NPP SO2 products, SO2 products created under the charter of the Making Earth System Data Records for Use in Research Environments (MEaSUREs) project, are also archived at GES DISC, through which NASA enacts to expand understanding the Earth system using consistent data records. The Land Processes Distributed Active Archive Center (LP DAAC) is another EOSDIS's DAAC that provides land data products and operates as a partnership with the U.S. Geological Survey (USGS). The LP DAAC has been archiving the satellite imagery from the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) aboard NASA's EOS-Terra satellite, a high spatial resolution (15 meters) and 14 band multispectral instrument. The ASTER imagery is one of the land products contributing to the application for monitoring hot spots and land terrain changes caused by volcanic eruption events. The data potential in GES DISC and LP DAAC to monitor volcanic sources of SO2 and the influence of wind fields on the gas plume spread will be demonstrated with the most recent 2018 May-July Kilauea Volcano eruption.

Zeng, Jian↗

Reconstruction of atmospheric pollutant concentrations from remote sensing data - An application of distributed parameter observer theory

The reconstruction of a concentration distribution from spatially averaged and noise-corrupted data is a central problem in processing atmospheric remote sensing data. Distributed parameter observer theory is used to develop reconstructibility conditions for distributed parameter systems having measurements typical of those in remote sensing. The relation of the reconstructibility condition to the stability of the distributed parameter observer is demonstrated. The theory is applied to a variety of remote sensing situations, and it is found that those in which concentrations are measured as a function of altitude satisfy the conditions of distributed state reconstructibility.

Koda, M.↗

Mission operations update for the restructured Earth Observing System (EOS) mission

The National Aeronautics and Space Administration's (NASA) Earth Observing System (EOS) will provide a comprehensive long term set of observations of the Earth to the Earth science research community. The data will aid in determining global changes caused both naturally and through human interaction. Understanding man's impact on the global environment will allow sound policy decisions to be made to protect our future. EOS is a major component of the Mission to Planet Earth program, which is NASA's contribution to the U.S. Global Change Research Program. EOS consists of numerous instruments on multiple spacecraft and a distributed ground system. The EOS Data and Information System (EOSDIS) is the major ground system developed to support EOS. The EOSDIS will provide EOS spacecraft command and control, data processing, product generation, and data archival and distribution services for EOS spacecraft. Data from EOS instruments on other Earth science missions (e.g., Tropical Rainfall Measuring Mission (TRMM)) will also be processed, distributed, and archived in EOSDIS. The U.S. and various International Partners (IP) (e.g., the European Space Agency (ESA), the Ministry of International Trade and Industry (MITI) of Japan, and the Canadian Space Agency (CSA)) participate in and contribute to the international EOS program. The EOSDIS will also archive processed data from other designated NASA Earth science missions (e.g., UARS) that are under the broad umbrella of Mission to Planet Earth.

Kelly, Angelita Castro↗