Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data migration”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS1993

Data management in NOAA

The NOAA archives contain 150 terabytes of data in digital form, most of which are the high volume GOES satellite image data. There are 630 data bases containing 2,350 environmental variables. There are 375 million film records and 90 million paper records in addition to the digital data base. The current data accession rate is 10 percent per year and the number of users are increasing at a 10 percent annual rate. NOAA publishes 5,000 publications and distributes over one million copies to almost 41,000 paying customers. Each year, over six million records are key entered from manuscript documents and about 13,000 computer tapes and 40,000 satellite hardcopy images are entered into the archive. Early digital data were stored on punched cards and open reel computer tapes. In the late seventies, an advanced helical scan technology (AMPEX TBM) was implemented. Now, punched cards have disappeared, the TBM system was abandoned, most data stored on open reel tapes have been migrated to 3480 cartridges, many specialized data sets were distributed on CD ROM's, special archives are being copied to 12 inch optical WORM disks, 5 1/4 inch magneto-optical disks were employed for workstation applications, and 8 mm EXABYTE tapes are planned for major data collection programs. The rapid expansion of new data sets, some of which constitute large volumes of data, coupled with the need for vastly improved access mechanisms, portability, and improved longevity are factors which will influence NOAA's future systems approaches for data management.

Callicott, William M.↗
NASA NTRS2011

Telemetry and Science Data Software System

The Telemetry and Science Data Software System (TSDSS) was designed to validate the operational health of a spacecraft, ease test verification, assist in debugging system anomalies, and provide trending data and advanced science analysis. In doing so, the system parses, processes, and organizes raw data from the Aquarius instrument both on the ground and while in space. In addition, it provides a user-friendly telemetry viewer, and an instant pushbutton test report generator. Existing ground data systems can parse and provide simple data processing, but have limitations in advanced science analysis and instant report generation. The TSDSS functions as an offline data analysis system during I&T (integration and test) and mission operations phases. After raw data are downloaded from an instrument, TSDSS ingests the data files, parses, converts telemetry to engineering units, and applies advanced algorithms to produce science level 0, 1, and 2 data products. Meanwhile, it automatically schedules upload of the raw data to a remote server and archives all intermediate and final values in a MySQL database in time order. All data saved in the system can be straightforwardly retrieved, exported, and migrated. Using TSDSS s interactive data visualization tool, a user can conveniently choose any combination and mathematical computation of interesting telemetry points from a large range of time periods (life cycle of mission ground data and mission operations testing), and display a graphical and statistical view of the data. With this graphical user interface (GUI), the data queried graphs can be exported and saved in multiple formats. This GUI is especially useful in trending data analysis, debugging anomalies, and advanced data analysis. At the request of the user, mission-specific instrument performance assessment reports can be generated with a simple click of a button on the GUI. From instrument level to observatory level, the TSDSS has been operating supporting functional and performance tests and refining system calibration algorithms and coefficients, in sync with the Aquarius/SAC-D spacecraft. At the time of this reporting, it was prepared and set up to perform anomaly investigation for mission operations preceding the Aquarius/SAC-D spacecraft launch on June 10, 2011.

Bates, Lakesha↗
NASA NTRSDate not supplied

Breaking Barriers: Integrating Geo-Leo Aerosol Data with an Open-Source Approach

The scientific community is still examining the novel data from geostationary satellite observations and evaluating methods for effectively fusing the polar observations with various spatial and temporal resolutions. However, the merged data will present a significant ""Big Data"" challenge, including processing, storage, data discoverability, accessibility, and migration within cloud computing environments. We have developed an open-source package to fuse aerosol optical depths (AOD) products from six satellite sensors in the past four years (2019~2023), and this presentation will update our recent progress. Using this Python-based package, we produced a level 3 global (AOD) product in a quarter-degree spatial resolution every half-hour, fusing the Level 2 AOD data with the Dark Target aerosol retrieval algorithm from six satellites: three geostationary (GOES-16/17 and Himawari-8) with high temporal resolution, and three polar orbiting (TERRA/MODIS, AQUA/MODIS, and SNPP-VIIRS) with global coverage. By integrating these observations, the diurnal cycle of global AOD in this fused product can be characterized at local, regional, and global scales. Furthermore, we are committed to openness and transparency by providing our package and its associated functionalities as open-source. Our dedication to adhering to the FAIR, CARE, and TRUST principles ensures that our users can rely on the integrity and ethical standards of our work. For instance of Interoperability, this package fuses remote sensing products on demand into desired temporal and spatial domains. It can be run in a central processing unit (CPU) or a Graphics processing unit (GPU) mode. This package will empower researchers and practitioners to use satellite and sensor data efficiently in various applications and research.

Xiaohua Pan↗
NASA NTRS2011

Determining the Completeness of the Nimbus Meteorological Data Archive

NASA launched the Nimbus series of meteorological satellites in the 1960s and 70s. These satellites carried instruments for making observations of the Earth in the visible, infrared, ultraviolet, and microwave wavelengths. The original data archive consisted of a combination of digital data written to 7-track computer tapes and on various film media. Many of these data sets are now being migrated from the old media to the GES DISC modern online archive. The process involves recovering the digital data files from tape as well as scanning images of the data from film strips. Some of the challenges of archiving the Nimbus data include the lack of any metadata from these old data sets. Metadata standards and self-describing data files did not exist at that time, and files were written on now obsolete hardware systems and outdated file formats. This requires creating metadata by reading the contents of the old data files. Some digital data files were corrupted over time, or were possibly improperly copied at the time of creation. Thus there are data gaps in the collections. The film strips were stored in boxes and are now being scanned as JPEG-2000 images. The only information describing these images is what was written on them when they were originally created, and sometimes this information is incomplete or missing. We have the ability to cross-reference the scanned images against the digital data files to determine which of these best represents the data set from the various missions, or to see how complete the data sets are. In this presentation we compared data files and scanned images from the Nimbus-2 High-Resolution Infrared Radiometer (HRIR) for September 1966 to determine whether the data and images are properly archived with correct metadata.

Johnson, James↗
NASA NTRS2016

An Enriched Shell Finite Element for Progressive Damage Simulation in Composite Laminates

A formulation is presented for an enriched shell nite element capable of progressive damage simulation in composite laminates. The element uses a discrete adaptive splitting approach for damage representation that allows for a straightforward model creation procedure based on an initially low delity mesh. The enriched element is veri ed for Mode I, Mode II, and mixed Mode I/II delamination simulation using numerical benchmark data. Experimental validation is performed using test data from a delamination-migration experiment. Good correlation was found between the enriched shell element model results and the numerical and experimental data sets. The work presented in this paper is meant to serve as a rst milestone in the enriched element's development with an ultimate goal of simulating three-dimensional progressive damage processes in multidirectional laminates.

McElroy, Mark W.↗
NASA NTRS1974

Evaluate ERTS imagery for mapping and detection of changes of snowcover on land and on glaciers

The author has identified the following significant results. The area of snowcover on 10 individual drainage basins in the North Cascades, Washington, has been determined by use of a semi-automatic radiance threshold technique. The result is a unique record of the changing water storage as snow in these important hydrologic units, the runoff of which is utilized for hydroelectric power, dilution of wastes and heat, support of salmon migration, and irrigation. These data allow a new type of hydrologic modelling to proceed which should permit more accurate forecasts of streamflow. A new technique has been developed for measuring snow-covered area or snowline altitude semi-automatically. This variable contour overlay method permits the snowcover to be matched efficiently to the best fit contour of altitude. The motion of the Yentna Glacier during the concluding phase of its surge was successfully measured by a flicker technique using images of two dates. It appears that displacements as small as 100 to 200 m can be measured. Motion of the Tweedsmuir Glacier in Alaska was measured using ERTS-1 images enlarged to 1:50,000. Changes detected included a shock wave moving down the glacier, the margin expanding, the moraine pattern deforming, and the marginal valley deepening.

Meier, M. F.↗
NASA NTRS2002

Reducing the Requirements and Cost of Astronomical Telescopes

Limits on astronomical telescope apertures are being rapidly approached. These limits result from logistics, increasing complexity, and finally budgetary constraints. In an historical perspective, great strides have been made in the area of aperture, adaptive optics, wavefront sensors, detectors, stellar interferometers and image reconstruction. What will be the next advances? Emerging data analysis techniques based on communication theory holds the promise of yielding more information from observational data based on significant computer post-processing. This paper explores some of the current telescope limitations and ponders the possibilities increasing the yield of scientific data based on the migration computer post-processing techniques to higher dimensions. Some of these processes hold the promise of reducing the requirements on the basic telescope hardware making the next generation of instruments more affordable.

Smith, W. Scott↗
NASA NTRS2023

Top‐Down Regulation by a Reindeer Herding System Limits Climate‐Driven Arctic Vegetation Change at a Regional Scale

Warming-driven growth of tall woody vegetation in the Arctic has the potential to accelerate climate change through multiple positive feedbacks. Local-scale evidence suggests that large herbivores limit this vegetation shift, but there is uncertainty at larger, regional scales whether current herbivory pressure is a major top-down control on ecosystem structure and functioning. Across a 67,000 km2 region of the Yamal Peninsula in West Siberia, we integrated satellite remote sensing with a novel data set mapping the migrations of herds comprising 151,000 domesticated reindeer. Where reindeer numbers varied over space, higher reindeer herbivory pressure was consistently linked with lower coverage of tall woody vegetation. Within areas dominated by this vegetation type, productivity and climate were increasingly decoupled where reindeer density was higher. Our spaceborne fingerprint detection suggests that large herbivores, at current population densities, counteract Arctic vegetation responses to climate change over large spatial scales.

Reindeer Herding System↗
NASA NTRS2018

NASA's Earth Observing Data and Information System - Near-Term Challenges

NASA's Earth Observing System Data and Information System (EOSDIS) has been a central component of the NASA Earth observation program since the 1990's. EOSDIS manages data covering a wide range of Earth science disciplines including cryosphere, land cover change, polar processes, field campaigns, ocean surface, digital elevation, atmosphere dynamics and composition, and inter-disciplinary research, and many others. One of the key components of EOSDIS is a set of twelve discipline-based Distributed Active Archive Centers (DAACs) distributed across the United States. Managed by NASA's Earth Science Data and Information System (ESDIS) Project at Goddard Space Flight Center, these DAACs serve over 3 million users globally. The ESDIS Project provides the infrastructure support for EOSDIS, which includes other components such as the Science Investigator-led Processing systems (SIPS), common metadata and metrics management systems, specialized network systems, standards management, and centralized support for use of commercial cloud capabilities. Given the long-term requirements, and the rapid pace of information technology and changing expectations of the user community, EOSDIS has evolved continually over the past three decades. However, many challenges remain. Challenges addressed in this paper include: growing volume and variety, achieving consistency across a diverse set of data producers, managing information about a large number of datasets, migration to a cloud computing environment, optimizing data discovery and access, incorporating user feedback from a diverse community, keeping metadata updated as data collections grow and age, and ensuring that all the content needed for understanding datasets by future users is identified and preserved.

Remote Sensing↗
NASA NTRSDate not supplied

The NASA Merra-2 Reanalysis Products: Data and Tools Used for Aerosol and Air Quality Studies

The NASA Modern-Era Retrospective analysis for Research and Applications Version 2 (MERRA-2) is atmospheric reanalysis data spanning 1980 to present. It has been produced by the NASA Global Modeling and Assimilation Office (GMAO) and is distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). MERRA-2 data includes 100 collections of Earth system variables, mainly from the atmospheric model, such as aerosol fields and meteorological fields, radiation fields, and aerosol fields, guided by the assimilation of as many as six million observations every six hours. MERRA-2 has been one of the most popular datasets from NASA and is widely used in interdisciplinary research and applications, with increasing numbers of new users. For example, at least 7000 users accessed MERRA-2 data at GES DISC in the year 2021, ~1000 more users than in the year 2020. In this presentation, we will introduce the MERRA-2 datasets associated with aerosol and air quality studies and use a wildfire case study to demonstrate the data tools developed at GES DISC to analyze and visualize MERRA-2 data, such as Giovanni and the level 3 and level 4 subsetter, and Jupyter Python notebook. We will also update the status of cloud migration of the MERRA-2 data to Amazon Web Services (AWS).

Xiaohua Pan↗
NASA NTRS2017

Data Recovery Effort of Nimbus Era Observations by the NASA GES DISC

NASA launched seven Nimbus meteorological satellites in the 1960s and 70s. These satellites carried instruments for making observations of the Earth in the visible, infrared, ultraviolet, and microwave wavelengths. The original data archive consisted of a combination of magnetic tapes and various film media. As these media are well past their expected end of life, the valuable data they contain are now being migrated to the GES DISC modern online archive. The process involves recovering the digital data files from the tapes as well as scanning images of the data from film strips. This presentation will address the status and challenges of recovering the Nimbus data. The old data products were written on now obsolete hardware systems and outdated file formats. They lack any metadata standards and each product is often written in its own proprietary file structure. This requires creating metadata by reading the contents of the old data files. The job is tedious and laborious, as documentation may be incomplete, data files and tapes are sometimes corrupted, or were improperly copied at the time they were created.

Nimbus↗
NASA NTRSDate not supplied

Cloud-Based Time Series Analysis of Extremes: Use Cases and Applications

"Extreme weather events, such as hurricanes, tornadoes, floods, droughts, heatwaves, and blizzards, can cause widespread damage, disrupting ecosystems, agricultural production, and economies. The frequency and intensity of these events have been increasing, likely due to climate change, raising concerns and the need for more accurate analysis and predictions. NASA's Goddard Earth Sciences Data and Information Services Center (GES DISC) has migrated its long-term historical datasets, including precipitation data from MERRA-2 reanalysis, GLDAS land data assimilation, and IMERG satellite observations, to the cloud. This cloud-stored data enables scientists and researchers to utilize cloud computing for advanced modeling and forecasting of extreme weather events, eliminating the need to download large datasets. In this presentation, we will provide an overview of the cloud-based data and services managed by GES DISC; demonstrate methods for accessing and analyzing time series data stored in the cloud; and compare results across various datasets to address critical questions related to extreme precipitation. We will present use cases including: 1. Determining the average total precipitation in California during January and February from 2000 to 2024, and identifying anomalous precipitation in 2021. 2. Calculating the 10, 20, 50, and 100-year return periods for maximum daily rainfall based on 25 years of historical precipitation data (2000-2024) for Maryland.

time series↗
NASA NTRS2007

NavP: Structured and Multithreaded Distributed Parallel Programming

We present Navigational Programming (NavP) -- a distributed parallel programming methodology based on the principles of migrating computations and multithreading. The four major steps of NavP are: (1) Distribute the data using the data communication pattern in a given algorithm; (2) Insert navigational commands for the computation to migrate and follow large-sized distributed data; (3) Cut the sequential migrating thread and construct a mobile pipeline; and (4) Loop back for refinement. NavP is significantly different from the current prevailing Message Passing (MP) approach. The advantages of NavP include: (1) NavP is structured distributed programming and it does not change the code structure of an original algorithm. This is in sharp contrast to MP as MP implementations in general do not resemble the original sequential code; (2) NavP implementations are always competitive with the best MPI implementations in terms of performance. Approaches such as DSM or HPF have failed to deliver satisfying performance as of today in contrast, even if they are relatively easy to use compared to MP; (3) NavP provides incremental parallelization, which is beyond the reach of MP; and (4) NavP is a unifying approach that allows us to exploit both fine- (multithreading on shared memory) and coarse- (pipelined tasks on distributed memory) grained parallelism. This is in contrast to the currently popular hybrid use of MP+OpenMP, which is known to be complex to use. We present experimental results that demonstrate the effectiveness of NavP.

navigational programming (NavP)↗
NASA NTRS1986

PCDS overview

A brief overview of the philosophy, design, and organization of the Pilot Climate Data System (PCDS) is presented. The design has evolved, over time, to deal with numerous data sets, provide appropriate tools for the manipulation and display of such data, and support a wide variety of users. The user interface to the system is provided by the Transportable Applications Executive (TAE). A user can choose one of the five major subsystems from the root menu display. These subsystems are CATALOG, INVENTORY, DATA ACCESS, DATA MANIPULATION, and GRAPHICS. Each subsystem was discussed and examples were given, demonstrating the flexibility provided by the existing PCDS software, Version 3.3. Three of these subsystems are presently being redesigned to provide a more flexible and friendly structure, to allow more consistency and facilitate the ability to support new data sets. Ongoing developmental efforts to migrate from the Version 3.3 prototype phase of PCDS to the operational Version 4.0 stress increased flexibility and data independence.

Treinish, L.↗
NASA NTRS2019

"Sensor Web Evolution - Webs of Webs for NASA Science - Focus on small Uninhabited Aerial Systems (sUAS)"

This paper will describe the evolution of information collection, derivation and delivery mechanisms in webs of NASA sensor webs, with a focus on recent advancements in small Uninhabited Aerial Systems (sUAS). I will discuss the movement to "Fog Computing", also known as Edge Computing. Fog Computing facilitates the distribution of common operations and networking between edge devices and cloud computing facilities, optimizing the production of actionable intelligence. Initially, sUASs utilized onboard data collection as standard, with minimal data downloaded directly. Information products were derived in conventional computational environments, generally desk top computers, and information products made available to the Science Community in weeks or months. With the increased availability, and increasingly lower costs, of beyond line of sight (BLOS) satellite based communication, transmission rates and data volumes increased, and processing migrated to Cloud based services. Contemporary sUASs are moving some of that information product derivation to on vehicle services, and are creating a distributed Cloud/Fog environment. I will describe the technological advances that have made this possible, including low power multi-core Central Processing Units (CPU), and, more recently, the availability of high end Graphical Processing Units (GPU) that consume only a few watts. Intelligent system software, leveraging these hardware advances, finally allows for information product generation on-board, rather than simple data collection. Additionally, intelligent flight control systems now support mutual vehicle to vehicle collaboration, allowing sUASs to create ad-hoc sensor webs on demand, as required. Also discussed will be the lessons learned by the Authors' development of data systems for NASA's large High Altitude Long Endurance (HALE) UASs like Predator and Global Hawk, and how those lessons are being applied to sUAS development. This paper will focus on application, rather a deep dive into the technology, and will highlight improving data management through these new technologies.

Sensor Web↗
NASA NTRSDate not supplied

UASs in the VOG/Edge/FOG Sensor Web Environment

This paper will describe the evolution of information collection, derivation and delivery mechanisms in sensor webs utilizing Uninhabited Aerial Systems (UAS).We will discuss the movement to "Fog Computing", also known as Edge Computing. Fog Computing facilitates the distribution of common operations and networking between edge devices and cloud computing facilities, optimizing the production of actionable intelligence. Initially, UASs utilized onboard data collection as standard, with minimal data downloaded directly. Information products were derived in conventional computational environments, generally desk top computers, and information products made available to the Science Community in weeks or months. With the increased availability, and increasingly lower costs, of beyond line of sight (BLOS) satellite based communication, transmission rates and data volumes increased, and processing migrated to Cloud based services. Contemporary UASs are moving some of that information product derivation to on vehicle services, and are creating a distributed Cloud/Fog environment. The Author will describe the technological advances that have made this possible, including low power multi-core Central Processing Units (CPU), and, more recently, the availability of high end Graphical Processing Units (GPU) that consume only a few watts. Intelligent system software, leveraging these hardware advances, finally allows for information product generation on-board, rather than simple data collection. Additionally, intelligent flight control systems now support mutual vehicle to vehicle collaboration, allowing UASs to create ad-hoc sensor webs on demand, as required. Also discussed will be the lessons learned by the Authors' development of data systems for NASA's large High Altitude Long Endurance (HALE) UASs like Predator and Global Hawk, and how those lessons are being applied to other UAS development This paper will focus on applications, rather a deep dive into the technology, and will highlight improving data management through these new technologies.

UAS↗