Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “postgresql”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

25 records · Page 2
All sourcesNASA NTRSDOE OSTI
NASA NTRS2004

A Low-Cost Clustered Archive Approach for Storing Remote Sensing Data

As part of NASA's Earth Observing System (EOS) Data and Information System (EOSDIS), the Moderate Resolution Imaging Spectrometer (MODIS) Data Processing System (MODAPS) is now processing data from two instruments on the EOS flag ship spacecraft Terra and Aqua. Between the two, MODAPS is generating over 1 Terrabyte of data per day and has surpassed 3 Petabytes of total data. The bulk of the data is stored near-line in StorageTek Powerhorn tape jukeboxes. Accessing data that has been moved to tape involves submitting an order, scheduling the tape, and waiting for the data to become available. I am developing a low cost clustered archive that could enable storing a very large amount of data such as the MODIS data described above in an organized fashion on a cluster of commodity hardware using low cost SATA hard drives such that the files are directly available online. The system takes full advantage of Open Source software, using the GNULinux operating system, PostgreSQL relational database and the Apache HTTP Server. This poster session will depict my approach, the interface to the archive, a brief discussion of the internals of the system and some performance numbers from my prototyping. I will also describe various costs and benefits of this approach versus the traditional large tape jukebox approach currently in use in the EOSDIS.

Tilmes, Curt↗
NASA NTRS2005

Lessons Learned From Developing Three Generations of Remote Sensing Science Data Processing Systems

The Biospheric Information Systems Branch at NASA s Goddard Space Flight Center has developed three generations of Science Investigator-led Processing Systems for use with various remote sensing instruments. The first system is used for data from the MODIS instruments flown on NASA s Earth Observing Systems @OS) Terra and Aqua Spacecraft launched in 1999 and 2002 respectively. The second generation is for the Ozone Measuring Instrument flying on the EOS Aura spacecraft launched in 2004. We are now developing a third generation of the system for evaluation science data processing for the Ozone Mapping and Profiler Suite (OMPS) to be flown by the NPOESS Preparatory Project (NPP) in 2006. The initial system was based on large scale proprietary hardware, operating and database systems. The current OMI system and the OMPS system being developed are based on commodity hardware, the LINUX Operating System and on PostgreSQL, an Open Source RDBMS. The new system distributes its data archive across multiple server hosts and processes jobs on multiple processor boxes. We have created several instances of this system, including one for operational processing, one for testing and reprocessing and one for applications development and scientific analysis. Prior to receiving the first data from OMI we applied the system to reprocessing information from the Solar Backscatter Ultraviolet (SBUV) and Total Ozone Mapping Spectrometer (TOMS) instruments flown from 1978 until now. The system was able to process 25 years (108,000 orbits) of data and produce 800,000 files (400 GiB) of level 2 and level 3 products in less than a week. We will describe the lessons we have learned and tradeoffs between system design, hardware, operating systems, operational staffing, user support and operational procedures. During each generational phase, the system has become more generic and reusable. While the system is not currently shrink wrapped we believe it is to the point where it could be readily adopted, with substantial cost savings, for other similar tasks.

Tilmes, Curt↗
NASA NTRS2009

DSN Beowulf Cluster-Based VLBI Correlator

The NASA Deep Space Network (DSN) requires a broadband VLBI (very long baseline interferometry) correlator to process data routinely taken as part of the VLBI source Catalogue Maintenance and Enhancement task (CAT M&E) and the Time and Earth Motion Precision Observations task (TEMPO). The data provided by these measurements are a crucial ingredient in the formation of precision deep-space navigation models. In addition, a VLBI correlator is needed to provide support for other VLBI related activities for both internal and external customers. The JPL VLBI Correlator (JVC) was designed, developed, and delivered to the DSN as a successor to the legacy Block II Correlator. The JVC is a full-capability VLBI correlator that uses software processes running on multiple computers to cross-correlate two-antenna broadband noise data. Components of this new system (see Figure 1) consist of Linux PCs integrated into a Beowulf Cluster, an existing Mark5 data storage system, a RAID array, an existing software correlator package (SoftC) originally developed for Delta DOR Navigation processing, and various custom- developed software processes and scripts. Parallel processing on the JVC is achieved by assigning slave nodes of the Beowulf cluster to process separate scans in parallel until all scans have been processed. Due to the single stream sequential playback of the Mark5 data, some ramp-up time is required before all nodes can have access to required scan data. Core functions of each processing step are accomplished using optimized C programs. The coordination and execution of these programs across the cluster is accomplished using Pearl scripts, PostgreSQL commands, and a handful of miscellaneous system utilities. Mark5 data modules are loaded on Mark5 Data systems playback units, one per station. Data processing is started when the operator scans the Mark5 systems and runs a script that reads various configuration files and then creates an experiment-dependent status database used to delegate parallel tasks between nodes and storage areas (see Figure 2). This script forks into three processes: extract, translate, and correlate. Each of these processes iterates on available scan data and updates the status database as the work for each scan is completed. The extract process coordinates and monitors the transfer of data from each of the Mark5s to the Beowulf RAID storage systems. The translate process monitors and executes the data conversion processes on available scan files, and writes the translated files to the slave nodes. The correlate process monitors the execution of SoftC correlation processes on the slave nodes for scans that have completed translation. A comparison of the JVC and the legacy Block II correlator outputs reveals they are well within a formal error, and that the data are comparable with respect to their use in flight navigation. The processing speed of the JVC is improved over the Block II correlator by a factor of 4, largely due to the elimination of the reel-to-reel tape drives used in the Block II correlator.

Rogstad, Stephen P.↗
NASA NTRS2013

Sally Ride EarthKAM - Automated Image Geo-Referencing Using Google Earth Web Plug-In

Sally Ride EarthKAM is an educational program funded by NASA that aims to provide the public the ability to picture Earth from the perspective of the International Space Station (ISS). A computer-controlled camera is mounted on the ISS in a nadir-pointing window; however, timing limitations in the system cause inaccurate positional metadata. Manually correcting images within an orbit allows the positional metadata to be improved using mathematical regressions. The manual correction process is time-consuming and thus, unfeasible for a large number of images. The standard Google Earth program allows for the importing of KML (keyhole markup language) files that previously were created. These KML file-based overlays could then be manually manipulated as image overlays, saved, and then uploaded to the project server where they are parsed and the metadata in the database is updated. The new interface eliminates the need to save, download, open, re-save, and upload the KML files. Everything is processed on the Web, and all manipulations go directly into the database. Administrators also have the control to discard any single correction that was made and validate a correction. This program streamlines a process that previously required several critical steps and was probably too complex for the average user to complete successfully. The new process is theoretically simple enough for members of the public to make use of and contribute to the success of the Sally Ride EarthKAM project. Using the Google Earth Web plug-in, EarthKAM images, and associated metadata, this software allows users to interactively manipulate an EarthKAM image overlay, and update and improve the associated metadata. The Web interface uses the Google Earth JavaScript API along with PHP-PostgreSQL to present the user the same interface capabilities without leaving the Web. The simpler graphical user interface will allow the public to participate directly and meaningfully with EarthKAM. The use of similar techniques is being investigated to place ground-based observations in a Google Mars environment, allowing the MSL (Mars Science Laboratory) Science Team a means to visualize the rover and its environment.

Andres, Paul M.↗
DOE OSTI2020

GBCGE Subsurface Database Explorer and APIs

This submission defines a DOI for the Great Basin Center for Geothermal Energy's (GBCGE) Subsurface Database Explorer web application and underlying data services, and acknowledges the INGENIOUS project as a major source of funding for data compilation and quality assurance. The GBCGE Subsurface Database Explorer is an interactive web mapping application that provides public access to the GBCGE Subsurface Database, and its collection of datasets pertinent to geothermal exploration, oil and gas exploration, critical mineral exploration, and other subsurface characterization for the Great Basin Region, western US. This is a living database, and will be continuously updated with new data and datasets as funding and motivations allow. The underlying database views that populate the web application are on an automated refresh schedule. Data sources and acknowledgements: We thank our partners with the Nevada Division of Minerals (NDOM), the Southern Methodist University (SMU), and Great Basin State Geological Surveys for their active efforts in data curation, schema design, and quality assurance. We also thank contributors among the USGS, Oregon Institute of Technology, State Divisions of Water Resources, State Divisions of Oil, Gas, and Minerals, and State Geological Surveys for open data availability and direct contributions made under the National Geothermal Data System (NGDS).

15 GEOTHERMAL ENERGY↗