Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data versions”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
NASA NTRS2000

Monthly Oceanic Rainfall from TRMM Microwave Imager (TMI) Data

We evaluated the performance of the Tropical Rainfall Measuring Mission (TRMM) Microwave Imager (TMI) at-launch algorithm for monthly oceanic rain rate using two years (January 1998 - December 1999) of TMI data. The TMI at-launch algorithm is based on Wilheit et al.'s technique for estimating monthly oceanic rainfall that relies on histograms of multichannel microwave measurements. Comparisons with oceanic monthly rain rates derived from the Defense Meteorological Satellite Program (DMSP) F-13 and F-14 Special Sensor Microwave Imager (SSM/I) data show the average rain rates over the TRMM region (between 400S and 40N) are 3.0, 2.85 and 2.89 mm/day, respectively for F-13, F-14 and TMI. Based on the latest version of TB data (version 5), both rainrate and freezing height derived from TMI are similar to those from the F-13 and F-14 SSM/I data. However, regionally the differences are statistically significant at the 95% confidence. Three hourly monthly rainrates are also computed from 3-hourly TB histograms to examine the diurnal cycle of precipitation. Over most of the oceanic TRMM area, a distinct early morning rainfall peak is found. A harmonic analysis shows that the amplitude of the 12h harmonic is significant and comparable to that of the 24h harmonic.

Chang, Alfred T. C.↗
NASA NTRS2006

Validation of CALIPSO Lidar Observations Using Data From the NASA Langley Airborne High Spectral Resolution Lidar

This poster focuses on preliminary comparisons of data from the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) instrument on the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) spacecraft with data acquired by the NASA Langley Airborne High Spectral Resolution Lidar (HSRL). A series of 20 aircraft validation flights was conducted from 14 June through 27 September 2006, under both day and night lighting conditions and a variety of aerosol and cloud conditions. This poster presents comparisons of CALIOP measurements of attenuated backscatter at 532 and 1064 nm and depolarization at 532 nm with near coincident measurements from the Airborne HSRL as a preliminary assessment of CALIOP calibration accuracy. Note that the CALIOP data presented here are the pre-release version. These data have known artifacts in calibration which have been corrected in the December 8 CALIPSO data release which was not available at the time the comparisons were conducted for this poster. The HSRL data are also preliminary. No artifacts are known to exist; however, refinements in calibration and algorithms are likely to be implemented before validation comparisons are made final.

Hostetler, Chris↗
NASA NTRS2019

Impact of Assimilating AIRS Cloud-Cleared Radiances on Atmospheric Dynamics and Boundary Layer Height at High Latitudes

High latitude weather forecasts, on scales ranging from mesoscale to synoptic, present difficulties due, in part, to the sparsity of conventional observations. In addition, the prevalence of extended low-level stratus cloud cover limits the use of infrared data, which are operationally assimilated only in areas unaffected by clouds. Use of cloud-cleared AIRS (Atmospheric Infrared Sounder) radiances (AIRS CCR), allows the assimilation of infrared information in cloudy regions, permitting data ingestion in regions usually undersampled. This study explores the sensitivity of planetary boundary layer height and related atmospheric dynamics to the assimilation of these data in the Goddard Earth Observing System (GEOS, version 5) data assimilation and forecast system during the boreal fall 2014 season using observing system experiments (OSEs). Examined here are comparisons between the current, operational approach of assimilating AIRS clear-sky radiances against the assimilation of CCR. Assimilation of hyperspectral infrared information from AIRS over the Arctic region slightly modifies the lower midtropospheric temperature structure, which in turn contributes to adjustments in geopotential height, affecting the baroclinic instability properties over the entire hemisphere and explaining the overall improvement in global forecast skill.

McGrath-Spangler, E. L.↗
NASA NTRS2004

Comparison of TOMS, SBW & SBUV/2 Version 8 Total Column Ozone Data with Data from Groundstations

The Nimbus-7 and Earth Probe Total Ozone Mapping Spectrometer (TOMS) data as well as SBUV and SBUV/2 data have been reprocessed with a new retrieval algorithm (Version 8) and an updated calibration procedure. An overview will be presented systematically comparing ozone values to an ensemble of Brewer and Dobson spectrophotometers. The comparisons were made as a function of latitude, solar zenith angle, reflectivity and total ozone. Results show that the accuracy of the TOMS retrieval has been improved when aerosols are present in the atmosphere, when snow/ice and sea glint are present, and when ozone in the northern hemisphere is extremely low. TOMS overpass data are derived from the single TOMS best match measurement, almost always located within one degree of the ground station and usually made within an hour of local noon. The Version 8 Earth Probe TOMS ozone values have decreased by an average of about 1% due to a much better understanding of the calibration of the instrument. N-7 SBUV as well as the series of NOAA SBUV/2 column ozone values have also been processed with the Version 8 algorithm and have been compared to values from an ensemble of groundstations. Results show that the SBW column ozone values agree well with the groundstations and the datasets are useful for trend studies.

Labow, G. J.↗
NASA NTRS2003

EP/TOMS Science Data Processing Programmer's Guide. Version 1.9

This EP/TOMS Programmer's Guide provides detailed information concerning the data, software, operations, and maintenance of the EP/TOMS Science Data Processing System. This system was developed for the NASA Goddard Space Flight Center (GSFC), Laboratory for Atmospheres, Atmospheric Chemistry Branch (Code 916) by SSAI. The description of the EP/TOMS Science Data Processing System starts with an overall system perspective (Section 3). A description of all data files ingested, stored, and/or produced follows in Section 4. Detailed descriptions of each major software subsystem are in Section 5. Section 6 presents processing procedures. Section 7 presents maintenance procedures. Procedures for updating this document are included in Section 7.

Source record↗
NASA NTRSDate not supplied

High Performance Access to Archival Data Stored in HDF4 and HDF5 on Cloud Object Stores Without Reformatting the Files

Cloud computing offers numerous advantages for users of extensive Earth science data collections. These benefits encompass direct online access to data files and granules from any location, scalable access supporting parallel computing workflows, and flexible computing tools enabling innovative experimentation with processing techniques. However, older archival file formats designed for distinct computing systems hinder efficient access to decade-long time-series data when compared to data stored in modern cloud-optimized formats like Web Object Stores (WOS), exemplified by Amazon Web Services’ Simple Storage Service (S3). We describe DMR++ (Dataset Metadata Response plus plus), a technology facilitating efficient access to HDF5 (Hierarchical Data Format, version 5) and HDF4 files stored on WOS systems without requiring data reformatting. DMR++ achieves performance comparable to technologies like Zarr while preserving the original file structure, a substantial benefit considering the vast quantity of archival files held by organizations such as NASA. Moreover, DMR++ typically outperforms cloud-optimized versions of HDF5. Essentially an XML (Extensible Markup Language) document usually stored alongside the described data, DMR++ can also be generated on-the-fly but is generally created during data staging to the WOS. Archival files that use HDF4/5 often store large arrays of numerical data. The data in these files is often compressed, typically reducing their size by a factor of four or more. To achieve efficient access to portions of those arrays, they are 'chunked' into smaller sub-arrays, each individually compressed. The chunk size is a compromise, where spinning disks can efficiently access data in smaller chunks while S3 favors larger chunks. A simple optimization of aggregating smaller chunks that are stored adjacently, transferring them in a single access and then individually decompressing them will improve performance. NASA data pose an additional challenge: special Application Programmer Interface (API) libraries are often needed to compute some variables. These libraries are incompatible with WOS environments. Our solution involves storing computed values in the DMR++ document or a companion file, making them accessible like other variables and eliminating the need for specialized APIs. We outline specific optimizations for both satellite grid and swath data stored in HDF4-EOS2 (Earth Observing System).

James Gallagher↗
NASA NTRS1994

Status of the Shuttle SBUV (SSBUV) calibration of the NOAA SBUV/2 operational ozone sounders and the detection of trends

The Shuttle Solar Backscatter Ultraviolet (SSBUV) experiment has flown four times since October 1989. The purpose of SSBUV is to perform calibration checks of the SBUV ozone sounding instruments on the Nimbus and NOAA satellites in order to remove calibration drift so that ozone trends in the middle stratosphere can be accurately derived. Calibration checks are performed by comparing coincident observations between SSBUV and the satellite instruments. Regular flights of about once per year and maintenance of the SSBUV calibration to 1 percent from flight to flight are the major challenges for SSBUV. To date the required flight frequency has been met and instrument calibration is known to about 1-2 percent for the first three flights. The first comparisons showed 30 percent differences between SSBUV and the original archived Nimbus SBUV data, but considerably smaller differences with the new SBUV 'Version 6' data. Differences between SSBUV and SBUV/2 instruments on NOAA-11 and NOAA-9 were of the order of 5-10 percent respectively. These differences have not been accounted for in the present NOAA data set since they contain initial calibration biases as well as long term instrument drift. With subsequent SSBUV comparisons, the satellite calibration can be corrected, which will then allow an accurate estimate of ozone trends in the upper stratosphere. In this initial study, 1989 Nimbus-7 SBUV data have been corrected using SSBUV observations and then compared to SBUV data for 1980. This comparison then leads to an ozone trend of 7 percent in the upper stratosphere over the tropics for the period 1980 to 1989.

Hilsenrath, Ernest↗
NASA NTRS2004

Processing Spacecraft Data Without Confusion

Producing multiple versions of the same data product for the same time frame with the same remotely sensed inputs can be a recipe for disaster. Yet, amidst the commotion of satellite launch and early operations (LEO), such data processing is needed. After LEO, the situation gets worse. Processing newly arriving data ("forward processing") is augmented with reprocessing and algorithm development, comparison, evaluation, and testing -- often happening all at the same time. The problem can be analyzed in three main parts -- maintaining multiple versions of algorithms and data so that end-product users are not overwhelmed,allocating computer resources efficiently, and simplifying production operations so that va st amounts of data can be processed with minimal staff and fewer errors. OMIDAPS provides a framework for execution of algorithms that transform lower level data acquired by OMI on NASA's Aura satellite into higher level science data products. In contrast to traditional science data processing systems, we address all parts of the problem with an innovative approach allowing multiple data processing to run within a single physical system. The data products, imports, exports, and execution planning are all segregated into distinct "ArchiveSets." This paper describes reasons for multiple concurrent productions on a typical satellite data processing project using OMI as an example. It describes the virtual data processing system concept and its advantages over separate physical processing strings. It explores the specific implementation of the virtual systems within OMIDAPS and discusses some of the implications of our approach and describes how virtual processing is used to accomplish the overall mission of OMI data processing.

Tilmes, Curt↗
NASA NTRS1997

An Investigation of Aerosol and Ozone Measurements from the Cryogenic Limb Array Etalon Spectrometer: Validation and Relation to Other Chemical Species

Throughout this study we focused on comparisons of CLAES and in situ measurements of ozone and aerosol extinction. Thus the comparison is between satellite data representative of large spatial regions and in situ data representative of nearly point samples. Both instruments provide vertical profiles, but the region of overlap is limited to between approximately 10 and 100 mb. CLAES Version 7 ozone measurements have been compared to electrochemical cell ozonesonde measurements over McMurdo Station, Antarctica (78 deg S, 167 deg E), Dumont d'Urville, Antarctica (66.7 deg S, 140 deg E), Laramie, Wyoming (41 deg N, 106 deg W), and Bear Island, Norway (74.3 deg N, 19 deg E). Comparisons were made between vertical ozone profiles, and between integrated column ozone over the region of overlap of the measurements. Comparisons using CLAES Version 8 data are underway. CLAES Version 8 aerosol extinction measurements at all wavelengths have also been compared to University of Wyoming aerosol extinctions over McMurdo Station, Antarctica, and over Laramie, Wyoming. Coincidences in all cases were determined by minimizing the distance between the CLAES observations and the surface station, and the time separation between the satellite and in situ measurements.

Deshler, Terry↗
NASA NTRS2006

NASA Tech Briefs, September 2006

Topics covered include: Improving Thermomechanical Properties of SiC/SiC Composites; Aerogel/Particle Composites for Thermoelectric Devices; Patches for Repairing Ceramics and Ceramic- Matrix Composites; Lower-Conductivity Ceramic Materials for Thermal-Barrier Coatings; An Alternative for Emergency Preemption of Traffic Lights; Vehicle Transponder for Preemption of Traffic Lights; Automated Announcements of Approaching Emergency Vehicles; Intersection Monitor for Traffic-Light-Preemption System; Full-Duplex Digital Communication on a Single Laser Beam; Stabilizing Microwave Frequency of a Photonic Oscillator; Microwave Oscillators Based on Nonlinear WGM Resonators; Pointing Reference Scheme for Free-Space Optical Communications Systems; High-Level Performance Modeling of SAR Systems; Spectral Analysis Tool 6.2 for Windows; Multi-Platform Avionics Simulator; Silicon-Based Optical Modulator with Ferroelectric Layer; Multiplexing Transducers Based on Tunnel-Diode Oscillators; Scheduling with Automated Resolution of Conflicts; Symbolic Constraint Maintenance Grid; Discerning Trends in Performance Across Multiple Events; Magnetic Field Solver; Computing for Aiming a Spaceborne Bistatic- Radar Transmitter; 4-Vinyl-1,3-Dioxolane-2-One as an Additive for Li-Ion Cells; Probabilistic Prediction of Lifetimes of Ceramic Parts; STRANAL-PMC Version 2.0; Micromechanics and Piezo Enhancements of HyperSizer; Single-Phase Rare-Earth Oxide/Aluminum Oxide Glasses; Tilt/Tip/Piston Manipulator with Base-Mounted Actuators; Measurement of Model Noise in a Hard-Wall Wind Tunnel; Loci-STREAM Version 0.9; The Synergistic Engineering Environment; Reconfigurable Software for Controlling Formation Flying; More About the Tetrahedral Unstructured Software System; Computing Flows Using Chimera and Unstructured Grids; Avoiding Obstructions in Aiming a High-Gain Antenna; Analyzing Aeroelastic Stability of a Tilt-Rotor Aircraft; Tracking Positions and Attitudes of Mars Rovers; Stochastic Evolutionary Algorithms for Planning Robot Paths; Compressible Flow Toolbox; Rapid Aeroelastic Analysis of Blade Flutter in Turbomachines; General Flow-Solver Code for Turbomachinery Applications; Code for Multiblock CFD and Heat-Transfer Computations; Rotating-Pump Design Code; Covering a Crucible with Metal Containing Channels; Repairing Fractured Bones by Use of Bioabsorbable Composites; Kalman Filter for Calibrating a Telescope Focal Plane; Electronic Absolute Cartesian Autocollimator; Fiber-Optic Gratings for Lidar Measurements of Water Vapor; Simulating Responses of Gravitational-Wave Instrumentation; SOFTC: A Software Correlator for VLBI; Progress in Computational Simulation of Earthquakes; Database of Properties of Meteors; Computing Spacecraft Solar-Cell Damage by Charged Particles; Thermal Model of a Current-Carrying Wire in a Vacuum; Program for Analyzing Flows in a Complex Network; Program Predicts Performance of Optical Parametric Oscillators; Processing TES Level-1B Data; Automated Camera Calibration; Tracking the Martian CO2 Polar Ice Caps in Infrared Images; Processing TES Level-2 Data; SmaggIce Version 1.8; Solving the Swath Segment Selection Problem; The Spatial Standard Observer; Less-Complex Method of Classifying MPSK; Improvement in Recursive Hierarchical Segmentation of Data; Using Heaps in Recursive Hierarchical Segmentation of Data; Tool for Statistical Analysis and Display of Landing Sites; Automated Assignment of Proposals to Reviewers; Array-Pattern-Match Compiler for Opportunistic Data Analysis; Pre-Processor for Compression of Multispectral Image Data; Compressing Image Data While Limiting the Effects of Data Losses; Flight Operations Analysis Tool; Improvement in Visual Target Tracking for a Mobile Robot; Software for Simulating Air Traffic; Automated Vectorization of Decision-Based Algorithms; Grayscale Optical Correlator Workbench; "One-Stop Shopping" for Ocean Remote-Sensing and Model Data; State Analysis Database Tool; Generating CAHV and CAHVOmages with Shadows in ROAMS; Improving UDP/IP Transmission Without Increasing Congestion; FORTRAN Versions of Reformulated HFGMC Codes; Program for Editing Spacecraft Command Sequences; Flight-Tested Prototype of BEAM Software; Mission Scenario Development Workbench; Marsviewer; Tool for Analysis and Reduction of Scientific Data; ASPEN Version 3.0; Secure Display of Space-Exploration Images; Digital Front End for Wide-Band VLBI Science Receiver; Multifunctional Tanks for Spacecraft; Lightweight, Segmented, Mostly Silicon Telescope Mirror; Assistant for Analyzing Tropical-Rain-Mapping Radar Data; and Anion-Intercalating Cathodes for High-Energy- Density Cells.

Source record↗
NASA NTRS2019

A Compilation of Global Bio-Optical in Situ Data for Ocean-Colour Satellite Applications – Version Two

A global compilation of in situ data is useful to evaluate the quality of ocean-colour satellite data records. Here we describe the data compiled for the validation of the ocean-colour products from the ESA Ocean Colour Climate Change Initiative (OC-CCI). The data were acquired from several sources (including, inter alia, MOBY, BOUSSOLE, AERONETOC, SeaBASS, NOMAD, MERMAID, AMT, ICES, HOT, GeP&CO) and span the period from 1997 to 2018. Observations of the following variables were compiled: spectral remote-sensing reflectances, concentrations of chlorophyll-a, spectral inherent optical properties, spectral diffuse attenuation coefficients and total suspended matter. The data were from multi-project archives acquired via open internet services or from individual projects, acquired directly from data providers. Methodologies were implemented for homogenisation, quality control and merging of all data. No changes were made to the original data, other than averaging of observations that were close in time and space, elimination of some points after quality control and conversion to a standard format. The final result is a merged table designed for validation of satellite-derived ocean-colour products and available in text format. Metadata of each in situ measurement (original source, cruise or experiment, principal investigator) were propagated throughout the work and made available in the final table. By making the metadata available, provenance is better documented, and it is also possible to analyse each set of data separately. This paper also describes the changes that were made to the compilation in relation to the previous version (Valente et al., 2016). The compiled data are available at https://doi.org/10.1594/PANGAEA.898188.

Andre Valente↗
NASA NTRS2019

Numerical Reanalyses as a Gateway to Arctic Synthesis

Reanalyses are regularly gridded, retrospective depictions of the physical earth system, which are produced through the correction of a short-term forecast to available observations. In the Arctic, reanalyses are particularly well suited to marshal the sparse observing network to provide a plausible, multivariate representation of conditions. Atmospheric reanalyses such as MERRA-2 (NASA Modern-Era Retrospective analysis for Research and Applications, version 2) and ocean reanalyses such as SODA3 (Univ. Maryland Simple Ocean Data Assimilation version 3) are widely used in Arctic research for diagnostic studies of circulation, model evaluation, and as boundary conditions for a variety of process models. Here, we provide examples that illustrate the utility of reanalyses for providing information on the spatial and temporal scales of recent, rapid changes in the Arctic. Recent trends in Arctic surface temperatures, surface melt over Greenland and Arctic glaciers, and evolving freshwater conditions in the Arctic Ocean are examples where reanalyses can provide information that cannot easily be obtained via other means. These examples provide information on the scale, magnitude, and the uncertainty of recent Arctic change and provide a context for future scenarios. We further quantify uncertainties in key reanalyses variables and approaches for addressing these issues.

Arctic↗
NASA NTRS1991

Digital solar system geology

All available synoptic maps of the solid-surface bodies of the Solar System were digitized for presentation in the planned Atlas of the Solar System by Greeley and Batson. Since the last report (Batson et al., 1990), preliminary Uranian satellite maps were replaced with improved versions, Galilean satellite geology was simplified and digitized, structure was added to many maps, and the maps were converted to a standard format, with corresponding standing colors for the mapped units. Following these changes, the maps were re-reviewed by their authors and are now undergoing final editing before preparation for publication. In some cases (for Mercury, Venus, and Mars), more detailed maps were digitized and then simplified for the Atlas. Other detailed maps are planned to be digitized in the coming year for the Moon and the Galilean satellites. For most of the remaining bodies such as the Uranian satellites, the current digitized versions contain virtually all the detail that can be mapped given the available data; those versions will be unchanged for the Atlas. These digital geologic maps are archived at the digital scale of 1/16 degree/ pixel, in sinusoidal format. The availability of geology of the Solar System in a digital database will facilitate comparisons and integration with other data: digitized lunar geologic maps have already been used in a comparison with Galileo SSI observations of the Moon.

Batson, R. M.↗
NASA NTRS2001

A Comparison of HALOE V19 with SAGE II V6.00 Ozone Observations using Trajectory Mapping

We apply trajectory mapping to an eight-year intercomparison of ozone observations from HALOE (V19) and SAGE II (V6.00) for the months March, May, June, September, October, and December from the period December 1991 - October 1999. Our results, which represent the most extensive such intercomparison of these two data sets to date, suggest a root-mean -square difference between the two data sets of greater than 15% below 22 km and of 4 - 12% throughout most of the rest of the stratosphere. In addition, we find a bias with HALOE ozone low relative to SAGE II by 5 - 20% below 22 km between 40degS and 40degN. Biases throughout most of the rest of the stratosphere are nearly nonexistent. Finally, our analysis suggests almost no drift in the bias between the data sets is observed over the period of study. In the course of our study, we also determine that the employment of the Wang-Cunnold criteria is still recommended with the V6.00 SAGE II ozone data. Results with the new versions of the data sets show significant improvement over previous versions, particularly in the elimination of mid-stratospheric biases and the elimination of the previously observed drifts in the biases between the data sets in the lower stratosphere. Since HALOE V19 and V18 ozone are very similar, these changes can likely be attributed to improvements in the SAGE II retrieval.

Morris, Gary A.↗
NASA NTRS2014

Past Changes in the Vertical Distribution of Ozone Part 1: Measurement Techniques, Uncertainties and Availability

Peak stratospheric chlorofluorocarbon (CFC) and other ozone depleting substance (ODS) concentrations were reached in the mid- to late 1990s. Detection and attribution of the expected recovery of the stratospheric ozone layer in an atmosphere with reduced ODSs as well as efforts to understand the evolution of stratospheric ozone in the presence of increasing greenhouse gases are key current research topics. These require a critical examination of the ozone changes with an accurate knowledge of the spatial (geographical and vertical) and temporal ozone response. For such an examination, it is vital that the quality of the measurements used be as high as possible and measurement uncertainties well quantified. In preparation for the 2014 United Nations Environment Programme (UNEP)/World Meteorological Organization (WMO) Scientific Assessment of Ozone Depletion, the SPARC/IO3C/IGACO-O3/NDACC (SI2N) Initiative was designed to study and document changes in the global ozone profile distribution. This requires assessing long-term ozone profile data sets in regards to measurement stability and uncertainty characteristics. The ultimate goal is to establish suitability for estimating long-term ozone trends to contribute to ozone recovery studies. Some of the data sets have been improved as part of this initiative with updated versions now available. This summary presents an overview of stratospheric ozone profile measurement data sets (ground and satellite based) available for ozone recovery studies. Here we document measurement techniques, spatial and temporal coverage, vertical resolution, native units and measurement uncertainties. In addition, the latest data versions are briefly described (including data version updates as well as detailing multiple retrievals when available for a given satellite instrument). Archive location information for each data set is also given.

stratospheric ozone layer measurement↗
NASA NTRS1994

Comparison of recalculated Dobson and TOMS total ozone at Hradec Kralove, Czechoslovakia, 1978-1990

The reevaluated Dobson total ozone data from Hradec Kralove, Czechoslovakia were compared with independent Total Ozone Mapping Spectrophotometer (TOMS) 'version 6' data set. The comparison was performed by means of the parallel daily averages of ground-based and satellite total ozone pairs of the period November 1978 to December 1990. The comparison showed slight differences between both data series. Their average relative difference is 0.48 percent. The similar results have been reached for subsets of direct sun and zenith types of measurements as well. Their relative differences are 0.61 percent and 0.11 percent respectively. These facts indicate not only good mutual relation of both data sources but also reliability and accuracy of the zenith charts of the spectrophotometer No. 74 used at Hradec Kralove. Preliminary assessment of seasonal MU-dependence of the differences between Dobson and TOMS data was made while using total ozones of winter and summer months representing values of MU=2.70-5.20 and MU = 1.12-1.30 respectively. The results did not show systematic underestimation or overestimation of total ozone due to MU-dependence of the instrument at Hradec Kralove in both seasons.

Stanek, Martin↗
NASA NTRS2019

Forming Aggregations using Virtual Sharding: Lessons Learned from Simple Scalable Storage (S3)

Data aggregation is the ability to combine separate datasets to form a single new logical dataset provides users with a powerful abstraction. The advantage of an aggregate dataset is that the users are freed from having to understand, and incorporate into their workflow, knowledge about the (ad hoc) organization of the constituent datasets. However, aggregating large numbers of files can be computationally complex with data server systems performing many repetitive operations. As part of the authors work on subsetting data stored on Amazon Web Service (AWS) Simple Storage Service (S3), we developed technology to read portions of otherwise monolithic data files. This enables the formation of virtual shards for user in subsetting data stored in HDF5 (hierarchical data format, version 5) files. This same tool can be used to form aggregations that combine data stored in many HDF5 files when those files are stored on S3. The nature of the virtual sharding and the algorithm that exploits it for subsetting is such that it can also be used for aggregation with the need for many of the repetitive operations required by the per file aggregation techniques. We will present timing information that demonstrates the flexibility of this approach. However, the lessons learned is that while this is a useful result in and of itself, these very same techniques can be applied in other contexts where data are stored in services and on media other than S3. For example, this same technique can be applied to data stored on spinning disk. Pushing the envelope for S3 forced a reexamination of our data access techniques which lead to unexpected positive benefits.

Gallagher, James↗