Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “End user”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,243 records · Page 69

4.4 Development of a 30-Year Soil Moisture Climatology for Situational Awareness and Public Health Applications

This paper provided a brief background on the work being done at NASA SPoRT and the CDC to create a soil moisture climatology over the CONUS at high spatial resolution, and to provide a valuable source of soil moisture information to the CDC for monitoring conditions that could favor the development of Valley Fever. The soil moisture climatology has multi-faceted applications for both the NOAA/NWS situational awareness in the areas of drought and flooding, and for the Public Health community. SPoRT plans to increase its interaction with the drought monitoring and Public Health communities by enhancing this testbed soil moisture anomaly product. This soil moisture climatology run will also serve as a foundation for upgrading the real-time (currently southeastern CONUS) SPoRT-LIS to a full CONUS domain based on LIS version 7 and incorporating real-time GVF data from the Suomi-NPP Visible Infrared Imaging Radiometer Suite (Vargas et al. 2013) into LIS-Noah. The upgraded SPoRT-LIS run will serve as a testbed proof-of-concept of a higher-resolution NLDAS-2 modeling member. The climatology run will be extended to near real-time using the NLDAS-2 meteorological forcing from 2011 to present. The fixed 1981-2010 climatology shall provide the soil moisture "normals" for the production of real-time soil moisture anomalies. SPoRT also envisions a web-mapping type of service in which an end-user could put in a request for either an historical or real-time soil moisture anomaly graph for a specified county (as exemplified by Figure 2) and/or for local and regional maps of soil moisture proxy percentiles. Finally, SPoRT seeks to assimilate satellite soil moisture data from the current Soil Moisture Ocean Salinity (SMOS; Blankenship et al. 2014) and the recently-launched NASA Soil Moisture Active Passive (SMAP; Entekhabi et al. 2010) missions, using the EnKF capability within LIS. The 9-km combined active radar and passive microwave retrieval product from SMAP (Das et al. 2011) has the potential to provide valuable information about the near-surface soil moisture state for improving land surface modeling output.

land surface modeling↗

Can Simulation Credibility Be Improved Using Sensitivity Analysis to Understand Input Data Effects on Model Outcome?

Model and simulation (MS) credibility is defined as, the quality to elicit belief or trust in MS results. NASA-STD-7009 [1] delineates eight components (Verification, Validation, Input Pedigree, Results Uncertainty, Results Robustness, Use History, MS Management, People Qualifications) that address quantifying model credibility, and provides guidance to the model developers, analysts, and end users for assessing the MS credibility. Of the eight characteristics, input pedigree, or the quality of the data used to develop model input parameters, governing functions, or initial conditions, can vary significantly. These data quality differences have varying consequences across the range of MS application. NASA-STD-7009 requires that the lowest input data quality be used to represent the entire set of input data when scoring the input pedigree credibility of the model. This requirement provides a conservative assessment of model inputs, and maximizes the communication of the potential level of risk of using model outputs. Unfortunately, in practice, this may result in overly pessimistic communication of the MS output, undermining the credibility of simulation predictions to decision makers. This presentation proposes an alternative assessment mechanism, utilizing results parameter robustness, also known as model input sensitivity, to improve the credibility scoring process for specific simulations.

Credibility↗

Development, Application, and Transition of Aerosol and Trace Gas Products Derived from Next-Generation Satellite Observations to Operations

NASA Short-term Prediction Research and Transition (SPoRT) Center has a history of successfully transitioning unique observations and research capabilities to the operational weather community to improve short-term forecasts. SPoRTstrives to bridge the gap between research and operations by maintaining interactive partnerships with end users to develop products that match specific forecast challenges, provide training, and assess the products in the operational environment. This presentation focuses on recent product development, application, and transition of aerosol and trace gas products to operations for specific forecasting applications. Recent activities relating to the SPoRT ozone products, aerosol optical depth composite product, sulfur dioxide, and aerosol index products are discussed.

remote sensing↗

No-Oven, No-Autoclave Composite Processing

Very large composite structures, such as those used in NASA's Space Launch System, push the boundaries imposed by current autoclaves. New technology is needed to maintain composite performance and free manufacturing engineers from the restraints of curing equipment size limitations. Recent efforts on a Phase II project by Cornerstone Research Group, Inc. (CRG), have advanced the technology and manufacturing readiness levels of a unique two-part epoxy resin system. Designed for room-temperature infusion of a dry carbon preform, the system includes a no-heat-added cure that delivers 350 F composite performance in a matter of hours. This no-oven, no-autoclave (NONA) composite processing eliminates part-size constraints imposed by infrastructure and lowers costs by increasing throughput and reducing capital-specific, process-flow bottlenecks. As a result of the Phase II activity, NONA materials and processes were used to make high-temperature composite tooling suitable for further production of carbon-epoxy laminates and honeycomb/ sandwich-structure composites with an aluminum core. The technology platform involves tooling design, resin infusion processing, composite part design, and resin chemistry. The various technology elements are combined to achieve a fully cured part. The individual elements are not unusual, but they are combined in such a way that enables proper management of the heat generated by the epoxy resin during cure. The result is a self-cured carbon/ epoxy composite part that is mechanically and chemically stable at temperatures up to 350 F. As a result of the successful SBIR effort, CRG has launched NONA Composites as a spinoff subsidiary. The company sells resin to end users, fabricates finished goods for customers, and sells composite tooling made with NONA materials and processes to composite manufacturers.

Rauscher, Michael D.↗

A Multifaceted Approach to Modernizing NASA's Advanced Multi-Mission Operations System (AMMOS) System Architecture

The Advanced Multi-Mission Operations Systems (AMMOS) is NASA's premier space mission operations product line offering for use in deep-space robotic and astrophysics missions. The general approach to AMMOS modernization over the course of its 29-year history exemplifies a continual, evolutionary approach with periods of sponsor investment peaks and valleys in between. Today, the Multimission Ground Systems and Services (MGSS) office-the program office that manages the AMMOS for NASA-actively pursues modernization initiatives and continues to evolve the AMMOS by incorporating enhanced capabilities and newer technologies into its end-user tool and service offerings. Despite the myriad of modernization investments that have been made over the evolutionary course of the AMMOS, pain points remain. These pain points, based on interviews with numerous flight project mission operations personnel, can be classified principally into two major categories: 1) information-related issues, and 2) process-related issues. By information-related issues, we mean pain points associated with the management and flow of MOS data across the various system interfaces. By process-related issues, we mean pain points associated with the MOS activities performed by mission operators (i.e., humans) and supporting software infrastructure used in support of those activities. In this paper, three foundational concepts-Timeline, Closed Loop Control, and Separation of Concerns-collectively form the basis for expressing a set of core architectural tenets that provides a multifaceted approach to AMMOS system architecture modernization intended to address the information- and process-related issues. Each of these architectural tenets will be further explored in this paper. Ultimately, we envision the application of these core tenets resulting in a unified vision of a future-state architecture for the AMMOS-one that is intended to result in a highly adaptable, highly efficient, and highly cost-effective set of multimission MOS products and services.

Estefan, Jeff A.↗

Engineering Model Propellant Feed System Development for an Iodine Hall Thruster Demonstration Mission

CUBESATS are relatively new spacecraft platforms that are typically deployed from a launch vehicle as a secondary payload, providing low-cost access to space for a wide range of end-users. These satellites are comprised of building blocks having dimensions of 10x10x10 cu cm and a mass of 1.33 kg (a 1-U size). While providing low-cost access to space, a major operational limitation is the lack of a propulsion system that can fit within a CubeSat and is capable of executing high (Delta)v maneuvers. This makes it difficult to use CubeSats on missions requiring certain types of maneuvers (i.e. formation flying, spacecraft rendezvous). Recently, work has been performed investigating the use of iodine as a propellant for Hall-effect thrusters (HETs) 2 that could subsequently be used to provide a high specific impulse path to CubeSat propulsion. 3, 4 Iodine stores as a dense solid at very low pressures, making it acceptable as a propellant on a secondary payload. It has exceptionally high ρIsp (density times specific impulse), making it an enabling technology for small satellite near-term applications and providing the potential for systems-level advantages over mid-term high power electric propulsion options. Iodine flow can also be thermally regulated, subliming at relatively low temperature (< 100 C) to yield I2 vapor at or below 50 torr. At low power, the measured performance of an iodine-fed HET is very similar to that of a state-of-the-art xenon-fed thruster. Just as importantly, the current-voltage discharge characteristics of low power iodine-fed and xenon-fed thrusters are remarkably similar, potentially reducing development and qualifications costs by making it possible to use an already-qualified xenon-HET PPU in an iodine-fed system. Finally, a cold surface can be installed in a vacuum test chamber on which expended iodine propellant can deposit. In addition, the temperature doesn't have to be extremely cold to maintain a low vapor pressure in the vacuum chamber (it is under 10(exp −6) torr at −75 C), making it possible to 'cryopump' the propellant with lower-cost recirculating refrigerant-based systems as opposed to using liquid nitrogen or low temperature gaseous helium cryopanels. In the present paper, we describe the design and testing of the engineering model propellant feed system for iSAT (see Fig. 1). The feed system is based around an iodine propellant reservoir and two proportional control valves (PFCVs) that meter the iodine flow to the cathode and anode. The flow is split upstream of the PFCVs to both components can be fed from a common reservoir. Testing of the reservoir is reported to demonstrate that the design is capable of delivering the required propellant flow rates to operate the thruster. The tubing and reservoir are fabricated from hastelloy to resist corrosion by the heated gaseous iodine propellant. The reservoir, tubing, and PFCVs are heated to ensure the sublimed propellant will not re-deposit within the feed system. Heating is accomplished through a number of individual zones to control the overall power expended on heating the system and insulation is employed to minimize the amount of power used to heat the system prior to thruster operation.

Polzin, Kurt A.↗

Exploring Cognition Using Software Defined Radios for NASA Missions

NASA missions typically operate using a communication infrastructure that requires significant schedule planning with limited flexibility when the needs of the mission change. Parameters such as modulation, coding scheme, frequency, and data rate are fixed for the life of the mission. This is due to antiquated hardware and software for both the space and ground assets and a very complex set of mission profiles. Automated techniques in place by commercial telecommunication companies are being explored by NASA to determine their usability by NASA to reduce cost and increase science return. Adding cognition the ability to learn from past decisions and adjust behavior is also being investigated. Software Defined Radios are an ideal way to implement cognitive concepts. Cognition can be considered in many different aspects of the communication system. Radio functions, such as frequency, modulation, data rate, coding and filters can be adjusted based on measurements of signal degradation. Data delivery mechanisms and route changes based on past successes and failures can be made to more efficiently deliver the data to the end user. Automated antenna pointing can be added to improve gain, coverage, or adjust the target. Scheduling improvements and automation to reduce the dependence on humans provide more flexible capabilities. The Cognitive Communications project, funded by the Space Communication and Navigation Program, is exploring these concepts and using the SCaN Testbed on board the International Space Station to implement them as they evolve. The SCaN Testbed contains three Software Defined Radios and a flight computer. These four computing platforms, along with a tracking antenna system and the supporting ground infrastructure, will be used to implement various concepts in a system similar to those used by missions. Multiple universities and SBIR companies are supporting this investigation. This paper will describe the cognitive system ideas under consideration and the plan for implementing them on platforms, including the SCaN Testbed. Discussions in the paper will include how these concepts might be used to reduce cost and improve the science return for NASA missions.

transmitters receivers↗

Towards a New Generation of Agricultural System Data, Models and Knowledge Products: Design and Improvement

This paper presents ideas for a new generation of agricultural system models that could meet the needs of a growing community of end-users exemplified by a set of Use Cases. We envision new data, models and knowledge products that could accelerate the innovation process that is needed to achieve the goal of achieving sustainable local, regional and global food security. We identify desirable features for models, and describe some of the potential advances that we envisage for model components and their integration. We propose an implementation strategy that would link a "pre-competitive" space for model development to a "competitive space" for knowledge product development and through private-public partnerships for new data infrastructure. Specific model improvements would be based on further testing and evaluation of existing models, the development and testing of modular model components and integration, and linkages of model integration platforms to new data management and visualization tools.

Agricultural systems↗

Analysis of Satellite and Airborne Imagery for Detection of Water Hyacinth and Other Invasive Floating Macrophytes and Tracking of Aquatic Weed Control Efficacy

Waterways of the Sacramento San Joaquin Delta have recently become infested with invasive aquatic weeds such as floating water hyacinth (Eichhoria crassipes) and water primrose (Ludwigia peploides). These invasive plants cause many negative impacts, including, but not limited to: the blocking of waterways for commercial shipping and boating; clogging of irrigation screens, pumps and canals; and degradation of biological habitat through shading. Zhang et al. (1997, Ecological Applications, 7(3), 1039-1053) used NASA Landsat satellite imagery together with field calibration measurements to map physical and biological processes within marshlands of the San Francisco Bay. Live green biomass (LGB) and related variables were correlated with a simple vegetation index ratio of red and near infra-red bands from Landsat images. More recently, the percent (water area) cover of water hyacinth plotted against estimated LGB of emergent aquatic vegetation in the Delta from September 2014 Landsat imagery showed an 80 percent overall accuracy. For the past two years, we have partnered with the U. S. Department of Agriculture (USDA) and the Department of Plant Sciences, University of California at Davis to conduct new validation surveys of water hyacinth and water primrose coverage and LGB in Delta waterways. A plan is underway to transfer decision support tools developed at NASA's Ames Research Center based on Landsat satellite images to improve Delta-wide integrated management of floating aquatic weeds, while reducing chemical control costs. The main end-user for this application project will be the Division of Boating and Waterways (DBW) of the California Department of Parks and Recreation, who has the responsibility for chemical control of water hyacinth in the Delta.

Agriculture↗

Preparations for Integrating Space-Based Total Lightning Observations into Forecast Operations

NASA's Short-term Prediction Research and Transition (SPoRT) Center has been a leader in collaborating with the United States National Weather Service (NWS) offices to integrate ground-based total lightning (intra-cloud and cloud-to-ground) observations into the real-time operational environment. For much of these collaborations, the emphasis has been on training, dissemination of data to the NWS AWIPS system, and focusing on the utility of these data in the warning decision support process. A shift away from this paradigm has occurred more recently for several reasons. For one, SPoRT's collaborations have expanded to new partners, including emergency managers and the aviation community. Additionally, and most importantly, is the impending launch of the GOES-R Geostationary Lightning Mapper (GLM). This has led to collaborative efforts to focus on additional forecast needs, new data displays, develop training for GLM uses based on the lessons learned from ground-based lightning mapping arrays, and ways to better relate total lightning data to other meteorological parameters. This presentation will focus on these efforts to prepare the operational end user community for GLM with an eye towards sharing lessons learned as EUMETSAT prepares for the Meteosat Third Generation Lightning Imager. This will focus on both software and training needs. In particular, SPoRT has worked closely with the Meteorological Development Laboratory to create the total lightning tracking tool. This software allows for NWS forecasters to manually track storms of interest and display a time series trend of observations. This tool also has been expanded to work on any gridded data set allowing for easy visual comparisons of multiple parameters in addition to total lightning. A new web display has been developed for the ground-based observations that can be easily extended to satellite observations. This paves the way for new collaborations outside of the NWS, both domestically and internationally, as the web display will be functional on PCs and mobile devices. Furthermore, SPoRT has helped developed the software plug-in to visualize GLM data. Examples using the official GLM proxy product will be used to provide a glimpse as to what real-time GLM and likely MTG-LI data will be in the near future.

GLM↗

Using Satellite Remote Sensing to Map Changes in Aquatic Invasive Plant Cover in the Sacramento-San Joaquin River Delta of California

Waterways of the Sacramento San Joaquin Delta have recently become infested with invasive aquatic weeds such as floating water hyacinth (Eichhoria crassipes) and water primrose (Ludwigia peploides). These invasive plants cause many negative impacts, including, but not limited to: the blocking of waterways for commercial shipping and boating; clogging of irrigation screens, pumps and canals; and degradation of biological habitat through shading. Zhang et al. (1997, Ecological Applications, 7(3), 1039-1053) used NASA Landsat satellite imagery together with field calibration measurements to map physical and biological processes within marshlands of the San Francisco Bay. Live green biomass (LGB) and related variables were correlated with a simple vegetation index ratio of red and near infra-red bands from Landsat images. More recently, the percent (water area) cover of water hyacinth plotted against estimated LGB of emergent aquatic vegetation in the Delta from September 2014 Landsat imagery showed a 80% overall accuracy. For the past two years, we have partnered with the U. S. Department of Agriculture (USDA) and the Department of Plant Sciences, University of California at Davis to conduct new validation surveys of water hyacinth and water primrose coverage and LGB in Delta waterways. A plan is underway to transfer decision support tools developed at NASA's Ames Research Center based on Landsat satellite images to improve Delta-wide integrated management of floating aquatic weeds, while reducing chemical control costs. The main end-user for this application project will be the Division of Boating and Waterways (DBW) of the California Department of Parks and Recreation, who has the responsibility for chemical control of water hyacinth in the Delta.

Map Changes↗

High Data Rate Architecture (HiDRA)

One of the greatest challenges in developing new space technology is in navigating the transition from ground based laboratory demonstration at Technology Readiness Level 6 (TRL-6) to conducting a prototype demonstration in space (TRL-7). This challenge is com- pounded by the relatively low availability of new spacecraft missions when compared with aeronautical craft to bridge this gap, leading to the general adoption of a low-risk stance by mission management to accept new, unproven technologies into the system. Also in consideration of risk, the limited selection and availability of proven space-grade components imparts a severe limitation on achieving high performance systems by current terrestrial technology standards. Finally from a space communications point of view the long duration characteristic of most missions imparts a major constraint on the entire space and ground network architecture, since any new technologies introduced into the system would have to be compliant with the duration of the currently deployed operational technologies, and in some cases may be limited by surrounding legacy capabilities. Beyond ensuring that the new technology is verified to function correctly and validated to meet the needs of the end users the formidable challenge then grows to additionally include: carefully timing the maturity path of the new technology to coincide with a feasible and accepting future mission so it flies before its relevancy has passed, utilizing a limited catalog of available components to their maximum potential to create meaningful and unprecedented new capabilities, designing and ensuring interoperability with aging space and ground infrastructures while simultaneously providing a growth path to the future. The International Space Station (ISS) is approaching 20 years of age. To keep the ISS relevant, technology upgrades are continuously taking place. Regarding communications, the state-of-the-art communication system upgrades underway include high-rate laser terminals. These must interface with the existing, aging data infrastructure. The High Data Rate Architecture (HiDRA) project is designed to provide networked store, carry, and forward capability to optimize data flow through both the existing radio frequency (RF) and new laser communications terminal. The networking capability is realized through the Delay Tolerant Networking (DTN) protocol, and is used for scheduling data movement as well as optimizing the performance of existing RF channels. HiDRA is realized as a distributed FPGA memory and interface controller that is itself controlled by a local computer running DTN software. Thus HiDRA is applicable to other arenas seeking to employ next-generation communications technologies, e.g. deep space. In this paper, we describe HiDRA and its far-reaching research implications.

DTN↗

Structure from Motion Photogrammetry and Micro X-Ray Computed Tomography 3-D Reconstruction Data Fusion for Non-Destructive Conservation Documentation of Lunar Samples

Our team is developing a modern, cross-disciplinary approach to documentation and preservation of astromaterials, specifically lunar and meteorite samples stored at the Johnson Space Center (JSC) Lunar Sample Laboratory Facility. Apollo Lunar Sample 60639, collected as part of rake sample 60610 during the 3rd Extra-Vehicular Activity of the Apollo 16 mission in 1972, served as the first NASA-preserved lunar sample to be examined by our team in the development of a novel approach to internal and external sample visualization. Apollo Sample 60639 is classified as a breccia with a glass-coated side and pristine mare basalt and anorthosite clasts. The aim was to accurately register a 3-dimensional Micro X-Ray Computed Tomography (XCT)-derived internal composition data set and a Structure-From-Motion (SFM) Photogrammetry-derived high-fidelity, textured external polygonal model of Apollo Sample 60639. The developed process provided the means for accurate, comprehensive, non-destructive visualization of NASA's heritage lunar samples. The data products, to be ultimately served via an end-user web interface, will allow researchers and the public to interact with the unique heritage samples, providing a platform to "slice through" a photo-realistic rendering of a sample to analyze both its external visual and internal composition simultaneously.

Beaulieu, K. R.↗

SpaceCubeX: A Framework for Evaluating Hybrid Multi-Core CPU FPGA DSP Architectures

The SpaceCubeX project is motivated by the need for high performance, modular, and scalable on-board processing to help scientists answer critical 21st century questions about global climate change, air quality, ocean health, and ecosystem dynamics, while adding new capabilities such as low-latency data products for extreme event warnings. These goals translate into on-board processing throughput requirements that are on the order of 100-1,000 more than those of previous Earth Science missions for standard processing, compression, storage, and downlink operations. To study possible future architectures to achieve these performance requirements, the SpaceCubeX project provides an evolvable testbed and framework that enables a focused design space exploration of candidate hybrid CPU/FPGA/DSP processing architectures. The framework includes ArchGen, an architecture generator tool populated with candidate architecture components, performance models, and IP cores, that allows an end user to specify the type, number, and connectivity of a hybrid architecture. The framework requires minimal extensions to integrate new processors, such as the anticipated High Performance Spaceflight Computer (HPSC), reducing time to initiate benchmarking by months. To evaluate the framework, we leverage a wide suite of high performance embedded computing benchmarks and Earth science scenarios to ensure robust architecture characterization. We report on our projects Year 1 efforts and demonstrate the capabilities across four simulation testbed models, a baseline SpaceCube 2.0 system, a dual ARM A9 processor system, a hybrid quad ARM A53 and FPGA system, and a hybrid quad ARM A53 and DSP system.

Hybrid Flight Architectures↗

Land Cover/Land Use Classification and Change Detection Analysis with Astronaut Photography and Geographic Object-Based Image Analysis

For over fifty years, NASA astronauts have taken exceptional photographs of the Earth from the unique vantage point of low Earth orbit (as well as from lunar orbit and surface of the Moon). The Crew Earth Observations (CEO) Facility is the NASA ISS payload supporting astronaut photography of the Earth surface and atmosphere. From aurora to mountain ranges, deltas, and cities, there are over two million images of the Earth's surface dating back to the Mercury missions in the early 1960s. The Gateway to Astronaut Photography of Earth website (eol.jsc.nasa.gov) provides a publically accessible platform to query and download these images at a variety of spatial resolutions and perform scientific research at no cost to the end user. As a demonstration to the science, application, and education user communities we examine astronaut photography of the Washington D.C. metropolitan area for three time steps between 1998 and 2016 using Geographic Object-Based Image Analysis (GEOBIA) to classify and quantify land cover/land use and provide a template for future change detection studies with astronaut photography.

Hollier, Andi B.↗

Modeling of Gaseous Oxygen Liquefaction Inside Mars Ascent Vehicle Propellant Tank

The In-Situ production of propellants for Mars missions has been considered to utilize the Carbon dioxide (CO2) in Mars atmosphere to produce Oxygen using a high temperature solid oxide electrolyzer. The oxygen then needs to be cooled, liquefied, and stored to be available for propulsion and other end users. The storage period could be up to two years either in the actual Mars ascent propulsion tanks or in a separate tank. Recent investigations have demonstrated the feasibility of both achieving zero-boil-off and controlling the pressure of oxygen within a tank using high efficiency cryocoolers. A representative configuration of tube on tank liquefaction using cryocooler is shown in Fig. 1.

Cryogenic fluids↗

Efficiently Serving HDF5 Products via OPeNDAP

Hyrax OPeNDAP services are widely used by the Earth Science data centers in NASA, NOAA and other organizations to serve end users. In this talk, we will present some key features added in the HDF5 Hyrax OPeNDAP handler that can help data centers to better serve the HDF5netCDF-4 data products. Among these new features, we will focus on the following:1.The DAP4 support 2.The memory cache and the disk cache support that can reduce the service access time 3.The enhancement that makes the swath-like HDF5 products visualized by CF-client tools. We will also discuss the role of the HDF5 handler in-depth in the recent study of the Hyrax service in the cloud environment.

DAP4↗

Re-Organizing Earth Observation Data Storage to Support Temporal Analysis of Big Data

The Earth Observing System Data and Information System archives many datasets that are critical to understanding long-term variations in Earth science properties. Thus, some of these are large, multi-decadal datasets. Yet the challenge in long time series analysis comes less from the sheer volume than the data organization, which is typically one (or a small number of) time steps per file. The overhead of opening and inventorying complex, API-driven data formats such as Hierarchical Data Format introduces a small latency at each time step, which nonetheless adds up for datasets with O(10^6) single-timestep files. Several approaches to reorganizing the data can mitigate this overhead by an order of magnitude: pre-aggregating data along the time axis (time-chunking); storing the data in a highly distributed file system; or storing data in distributed columnar databases. Storing a second copy of the data incurs extra costs, so some selection criteria must be employed, which would be driven by expected or actual usage by the end user community, balanced against the extra cost.

data storage↗