Solar Irradiance on Equator Facing Tilted Surfaces: An extension of the NASA POWER Project Solar Energy Dataset
Explore the source record for details and available documents.
solar irradiance↗
SEARCH · Engineering Papers
Results for “validation dataset”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
From Text to Maps: LLM-Driven Extraction and Geotagging of Epidemiological Data
Epidemiological datasets are essential for public health analysis and decision-making, yet they remain scarce and often difficult to compile due to inconsistent data formats, language barriers, and evolving political boundaries. Traditional methods of creating such datasets involve extensive manual effort and are prone to errors in accurate location extraction. To address these challenges, we propose utilizing large language models (LLMs) to automate the extraction and geotagging of epidemiological data from textual documents. Our approach significantly reduces the manual effort required, limiting human intervention to validating a subset of records against text snippets and verifying the geotagging reasoning, as opposed to reviewing multiple entire documents manually to extract, clean, and geotag. Additionally, the LLMs identify information often overlooked by human annotators, further enhancing the dataset’s completeness. Our findings demonstrate that LLMs can be effectively used to semi-automate the extraction and geotagging of epidemiological data, offering several key advantages: (1) comprehensive information extraction with minimal risk of missing critical details; (2) minimal human intervention; (3) higher-resolution data with more precise geotagging; and (4) significantly reduced resource demands compared to traditional methods.
Harrod, Karly↗
Compaction of crushed salt for safe containment – overview of the KOMPASS project
Abstract. In Germany, rock salt formations are possible host rock candidates for a repository for heat-emitting radioactive waste. The safety concept of a repository in salt bases on a multibarrier system consisting mainly of the geological barrier salt and geotechnical seals ensuring safe containment. Crushed salt will be used for backfilling of cavities and sealing measures in drifts and shafts due to its favourable properties and its easy availability (mined-off material). The creep of the rock salt leads to crushed salt compaction with time. Thereby, the crushed salts' porosity is reduced from the initial porosity of 30 %–40 % to a value comparable to the porosity of undisturbed rock salt (≤1 %). In such low porosity ranges, technical impermeability is assumed. The compaction behaviour of crushed salt is rather complex and involves several coupled THM processes (Kröhn et al., 2017; Hansen et al., 2014). It is influenced by internal properties like humidity and grain size distribution, as well as boundary conditions such as temperature, compaction rate or stress state. However, the current process understanding has some important gaps referring to the material behaviour, experimental database and numerical modelling. It needs to be extended and validated, especially in the low porosity range. The objective of the KOMPASS project was development of methods and strategies for the reduction of deficits in the prediction of crushed salt compaction leading to an improvement of the prognosis quality. Key results are as follows (KOMPASS Phase 1, 2020): selection of an easily available and permanently producible synthetic crushed salt mixture, acting as a reference material for generic investigations; development and proof of different techniques for producing pre-compacted samples for further investigations; establishment of a tool of microstructure investigation methods to demonstrate the comparability of grain structures of pre-compacted samples with in-situ compacted material for future investigations; execution of various laboratory experiments using pre-compacted samples, e.g. long-term creep tests which deliver reliable information about time- and stress-dependent compaction behaviour; development of a complex experimental investigation strategy to derive necessary model parameters considering individual functional dependencies. Its technical feasibility was successfully verified; benchmarking with various existing numerical models using datasets from three different triaxial long-term tests. The result was not entirely satisfactory; however, the number of influencing factors is small and further validation work has to be done. Overall, the KOMPASS project has made significant progress in the approaches to solving the outstanding question, building the basis for further investigations.
Friedenberg, Larissa↗
Seismic signal augmentation to improve generalization of deep neural networks
Deep learning has emerged as an effective approach for seismic data processing in general, and for earthquake monitoring in particular. The ability of deep learning models to generalize beyond the training and validation data is important for comprehensive earthquake monitoring; this ability furthermore depends on the availability of a sufficiently large and complete training dataset. However, this requirement can prove challenging to meet due to significant effort and time for data collection and labeling. Data augmentation provides an efficient and effective approach for increasing the dimension of training samples and improving generalization to unseen samples. In this paper, we present augmentation methods appropriate for seismic waveforms and demonstrate their ability to reduce bias and increase performance. Furthermore, these augmentation methods can be applied to a wide range of deep learning applications designed for seismic data.
58 GEOSCIENCES↗
Overview of the SCEC/USGS Community Stress Drop Validation Study Using the 2019 Ridgecrest Earthquake Sequence
We present initial findings from the ongoing Community Stress Drop Validation Study to compare spectral stress-drop estimates for earthquakes in the 2019 Ridgecrest, California, sequence. This study uses a unified dataset to independently estimate earthquake source parameters through various methods. Stress drop, which denotes the change in average shear stress along a fault during earthquake rupture, is a critical parameter in earthquake science, impacting ground motion, rupture simulation, and source physics. Spectral stress drop is commonly derived by fitting the amplitude-spectrum shape, but estimates can vary substantially across studies for individual earthquakes. Sponsored jointly by the U.S. Geological Survey and the Statewide (previously, Southern) California Earthquake Center our community study aims to elucidate sources of variability and uncertainty in earthquake spectral stress-drop estimates through quantitative comparison of submitted results from independent analyses. The dataset includes nearly 13,000 earthquakes ranging from M 1 to 7 during a two-week period of the 2019 Ridgecrest sequence, recorded within a 1° radius. Here, in this article, we report on 56 unique submissions received from 20 different groups, detailing spectral corner frequencies (or source durations), moment magnitudes, and estimated spectral stress drops. Methods employed encompass spectral ratio analysis, spectral decomposition and inversion, finite-fault modeling, ground-motion-based approaches, and combined methods. Initial analysis reveals significant scatter across submitted spectral stress drops spanning over six orders of magnitude. However, we can identify between-method trends and offsets within the data to mitigate this variability. Averaging submissions for a prioritized subset of 56 events shows reduced variability of spectral stress drop, indicating overall consistency in recovered spectral stress-drop values.
58 GEOSCIENCES↗
WFIP3 Lidar / Processed Data
This dataset contains preliminary processed data from the Leosphere WindCube V2.1 deployed at Nantucket, MA for WFIP-3. The lidar was tall tower validated in May 2023.
17 WIND ENERGY↗
WFIP3 Lidar / Processed Data
This dataset contains standardized, processed data from the Leosphere WindCube V2.1 deployed at Nantucket, MA for WFIP-3. The lidar was tall tower validated in May 2023.
17 WIND ENERGY↗
Better Weather Prediction and Climate Diagnostics Using Rainfall Measurements from Space
Progress in understanding of the role of water in global weather and climate is currently limited by our knowledge of the spatial and temporal variability of primary hydrological fields such as precipitation and evaporation. The Tropical Rainfall Measuring Mission (TRMM) has recently demonstrated that use of microwave-based rainfall observations from space in data assimilation can provide better climate data sets and improve short-range weather forecasting. At NASA, we have been exploring non-traditional approaches to assimilating TRMM Microwave Imager (TMI) and Special Sensor Microwavehager (SSM/I) surface rain rate and latent heating profile information in global systems. In this talk we show that assimilating microwave rain rates using a continuous variational assimilation scheme based on moisture tendency corrections improves quantitative precipitation estimates (QPE) and related clouds, radiation energy fluxes, and large-scale circulations in the Goddard Earth Observing System (GEOS) reanalyses. Short-range forecasts initialized with these improved analyses also yield better QPE scores and storm track predictions for Hurricanes Bonnie and Floyd. We present a status report on current efforts to assimilate convective and stratiform latent heating profile information within the general variational framework of model parameter estimation to seek further improvements. Within the next 5 years, there will be a gradual increase in microwave rain products available from operational and research satellites, culminating to a target constellation of 9 satellites to provide global rain measurements every 3 hours with the proposed Global Precipitation Measurement (GPM) mission in 2007/2008. Based on what has been learned from TRMM, there is a high degree of confidence that these observations can play a'major role in improving weather forecasts and producing better global datasets for understanding the Earth's water and energy cycle. The key to success is to adopt an integrated approach to retrieval, validation, modeling, and data assimilation in a coordinated end-to-end observation-application program.
Hou, Arthur↗
ICESat-2 Simulated Data from Airborne Altimetery
Ice, Cloud, and land Elevation Satellite-2 (ICESat-2) is scheduled to launch in 2015 and will carry onboard the Advanced Topographic Laser Altimeter System (ATLAS), which represents a new approach to spaceborne determination of surface elevations. Specifically, the current ATLAS design is for a micropulse, multibeam, photon-counting laser altimeter with lower energy, a shorter pulse width, and a higher repetition rate relative to the Geoscience Laser Altimeter (GLAS), the instrument that was onboard ICESat. Given the new and untested technology associated with ATLAS, airborne altimetry data is necessary (1) to test the proposed ATLAS instrument geometry, (2) to validate instrument models, and (3) to assess the atmospheric effects on multibeam altimeters. We present an overview of the airborne instruments and datasets intended to address the ATLAS instrument concept, including data collected over Greenland (July 2009) using an airborne SBIR prototype 100 channel, photon-counting, terrain mapping altimeter, which addresses the first of these 3 scientific concerns. Additionally, we present the plan for further simulator data collection over vegetated and ice covered regions using Multiple Altimeter Beam Experimental Lidar (MABEL), intended to address the latter two scientific concerns. As the ICESAT-2 project is in the design phase, the particular configuration of the ATLAS instrument may change. However, we expect this work to be relevant as long as ATLAS pursues a photon-counting approach.
Brunt, Kelly M.↗
NASA IMAGESEER: NASA IMAGEs for Science, Education, Experimentation and Research
A number of web-accessible databases, including medical, military or other image data, offer universities and other users the ability to teach or research new Image Processing techniques on relevant and well-documented data. However, NASA images have traditionally been difficult for researchers to find, are often only available in hard-to-use formats, and do not always provide sufficient context and background for a non-NASA Scientist user to understand their content. The new IMAGESEER (IMAGEs for Science, Education, Experimentation and Research) database seeks to address these issues. Through a graphically-rich web site for browsing and downloading all of the selected datasets, benchmarks, and tutorials, IMAGESEER provides a widely accessible database of NASA-centric, easy to read, image data for teaching or validating new Image Processing algorithms. As such, IMAGESEER fosters collaboration between NASA and research organizations while simultaneously encouraging development of new and enhanced Image Processing algorithms. The first prototype includes a representative sampling of NASA multispectral and hyperspectral images from several Earth Science instruments, along with a few small tutorials. Image processing techniques are currently represented with cloud detection, image registration, and map cover/classification. For each technique, corresponding data are selected from four different geographic regions, i.e., mountains, urban, water coastal, and agriculture areas. Satellite images have been collected from several instruments - Landsat-5 and -7 Thematic Mappers, Earth Observing-1 (EO-1) Advanced Land Imager (ALI) and Hyperion, and the Moderate Resolution Imaging Spectroradiometer (MODIS). After geo-registration, these images are available in simple common formats such as GeoTIFF and raw formats, along with associated benchmark data.
Le Moigne, Jacqueline↗
Mapping Global Forest Age from Forest Inventories, Biomass and Climate Data
Forest age can determine the capacity of a forest to uptake carbon from the atmosphere. However, a lack of global diagnostics that reflect the forest stage and associated disturbance regimes hampers the quantification of age-related differences in forest carbon dynamics. This study provides a new global distribution of forest age circa 2010, estimated using a machine learning approach trained with more than 40 000 plots using forest inventory, biomass and climate data. First, an evaluation against the plot-level measurements of forest age reveals that the data-driven method has a relatively good predictive capacity of classifying old-growth vs. non-old-growth (precision = 0.81 and 0.99 for old-growth and non-old-growth, respectively) forests and estimating corresponding forest age estimates (NSE = 0.6 – Nash–Sutcliffe efficiency – and RMSE = 50 years – root-mean-square error). However, there are systematic biases of overestimation in young- and underestimation in old-forest stands, respectively. Globally, we find a large variability in forest age with the old-growth forests in the tropical regions of Amazon and Congo, young forests in China, and intermediate stands in Europe. Furthermore, we find that the regions with high rates of deforestation or forest degradation (e.g. the arc of deforestation in the Amazon) are composed mainly of younger stands. Assessment of forest age in the climate space shows that the old forests are either in cold and dry regions or warm and wet regions, while young–intermediate forests span a large climatic gradient. Finally, comparing the presented forest age estimates with a series of regional products reveals differences rooted in different approaches and different in situ observations and global-scale products. Despite showing robustness in cross-validation results, additional methodological insights on further developments should as much as possible harmonize data across the different approaches. The forest age dataset presented here provides additional insights into the global distribution of forest age to better understand the global dynamics in the forest water and carbon cycles. The forest age datasets are openly available at https://doi.org/10.17871/ForestAgeBGI.2021 (Besnard et al., 2021).
Simon Besnard↗
Optimal Estimation Framework for Ocean Color Atmospheric Correction and Pixel-level Uncertainty Quantification
Ocean color remote sensing requires compensation for atmospheric scattering and absorption (aerosol, Rayleigh, and trace gases), referred to as atmospheric correction (AC). AC allows inference of parameters such as spectrally resolved remote sensing reflectance ( R rs )(λ) ; sr 1 ) at the ocean surface from the top-of-atmosphere reflectance. Often, the uncertainty of this process is not fully explored. Bayesian inference techniques provide a simultaneous AC and uncertainty assessment via a full posterior distribution of the relevant variables, given the prior distribution of those variables and the radiative transfer (RT) likelihood function. Given uncertainties in the algorithm inputs, the Bayesian framework enables better constraints on the AC process by using the complete spectral information compared to traditional approaches that use only a subset of bands for AC. This paper investigates a Bayesian inference research method (Optimal Estimation, OE) for ocean color AC by simultaneously retrieving atmospheric and ocean properties using all visible and near-infrared spectral bands. The OE algorithm analytically approximates the posterior distribution of parameters based on normality assumptions and provides a potentially viable operational algorithm with a reduced computational expense. We developed a Neural Network (NN) RT forward model look-up-table-based emulator to increase algorithm efficiency further and thus speed up the likelihood computations. We then applied the OE algorithm to synthetic data and observations from the MODerate resolution Imaging Spectroradiometer (MODIS) on NASA’s Aqua spacecraft. We compared the R rs )(λ) retrieval and its uncertainty estimates from the OE method with in-situ validation data from the SeaWiFS Bio-optical Archive and Storage System (SeaBASS) and Aerosol Robotic Network Ocean Color (AERONET-OC) datasets. The OE algorithm improved R rs )(λ) estimates relative to the NASA standard operational algorithm by improving all statistical metrics at 443, 555, and 667 nm. Unphysical negative R rs )(λ) , which often appear in complex water conditions, was reduced by a factor of 3. The OE-derived pixel-level R rs )(λ) uncertainty estimates were also assessed relative to in-situ data and were shown to have skill.
Atmospheric correction↗
The Transition of Satellite Observations Assimilated in GEOS to JEDI
The Goddard Earth Observing System (GEOS) is used by NASA’s Global Modeling and Assimilation Office (GMAO) to produce weather, climate, and air quality forecasts and reanalysis datasets. In order to incorporate the Joint Effort for Data assimilation Integration (JEDI) system into GEOS it is necessary to provide a validated set of input observations in JEDI. The GMAO, in collaboration with the Joint Center for Satellite Data Assimilation (JCSDA), is developing the Unified Forward Operator (UFO) in JEDI and adding all the necessary features to replicate the existing capability of the Gridpoint Statistical Interpolation (GSI)–based GEOS atmospheric data assimilations system. GMAO has been adding, validating, and updating procedures to assimilate the vast array of satellite and conventional observations assimilated in GEOS, including the GEOS all-sky microwave radiance assimilation framework to assimilate those observations in the UFO. Robust tests have been conducted to ensure correct configurations of observational data bias correction, quality control, and observation error in UFO and good agreements between UFO and GSI results. Our work on satellite observations will be reported in this presentation.
Jianjun Jin↗
Validity of the Landsat Surface Reflectance Archive for Aquatic Science: Implications for Cloud-Based Analysis
Originally developed for terrestrial science and applications, the US Geological Survey Landsat surface reflectance (SR) archive spanning ~ 40 yr of observations has been increasingly utilized in large‐scale water‐quality studies. These products, however, have not been rigorously validated using in situ measured reflectance. This letter quantifies and demonstrates the quality of the SR products by harnessing a sizeable global dataset ( N = 1100). We found that the Landsat 8/9 SR in the green and red bands marginally meet the targeted accuracy requirements (30%), whereas the uncertainties in the blue and coastal‐aerosol bands ranged from 48% to 110%. We further observed > +25% biases in the visible bands of Landsat 5/7 SR, which can introduce an apparent downward trend when applied in time‐series analyses combined with Landsat 8/9. Users must exercise caution when using this archive for trend analyses, and progress in atmospheric correction is required to foster advanced applications of the Landsat archive for aquatic science.
Landsat, lakes, harmful algal blooms↗
How does uncertainty of soil organic carbon stock affect the calculation of carbon budgets and soil carbon credits for croplands in the U.S. Midwest?
Cropland carbon budget depicts the amount of carbon flowing in and out of agroecosystems and the changes in carbon stocks of soil and living biomass during the same period. Soil carbon credit is the additional change in soil carbon stock under certain farming practices compared with the business-as-usual practices. Accurately calculating cropland carbon budget and soil carbon credit is critical to assessing climate change mitigation potential in agroecosystems. The calculation of cropland carbon budget and soil carbon credit is sensitive to local soil and climatic conditions, especially initial soil organic carbon (SOC) stock, which is determined by both SOC concentration (SOC%) and bulk density (Bulk_Density). SOC stock data are either from soil sampling or gridded public survey data. In agroecosystem models, SOC stock data are a key model input for quantifying cropland carbon budget and soil carbon credit. However, various types and degrees of uncertainties exist in SOC stock datasets, which propagate to the quantification of SOC stock change. In particular, a large discrepancy is found in two widely used SOC stock datasets — Rapid Carbon Assessment dataset (RaCA) and Gridded Soil Survey Geographic Database (gSSURGO) — in the U.S. Midwest, with a relative difference (quantified using Normalized Root Mean Square Error, NRMSE) of 48.0% for 0–30 cm SOC stock between the two datasets. It remains largely unclear how uncertainty in SOC stocks affects the calculation of cropland carbon budget and soil carbon credit. To address this question, we used a well-validated process-based agroecosystem model, ecosys, to assess the impacts of SOC stock uncertainty on carbon budget and soil carbon credit calculation in the U.S. Midwestern corn-soybean rotation systems. Our results reveal the following findings: (1) A sizable discrepancy exists in simulated cropland carbon budget between using gSSURGO and using RaCA for their SOC% and Bulk_Density as model inputs, with a Pearson correlation coefficient (r) of only 0.4 for simulated change of SOC stock (ΔSOC) using these two different soil datasets. (2) Simulated cropland carbon budget components were more sensitive to initial SOC% than to Bulk_Density. For example, the upper and lower quartiles of multi-year averaged ΔSOC were –29.8 and 4.8 gC/m 2 /year for the selected counties respectively, with an uncertainty of 13.7 and 0.7 gC/m 2 /year induced by uncertainties in initial SOC% and Bulk_Density, respectively. (3) Both simulated ΔSOC and its uncertainty were negatively correlated with initial SOC%, whereas ΔSOC was negatively correlated with air temperature, and ΔSOC uncertainty was positively correlated with air temperature. (4) The uncertainty of calculated soil carbon credits was much smaller compared with the uncertainty of calculated absolute carbon budgets assuming the same SOC stock uncertainty level in the inputs. Specifically, in our assessment comparing planting cover crops vs no cover crop, the uncertainty of calculated soil carbon credits induced by initial SOC% uncertainty was less than 4% (relative to the quantified value of the soil carbon credits) for 90% of the cases. Our analysis highlights that high accuracy measurement of SOC% as inputs is needed for the calculation of cropland carbon budgets; however, soil carbon credit quantification is much less sensitive to the initial SOC% inputs, and the current publicly available soil datasets (e.g., gSSURGO) are largely suitable for the calculation of soil
54 ENVIRONMENTAL SCIENCES↗
Dataset for manuscript "Rotational Memory Function of SPC/E water"
Memory effect are essential for dynamics of condensed materials and are responsible for non-exponential relaxation of correlation functions of dynamic variables through the memory function entering the memory equation. Memory functions of dipole rotations for polar liquids have never been calculated. We present here calculations of memory functions and single-dipole rotations and of the overall system dipole moment for SPC/E water measured by dielectric spectroscopy. The memory functions for single-particle and collective dynamics turn out to be nearly identical. This result validates theories of dielectric spectroscopy in terms of single-particle time correlation function and the connection between the collective and single-particle relaxation times in terms of the Kirkwood factor. The dataset includes single particle and system dipole moments, including their time-dependence.
74 ATOMIC AND MOLECULAR PHYSICS↗
Multi-modal Dataset of a Polycrystalline Metallic Material: 3D Microstructure and Deformation Fields
The development of high-fidelity mechanical property prediction models for the design of polycrystalline materials relies on large volumes of microstructural feature data. Concurrently, at these same scales, the deformation fields that develop during mechanical loading can be highly heterogeneous. Spatially correlated measurements of 3D microstructure and the ensuing deformation fields at the micro-scale would provide highly valuable insight into the relationship between microstructure and macroscopic mechanical response. They would also provide direct validation for numerical simulations that can guide and speed up the design of new materials and microstructures. However, to date, such data have been rare. Here, a one-of-a-kind, multi-modal dataset is presented that combines recent state-of-the-art experimental developments in 3D tomography and high-resolution deformation field measurements.
36 MATERIALS SCIENCE↗
HarDWR - Harmonized Water Rights Records
A dataset within the Harmonized Database of Western U.S. Water Rights (HarDWR). For a detailed description of the database, please see the meta-record v2.0. Changelog v2.0 - Recalculated based on data sourced from WestDAAT - Changed using a Site ID column to identify unique records to using aa combination of Site ID and Allocation ID - Removed the Water Management Area (WMA) column from the harmonized records. The replacement is a separate file which stores the relationship between allocations and WMAs. This allows for allocations to contribute to water right amounts to multiple WMAs during the subsequent cumulative process. - Added a column describing a water rights legal status - Added "Unspecified" was a water source category - Added an acre-foot (AF) column - Added a column for the classification of the right's owner v1.02 - Added a .RData file to the dataset as a convenience for anyone exploring our code. This is an internal file, and the one referenced in analysis scripts as the data objects are already in R data objects. v1.01 - Updated the names of each file with an ID number less than 3 digits to include leading 0s v1.0 - Initial public release Description Here we present an updated database of Western U.S. water right records. This database provides consistent unique identifiers for each water right record, and a consistent categorization scheme that puts each water right record into one of seven broad use categories. These data were instrumental in conducting a study of the multi-sector dynamics of inter-sectoral water allocation changes though water markets (Grogan et al., *in review*). Specifically, the data were formatted for use as input to a process-based hydrologic model, Water Balance Model (WBM), with a water rights module (Grogan et al., *in review*). While this specific study motivated the development of the database presented here, water management in the U.S. West is a rich area of study (e.g., Anderson and Woosly, 2005; Tidwell, 2014; Null and Prudencio, 2016; Carney et al., 2021) so releasing this database publicly with documentation and usage notes will enable other researchers to do further work on water management in the U.S. West. We produced the water rights database presented here in four main steps: (1) data collection, (2) data quality control, (3) data harmonization, and (4) generation of cumulative water rights curves. Each of steps (1)-(3) had to be completed in order to produce (4), the final product that was used in the modeling exercise in Grogan et al. (*in review*). All data in each step is associated with a spatial unit called a Water Management Area (WMA), which is the unit of water right administration utilized by the state in which the right came from. Steps (2) and (3) required use to make assumptions and interpretation, and to remove records from the raw data collection. We describe each of these assumptions and interpretations below so that other researchers can choose to implement alternative assumptions an interpretation as fits their research aims. Motivation for Changing Data Sources The most significant change has been a switch from collecting the raw water rights directly from each state to using the water rights records presented in WestDAAT, a product of the Water Data Exchange (WaDE) Program under the Western States Water Council (WSWC). One of the main reasons for this is that each state of interest is a member of the WSWC, meaning that WaDE is partially funded by these states, as well as many universities. As WestDAAT is also a database with consistent categorization, it has allowed us to spend less time on data collection and quality control and more time on answering research questions. This has included records from water right sources we had previously not known about when creating v1.0 of this database. The only major downside to utilizing the WestDAAT records as our raw data is that further updates are tied to when WestDAAT is updated, as some states update their public water right records daily. However, as our focus is on cumulative water amounts at the regional scale, it is unlikely most records updates would have a significant effect on our results. The structure of WestDAAT led to several important changes to how HarWR is formatted. The most significant change is that WaDE has calculated a field known as `SiteUUID`, which is a unique identifier for the Point of Diversion (POD), or where the water is drawn from. This separate from `AllocationNativeID`, which is the identifier for the allocation of water, or the amount of water associated with the water right. It should be noted that it is possible for a single site to have multiple allocations associated with it and for an allocation to be able to be extracted from multiple sites. The site-allocation structure has allowed us to adapt a more consistent, and hopefully more realistic, approach in organizing the water right records than we had with HarDWR v1.0. This was incredibly helpful as the raw data from many states had multiple water uses within a single field within a single row of their raw data, and it was not always clear if the first water use was the most important, or simply first alphabetically. WestDAAT has already addressed this data quality issue. Furthermore, with v1.0, when there were multiple records with the same water right ID, we selected the largest volume or flow amount and disregarded the rest. As WestDAAT was already a common structure for disparate data formats, we were better able to identify sites with multiple allocations and, perhaps more importantly, allocations with multiple sites. This is particularly helpful when an allocation has sites which cross WMA boundaries, instead of just assigning the full water amount to a single WMA we are now able to divide the amount of water between the number of relevant WMAs. As it is now possible to identify allocations with water used in multiple WMAs, it is no longer practical to store this information within a single column. Instead the stAllocationToWMATab.csv file was created, which is an allocation by WMA matrix containing the percent Place of Use area overlap with each WMA. We then use this percentage to divide the allocation's flow amount between the given WMAs during the cumulation process to hopefully provide more realistic totals of water use in each area. However, not every state provides areas of water use, so like HarDWR v1.0, a hierarchical decision tree was used to assign each allocation to a WMA. First, if a WMA could be identified based on the allocation ID, then that WMA was used; typically, when available, this applied to the entire state and no further steps were needed. Second was the spatial analysis of Place of Use to WMAs. Third was a spatial analysis of the POD locations to WMAs, with the assumption that allocation's POD is within the WMA it should belong to; if an allocation still had multiple WMAs based on its POD locations, then the allocation's flow amount would be divided equally between all WMAs. The fourth, and final, process was to include water allocations which spatially fell outside of the state WMA boundaries. This could be due to several reasons, such as coordinate errors / imprecision in the POD location, imprecision in the WMA boundaries, or rights attached with features, such as a reservoir, which crosses state boundaries. To include these records, we decided for any POD which was within one kilometer of the state's edge would be assigned to the nearest WMA. Other Changes WestDAAT has Allowed In addition to a more nuanced and consistent method of assigning water right's data to WMAs, there are other benefits gained from using the WestDAAT dataset. Among those is a consistent categorization of a water right's legal status. In HarDWR v1.0, legal status was effectively ignored, which led to many valid concerns about the quality of the database related to the amounts of water the rights allowed to be claimed. The main issue was that rights with legal status' such as "application withdrawn", "non-active", or "cancelled" were included within HarDWR v1.0. These, and other water rights status' which were deemed to not be in use have been removed from this version of the database. Another major change has been the addition of the "unspecified water source category. This is water that can come from either surface water or groundwater, or the source of which is unknown. The addition of this source category brings the total number of categories to three. Due to reviewer feedback, we decided to add the acre-foot (AF) column so that the data may be more applicable to a wider audience. We added the ownerClassification column so that the data may be more applicable to a wider audience. File Descriptions The dataset is a series of various files organized by state sub-directories. In addition, each file begins with the state's name, in case the file is separate from its sub-directory for some reason. After the state name is the text which describes the contents of the file. Here is each file described in detail. Note that st is a placeholder for the state's name. stFullRecords_HarmonizedRights.csv: A file of the complete water records for each state. The column headers for each of this type of file are: state - The name of the state to which the allocations belong to. FIPS - The two digit numeric state ID code. siteID - The site location ID for POD locations. A site may have multiple allocations, which are the actual amount of water which can be drawn. In a simplified hypothetical, a farm stead may have an allocation for "irrigation" and an allocation for "domestic" water use, but the water is drawn from the same pumping equipment. It should be noted that many of the site ID appear to have been added by WaDE, and therefore may not be recognized by a given state's water rights database. allocationID - The allocation ID for the water right. For most states this is the water right ID, and what is recommended to use should a right be looked up on a given state's water rights database. The water amounts associated with these IDs tend to be finer scaled than those associated with siteID. It should be noted that some allocations may be extracted from multiple sites, particularly for larger Places of Use. ownerClassification - A classification of the types of owners for water rights. The most common is `Private` which incorporates a wide range of entities. Several classifications would be grouped into a government category, most of which are for the U.S. Federal Government. These allocations could be listed as "Federal", "United States of America", or as the names of any number of federal agencies. The last major grouping of entities is for "Native American"s. priorityDate - The date we use as the water right priority date for our modeling analysis. This is the legal priority date when it is available. However, for some rights, specifically from California and New Mexico, we used a pseudo priority date (e.g. well completion date or start of well drilling date) when a legal priority date was not available. The most questionable dates come from New Mexico, where the only date associated with certain water right records was the date the allocation was recorded in the database. As the allocation record creation tended to be within a few months of the filing of the application of the water right, from manually double checking the water rights, and our analysis focuses on aggregating water rights on the timescale of years, we determined it was acceptable to use such dates to include as many records as possible. primaryBeneficialUse - From the numerous state water use categories, WaDE categorized them into 21 categories WestDAAT. This column is the original WaDE category for the primary water use at the PoD site. allocationBeneficialUse - From the numerous state water use categories, WaDE categorized them into 21 categories for WestDAAT. This column is the original WaDE category
Economics↗