Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data formats”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 973 records · Page 54
All sourcesNASA NTRSDOE OSTI
NASA NTRS2003

Modified Polar-Format Software for Processing SAR Data

HMPF is a computer program that implements a modified polar-format algorithm for processing data from spaceborne synthetic-aperture radar (SAR) systems. Unlike prior polar-format processing algorithms, this algorithm is based on the assumption that the radar signal wavefronts are spherical rather than planar. The algorithm provides for resampling of SAR pulse data from slant range to radial distance from the center of a reference sphere that is nominally the local Earth surface. Then, invoking the projection-slice theorem, the resampled pulse data are Fourier-transformed over radial distance, arranged in the wavenumber domain according to the acquisition geometry, resampled to a Cartesian grid, and inverse-Fourier-transformed. The result of this process is the focused SAR image. HMPF, and perhaps other programs that implement variants of the algorithm, may give better accuracy than do prior algorithms for processing strip-map SAR data from high altitudes and may give better phase preservation relative to prior polar-format algorithms for processing spotlight-mode SAR data.

Chen, Curtis↗
DOE OSTI2021

Photoelectrochemically Self Improving Si/GaN Photocathode: Figure 4a Raw Data

Calculated formation energy of (0001 ̅) c-plane GaN surfaces. The density-functional theory (DFT) calculations were performed with the generalized gradient approximation, using the plane-wave pseudopotential quantum-ESPRESSO package. Ultrasoft pseudopotentials were used to describe the interaction between valence electrons and ionic cores, where Ga 3d electrons were explicitly treated as valence electrons. The wave functions and electronic density were expanded in a plane-wave basis set truncated at a cutoff energy of 30 Ry and 240 Ry, respectively. The surface was modeled using a repeated slab geometry using a (2×2) unit cell with a thickness of seven GaN bilayers, where the Ga-terminated bottom side was passivated by pseudo-hydrogen with charge of 1.25e. In addition, the (1010) surface was modeled using 8 GaN atomic layers with a lateral dimension of 9.57 Å x 10.38 Å. A vacuum width of at least 15.0 Å was introduced between consecutive slabs, and a k-point mesh of 5x5x1 and 2x2x1 was used to sample the surface Brillouin zone for the polar and non-polar surfaces, respectively. Calculated formation energy of (0001 ̅) c-plane GaN surfaces with different oxygen configuration within the anion and cation-rich limits: ideal GaN surfaces of c-plane and m-plane; configurations with 100% replacement of nitrogen by oxygen on the surface c-plane and m-plane; 100% replacement of nitrogen by oxygen in the subsurface bilayer c-plane and m-plane; and configurations with 50% replacement of nitrogen of the surface and subsurface bilayers by oxygens c-plane and m-plane. In both cases, the ideal surface is chosen as the reference.

photocathode↗
DOE OSTI2021

Photoelectrochemically Self Improving Si/GaN Photocathode: Figure 4b Raw Data

Calculated formation energy of (1010 ̅) m-plane GaN surfaces. The density-functional theory (DFT) calculations were performed with the generalized gradient approximation, using the plane-wave pseudopotential quantum-ESPRESSO package. Ultrasoft pseudopotentials were used to describe the interaction between valence electrons and ionic cores, where Ga 3d electrons were explicitly treated as valence electrons. The wave functions and electronic density were expanded in a plane-wave basis set truncated at a cutoff energy of 30 Ry and 240 Ry, respectively. The surface was modeled using a repeated slab geometry using a (2×2) unit cell with a thickness of seven GaN bilayers, where the Ga-terminated bottom side was passivated by pseudo-hydrogen with charge of 1.25e. In addition, the (1010) surface was modeled using 8 GaN atomic layers with a lateral dimension of 9.57 Å x 10.38 Å. A vacuum width of at least 15.0 Å was introduced between consecutive slabs, and a k-point mesh of 5x5x1 and 2x2x1 was used to sample the surface Brillouin zone for the polar and non-polar surfaces, respectively. Calculated formation energy of (1010 ̅) m-plane GaN surfaces with different oxygen configuration within the anion and cation-rich limits: ideal GaN surfaces of c-plane and m-plane; configurations with 100% replacement of nitrogen by oxygen on the surface c-plane and m-plane; 100% replacement of nitrogen by oxygen in the subsurface bilayer c-plane and m-plane; and configurations with 50% replacement of nitrogen of the surface and subsurface bilayers by oxygens c-plane and m-plane. In both cases, the ideal surface is chosen as the reference.

photocathode↗
DOE OSTI2019

Eddy Flux and Meteorology over Deciduous Forest, Prairie, and Soybean Ecosystems in Missouri, USA, during the Total Solar Eclipse of 2017

Eddy fluxes and meteorology data are reported at high temporal resolution for three flux tower sites in Mid-Missouri before, during, and after the Total Solar Eclipse of August 21, 2017. Mid-Missouri experienced up to 2 minutes and 40 seconds of totality at around solar noon during the eclipse (Fig. 1). The sites are located in deciduous oak-hickory forest, native prairie, and soybean ecosystems (Table 1 and Fig. 3). Eddy fluxes were computed using a wavelet-based approach that permitted the calculation of two-minute mean fluxes without losing low frequency flux contributions (Fig. 2). During the eclipse, standard meteorological variables were sampled and recorded at 5 second intervals. The two-minute means of fluxes and meteorology are reported. This data product includes three data files in NetCDF (*.nc) format, three data dictionary files and a file-level metadata file in comma separate (*.csv) format, and a user guide in PDF (*.pdf) format.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2018

Extracting the Data From the LCM vk4 Formatted Output File

These are slides about extracting the data from the LCM vk4 formatted output file. The following is covered: vk4 file produced by Keyence VK Software, custom analysis, no off the shelf way to read the file, reading the binary data in a vk4 file, various offsets in decimal lines, finding the height image data, directly in MATLAB, binary output beginning of height image data, color image information, color image binary data, color image decimal and binary data, MATLAB code to read vk4 file (choose a file, read the file, compute offsets, read optical image, laser optical image, read and compute laser intensity image, read height image, timing, display height image, display laser intensity image, display RGB laser optical images, display RGB optical images, display beginning data and save images to workspace, gamma correction subroutine), reading intensity form the vk4 file, linear in the low range, linear in the high range, gamma correction for vk4 files, computing the gamma intensity correction, observations.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
NASA NTRS2007

Improving Access to Precipitation Data for GIS Users: Designing for Ease of Use

The Global Precipitation Measurement Mission (GPM) is a NASA/JAXA led international mission to configure a constellation of space-based radiometers to monitor precipitation over the globe. The GPM goal of making global 3-hour precipitation products available in near real-time will make such global products more useful to a broader community of modelers and Geographic Information Systems (GIS) users than is currently the case with remote sensed precipitation products. Based on the existing interest to make Tropical Rainfall Measuring Mission (TRMM) data available to a growing community of GIS users as well as what will certainly be an expanded community during the GPM era, it is clear that data systems must make a greater effort to provide data in formats easily used by GIS. We describe precipitation GIS products being developed for TRMM data. These products will serve as prototypes for production efforts during the GPM era. We describe efforts to convert TRMM precipitation data to GeoTIFF, Shapefile, and ASCII grid. Clearly, our goal is to format GPM data so that it can be easily used within GIS applications. We desire feedback on these efforts and any additions or direction changes that should be undertaken by the data system.

Stocker, Erich F.↗
NASA NTRS1994

The Rings Node for the Planetary Data System

The Planetary Data System's Rings Node is devoted to the archiving and distributing of scientific data sets relevant to planetary ring systems. The two major classes of ring data are images and occultation profiles, although a variety of additional data types (e.g. spectra, particle absorption signatures, etc.) are also of interest. A large fraction of our data sets are from the Voyager missions to the outer planets, but Earth-based and Hubble Space Telescope data sets are also represented. Archiving work often includes re-formatting the data into standardized formats and reconstructing some of the data processing steps. The Rings Node also performs a variety of services to support research into these data sets. These services include developing on-line catalogs and information systems, filling orders for data, developing software tools, and coordinating special observing campaigns.

Showalter, Mark R.↗
DOE OSTI2024

HarDWR - Harmonized Water Rights Records

For a detailed description of the database of which this record is only one part, please see the HarDWR meta-record. Here we present a new dataset of western U.S. water rights records. This dataset provides consistent unique identifiers for each spatial unit of water management across the domain, unique identifiers for each water right record, and a consistent categorization scheme that puts each water right record into one of 7 broad use categories. These data were instrumental in conducting a study of the multi-sector dynamics of intersectoral water allocation changes through water markets (Grogan et al., in review). Specifically, the data were formatted for use as input to a process-based hydrologic model, WBM, with a water rights module (Grogan et al., in review). While this specific study motivated the development of the database presented here, U.S. west water management is a rich area of study (e.g., Anderson and Woosly, 2005; Tidwell, 2014; Null and Prudencio, 2016; Carney et al, 2021) so releasing this database publicly with documentation and usage notes will enable other researchers to do further work on water management in the U.S. west. The raw downloaded data for each state is described in Lisk et al. (in review), as well as here. The dataset is a series of various files organized by state sub-directories. The first two characters of each file name is the abbreviation for the state the in which the file contains data for. After the abbreviation is the text which describes the contents of the file. Here is each file type described in detail: XXFullHarmonizedRights.csv: A file of the combined groundwater and surface water records for each state. Essentially, this file is the merging of XXGroundwaterHarmonizedRights.csv and XXSurfaceWaterHarmonizedRights.csv by state. The column headers for each of this type of file are: state - The name of the state the data comes from. FIPS - The two-digit numeric state ID code. waterRightID - The unique identifying ID of the water right, the same identifier as its state uses. priorityDate - The priority date associated with the right. origWaterUse - The original stated water use(s) from the state. waterUse - The water use category under the unified use categories established here. source - Whether the right is for surface water or groundwater. basinNum - The alpha-numeric identifier of the WMA the record belongs to. CFS - The maximum flow of the allocation in cubic feet per second (ft3s-1). Arizona is unique among the states, as its surface and groundwater resources are managed with two different sets of boundaries. So, for Arizona, the basinNum column is missing and instead there are two columns: surBasinNum - The alpha-numeric identifier of the surface water WMA the record belongs to. grdBasinNum - The alpha-numeric identifier of the groundwater WMA the record belongs to. XXStatePOD.shp: A shapefile which identifies the location of the Points of Diversion for the state's water rights. It should be noted that not all water right records in XXFullHarmonizedRights.csv have coordinates, and therefore may be missing from this file. XXStatePOU.shp: A shapefile which contains the area(s) in which each water right is claimed to be used. Currently, only Idaho and Washington provided valid data to include within this file. XXGroundwaterHarmonizedRights.csv: A file which contains only harmonized groundwater rights collected from each state. See XXFullHarmonizedRights.csv for more details on how the data is formatted. XXSurfaceWaterHarmonizedRights.csv: A file which contains only harmonized surface water rights collected from each state. See XXFullHarmonizedRights.csv for more details on how the data is formatted. Additionally, one file, stateWMALabels.csv, is not stored within a sub-directory. While we have referred to the spatial boundaries that each state uses to manage its water resources as WMAs, this term is not shared across all states. This file lists the proper name for each boundary set, by state. For those whom may be interested in exploring our code more in depth, we are also making available an internal data file for convenience. The file is in .RData format and contains everything described above as well as some minor additional objects used within the code calculating the cumulative curves. For completeness, here is a detailed description of the various objects which can be found within the .RData file: states: A character vector containing the state names for those states in which data was collected for. More importantly, the index of the state name is also the index in which that state's data can be found in the various following list objects. For example, if California is the third index in this object, the data for California will also be in the third index for each accompanying list. rightsByState_ground: A list of data frames with the cleaned ground water rights collected from each state. This object holds the the data that is exported to created the xxGroundwaterHarmonizedRights.csv files. rightsByState_surface: A list of data frames with the cleaned surface water rights collected from each state. This object holds the the data that is exported to created the xxSurfaceWaterHarmonizedRights.csv files. fullRightsRecs: A list of the combined groundwater and surface water records for each state. This object holds the the data that is exported to created the xxFullHarmonizedRights.csv files. projProj: The spatial projection used for map creation in the beginning of the project. Specifically, the World Geodetic System (WGS84) as a coordinate reference system (CRS) string in PROJ.4 format. wmaStateLabel: The name and/or abbreviation for what each state legally calls their WMAs. h2oUseByState: A list of spatial polygon data frames which contain the area(s) in which each water right is claimed to be used. It should be noted that not all water right records have a listed area(s) of use in this object. Currently, only Idaho and Washington provided valid data to be included in this object. h2oDivByState: A list of spatial points data frames which identifies the location of the Point of Diversion for the state's water rights. It should be noted that not all water right records have a listed Point of Diversion in this object. spatialWMAByState: A list of spatial polygon data frames which contain the spatial WMA boundaries for each state. The only data contained within the table are identifiers for each polygon. It is worth reiterating that Arizona is the only state in which the surface and groundwater WMA boundaries are not the same. wmaIDByState: A list which contains the unique ID values of the WMAs for each state. plottingDim: A character vector used to inform mapping functions for internal map making. Each state is classified as either "tall" or "wide", to maximize space on a typical 8x11 page. The code related to the creation of this dataset can be viewed within HarDWR GitHub Repository/dataHarmonization.

Economics↗
NASA NTRS1972

The fluid mechanics of thrombus formation

Experimental data are presented for the growth of thrombi (blood clots) in a stagnation point flow of fresh blood. Thrombus shape, size and structure are shown to depend on local flow conditions. The evolution of a thrombus is described in terms of a physical model that includes platelet diffusion, a platelet aggregation mechanism, and diffusion and convection of the chemical species responsible for aggregation. Diffusion-controlled and convection-controlled regimes are defined by flow parameters and thrombus location, and the characteristic growth pattern in each regime is explained. Quantitative comparisons with an approximate theoretical model are presented, and a more general model is formulated.

Source record↗
NASA NTRS2000

BOREAS RSS-18 Level-1B AVIRIS Imagery: At-Sensor Radiance in BIL Format

These data were collected and processed by the BOREAS RSS-18 team at NASA JPL. Data were acquired for BOREAS with NASA's AVIRIS. This optical sensor measures images that consist of spectra from 400 to 2500 nm at 10-nm sampling. These spectra are acquired as images with 20-meter spatial resolution, 11-km swath width and up to 800-km length. The measurements are spectrally, radiometrically, and geometrically calibrated. Spatially, the data are focused on the BOREAS NSA and SSA near Thompson, Manitoba, and Candle Lake, Saskatchewan, Canada, respectively. AVIRIS data were collected in 1994 during the Thaw campaign at the NSA and SSA, at the SSA in IFC-1, and at the NSA and SSA in both IFC-2 and IFC-3. In 1996, AVIRIS was deployed in the winter and summer campaigns in the SSA only. The data files are available on a CD-ROM (see document number 20010000884) or from the Oak Ridge National Laboratory (ORNL) Distributed Active Archive Center (DAAC).

Newcomer, Jeffrey A.↗
DOE OSTI2023

HarDWR - Raw Water Rights Records

For a detailed description of the database of which this record is only one part, please see the HarDWR meta-record. In order to hold a water right in the western United States, an entity, (e.g., an individual, corporation, municipality, sovereign government, or non-profit) must register a physical document with the state's water regulatory agency. State water agencies each maintain their own database containing all registered water right documents within the state, along with relevant metadata such as the point of diversion and place of use of the water. All western U.S. states have digitized their individual water rights databases, along with the geospatial data describing the spatial units where water rights are managed. Each state maintains and provides their own water rights data in accordance with individual state regulations and standards. We collected water rights databases from 11 western United States states either by downloading them from publicly accessible web portals, or by contacting state water management representatives; detailed descriptions of where and when the data was collected is provided in the README.txt, as well as Lisk et al.(in review). This collection of data are those raw water rights. Each state formats their data differently, meaning that file types, field availability, and names vary from state to state. Note, the data provided here reflects the state of the water rights databases at the time we collected the data; updates have likely occurred in many states. Some pieces of information are common among all states. These are: priority date, volume or flow of water allowed by the right, stated water use of the right, and some means of identifying the geography and source of the water pertaining to the right - typically the coordinates of the Point of Diversion (PoD) of a waterbody or well. Arizona regulates water in a different way than the other 10 states. Outside of some relatively small critical agricultural areas called Active Management Areas (AMAs), Arizona does not maintain any water rights. However, the state does require registration of surface and groundwater pumping devices, which includes disclosing the mechanical specifics of the devices. We used these records as a proxy for water rights. Each state, and their respective water right authorities, have made their water right records available for non-commercial reference uses. In addition, the states make no guarantees as to the completeness, accuracy, or timeliness of their respective databases, let alone the modifications which we, the authors of this paper, have made to the collected records. None of the states should be held liable for using this data outside of its intended use. In addition, the following states have requested specifically worded disclaimers to be included with their data. Colorado: "The data made available here has been modified for use from its original source, which is the State of Colorado. THE STATE OF COLORADO MAKES NO REPRESENTATIONS OR WARRANTY AS TO THE COMPLETENESS, ACCURACY, TIMELINESS, OR CONTENT OF ANY DATA MADE AVAILABLE THROUGH THIS SITE. THE STATE OF COLORADO EXPRESSLY DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS OR IMPLIED, INCLUDING ANY IMPLIED WARRANTIES OF MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. The data is subject to change as modifications and updates are complete. It is understood that the information contained in the Web feed is being used at one's own risk." Montana: "The Montana State Library provides this product/service for informational purposes only. The Library did not produce it for, nor is it suitable for legal, engineering, or surveying purposes. Consumers of this information should review or consult the primary data and information sources to ascertain the viability of the information for their purposes. The Library provides these data in good faith but does not represent or warrant its accuracy, adequacy, or completeness. In no event shall the Library be liable for any incorrect results or analysis; any direct, indirect, special, or consequential damages to any party; or any lost profits arising out of or in connection with the use or the inability to use the data or the services provided. The Library makes these data and services available as a convenience to the public, and for no other purpose. The Library reserves the right to change or revise published data and/or services at any time." Oregon: "This product is for informational purposes and may not have been prepared for, or be suitable for legal, engineering, or surveying purposes. Users of this information should review or consult the primary data and information sources to ascertain the usability of the information." The available data is provided as a series of compressed files, which each containing the full data collected from each state. Some of the files have been renamed, to more easily know which state the data belongs to. The file renaming was also required as some files from different states had the same name. In other cases, the data for a state has been placed in a folder indicating which state it belongs to - as the state organized its data by selected subregions. Below is a brief description of the format of the collected data from each state. ArizonaRights_StatementOfClaimants: A folder containing a database of interconnected CSV files. The soc_erd.pdf file contains a visual flowchart of how the various files are connected, beginning with SOC_MAIN.csv in the center of the page. ArizonaRights_SurfaceWaterRightsData: A folder containing a database of a single Shapefile and 10 associated CSVs. SurfaceWater.pdf contains a visual flowchart of how the various files are connected, beginning with ADWR_SW_APPL_REGRY.csv. ArizonaRights_Well55Registry: A folder containing a database of a single Shapefile and 59 associated CSVs. Wells55.pdf contains a visual flowchart of how the various files are connected, beginning with WellRegistry.shp. CaliforniaRights_eWRIMS_directDatabase: A folder containing a collection of four "series" Microsoft Excel files, as either XLS or XLSX. The four "series": byCounty, byEntity (what type of legal entity holds the right), byUse (stated water use), and byWatershed, are various methods by which the California water rights are organized within the state's database. However, it was observed that by only collecting a single series, not all water rights were being provided. So, essentially, the majority of records within each "series" are copies of each other, with each "series" containing some unique records. ColoradoRights_NetAmounts: A folder containing 78 CSV files, with one file per Colorado Water District. IdahoRights_PointOfDiversion: A Shapefile containing the Points of Diversion for the entire state of Idaho. IdahoRights_PlaceOfUse: A Shapefile containing the Place of Use polygons for the entire state of Idaho. MontanaRights_WaterRights: A Geodatabase file containing the Points of Diversion and Places of Use for the entire state of Montana. The name of the Points of Diversion Feature Layer within the Geodatabase is "WRDIV", and the name of the Places of Use Feature Layer is "WRPOU". NevadaRights_POD_Sites: A Shapefile containing the Points of Diversion for the entire state of Nevada. NewMexicoRights_Points_of_Diversion: A Shapefile containing the Points of Diversion for the entire state of New Mexico. OregonRights_state_shp: A folder containing 36 Shapefiles and are split between "pod" (Point of Diversion) and "pou" (Place of Use) for each water management basin within Oregon. In other words, each basin has one "pod" file and one "pou" file. The "pod" files are point shapes, and the "pou" files are polygons. UtahRights_Points_of_Diversion: A Shapefile containing the Points of Diversion for the entire state of Utah. WashingtonRights_WaterDiversions_ECY_NHD: A Geodatabase file containing both the Points of Diversion for the entire state of Washington. The name of the Feature Layer within the Geodatabase is "WaterDiversions_ECY_NHD". WyomingRights: A folder containing four subdirectories, one for each Wyoming Water Division. Each Division directory includes a varying number of subdirectories for each Wyoming Water District. Each District folder contains two copies of the Point of Diversion records for that area, with one copying being in CSV and one copy in Microsoft Excel XLS format.

Lisk, Matthew↗
NASA NTRS1988

Programmable Data Formatter

System adapts data in diverse formats for transmission. Microprocessor controls complex activities of data formatting, error checking, and transmission-priority arbitration. System developed for formatting signals from and to telemetry units on spacecraft for widespread telecommunication network.

Martin, Robert E.↗
NASA NTRS1988

File-Format Program For Transferable Output ASCII Data

TOAD utilities machine-independent and require minimal central memory. Transferable Output ASCII Data (TOAD) file-format computer program facilitates transfer of data files from one computer installation to another. TOAD files preferred type and record length, easy to edit, read, and write on magnetic tape or transfer across communications networks. Applications programs write TOAD files directly and conform to all ANSI FORTRAN 77 standards.

Bingle, Bradford↗
NASA NTRS2006

Inter-calibrating Brightness Temperatures of a Constellation of Radiometers

In 2010, the National Aeronautics and Space Agency (NASA) of the U.S.A. and the Japanese Exploration Agency (JAXA) of Japan in cooperation with other U.S. and international partners will launch the Global Precipitation Measurement (GPM) mission. The mission center-piece is a core U.S. provided satellite holding a scanning microwave imager provided by the U.S. A. and a dual-frequency precipitation radar provided by Japan. The core satellite is in a 65 deg inclination (the current Tropical Rainfall Measuring Mission -TRMM is in a 35 deg inclination). Joining the core satellite are a constellation of approximately 8 satellites containing scanning radiometers. The purpose of the constellation is to increase the global sampling capability of the mission. One constellation satellite will be provided by NASA. Some of the constellation radiometers will provided by other U.S. agencies with existing (e.g. SSMI/S) or planned (NPOESS) radiometer assets. International groups have also expressed interest in contributing to the GPM mission including providing radiometer data for the GPM constellation. The use of a heterogeneous group of scanning radiometers each with its own unique purpose, characteristics and calibration offers a significant challenge for combining brightness temperatures or rain retrievals to create meaningful combined global radiometer products. However, the availability of active dual precipitation radar on the GPM core in combination with a well-calibrated radiometer on the same platform offers the possibility of inter-calibrating the constellation radiometers using the core satellite as a calibrator. This paper describes a joint NASA/GSFC and Colorado State University prototype effort at inter-calibrating existing radiometers using such a core calibrator approach. In the prototype, existing radiometers (i.e. SSM/I and AMSR-E) are intercalibrated, as required, using the TRMM Precipitation Radar (PR) and TRMM Microwave Imager (TMI) as the calibration core. This prototype includes four basic steps: provide the data in a normalized-common logical radiometer format, convert all data to a granule that stops and starts at the southernmost latitude rather than at the equator, perform basic processing and finally inter-calibrate the radiometers with the TRMM core as appropriate. Basic processing includes quality control, cross-track bias correction as well as the core action of converting T(sub a), to T(sub b). The purpose of the inter-calibration step is to provide consistent brightness temperatures for the various SSMA sensors, TMI and AMSR-E. The C appended to the level of the product indicates common brightness temperature (T(sub c)). The paper provides a summary of each of these actions. The details of the processing as well as data format and tools can be obtained by connecting to the website at mrain.atmos.colostate.eduLEVEL1C .

Stocker, Erich Franz↗
DOE OSTI2023

HarDWR - Harmonized Water Rights Records

For a detailed description of the database of which this record is only one part, please see the HarDWR meta-record. Here we present a new dataset of western U.S. water rights records. This dataset provides consistent unique identifiers for each spatial unit of water management across the domain, unique identifiers for each water right record, and a consistent categorization scheme that puts each water right record into one of 7 broad use categories. These data were instrumental in conducting a study of the multi-sector dynamics of intersectoral water allocation changes through water markets (Grogan et al., in review). Specifically, the data were formatted for use as input to a process-based hydrologic model, WBM, with a water rights module (Grogan et al., in review). While this specific study motivated the development of the database presented here, U.S. west water management is a rich area of study (e.g., Anderson and Woosly, 2005; Tidwell, 2014; Null and Prudencio, 2016; Carney et al, 2021) so releasing this database publicly with documentation and usage notes will enable other researchers to do further work on water management in the U.S. west. The raw downloaded data for each state is described in Lisk et al. (in review), as well as here. The dataset is a collection of objects stored within an RDate file, stateWaterRightsHarmonized.RData. Each object either describes how one should interact with the other objects, or contains the harmonized water rights data. Here is each object described in detail: states: A character vector containing the state names for those states in which data was collected for. More importantly, the index of the state name is also the index in which that state's data can be found in the various following list objects. For example, if California is the third index in this object, the data for California will also be in the third index for each accompanying list. rightsByState_ground: A list of data frames with the cleaned ground water rights collected from each state. The column headers for each data frame are: waterRightID - The unique identifying ID of the water right, the same identifier as its state uses. basinNum - The alpha-numeric identifier of the WMA the record belongs to. priorityDate - The priority date associated with the right. allocatedFlow - The maximum flow of the allocation in cubic feet per second (ft3s-1). source - Whether the right is for surface water or groundwater. origWaterUse - The original stated water use from the state. waterUse - The water use category under the unified use categories established here. rightsByState_surface: A list of the cleaned surface water rights collected from each state. See rightsByState_ground for more details on how the data is formatted. fullRightsRecs: A list of the combined groundwater and surface water records for each state. Essentially, these lists are the merging of rightsByState_ground and rightsByState_surface by state. See rightsByState_ground for more details on how the data is formatted. projProj: The spatial projection used for map creation in the beginning of the project. Specifically, the World Geodetic System (WGS84) as a coordinate reference system (CRS) string in PROJ.4 format. wmaStateLabel: The name and/or abbreviation for what each state legally calls their WMAs. h2oUseByState: A list of spatial polygon data frames which contain the area(s) in which each water right is claimed to be used. It should be noted that not all water right records have a listed area(s) of use in this object. Currently, only Idaho and Washington provided valid data to be included in this object. h2oDivByState: A list of spatial points data frames which identifies the location of the Point of Diversion for the state's water rights. It should be noted that not all water right records have a listed Point of Diversion in this object. spatialWMAByState: A list of spatial polygon data frames which contain the spatial WMA boundaries for each state. The only data contained within the table are identifiers for each polygon. It is worth reiterating that Arizona is the only state in which the surface and groundwater WMA boundaries are not the same. wmaIDByState: A list which contains the unique ID values of the WMAs for each state. plottingDim: A character vector used to inform mapping functions for internal map making. Each state is classified as either "tall" or "wide", to maximize space on a typical 8x11 page.

Economics↗
DOE OSTI2020

Striped Data Analysis Framework

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Gutsche, Oliver↗
DOE OSTI2018

Striped Data Server for Scalable Parallel Data Analysis

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. As a result, we have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2024

HarDWR - Harmonized Water Rights Records

For a detailed description of the database of which this record is only one part, please see the HarDWR meta-record. Here we present a new dataset of western U.S. water rights records. This dataset provides consistent unique identifiers for each spatial unit of water management across the domain, unique identifiers for each water right record, and a consistent categorization scheme that puts each water right record into one of 7 broad use categories. These data were instrumental in conducting a study of the multi-sector dynamics of intersectoral water allocation changes through water markets (Grogan et al., in review). Specifically, the data were formatted for use as input to a process-based hydrologic model, WBM, with a water rights module (Grogan et al., in review). While this specific study motivated the development of the database presented here, U.S. west water management is a rich area of study (e.g., Anderson and Woosly, 2005; Tidwell, 2014; Null and Prudencio, 2016; Carney et al, 2021) so releasing this database publicly with documentation and usage notes will enable other researchers to do further work on water management in the U.S. west. The raw downloaded data for each state is described in Lisk et al. (in review), as well as here. The dataset is a series of various files organized by state sub-directories. The first two characters of each file name is the abbreviation for the state the in which the file contains data for. After the abbreviation is the text which describes the contents of the file. Here is each file type described in detail: XXFullHarmonizedRights.csv: A file of the combined groundwater and surface water records for each state. Essentially, this file is the merging of XXGroundwaterHarmonizedRights.csv and XXSurfaceWaterHarmonizedRights.csv by state. The column headers for each of this type of file are: state - The name of the state the data comes from. FIPS - The two digit numeric state ID code. waterRightID - The unique identifying ID of the water right, the same identifier as its state uses. priorityDate - The priority date associated with the right. origWaterUse - The original stated water use from the state. waterUse - The water use category under the unified use categories established here. source - Whether the right is for surface water or groundwater. basinNum - The alpha-numeric identifier of the WMA the record belongs to. CFS - The maximum flow of the allocation in cubic feet per second (ft3s-1). Arizona is unique among the states, as its surface and groundwater resources are managed with two different sets of boundaries. So, for Arizona, the basinNum column is missing and instead there are two columns: surBasinNum - The alpha-numeric identifier of the surface water WMA the record belongs to. grdBasinNum - The alpha-numeric identifier of the groundwater WMA the record belongs to. XXStatePOD.shp: A shapefile which identifies the location of the Points of Diversion for the state's water rights. It should be noted that not all water right records in XXFullHarmonizedRights.csv have coordinates, and therefore may be missing from this file. XXStatePOU.shp: A shapefile which contains the area(s) in which each water right is claimed to be used. Currently, only Idaho and Washington provided valid data to included within this file. XXGroundwaterHarmonizedRights.csv: A file which contains only harmonized groundwater rights collected from each state. See XXFullHarmonizedRights.csv for more details on how the data is formatted. XXSurfaceWaterHarmonizedRights.csv: A file which contains only harmonized surface water rights collected from each state. See XXFullHarmonizedRights.csv for more details on how the data is formatted. Additionally, one file, stateWMALabels.csv, is not stored within a sub-directory. While we have referred to the spatial boundaries that each state uses to manage its water resources as WMAs, this term is not shared across all states. This file lists the proper name for each boundary set, by state.

Economics↗