Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “metadata”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 937 records · Page 52
All sourcesNASA NTRSDOE OSTI
DOE OSTI2014

Daymet: Daily Surface Weather Data on a 1-km Grid for North America, Version 2.

More information: http://daymet.ornl.gov Presenter: Ranjeet Devarakonda Environmental Sciences Division Oak Ridge National Laboratory (ORNL) Daymet: Daily Surface Weather Data and Climatological Summaries provides gridded estimates of daily weather parameters for North America, including daily continuous surfaces of minimum and maximum temperature, precipitation occurrence and amount, humidity, shortwave radiation, snow water equivalent, and day length. The current data product (Version 2) covers the period January 1, 1980 to December 31, 2013 [1]. The prior product (Version 1) only covered from 1980-2008. Data are available on a daily time step at a 1-km x 1-km spatial resolution in Lambert Conformal Conic projection with a spatial extent that covers the conterminous United States, Mexico, and Southern Canada as meteorological station density allows. Daymet data can be downloaded from 1) the ORNL Distributed Active Archive Center (DAAC) search and order tools (http://daac.ornl.gov/cgi-bin/cart/add2cart.pl?add=1219) or directly from the DAAC FTP site (http://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=1219) and 2) the Single Pixel Tool [2] and THREDDS (Thematic Real-time Environmental Data Services) Data Server [3]. The Single Pixel Data Extraction Tool allows users to enter a single geographic point by latitude and longitude in decimal degrees. A routine is executed that translates the (lon, lat) coordinates into projected Daymet (x,y) coordinates. These coordinates are used to access the Daymet database of daily-interpolated surface weather variables. Daily data from the nearest 1 km x 1 km Daymet grid cell are extracted from the database and formatted as a table with one column for each Daymet variable and one row for each day. All daily data for selected years are returned as a single (long) table, formatted for display in the browser window. At the top of this table is a link to the same data in a simple comma-separated text format, suitable for import into a spreadsheet or other data analysis software. The Single Pixel Data Extraction Tool also provides the option to download multiple coordinates programmatically. A multiple extractor script is freely available to download at http://daymet.ornl.gov/files/daymet.zip. The ORNL DAAC s THREDDS data server (TDS) provides customized visualization and access to Daymet time series of North American mosaics. Users can subset and download Daymet data via a variety of community standards, including OPeNDAP, NetCDF Subset service, and Open Geospatial Consortium (OGC) Web Map/Coverage Service. The ORNL DAAC TDS also exposes Daymet metadata through its ncISO service to facilitate harvesting Daymet metadata records into 3rd party catalogs. References: [1] Thornton, P.E., M.M. Thornton, B.W. Mayer, N. Wilhelmi, Y. Wei, R. Devarakonda, and R.B. Cook. 2014. Daymet: Daily Surface Weather Data on a 1-km Grid for North America, Version 2. Data set. Available on-line [http://daac.ornl.gov] from Oak Ridge National Laboratory Distributed Active Archive Center, Oak Ridge, Tennessee, USA. [2] Devarakonda R., et al. 2012. Daymet: Single Pixel Data Extraction Tool. Available on-line [http://daymet.ornl.go/singlepixel.html]. [3] Wei Y., et al. 2014. Daymet: Thematic Real-time Environmental Data Services. Available on-line [http://daymet.ornl.gov/thredds_tiles.html].

Thornton, Peter E↗
DOE OSTI2014

Storing files in a parallel computing system based on user-specified parser function

Techniques are provided for storing files in a parallel computing system based on a user-specified parser function. A plurality of files generated by a distributed application in a parallel computing system are stored by obtaining a parser from the distributed application for processing the plurality of files prior to storage; and storing one or more of the plurality of files in one or more storage nodes of the parallel computing system based on the processing by the parser. The plurality of files comprise one or more of a plurality of complete files and a plurality of sub-files. The parser can optionally store only those files that satisfy one or more semantic requirements of the parser. The parser can also extract metadata from one or more of the files and the extracted metadata can be stored with one or more of the plurality of files and used for searching for files.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Using GTO-Velo to Facilitate Communication and Sharing of Simulation Results in Support of the Geothermal Technologies Office Code Comparison Study

The Geothermal Technologies Office Code Comparison Study (GTO-CCS) aims to support the DOE Geothermal Technologies Office in organizing and executing a model comparison activity. This project is directed at testing, diagnosing differences, and demonstrating modeling capabilities of a worldwide collection of numerical simulators for evaluating geothermal technologies. Teams of researchers are collaborating in this code comparison effort, and it is important to be able to share results in a forum where technical discussions can easily take place without requiring teams to travel to a common location. Pacific Northwest National Laboratory has developed an open-source, flexible framework called Velo that provides a knowledge management infrastructure and tools to support modeling and simulation for a variety of types of projects in a number of scientific domains. GTO-Velo is a customized version of the Velo Framework that is being used as the collaborative tool in support of the GTO-CCS project. Velo is designed around a novel integration of a collaborative Web-based environment and a scalable enterprise Content Management System (CMS). The underlying framework provides a flexible and unstructured data storage system that allows for easy upload of files that can be in any format. Data files are organized in hierarchical folders and each folder and each file has a corresponding wiki page for metadata. The user interacts with Velo through a web browser based wiki technology, providing the benefit of familiarity and ease of use. High-level folders have been defined in GTO-Velo for the benchmark problem descriptions, descriptions of simulator/code capabilities, a project notebook, and folders for participating teams. Each team has a subfolder with write access limited only to the team members, where they can upload their simulation results. The GTO-CCS participants are charged with defining the benchmark problems for the study, and as each GTO-CCS Benchmark problem is defined, the problem creator can provide a description using a template on the metadata page corresponding to the benchmark problem folder. Project documents, references and videos of the weekly online meetings are shared via GTO-Velo. A results comparison tool allows users to plot their uploaded simulation results on the fly, along with those of other teams, to facilitate weekly discussions of the benchmark problem results being generated by the teams. GTO-Velo is an invaluable tool providing the project coordinators and team members with a framework for collaboration among geographically dispersed organizations.

Collaborative Tools↗
DOE OSTI2015

Minimally buffered data transfers between nodes in a data communications network

Methods, apparatus, and products for minimally buffered data transfers between nodes in a data communications network are disclosed that include: receiving, by a messaging module on an origin node, a storage identifier, a origin data type, and a target data type, the storage identifier specifying application storage containing data, the origin data type describing a data subset contained in the origin application storage, the target data type describing an arrangement of the data subset in application storage on a target node; creating, by the messaging module, origin metadata describing the origin data type; selecting, by the messaging module from the origin application storage in dependence upon the origin metadata and the storage identifier, the data subset; and transmitting, by the messaging module to the target node, the selected data subset for storing in the target application storage in dependence upon the target data type without temporarily buffering the data subset.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Sensor-agnostic photogrammetric image registration with applications to population modeling

Photogrammetric registration of airborne and spaceborne imagery is a crucial prerequisite to many data fusion tasks. While embedded sensor models provide a rough geolocation estimate, these metadata may be incomplete or imprecise. Manual solutions are appropriate for small-scale projects, but for rapid streams of cross-modal, multi-sensor, multi-temporal imagery with varying metadata standards, an automated approach is required. We present a high-performance image registration workflow to address this need. This paper outlines the core development concepts and demonstrates its utility with respect to the 2016 data fusion contest imagery. In particular, Iris ultra-HD video is georeferenced to the Earth surface via registration to DEIMOS-2 imagery, which serves as a trusted control source. Geolocation provides opportunity to augment the video with spatial context, stereo-derived disparity, spectral sensitivity, change detection, and numerous ancillary geospatial layers. We conclude by leveraging these derivative data layers towards one such fusion application: population distribution modeling.

White, Devin A↗
DOE OSTI2017

Active non-volatile memory post-processing

A computing node includes an active Non-Volatile Random Access Memory (NVRAM) component which includes memory and a sub-processor component. The memory is to store data chunks received from a processor core, the data chunks comprising metadata indicating a type of post-processing to be performed on data within the data chunks. The sub-processor component is to perform post-processing of said data chunks based on said metadata.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Efficient accesses of data structures using processing near memory

Systems, apparatuses, and methods for implementing efficient queues and other data structures. A queue may be shared among multiple processors and/or threads without using explicit software atomic instructions to coordinate access to the queue. System software may allocate an atomic queue and corresponding queue metadata in system memory and return, to the requesting thread, a handle referencing the queue metadata. Any number of threads may utilize the handle for accessing the atomic queue. The logic for ensuring the atomicity of accesses to the atomic queue may reside in a management unit in the memory controller coupled to the memory where the atomic queue is allocated.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Systems and methods for generating slice files from native CAD geometries

A system and method for improving the production of objects with fully defined CAD models by generating CAD slice files from the native CAD geometries for use by AM machines. An electronic processing element receives the CAD model having a native and/or neutral format with metadata, determines a desired resolution, slices the CAD model to create the slice files having a slice format (e.g., point, edge, surface, volume) and retaining the metadata, and evaluates the slice files to determine whether the resolution has been achieved. If so, the slice files are sent to the AM machine. If not, a new desired resolution is determined and the CAD model is re-sliced, which may include combining the slice files to regenerate the CAD model. A support structure for the object may be analyzed, and if distortion is predicted, a modified CAD model may be created and sliced to create modified slice files.

Vernon, Gregory J.↗
DOE OSTI2020

Apparatus and method for neighborhood-aware virtual to physical address translations

An apparatus and method performs neighborhood-aware virtual to physical address translations. A coalescing opportunity for a first virtual address is determined, based on completing a memory access corresponding to a page walk for a second virtual address. Metadata corresponding to the first virtual address is provided to a page table walk buffer based on the coalescing opportunity and a page walk for the first virtual address is performed based on the metadata corresponding to the first virtual address.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Solar collector installation design system including exceptional condition management and display

Embodiments may include systems and methods to create and edit a representation of a worksite, to create various data objects, to classify such objects as various types of pre-defined “features” with attendant properties and layout constraints. As part of or in addition to classification, an embodiment may include systems and methods to create, associate, and edit intrinsic and extrinsic properties to these objects. A design engine may apply of design rules to the features described above to generate one or more solar collectors installation design alternatives, including generation of on-screen and/or paper representations of the physical layout or arrangement of the one or more design alternatives. In some embodiments, metadata about the design process, including the process of classifying features and providing user input, generating layouts, and then modifying those layouts, may be generated. The metadata may include information about exceptional conditions in the project state information or design. A list of exceptions corresponding to exceptional conditions may be generated, and the designer may interact with these exceptions in a variety of ways, such as by complying with rules to remove an item from the exceptions list or overriding the application of the rules. The exceptions may be non-blocking relative to other user actions.

Wayne, Gary↗
DOE OSTI2021

Grand unified file indexing

Systems and methods are disclosed for a unified file index for a file system. In one example, a Grand Unified File Index (GUFI) includes a tree replicating the directory hierarchy of one or more primary filesystems, and individual metadata stores for each directory. The GUFI tree permits fast traversal, efficient user space access controls, and simple tree directed operations such as renames, moves, or permission changes. In some examples, the individual metadata stores can be implemented as embedded databases on flash storage for speed. In some examples, use of summary tables at the directory or subtree level can eliminate wasteful executions, prune tree traversal, and further improve performance. In various examples, efficient operation can be achieved from laptop to supercomputer scale, across a wide mix of file distributions and filesystems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Caching data from remote memories

An approach is disclosed that caches distant memories within the storage a local node. The approach provides a memory caching infrastructure that supports virtual addressing by utilizing memory in the local node as a cache of distant memories for data granules. The data granules are accessed along with metadata and an ECC associated with the data granule. The metadata is updated to indicate storage of the selected data granule in the cache.

Johns, Charles Ray↗
DOE OSTI2026

Evaluation of Best Practices in Mitigating Startup Costs on Leadership-Class Supercomputers

Supercomputers at Department of Energy (DOE) National Laboratories face a widening range of workloads, from traditional modeling and simulation to Artificial Intelligence model training or complex multi-stage workflows, and beyond. At DOE Leadership Computing Facilities like the Oak Ridge Leadership Computing Facility (OLCF), these workloads demand concurrent access to large portions of the supercomputer’s resources. Launching a job across massive supercomputers is challenging from the start; the file system struggles with a large backlog of metadata requests as tens of thousands of processes read thousands of the same files, and the compute job cannot start until this is completed. There are multiple existing approaches to calm this metadata storm, ranging from vendor-developed tools like sbcast to National Laboratory-developed tools like Spindle and Copper. In this paper, we benchmark and discuss three common approaches to improving compute job launch latencies on Frontier: Slurm’s sbcast tool, Spindle, and Copper. We evaluate these tools by measuring the launch latencies of four workloads: OSU Microbenchmark’s osu_init, Pynamic, Python import mpi4py, and Python import torch. We provide discussion of the results, highlighting data that meet expectations and that do not meet expectations.

Hagerty, Nick [ORNL] (ORCID:0000000330014414)↗
DOE OSTI2021

TriPoll

TriPoll is software to perform surveys of triangles within large graphs containing metadata. In TriPoll, users define their own callback functions that get executed on the metadata of each triangle within a graph as triangles are identified. This functionality makes it possible for users to perform advanced analyses and gather statistics about specific classes of triangles of interest.

Steil, TrevorW↗
DOE OSTI2025

Time-lapse imagery in 2017 and 2018 at the Lower Montane site in the East River Watershed, Colorado

Time-lapse imagery was collected using an automated RGB camera mounted on a pole at the base of the northeast-facing hillslope at the Lower Montane site in the East River Watershed, Colorado. The imagery was intended to support a better understanding of plant dynamics and their controls during the growing season. The dataset includes RGB images archived in four zip files (containing imagery in JPEG format), corresponding to photos taken from the hillslope and the adjacent floodplain during 2017 and 2018. A fifth zip file contains a few AVI movies that compare imagery between the two years. The AVI files can be read with most media players applications. The archive contains a total of five *.zip files and three csv metadata files (flmd.csv, dd.csv, and locations.csv).This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Processed sap flow and fine-root trait data associated with summer drought responses in temperate trees in Lisle, Illinois, USA (2019–2021)

These data support the manuscript “Acquisitive root exploration strategies help maintain higher peak sap flux rates during summer drought, but more root biomass does not”. The dataset includes processed sap flow measurements and fine-root trait data collected between 2019 and 2021 from temperate monodominant tree plots established in the 1920s to 1930s ranging in size from 0.05 to 0.8 ha at The Morton Arboretum in Lisle, IL. Sap flow was measured with ICT sap flow sensors using the heat ratio method. Fine-root traits were measured from soil cores which includes specific root length (SRL), specific root area (SRA), diameter, biomass, and length for diameter classes ≤1 mm and ≤2 mm. The package contains comma separated value (CSV) data files and associated metadata that can be viewed and analyzed using common software such as spreadsheet programs, R, and Python. These data are used to investigate how variation in fine-root traits relate to tree water use and drought response during summer drought linking belowground root traits and aboveground physiological responses.

drought↗
DOE OSTI2021

The future low-temperature geochemical data-scape as envisioned by the U.S. geochemical community

Data sharing benefits the researcher, the scientific community, and the public by allowing the impact of data to be generalized beyond one project and by making science more transparent. However, many scientific communities have not developed protocols or standards for publishing, citing, and versioning datasets. One community that lags in data management is that of low-temperature geochemistry (LTG). This paper resulted from an initiative from 2018 through 2020 to convene LTG and data scientists in the U.S. to strategize future management of LTG data. Through webinars, a workshop, a preprint, a townhall, and a community survey, the group of U.S. scientists discussed the landscape of data management for LTG – the data-scape. Currently this data-scape includes a “street bazaar” of data repositories. This was deemed appropriate in the same way that LTG scientists publish articles in many journals. The variety of data repositories and journals reflect that LTG scientists target many different scientific questions, produce data with extremely different structures and volumes, and utilize copious and complex metadata. Nonetheless, the group agreed that publication of LTG science must be accompanied by sharing of data in publicly accessible repositories, and, for sample-based data, registration of samples with globally unique persistent identifiers. LTG scientists should use certified data repositories that are either highly structured databases designed for specialized types of data, or unstructured generalized data systems. Recognizing the need for tools to enable search and cross-referencing across the proliferating data repositories, the group proposed that the overall data informatics paradigm in LTG should shift from “build data repository, data will come” to “publish data online, cybertools will find”. Funding agencies could also provide portals for LTG scientists to register funded projects and datasets, and forge approaches that cross national boundaries. Finally, the needed transformation of the LTG data culture requires emphasis in student education on science and management of data.

58 GEOSCIENCES↗