Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Open data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,045 records · Page 58
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP

With ever-increasing volumes of scientific data produced by high-performance computing applications, significantly reducing data size is critical because of limited capacity of storage space and potential bottlenecks on I/O or networks in writing/reading or transferring data. SZ and ZFP are two leading BSD licensed open source C/C++ libraries for compressed floating-point arrays that support high throughput read and write random access. However, their performance is not consistent across different data sets and across different fields of some data sets, which raises the need for an automatic online (during compression) selection between SZ and ZFP, with minimal overhead. In this paper, the automatic selection optimizes the rate-distortion, an important statistical quality metric based on the signal-to-noise ratio. To optimize for rate-distortion, we investigate the principles of SZ and ZFP. We then propose an efficient online, low-overhead selection algorithm that predicts the compression quality accurately for two compressors in early processing stages and selects the best-fit compressor for each data field. Here, we implement the selection algorithm into an open-source library, and we evaluate the effectiveness of our proposed solution against plain SZ and ZFP in a parallel environment with 1,024 cores. Evaluation results on three data sets representing about 100 fields show that our selection algorithm improves the compression ratio up to 70 percent with the same level of data distortion because of very accurate selection (around 99 percent) of the bestfit compressor, with little overhead (less than 7 percent in the experiments).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Cosmology intertwined: A review of the particle physics, astrophysics, and cosmology associated with the cosmological tensions and anomalies

The standard Λ Cold Dark Matter (ΛCDM) cosmological model provides a good description of a wide range of astrophysical and cosmological data. However, there are a few big open questions that make the standard model look like an approximation to a more realistic scenario yet to be found. In this paper, we list a few important goals that need to be addressed in the next decade, taking into account the current discordances between the different cosmological probes, such as the disagreement in the value of the Hubble constant H 0 , the σ 8 – S 8 tension, and other less statistically significant anomalies. While these discordances can still be in part the result of systematic errors, their persistence after several years of accurate analysis strongly hints at cracks in the standard cosmological scenario and the necessity for new physics or generalisations beyond the standard model. In this paper, we focus on the 5.0 σ tension between the Planck CMB estimate of the Hubble constant H 0 and the SH0ES collaboration measurements. After showing the H 0 evaluations made from different teams using different methods and geometric calibrations, we list a few interesting new physics models that could alleviate this tension and discuss how the next decade's experiments will be crucial. Moreover, we focus on the tension of the Planck CMB data with weak lensing measurements and redshift surveys, about the value of the matter energy density Ω m , and the amplitude or rate of the growth of structure ( σ 8 , f σ 8 ). We list a few interesting models proposed for alleviating this tension, and we discuss the importance of trying to fit a full array of data with a single model and not just one parameter at a time. Additionally, we present a wide range of other less discussed anomalies at a statistical significance level lower than the H 0 – S 8 tensions which may also constitute hints towards new physics, and we discuss possible generic theoretical approaches that can collectively explain the non-standard nature of these signals. Finally, we give an overview of upgraded experiments and next-generation space missions and facilities on Earth that will be of crucial importance to address all these open questions.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2019

wavepy - Python package for x-ray grating interferometry with applications in imaging and wavefront characterization

X-ray grating interferometry has been employed for a wide range of imaging applications by using the Talbot self-images of a grating. At synchrotron facilities, it has also been exploited to characterize the coherence and wavefront of the x-ray beam. The method is relatively straightforward, easy to implement, and provides accurate quantitative data. However, to exploit its full potential, it requires highly specialized, robust data analysis. It this work, we have set the framework for the development and dissemination of a robust open-source code that is tailored to the data analysis for imaging and beam characterization using grating interferometry.

Grizolli, Walan↗
DOE OSTI2026

GenomeDepot: data management system for microbial comparative genomics

Summary GenomeDepot is an open-source web-based platform for annotation, management, and comparative analysis of microbial genomic sequences and associated data including ortholog families, protein domains, operons, regulatory interactions, strain taxonomy, and sample metadata. GenomeDepot supports rapid creation of websites for user-defined genome collections that include bioinformatic tools for interactive genome browsing, Basic Local Alignment Search Tool (BLAST) search, annotation search, comparative genomic neighborhood visualization, and sequence download. Gene function annotations are generated by a customizable annotation pipeline. The pipeline runs annotation tools in Conda environments and can be easily extended with additional user-specified tools. Availability and implementation GenomeDepot is open source and distributed under the GNU General Public License via GitHub (https://github.com/aekazakov/genome-depot). GenomeDepot is implemented in Python and was tested in Ubuntu Linux. Full installation instructions and documentation are available at https://aekazakov.github.io/genome-depot/. GenomeDepot demo server is freely accessible at https://iseq.lbl.gov/demogd/.

Kazakov, Alexey [Lawrence Berkeley National Labora↗
DOE OSTI2022

Learning Latent Interactions for Event Identification via Graph Neural Networks and PMU Data

Phasor measurement units (PMUs) are being widely installed on power systems, providing a unique opportunity to enhance wide-area situational awareness. One essential application is the use of PMU data for real-time event identification. However, how to take full advantage of all PMU data in event identification is still an open problem. Thus, we propose a novel method that performs event identification by mining interaction graphs among different PMUs. The proposed interaction graph inference method follows an entirely data-driven manner without knowing the physical topology. Moreover, unlike previous works that treat interactive learning and event identification as two different stages, our method learns interactions jointly with the identification task, thereby improving the accuracy of graph learning and ensuring seamless integration between the two stages. Moreover, to capture multi-scale event patterns, a dilated inception-based method is investigated to perform feature extraction of PMU data. To test the proposed data-driven approach, a large real-world dataset from tens of PMU sources and the corresponding event logs have been utilized in this work. We report numerical results validate that our method has higher classification accuracy compared to previous methods.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2015

2015 ESGF Progress Report

The Earth System Grid Federation (ESGF) is a multi-agency, international collaboration whose purpose is to develop the software infrastructure needed to facilitate and empower the study of climate change on a global scale. ESGF’s architecture employs a system of geographically distributed peer nodes that are independently administered yet united by common federation protocols and application programming interfaces. The cornerstones of its interoperability are the peer-to-peer messaging, which is continuously exchanged among all nodes in the federation; a shared architecture for search and discovery; and a security infrastructure based on industry standards. ESGF integrates popular application engines available from the open-source community with custom components (for data publishing, searching, user interface, security, and messaging) that were developed collaboratively by the team. The full ESGF infrastructure has now been adopted by multiple Earth science projects and allows access to petabytes of geophysical data, including the Coupled Model Intercomparison Project (CMIP)—output used by the Intergovernmental Panel on Climate Change assessment reports. ESGF is a successful example of integration of disparate open-source technologies into a cohesive functional system that serves the needs of the global climate science community.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Using Atmospheric Inverse Modelling of Methane Budgets with Copernicus Land Water and Wetness Data to Detect Land Use-Related Emissions

Climate change mitigation requires countries to report their annual greenhouse gas (GHG) emissions and sinks, including those from land use, land use change, and forestry (LULUCF). In Finland, the LULUCF sector plays a crucial role in achieving net-zero GHG emissions, as the sector is expected to be a net sink. However, accurate estimates of LULUCF-related GHG emissions, such as methane (CH 4 ), remain challenging. We estimated LULUCF-related CH 4 emissions in Finland in 2013–2020 by combining national land cover and remote-sensed surface wetness data with CH 4 emissions estimated by an inversion model. According to our inversion model, most of Finland’s CH 4 emissions were attributed to natural sources such as open pristine peatlands. However, our research indicated that forests with thin tree cover surrounding open peatlands may also be a significant source of CH 4 . Unlike open pristine peatlands and pristine peatlands with thin tree cover, surrounding transient forests are included in the Finnish GHG inventory if they meet the criteria used for forest land. The current Finnish national GHG inventory may therefore underestimate CH 4 emissions from forested organic soils surrounding open peatlands, although more precise methods and data are needed to verify this. Given the potential impact on net GHG emissions, CH 4 emissions from transitional forests on organic soils should be further investigated. Furthermore, the results demonstrate the potential of combining atmospheric inversion modelling of GHGs with diverse data sources and highlight the need for methods to more easily combine atmospheric inversions with national GHG inventories.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Generic and ML Workloads in an HPC Datacenter: Node Energy, Job Failures, and Node-Job Analysis

HPC datacenters offer a backbone to the modern digital society. Increasingly, they run Machine Learning (ML) jobs next to generic, compute-intensive workloads, supporting science, business, and other decision-making processes. However, understanding how ML jobs impact the operation of HPC datacenters, relative to generic jobs, remains desirable but understudied. In this work, we leverage long-term operational data, collected from a national-scale production HPC datacenter, and statistically compare how ML and generic jobs can impact the performance, failures, resource utilization, and energy consumption of HPC datacenters. Our study provides key insights, e.g., ML-related power usage causes GPU nodes to run into temperature limitations, median/mean runtime and failure rates are higher for ML jobs than for generic jobs, both ML and generic jobs exhibit highly variable arrival processes and resource demands, significant amounts of energy are spent on unsuccessfully terminating jobs, and concurrent jobs tend to terminate in the same state. We open-source our cleaned-up data traces on Zenodo (https://doi. org/10.5281/zenodo.13685426), and provide our analysis toolkit as software hosted on GitHub (https://github.com/atlarge-research/2024-icpads-hpc-workload-characterization). This study offers multiple benefits for data center administrators, who can improve operational efficiency, and for researchers, who can further improve system designs, scheduling techniques, etc.

crossanalysis↗
DOE OSTI2026

Location Identifiers, Metadata, and Map for Field Measurements at the East-Taylor Watershed Community Observatory, Colorado, USA (Version 3.3)

This dataset contains identifiers, metadata, and a map of the locations where field measurements have been conducted at the East-Taylor Watershed Community Observatory located in the Upper Colorado River Basin, United States. This is version 3.3 of the dataset and replaces the prior version 3.2 (see below for details on changes between the versions). Dataset description: The East River-Taylor Watershed is the primary field site of the Watershed Function Scientific Focus Area (WFSFA) and the Rocky Mountain Biological Laboratory. Researchers from several institutions generate highly diverse hydrological, biogeochemical, climate, vegetation, geological, remote sensing, and model data at the East-Taylor Watershed in collaboration with the WFSFA. Thus, the purpose of this dataset is to maintain an inventory of the field locations and instrumentation to provide information on the field activities in the East-Taylor Watershed and coordinate data collected across different locations, researchers, and institutions. The dataset contains (1) a README file with information on the various files, (2) three csv files describing the metadata collected for each surface point location, plot and region registered with the WFSFA, (3) csv files with metadata and contact information for each surface point location registered with the WFSFA, (4) a csv file with with metadata and contact information for plots, (5) a csv file with metadata for geographic regions and sub-regions within the watershed, (6) a compiled xlsx file with all the data and metadata which can be opened in Microsoft Excel, (7) a kml map of the locations plotted in the watershed which can be opened in Google Earth, (8) a jpg image of the kml map which can be viewed in any photo viewer, and (9) a zipped file with the registration templates used by the SFA team to collect location metadata. The zipped template file contains two csv files with the blank templates (point and plot), two csv files with instructions for filling out the location templates, and one compiled xlsx file with the instructions and blank templates together. Additionally, the templates in the xlsx include drop down validation for any controlled metadata fields. Persistent location identifiers (Location_ID) are determined by the WFSFA data management team and are used to track data and samples across locations. Dataset uses: This location metadata is used to update the Watershed SFA’s publicly accessible Field Information Portal (an interactive field sampling metadata exploration tool; https://wfsfa-data.lbl.gov/watershed/), the kml map file included in this dataset, and other data management tools internal to the Watershed SFA team. Version Information: The latest version of this dataset publication is version 3.3. This version contains 167 new point locations, 1 new plot, and 2 new geographic regions. Overall, there are a total of 1439 point locations, 75 plots, and 54 geographic regions. Additionally, the kml map of locations and image now includes two boundaries (Upper Ohio Creek (UO) and Carbon Creek (CA)) outside of the East River watershed (USGS HUC-10) and accompanying stream network that represents areas of focus. Refer to methods for further details on the version history. This dataset will be updated on a periodic basis with new measurement location information. Researchers interested in having their East-Taylor Watershed measurement locations added to this list should reach out to the WFSFA data management team at wfsfa-data@googlegroups.com. Acknowledgments: Please cite this dataset if using any of the location metadata in other publications or derived products. If using the location metadata for the 2018 NEON hyperspectral campaign, additionally cite Chadwick et al. (2020). doi:10.15485/1618130. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231. Part of this work was performed at SLAC Accelerator Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-76SF00515.

2018 NEON and 2025 CHESS Campaigns↗
DOE OSTI2017

Framework for Analysis and Visualization of Simulation Data (VizAly)

As supercomputing resources increase, cosmological scientists are able to run more detailed and larger simulations generating massive amounts of data. Analyzing these simulations with an available open-source toolkit is important for collaborative Department of Energy scientific discovery across labs, universities, and other partners. Developed software as a part of this collection include: comparing data with other existing simulations, verifying and validating results with observation databases, new halo finder algorithms, and using analytical tools to get insights into the physics of the cosmological universe. The goal of this software project is to provide a set of open-source libraries, tools, and packages for large-scale cosmology that allows scientists to visualize, analyze, and compare large-scale simulation and observational data sets. Developed software will provide a variety of methods for processing, visualization, and analysis of astronomical observation and cosmological simulation data. These tools are intended for deployment on multiple scientific computing platforms, including but not limited to personal computers, cloud computing, experimental sites (telescopes) and high performance supercomputers.

Wendelberger, Joanne↗
DOE OSTI2022

The Open Cluster Chemical Abundances and Mapping Survey. VI. Galactic Chemical Gradient Analysis from APOGEE DR17

The goal of the Open Cluster Chemical Abundances and Mapping (OCCAM) survey is to constrain key Galactic dynamic and chemical evolution parameters by the construction and analysis of a large, comprehensive, uniform data set of infrared spectra for stars in hundreds of open clusters. This sixth contribution from the OCCAM survey presents analysis of SDSS/APOGEE Data Release 17 (DR17) results for a sample of stars in 150 open clusters, 94 of which we designate to be "high-quality" based on the appearance of their color–magnitude diagram. We find the APOGEE DR17-derived [Fe/H] values to be in good agreement with those from previous high-resolution spectroscopic open cluster abundance studies. Using a subset of the high-quality sample, the Galactic abundance gradients were measured for 16 chemical elements, including [Fe/H], for both Galactocentric radius ($R$ GC ) and guiding center radius ($R$ guide ). We find an overall Galactic [Fe/H] versus RGC gradient of -0.073 ± 0.002 dex kpc -1 over the range of 6 > $R$ GC < 11.5 kpc, and a similar gradient is found for [Fe/H] versus $R$ guide . Significant Galactic abundance gradients are also noted for O, Mg, S, Ca, Mn, Na, Al, K, and Ce. Our large sample additionally allows us to explore the evolution of the gradients in four age bins for the remaining 15 elements.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2016

Big, Deep, and Smart Data in Scanning Probe Microscopy

Scanning probe microscopy techniques open the door to nanoscience and nanotechnology by enabling imaging and manipulation of structure and functionality of matter on nanometer and atomic scales. We analyze the discovery process by SPM in terms of information flow from tip-surface junction to the knowledge adoption by scientific community. Furthermore, we discuss the challenges and opportunities offered by merging of SPM and advanced data mining, visual analytics, and knowledge discovery technologies.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2017

Trinity Phase 2 Open Science: CTH

CTH is an Eulerian hydrocode developed by Sandia National Laboratories (SNL) to solve a wide range of shock wave propagation and material deformation problems. Adaptive mesh refinement is also used to improve efficiency for problems with a wide range of spatial scales. The code has a history of running on a variety of computing platforms ranging from desktops to massively parallel distributed-data systems. For the Trinity Phase 2 Open Science campaign, CTH was used to study mesoscale simulations of the hypervelocity penetration of granular SiC powders. The simulations were compared to experimental data. A scaling study of CTH up to 8192 KNL nodes was also performed, and several improvements were made to the code to improve the scalability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Developing Flexible, Networked Lighting Control Systems That Reliably Save Energy in California Buildings

An important strategy to meet California's ambitious energy efficiency goals is to use innovative wireless communications, embedded sensors, data analytics and controls to significantly reduce lighting energy use in commercial buildings. This project developed a suite of networked lighting solutions to further this goal. The technologies include a platform for low-cost sensing, distributed intelligence and communications, the “PermaMote,” which is a self-powered sensor and controller for lighting applications. The project team also developed a task ambient daylighting system that integrates sensors with data-driven daylighting control using an open communication interface, called the “Readings-At-Desk” (RAD) system. To address the problem of building occupants being confused about how to operate traditional lighting control systems, the research team created content that could be the basis for a user interface standard for lighting controls. Finally, to address the difficulty of ensuring that advanced lighting control systems actually deliver their promised energy savings, the project team developed a new method for evaluating and specifying lighting systems’ performance. The research team validated these technologies in the laboratory, showing significant lighting energy savings, up to 73% for the PermaMote sensor system from occupancy control and daylight dimming features, compared to the same light source (LED replacement lamps) operated via simple on/off scheduling. The project team also developed a proposed standard lighting data model and user interface elements, which were contributed to the ANSI Lighting Systems Committee (C137) for standardization. Existing data models are incomplete and inconsistent, whereas the lighting-specific data model developed here is clear and comprehensive, to serve as a starting point for creating common, universally agreed upon semantic definitions of key lighting parameters, to promote interoperability. For the task on verifiable performance of lighting systems, the project team developed a more effective metric for capturing the actual energy impact of a lighting system over time — the energy usage intensity (kWh/ft2/year). Three commercial lighting systems were tested in FLEXLAB® using this new metric, and the tests show a wide range in the accuracy of the self-reported energy-use metric, from 0.5% to 28% error compared to direct measurement of lighting energy using dedicated submeters. Overall, the project team estimates that these advanced technologies can reduce California office lighting energy use by 20% (above and beyond normal advanced lighting controls mandated by Title 24), resulting in about 1,600 GWh/year in savings.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Advances in building data management for building performance standards using the SEED platform

Reducing energy consumption and greenhouse gas emissions in the built environment is a critical step in achieving emission goals to mitigate climate change impacts. Local, federal, and international jurisdictions are deploying several methods to reduce energy and emissions such as voluntary and mandatory benchmarking and building performance standards, requiring building owners to reach energy and emission targets. Jurisdictions leveraging benchmarking and building performance standards require knowledge of the buildings covered; which is a large task due to staffing constraints, limited information on building characteristics and tax parcel data, and the need for advanced data management techniques to align datasets. This paper describes an open-source platform's recent advances to create consistent taxonomies, identify erroneous data, enable auditability, and track building performance. The paper concludes with two use cases on how the platform has been used by jurisdictions.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2016

The Monarch Initiative: an integrative data and analytic platform connecting phenotypes to genotypes across species

The correlation of phenotypic outcomes with genetic variation and environmental factors is a core pursuit in biology and biomedicine. Numerous challenges impede our progress: patient phenotypes may not match known diseases, candidate variants may be in genes that have not been characterized, model organisms may not recapitulate human or veterinary diseases, filling evolutionary gaps is difficult, and many resources must be queried to find potentially significant genotype-phenotype associations. Nonhuman organisms have proven instrumental in revealing biological mechanisms. Advanced informatics tools can identify phenotypically relevant disease models in research and diagnostic contexts. Large-scale integration of model organism and clinical research data can provide a breadth of knowledge not available from individual sources and can provide contextualization of data back to these sources. The Monarch Initiative (monarchinitiative.org) is a collaborative, open science effort that aims to semantically integrate genotype-phenotype data from many species and sources in order to support precision medicine, disease modeling, and mechanistic exploration. Our integrated knowledge graph, analytic tools, and web services enable diverse users to explore relationships between phenotypes and genotypes across species.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Annual Technology Baseline: ATB-calc Open Source Tools [Slides]

The Annual Technology Baseline (ATB) team will introduce the new open-source Python tools for processing and interacting with electricity ATB data and provide helpful user demonstrations, including how to: extract data from the ATB workbook and calculate levelized cost of electricity (LCOE) in Python; programmatically interact with and modify ATB data; and use the tools to modify the tax credit assumptions of the ATB to account for bonus credits from the Inflation Reduction Act of 2022.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗