Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “scientific information systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Simulation of hydropower at subcontinental to global scales: a state-of-the-art review

Abstract Hydroelectric power is playing a new and often expanded role in the world’s major power grids, offering low carbon generating capacity in industrializing, dam-building economies while providing reserve and flexibility to co-manage fledgling wind and solar resources in high income countries. Driven by river flows, conventional hydropower is exposed to the vagaries of weather and climate, motivating drought and climate change hydropower impact studies at large spatial scales. Here we review methods of climate-driven hydropower simulation at large spatial scales, specifically multi-basin regions to global. We identify four types of approach based on complexity of tools and richness of data applied to the problem. Since the earliest attempts to model climate-driven hydropower at continental scale almost two decades ago, the field has transitioned from one of scientific curiosity to practical application, with studies increasingly motivated by the need to inform power grid expansion planning and operation. As the hydrological and water management models used in large-scale hydropower studies become more sophisticated, new opportunities will emerge to study the impacts of changing hydropower on power system reliability and performance at large power grid scale. To grasp these opportunities, the water resources community must continue to enhance data and models for representing river flows and anthropogenic water use and management at subcontinental to global scales.

13 HYDRO ENERGY↗
DOE OSTI2021

Recent advances in nano-scale spatial resolution x-ray microscopy instrumentation at NSLS-II

X-ray microscopy is an invaluable and powerful characterization tool applied in many scientific fields, such as materials science, biology, environmental science, and energy research. In recent years it has been driven by rapid developments of novel technologies and systems resulting in imaging experiments elucidating structural inhomogeneities and chemical reactions at the nanometer scale. To obtain high spatial resolution comprehensive chemical and structural information, an X-ray microscope must be equipped with adequate capabilities and allow for simultaneous acquisition of multiple datasets. In recent years, a number of X-ray microscopes have been designed, constructed, and commissioned at NSLS-II. Here we provide an overview of the microscopy instrumentation development program at NSLS-II and specifically focus on the multilayer Laue lens-based hard X-ray nanoprobe optimized for ~10 nm spatial resolution imaging, its current status, and future upgrades along with recently constructed Kirkpatrick-Baez based scanning microscope designed for ~100 nm spatial resolution experiments.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Towards provision of regularly updated climate data from the Coupled Model Intercomparison Project

The Coupled Model Intercomparison Project (CMIP) is a flagship of the World Climate Research Programme (WCRP). CMIP has become a recognised ‘brand’ in climate circles evolving over the last thirty years from a targeted research activity by a small number of climate modelling centres intercomparing their Earth System Model (ESM) simulations to a broad international coordinated research effort (Durack et al, 2025). CMIP is organized as a research activity leveraging funded and in-kind contributions from experts within modelling centres and the broader scientific community supported more recently by a fully-funded International Project Office. Within CMIP, Model Intercomparison Projects (MIPs) are community-designed to understand past, present and future climate. CMIP data provides a valuable resource for climate research and is routinely used to assess model representation of climate processes and test scientific hypotheses in the context of model uncertainty and (forced and internal) variability as evident from its prolific use in scientific publications1 . The impact relies on enabling infrastructure (most prominently via the Earth System Grid Federation (ESGF)), which allows sharing of simulation output, provision of the boundary conditions used in each simulation, and definition of the data standards that are essential to facilitating wide use of the data. The impact is supplemented by the wide-ranging scrutiny to which model simulations are subjected. Beyond its use in research, CMIP data is a key resource for communities producing derived climate information from downscaling and impact studies, such as the Coordinated Regional Downscaling Experiment (CORDEX; Gutowski et al., 2016) and the Intersectoral Impacts MIP (ISIMIP; Frieler et al., 2024). Government, academic and commercial entities also increasingly rely on CMIP and its downstream data for climate risk assessments and climate services (for example, Copernicus Climate Change Service and World Bank portal). This means that, although CMIP is a research activity, it increasingly serves a secondary and very relevant role as a provider of climate data – a long-recognised dichotomy (Stevens, 2024). Research and applications have distinct needs, with the former requiring flexibility and generality and the latter consistency. Here we explain how the design of the research activity has been adapted to reduce the burdens imposed by applications and how the research infrastructure might evolve to further enable scientific inquiry. We propose one possible approach to consistently providing model information and projections for applications in the future.

Environmental sciences↗
DOE OSTI2023

ET-AL: Entropy-targeted active learning for bias mitigation in materials data

Growing materials data and data-driven informatics drastically promote the discovery and design of materials. While there are significant advancements in data-driven models, the quality of data resources is less studied despite its huge impact on model performance. In this work, we focus on data bias arising from uneven coverage of materials families in existing knowledge. Observing different diversities among crystal systems in common materials databases, we propose an information entropy-based metric for measuring this bias. To mitigate the bias, we develop an entropy-targeted active learning (ET-AL) framework, which guides the acquisition of new data to improve the diversity of underrepresented crystal systems. We demonstrate the capability of ET-AL for bias mitigation and the resulting improvement in downstream machine learning models. This approach is broadly applicable to data-driven materials discovery, including autonomous data acquisition and dataset trimming to reduce bias, as well as data-driven informatics in other scientific domains.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Physics‐Informed Machine Learning for Inverse Design of Optical Metamaterials

Optical metamaterials manipulate light through various confinement and scattering processes, offering unique advantages like high performance, small form factor and easy integration with semiconductor devices. However, designing metasurfaces with suitable optical responses for complex metamaterial systems remains challenging due to the exponentially growing computation cost and the ill‐posed nature of inverse problems. To expedite the computation for the inverse design of metasurfaces, a physics‐informed deep learning (DL) framework is used. A tandem DL architecture with physics‐based learning is used to select designs that are scientifically consistent, have low error in design prediction, and accurate reconstruction of optical responses. The authors focus on the inverse design of a representative plasmonic device and consider the prediction of design for the optical response of a single wavelength incident or a spectrum of wavelength in the visible light range. The physics‐based constraint is derived from solving the electromagnetic wave equations for a simplified homogenized model. The model converges with an accuracy up to 97% for inverse design prediction with the optical response for the visible light spectrum as input, and up to 96% for optical response of single wavelength of light as input, with optical response reconstruction accuracy of 99%.

Sarkar, Sulagna↗
DOE OSTI2021

Downscaled precipitation and mean air temperature datasets; East-Taylor subbasin; 2008-2019; daily temporal resolution; 400 m spatial resolution

This dataset provides gridded meteorological forcing data (specifically, daily precipitation and daily mean air temperature). The dataset has been generated by downscaling the Parameter-elevation Regressions on Independent Slopes Model (PRISM) dataset from a spatial resolution of 800 m to 400 m. The time period is 2008-2019, and the mapped area is East Taylor subbasin in Upper Colorado. The data are in the form of NetCDF files, arranged by year. Compared to the PRISM dataset, the dates of the downscaled dataset are shifted backwards by one (e.g., downscaled data for May 25 corresponds to PRISM data for May 26). This temporal shifting makes the meteorological forcing correspond more closely to the prescribed date. The NetCDF format is a standard raster format that can be read using any Geographic Information System (GIS) software; plenty of modules exist in popular scripting languages (such as Python, R and Matlab) that can also be used to read NetCDF files.East_Taylor_Tavg_PRISM400 corresponds to mean air temperature.East_Taylor_Precip_PRISM400.zip corresponds to precipitation.The proprietary PRISM data (800 m resolution) were purchased with funding from the Watershed Function Scientific Focus Area supported by U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research under award no. DE-AC02-05CH11231.Dataset update on March 9, 2022:Uploaded new data files that are identical to the earlier files but are now in the NetCDF format, arranged by year. Earlier, the files were in the GeoTiff format, arranged by date.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Optical Test Bench (SYSM 5620 Final Report)

The National Ignition Facility (NIF) is a research facility that focuses on High Energy Density Science and Inertial Confinement Fusion. In the NIF, 192 laser beams are directed into a miniscule Target Capsule to generate an implosion – this is called a NIF Shot. Many NIF Users conduct experiments using different Targets and laser inputs to generate reactions that are of interest to the scientific community. Those reactions are monitored using various Target Diagnostic instruments, including Shot Time Photography – which provides crucial images of the Target as the implosion occurs. A subset of these NIF Users is less interested in the reaction itself and more interested in subjecting materials or devices to the intense environment that these reactions can create. Using the Target as a source of x-rays, neutrons, or blast loading for particular materials has been significant to the shock physics community and to those who work in extreme design spaces. However, the Material and Radiation Effects team – that performs these experiments on behalf of the NIF Users – has not been able to make good use of the existing Shot Time Photography Diagnostics. The STP cameras are focused on the Targets rather than the MRE Customers’ materials, the current system is not oriented to have a good view of the MRE Customers’ materials, and the cameras are limited in their capabilities (black and white, poor resolution, etc.). As MRE experiments increase in quantity and importance to the scientific community, the desire for visual monitoring of the experiments to substantiate models and inform other data is also rising.

42 ENGINEERING↗
DOE OSTI2025

U-Pu and Ba-Cs isotopic measurements on Trinitite by laser ablation sampling on the Neoma MC-ICP-MS

In this study we present the results of combined U-Pu and Ba-Cs isotope measurements obtained by laser ablation (LA) sampling of two glassy debris fragments (‘Trinitite’) from the world's first atomic bomb detonation conducted in New Mexico on July 16, 1945. Our primary goal in conducting these measurements was to understand whether examination of the U-Pu and Ba-Cs systematics by direct sampling (e.g. without any chemical separation or purification prior to isotope ratio measurement) could yield meaningful information that would differentiate the Trinitite fragments from glassy material lacking a nuclear fission signature. These measurements were conducted on a ThermoFisher Scientific Neoma multi collector – inductively coupled plasma – mass spectrometer (MC-ICP-MS), which is a relatively new MC-ICP-MS platform, so we also examine the behavior of these isotope systems in standards sampled in solution and via LA. Unsurprisingly, the measurements made on purified solutions of the U, Pu, and Ba isotopic standards produce high precision isotope ratios. Furthermore, this extends to the U-Pu measurements made by LA sampling, with the expected degradation in precision and accuracy related to matrix effects and signal intensity fluctuation. However, the Ba-Cs data acquired by LA is of low precision across all of the matrices examined and bears evidence of complex mass fractionation that will require further investigation to resolve. In total, our results indicate that the observed U-Pu isotope data are of sufficient quality to accurately constrain the U and Pu isotopic composition of glass containing sub-ppm levels of these elements which in turn could be used to differentiate glass containing anthropogenic fission products from natural glass whereas the Ba-Cs LA data cannot be used for this purpose until further methodological refinement is performed.

Ba-Cs↗
DOE OSTI2024

Error and Correction Analysis for the FFA@CEBAF Energy Upgrade

An energy upgrade design for the Continuous Electron Beam Accelerator Facility (CEBAF) is under development, using fixed field alternating gradient (FFA) return arcs to recirculate electron beam up to an additional five times through the accelerating structures at CEBAF. A necessary component of any large accelerator is a beam steering and optical correction system. Small environmental changes and system errors can lower beam quality or even shut down the machine; and in pursuit of the scientific mission of JLab, high quality electron beams must be delivered to the experimental halls on a predictable schedule. Correction in the novel FFA arcs of the current upgrade design is complicated by several factors. These complexities inform the choice of correction algorithm structure and parameter values. A baseline algorithm in addition to diagnostic and correction hardware configuration is presented. The effect of this correction protocol is shown with respect to estimated errors, and several possible extensions of the algorithm are discussed. This work presents an important proof of concept for the FFA@CEBAF design effort, and provides a functional correction strategy which may be simply adjusted and optimized for future design changes.

Coxe, Alex [Old Dominion Univ., Norfolk, VA (Unite↗
DOE OSTI2020

Joint Conference Support (ISPP and ICMP)

Funds are requested to support the attendance of junior scientists at two international conferences: the International Symposium on Phototrophic Prokaryotes (ISPP) and the International Conference on Microbial Photosynthesis (ICMP). Both will be held on the campus of the University of British Colombia in Vancouver in August of 2018. The primary goal of the ISPP is to provide a forum for students and senior investigators to gather in an informal setting to share new scientific findings on all aspects of bacterial phototrophs (both anoxygenic and oxygenic). The ICMP will appeal to those working on photosynthetic processes in these organisms as well as in eukaryotic phototrophs for other reasons (e.g. as major producers in the oceans, as models for marine phytoplankton, as biofuel-producing systems, etc.). The ICMP was designed to follow the ISPP such that students and postdocs could attend two conferences for the travel costs of one. The funds will be used to defray the registration costs of graduate student and postdoctoral attendees to make it even more affordable for junior scientists.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

The Analysis and Support for LANL Technology Testbed: High Altitude Balloon Gondola and Orion Eagle Gamma Ray Detection Instrument

The Analysis and Support for LANL Technology Testbed: High Altitude Balloon Gondola and Orion Eagle Gamma-Ray Detection Instrument is a project where Los Alamos National Laboratory (LANL), in conjunction with the University of Michigan, are looking into gamma-ray detection in a space-like environment. The purpose of the detection is to work on instruments that can assist with planetary exploration and nuclear deterrents using a High-Altitude Balloon (HAB). For this paper, the main focus is on the mechanical aspects related to the structural analysis of the gondola. It is vital to understand the structural analysis because it helps determine if the gondola can survive the changes in the environment and the different forces or shocks that it will experience when the parachute is deployed and at landing. Additionally, NASA's Balloon Program Office (BPO) has specific requirements to meet to achieve flight certifications. Part of those certifications requires the gondola to simulate different scenarios at specific conditions with various forces applied and orientations. Therefore, it is necessary to adjust the calculations of each method and the force applied to get the right results. Once that is done, the information is inputted into the structural analysis modeling system. The system will aid the mechanical team in assessing the conditions of the gondola at each scenario and determine its margin of safety. After the gondola and instrument are certified, the payload will be able to fly. The HAB and payload are launched from NASA's Columbia Scientific Balloon Facility (CSBF) at Fort Sumner, NM. The official flight was on September 26th, 2021.

42 ENGINEERING↗
DOE OSTI2025

WRS Capabilities Booklet [Slides]

WRS is the digital backbone of the Weapons Program—delivering trusted data assets, cyber-assured software and systems, and AI-enabling software—that transform insights into decisive action. We empower physicists, engineers, researchers, and scientists to think faster, act strategically, and stay ahead in an ever-evolving threat landscape. Our efforts ensure critical nuclear weapons data remains secure, accessible, and usable—supporting mission-critical work, informed decision making, and scientific advancement at LANL and across the Nuclear Security Enterprise (NSE).

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2025

COMPASS-FME Synoptic Sites Level 1 Sensor Data v2-1

This is the version 2-1 Level 1 (L1) data release for COMPASS-FME environmental sensors located at our synoptic field sites. COMPASS-FME is studying sites in two distinct regions, the Chesapeake Bay and the Western Lake Erie Basin. We established the network at seven "synoptic" (observational) sites along the Chesapeake Bay and Lake Erie coastlines, collectively generating over three million observations per month, to track and comprehend environmental changes where land and water intersect. Additionally, the two regions provide an interesting contrast of saltwater and freshwater coasts that allow us to differentiate the impacts of inundation and coastal water chemistries in two nationally important coastal systems. L1 data are close to raw, but are units-transformed and have out-of-instrument-bounds, out-of-service, and outlier flags added. Duplicates and missing data are removed but otherwise these data are not filtered, and have not been subject to any additional algorithmic or human QA/QC. Any scientific analyses of L1 data should be performed with care. **This dataset will be updated quarterly with new data for the duration of the project** This dataset includes: - An overall dataset README file that describes the current version, gives citation and contact information, etc. - Site- and year-specific folders, each holding up to 12 CSV (comma separated value) data files for each site and plot in that year. - Metadata files within each site-year folder provide full information on data units, expected ranges, contact information, as well as a general description of the site. - Environmental sensor types that appear in the data files include weather (ClimaVUE50, CS, RM Young, and LI instruments in the graphs below); soil conditions (TEROS12); soil redox state (Redox); groundwater variables (AquaTROLL200 and AquaTROLL600); open water sondes (Exo); tree sap velocity (Sapflow); and system voltage and state (Datalogger). Data are normally logged every 15 minutes. Please see v2-0 Synoptic L1 Sensor Package Quick Start.pdf for detailed information on data package structure, temporal coverage, and versioning. This dataset was updated 2026-03-12: (i) data now go through 2025-12-31 (previous end was 2025-06-30) and (ii) dataset and file names updated to “…v2-1” (previously was “v2-0”).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Probing Electron Correlations in 1D Electronic Materials Using Single Quantum Channels (Final Technical Report)

This research focuses on the investigation of strong electron correlations in quasi-one-dimensional oxide nanostructures. The archetypical system consisting of heterostructures formed from an ultrathin layer of LaAlO 3 and TiO 2 -terminated SrTiO 3 offers a unique laboratory for probing strong correlations due to intrinsic richness of the physical system and its ability to be modulated at extreme nanoscale dimensions. Under suitable conditions, SrTiO 3 is known to exhibit a variety of important properties associated with strong electronic correlations, especially superconductivity, magnetism. Understanding the nature of these electronic correlations at fundamental scales, and in reduced spatial dimensions, will provide the scientific basis for new energy technologies, information technologies, and materials. The research program is motivated by recent discoveries and technical advances in the PIs' lab, specifically, (1) pioneering work to create conductive nanostructures to be created in LaAlO 3 /SrTiO 3 heterointerfaces with 2 nm spatial resolution, (2) the discovery of a novel correlated electronic phases in which electrons pair without forming a superconductor, and (3) the demonstrated ability to form high-quality quantum point contacts (QPCs) and achieve ballistic electron transport over micrometer-scales in LaAlO 3 /SrTiO 3 nanowires. The program has three major components. First, we have press forward to develop a full microscopic understanding of the mechanism for electron pairing in LaAlO 3 /SrTiO 3 nanostructures. Understanding the “glue” that leads to pairing and superconductivity in a doped semiconductor will help to form a scientific basis for the correlated oxide nanoelectronics platform being developed. Second, we have used their ability to create well-defined quantum channels in nanowires allows to allow novel “scattering” experiments to be performed that probe fundamentally new quasiparticle excitations using an unprecedented combination of experimental methodologies. Experiments already underway with artificially defined superlattices (5 nm period) demonstrate the feasibility of this approach. Third, we have investigated 1D correlations in proximally coupled nanowires, which are predicted exhibit Majorana-like zero modes in the absence of superconductivity or spin-orbit coupling. These studies provide insight not only into new and emergent phenomena in the rich class of complex oxides, but also fundamental mechanisms of superconductivity and magnetism. Thematically, the research aims to combine some of the most important and interesting challenges in the physics of semiconductor nanostructures and correlated electronic materials. The increased understanding of electron correlations can pave the way to the development of new materials with outstanding or novel properties, which could be relevant for energy-related technologies such as low-power high performance computing, quantum computation, or the development of novel materials.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2025

LCLS Big Data Handling – How I Learned to Stop Worrying and Love the Data Deluge

Advanced data and computing systems are vital to Linac Coherent Light Source (LCLS) operations, data interpretation and overall scientific productivity. The transition to MHz-era operation marks a fundamental change in scale that requires new infrastructure and architectures to link LCLS to the required scale of computing needed for scientific interpretation. The LCLS-II Data System meets big data challenges by implementing configurable data reduction that can adapt to multiple science areas, real-time analysis frameworks to provide visualization and fast feedback, and the ability to transfer data to local and remote computational facilities for near real time analysis at the appropriate scale. Feature extracted information generated in the data analysis pipeline - at the edge, local compute, or remote High-Performance Computing (HPC) resources - can be used to steer experiments and inform user decisions during beam time. Artificial Intelligence and Machine Learning (AI/ML) techniques present new opportunities to rapidly analyse large datasets and direct experiments, but create new challenges in scaling, adaptability, complexity, and trustworthiness. We describe how the LCLS-II Data System architecture addresses its data-driven challenges in the areas of data acquisition, data processing, data management, and workflow orchestration to decrease the overall time-to-science and provide a vision for future developments.

artificial intelligence↗
DOE OSTI2025

Literature Review of Selected Publications Relevant to Carbon Dioxide Pipeline and Storage Systems

The annotated bibliographies provided in this document focus on identifying and reviewing environmental impact statements (EIS), environmental assessments (EA), and scientific literature relevant to aspects of CO 2 pipeline and storage construction and operation. The purpose of these summaries is to aid stakeholders responsible for the preparation of documents compliant with the National Environmental Policy Act (NEPA) requirements to have access to a quick and comprehensive guide of the literature that can inform their activities. They aim to inform the development of EISs and EAs by identifying potential environmental concerns and mitigation strategies. The annotated bibliographies captured in this document cover the general environmental assessment and impact topics relevant to CO 2 pipeline and storage construction and operation activities, with the subject of waste creation and handling being specifically pulled out into its own section. The reasoning behind a dedicated section for waste is that many EAs and EISs focus on the description of waste and its impacts, be it caused in routine operation or as a result of an accident.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

SMC 2021 : Analyzing Resource Utilization and User Behavior on Titan Supercomputer

Resource utilization statistics of submitted jobs on a supercomputer can help us understand how users from various scientific domains use HPC platforms and better design a job scheduler. We explore to generate insight regarding workload distribution and usage pattern domains from job scheduler trace, GPU failure information, and project-specific information collected from Titan supercomputer. Furthermore, we want to know how the scheduler performance varies over time and how the users' scheduling behavior changes following a system failure. These observations have the potential to provide valuable insight, which is helpful to prepare for system failures. These practices will help us develop and apply novel machine learning algorithms in understanding system behavior, requirement, and better scheduling of HPC systems. There are two datasets, RUR and GPU. RUR: This dataset is the job scheduler traces collected from the Titan supercomputerfrom 01/01/2015 to 07/31/2019 (2015.csv - 2019.csv). These were collected usingResource Utilization Report (RUR), a Cray-developed resource-usage data collectionand reporting system. It contains the usage information of its critical resources (CPU,Memory, GPU, and I/O) of each running job on Titan during that period [2]. ProjectAreas: Every job is associated with a project ID. TheProjectAreas.csvdatasetprovides a mapping of the project ID to its domain science. GPU: There have been some hardware-related issues in the GPUs in Titan that caused some GPUs to fail, sometimes irrecoverably during some job runs. This dataset provides information regarding these failures during the execution of the submitted jobs. GPUs on Titan are uniquely identified by a serial number (SN), and they are installed in a location. A GPU can be installed in a location, then removed from that location following a failure, and then re-installed in a different location after fixing the problem. If the failure can't be recovered, the GPU might be removed entirely from Titan. There are two prominent types of failures that resulted in the removal of GPUs from Titan: Double Bit Error (DBE) and Out of the Bus (OTB). The dataset (gc_full.csv) has the following fields: 1. SN : Serial number of a GPU 2. location : The location where it is installed 3. insert : The time when it was inserted into that location 4. remove : The time when it was removed from that location 5. duration : Amount of time the GPU spent in this location 6. out : If the device was taken out entirely w/o a re-installment into a new location. 7. event : If the GPU was taken out entirely, the reason for its removal. To learn more about this dataset, please refer to the git repositoryhttps://github.com/olcf/TitanGPULifeand the related publication [1]. References [1] George Ostrouchov, Don Maxwell, Rizwan A Ashraf, Christian Engelmann, MallikarjunShankar, and James H Rogers. Gpu lifetimes on titan supercomputer: Survival analysisand reliability. InSC20: International Conference for High Performance Computing,Networking, Storage and Analysis, pages 1-14. IEEE, 2020. [2] Feiyi Wang, Sarp Oral, Satyabrata Sen, and Neena Imam. Learning from five-yearresource-utilization data of titan system. In2019 IEEE International Conference onCluster Computing (CLUSTER), pages 1-6. IEEE, 2019.

42 ENGINEERING↗
DOE OSTI2021

SMC 2021 Data Challenge: Analyzing Resource Utilization and User Behavior on Titan Supercomputer

Resource utilization statistics of submitted jobs on a supercomputer can help us understand how users from various scientific domains use HPC platforms and better design a job scheduler. We explore to generate insight regarding workload distribution and usage pattern domains from job scheduler trace, GPU failure information, and project-specific information collected from Titan supercomputer. Furthermore, we want to know how the scheduler performance varies over time and how the users' scheduling behavior changes following a system failure. These observations have the potential to provide valuable insight, which is helpful to prepare for system failures. These practices will help us develop and apply novel machine learning algorithms in understanding system behavior, requirement, and better scheduling of HPC systems. There are two datasets, RUR and GPU: RUR dataset is the job scheduler traces collected from the Titan supercomputer from 01/01/2015 to 07/31/2019 (2015.csv - 2019.csv). These were collected using resource Utilization Report (RUR), a Cray-developed resource-usage data collection and reporting system. It contains the usage information of its critical resources (CPU, Memory, GPU, and I/O) of each running job on Titan during that period (https://ieeexplore.ieee.org/abstract/document/8891001). It includes ProjectAreas as additional information, every job is associated with a project ID. TheProjectAreas.csv dataset provides a mapping of the project ID to its domain science. GPU dataset has information regarding GPU failure on Titan. There have been some hardware-related issues in the GPUs in Titan that caused some GPUs to fail, sometimes irrecoverably during some job runs. This dataset provides information regarding these failures during the execution of the submitted jobs. GPUs on Titan are uniquely identified by a serial number (SN), and they are installed in a location. A GPU can be installed in a location, then removed from that location following a failure, and then re-installed in a different location after fixing the problem. If the failure can't be recovered, the GPU might be removed entirely from Titan. There are two prominent types of failures that resulted in the removal of GPUs from Titan: Double Bit Error (DBE) and Out of the Bus (OTB). The dataset (gc_full.csv) has seven attributes, we provided a short description of these attributes in the ReadMe file. To learn more about this dataset, please refer to the git repository https://github.com/olcf/TitanGPULife and the related publication (https://ieeexplore.ieee.org/abstract/document/9355319).

42 ENGINEERING↗