Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Jupyter notebook”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Data, scripts, and figures associated with a manuscript studying impact of climate and topography on post-fire vegetation recovery.

This data package is associated with the publication “Impact of Topography and Climate on Post-fire Vegetation Recovery Across Different Burn Severity and Land Cover Types through Machine Learning” submitted to Remote Sensing of Environment (Zahura et al. 2023). In this research, a machine learning algorithm, random forest (RF), was utilized to examine the impact of climate and topography on post-fire vegetation recovery. We used enhanced vegetation index (EVI) to examine varying burn severity and land cover types. The data package includes the input files for RF model training, outputs from model predictions and analysis, and python scripts to run the model, analyze the results to understand model performance and interpretability, and plot manuscript figures. This data package contains three folders (Data, Scripts, and Figures), a file-level metadata (FLMD) csv, and a data dictionary (dd) csv. Please see Postfire_recovery_flmd.csv for a list of all files contained in this data package and descriptions for each. The data dictionary (Postfire_recovery_dd.csv) describes the csv column headers. The “Data” folder provides all the inputs and outputs to train the RF model, evaluate performance, and interpret predictions. The “Scripts” folder contains python scripts and jupyter notebooks for model training and result analysis. The “Figures” folder includes the figures used in the manuscript in “.png” and “.jpg” format.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Improving the Estimation of the Atmospheric Water Vapor Pressure Using Interpretable Long Short-Term Memory Networks: Dataset, Python code, and trained models

Atmospheric water vapor pressure is an essential meteorological control on land surface and hydrologic processes. It is not as frequently observed as other meteorologic conditions, but often inferred through the August–Roche–Magnus formula by simply assuming dew point and daily minimum temperatures are equivalent or by empirically correlating the two temperatures using an aridity correction. The performance of both methods varies considerably across different regions and during different time periods; obtaining consistently accurate estimates across space and time remains a great challenge. We applied an interpretable Long Short-Term Memory (iLSTM) network conditioned on static, location specific attributes to estimate daily vapor pressure for 83 FLUXNET sites in the United States and Canada. This data package includes all raw data of the 83 FLUXNET sites, input data for model training/validation/test, trained models and results, and python codes for the manuscript "Improving the Estimation of the Atmospheric Water Vapor Pressure Using an Interpretable Long Short-term Memory Network". Specifically, it consists of five parts. - First, "1_Daymet_data_83sites.zip" includes raw data downloaded from Daymet for the 83 sites used in the paper according to their longitude and latitude, in which vapor pressure is used. It also includes a pre-processed CSV data file combining all data from the 83 sites which is specifically used for the paper. - Second, "2_Fluxnet2015_data_83sites.zip" includes raw half hourly data of the 83 sites downloaded from FLUXNET2015 data portal, pre-processed daily data of the 83 sites, a CSV file including combined pre-processed daily data of the 83 sites, and a CSV file including the information (site ID, site name, latitude, longitude, data available period) of the 83 sites. - Third, "3_MODIS_LAI_data_83sites_raw.zip" includes raw leaf area index (LAI) data downloaded from the AppEEARs data portal. - Fourth, "4_Scripts.zip" includes all scripts related to model training and post-processing of a trained model, and a jupyter notebook showing an example for model post-processing. Two typo errors in files titled "run2get_args.py" and "postprocess.py" were corrected on March 27, 2024 to avoid confusions. - Finally, "Trained_models_and_results.zip" includes three folders and three files with suffix ".npy", and each folder corresponds to one file with suffix ".npy" with the same title. Each of the three folders include all trained models associated with one iLSTM model configuration (35 models for each configuration, details are described in the paper). Each file with suffix ".npy" includes the post-processed results of the corresponding 35 models under one iLSTM model configuration.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Water chemistry in flume channel and hyporheic zone (i.e., porewater) associated with: “Rethinking Aerobic Respiration in the Hyporheic Zone Under Variation in Carbon and Nitrogen Stoichiometry”

Dissolved oxygen (DO), total organic carbon (TOC), total nitrogen (TN), molecular data for organic matter, and biochemical reactions for surface water and porewater (i.e., hyporheic zone) collected from a water recirculating flume located at the University of Texas, Austin. The flume contained real river water from Lower Colorado River(Austin, TX) and clean sand. Hyporheic exchange in the flume was induced through The study aims to understand relationships between aerobic metabolism of organic matter and molecular characteristics of organic matter, such as thermodynamic signature and nitrogen content, through the extent of the hyporheic zone at 10 cm- resolution, and through time. During the experiment, organic matter (dry leaves) was added to the flume and removed after 24 hours. The water samples were collected before the addition of leaves, at the time of removal of leaves, and at hour 72. The water samples were analyzed using ultrahigh resolution Fourier transform ion cyclotron resonance mass spectrometry (FTICR-MS) and total organic carbon (TOC) and total nitrogen (TN) analysis. Dissolved oxygen content throughout the surface water and the hyporheic zone of the flume was measured with a large planar optode. This data package is associated with the publication ’ Rethinking Aerobic Respiration in the Hyporheic Zone Under Variation in Carbon and Nitrogen Stoichiometry’ published in Environmental Science and Technology (Turețcaia et al., 2023 https://doi.org/10.1021/acs.est.3c04765). The dataset is comprised of five folders (1) Diss_O2_pic, (2) input_files (3) output_files; (4) python_code; and (5) R_code . Diss_O2_pic contains siximages of dissolved oxygen distribution in a bedform at hours 0, 24, and 72 of the experiment conducted in a large recirculation flume. Images are in separate R and G channels (i.e., RGB). The input_files contains (1) a csv file with FTICR peaks identified within each sample, (2) a csv file with molecular information pertinent to FTICR data with Gibbs free energy calculations adjusted for environmental temperature, (3) a csv file containing concentrations of non-purgeable organic carbon measured throughout the experiment , (4) a csv file containing concentrations of total nitrogen measured throughout the experiment, (5) a csv file containing total biochemical reactions (i.e., transformations) identified in the dataset, (6) a csv containing transformation profiles, and (7) a csv file containing transformations with formulas, and (8) a jpg file with schematic representation of locations for sample collection. The output_files contains (1) and xlsx file containing percent biochemical reactions containing nitrogen identified across all 39 sample, (2) a csv file of merged FTICR data and molecular information files, (3) a csv files containing average Gibbs free energy within sampling domains and at each sampling location, (4) a csv file with average concentrations of dissolved oxygen across sampling locations at hour 0, (5) a csv file with average concentrations of dissolved oxygen across sampling locations at hour 24, (6) a csv file with average concentrations of dissolved oxygen across sampling locations at hour 72, (7) a csv file with percent chemical classes identified across sampling locations at hour 0, (8) a csv file with percent chemical classes identified across sampling locations at hour 24, (9) a csv file with percent chemical classes identified across sampling locations at hour 72, and (10) a csv file containing percent nitrogen containing biochemical reactions identified across sampling locations at hours 0, 24, and 72. The python_code contains seven ipynb files which are Jupyter Notebooks used for data analysis and figures generation. The R_code contains 3 R files with R code used for data analysis and figures generation. This data package contains the processed data used in the associated manuscript. This data has not been previously published.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Site and endmember spectra of terrestrial vegetation and soils for the Colorado Headwaters Ecological Spectroscopy Study, June-July 2025

This dataset provides site and endmember spectra collected during the 2025 Colorado Headwaters Ecological Spectroscopy Study (CHESS) campaign. The site spectra were collected to help validate airborne hyperspectral data acquired by the National Ecological Observatory Network's aerial observation platform (NEON AOP). Endmember spectra were collected to augment existing spectral libraries with additional samples of bare surfaces and non-photosynthetic vegetation. All measurements were acquired with an Analytical Spectral Devices (ASD) FieldSpec4 Hi-Res NG (Next Generation) spectroradiometer, which records radiance at 1nm (nanometer) intervals from the ultraviolet to the short-wave infrared (350-2500 nm). The dataset includes spectra measured at meadow sites where the CHESS team also collected vegetation samples for trait analyses. The site spectra were collected with the ASD FieldSpec4 palm grip attachment using an 8° field-of-view foreoptic. Site spectra are integrated measurements of the entire surface within the foreoptic’s field of view. For site-level spectra, the sun is the illumination source. A Spectralon panel mounted on a tripod was used for instrument optimization and white reference measurements for all site spectra. Site spectra were acquired within two hours of solar noon and within 48 hours of a NEON AOP overflight. Site spectra are labeled by date, sampling area, and site number according to the naming conventions of the CHESS campaign’s data management plan. The dataset also contains endmember spectra in the following categories: photosynthetic vegetation (PV), non-photosynthetic vegetation (NPV), bare (soil/rock), and flowers. Endmember measurements were acquired using either the contact probe or the leaf clip attachments of the ASD FieldSpec4. In these configurations, the bulb inside the spectrometer provides the light source for the measurements. The spectrometer was optimized and white reference measurements were recorded using the circular white pucks attached to the contact probe and leaf clip. Because they do not rely on solar illumination, contact probe and leaf clip measurements were collected during a broader time frame than the palm grip site spectra. Some endmembers were measured at CHESS meadow sites, while others were collected within the larger sampling area or in nearby locations (e.g. Gothic Townsite) with similar characteristics. Radiance, reflectance, and metadata files are split into three subfolders according to measurement type: proximal/palm grip (prx), contact probe (cp), and leaf clip (lc). Radiance spectra are provided in ASD file format (.asd file extension). All ASD files can be opened using the provided scripts. Metadata is provided in two formats: CSV file format (no geolocation) and GEOJSON file format (includes geolocation for each spectra). The dataset includes a set of pre-processed reflectance spectra as CSV files (yyyymmdd_rfl.csv). The python scripts and jupyter notebook used to calculate reflectance spectra from the ASD radiance data is included here and was previously published at: https://doi.org/10.3334/ORNLDAAC/2446. There is also a folder of JPEG photographs corresponding to selected spectra. We include a protocol document with detailed steps for ASD FieldSpec4 assembly and operations. This data additionally contains a file level metadata (flmd.csv) and data dictionary (dd.csv) file. Geospatial information: Geospatial data for mapping measurement site locations are in the files CHESS_polygons_lai_UTM.geojson, CHESS_polygons_shrub_UTM.geojson, and CHESS_polygons_meadow_UTM.geojson in the companion geospatial package for the 2025 CHESS campaign, ‘CHESS 2025: Location data for field observations and sampling’ (Henderson et al., 2026). CHESS Project Description: The Colorado Headwaters Ecological Spectroscopy Study (CHESS) comprised a multi-week airborne remote sensing and field observation campaign in the Upper Gunnison Basin, Colorado, conducted in June and July of 2025. Airborne remote sensing was conducted by the National Ecological Observatory Network Airborne Observation Platform (NEON AOP), concurrent with a field campaign run by the Rocky Mountain Biological Laboratory (RMBL), the Lawrence Berkeley National Laboratory (LBNL) and SLAC National Accelerator Laboratory Watershed Function Science Focus Area (SFA), and NASA-JPL (Jet Propulsion Laboratory) Earth Surface Mineral Dust Source Investigation (EMIT) program. Between June 10 and July 18, 2025, the NEON AOP flight team collected high-resolution aerial imaging spectroscopy and Light Detection and Ranging (LiDAR) data over three domains: the Upper East River (CRBU), Almont Triangle (ALMO), and the Upper Taylor Basin (UPTA). In coordination with the flights, a field campaign acquired ground-truth observations, including observations of vegetation composition, foliar traits, forest demography, and subsurface properties in 18 core sampling areas within the domains. Additional surface water observations were taken at over 380 point locations. All CHESS campaign datasets can be found within the CHESS ESS-DIVE data portal: https://data.ess-dive.lbl.gov/portals/chess. Funding Acknowledgment: This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration (80NM0018D0004) and was funded by EMIT Extended Mission Phase E Science.

2018 NEON and 2025 CHESS Campaigns↗
DOE OSTI2025

Hyporheic-zone Processes and Stream Oxygen Dynamics: Insights from a Multiscale Reactive Transport Model: Modeling Archive

This archive contains the data and Python scripts required to reproduce the analyses and figures in the study: Gomez-Velez, J. D., Rathore, S. S., Cohen, M. J., & Painter, S. L. (2025). Hyporheic-zone Processes and Stream Oxygen Dynamics: Insights from a Multiscale Reactive Transport Model. Submitted to Water Resources Research. The analysis utilizes the subgrid model Advection Dispersion Equation with Lagrangian Subgrids (ADELS) implemented in the Advanced Terrestrial Simulator (ATS; https://amanzi.github.io/ats/stable/). In this case, the ATS and Amanzi versions are (1) ATS version 1.5.1_f5ba18f8 and (2) Amanzi version 1.6-dev_53444cca4. The repository includes a Jupyter Notebook and the necessary data (Pandas DataFrames stored as pickle files) to generate the figures for the manuscript. Additionally, it contains Python scripts to create ATS input files, run the ATS simulations, and post-process the results. Finally, it provides routines for parameter estimation using the Single-Station Metabolism (SSM) model with the Differential Evolution Adaptive Metropolis (DREAM) Markov Chain Monte Carlo (MCMC) algorithm with ZS enhancements (DREAM-ZS).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Model data for infrastructure-aware simulation of compound flooding at Alligator Bayou Watershed, southeast Texas

This dataset supports infrastructure-aware hydrologic modeling and flood scenario analysis for the Alligator Bayou Watershed, a highly managed urban watershed in Southeast Texas. It includes Jupyter notebooks for figure reproduction, model configuration files, simulation outputs, and derived products used to quantify the influence of engineered stormwater infrastructure on flood behavior across multiple spatial scales. The dataset was generated using the Watershed Workflow Python package and the Advanced Terrestrial Simulator (ATS), enabling integrated surface–subsurface hydrologic simulations on a channel-aligned mesh with explicit representations of pump stations, gate structures, detention basins, and impervious surfaces. Outputs include time series of gate and pump flows, stage observations, and water balance components, as well as spatially explicit fields of peak ponded depth and flood duration across multiple infrastructure scenarios spanning a single-location detention basin expansion, distributed drainage limitations, and compound coastal flooding. These data facilitate full reproducibility of the manuscript figures and support further research on urban flood dynamics and the role of stormwater infrastructure in shaping watershed-scale flood response.

EARTH SCIENCE > OCEANS > COASTAL PROCESSES↗
DOE OSTI2022

FATES crown damage simulation outputs 2022

This dataset contains outputs from the Functionally Assembled Terrestrial Ecosystem Simulator (FATES) and accompanies the paper "Needham, J.F., Arellano, A., Davies, S.J., Fisher, R.A., Hammer, V., Knox, R., Mitre, D., Muller-Landau, H.C., Zuleta, D., Koven, C.D. Tree crown damage and its effects on forest carbon cycling in a tropical forest, 2022, Global Change Biology". Data are unprocessed netcdf file outputs from simulations that were run to test the effect of a new crown damage module in FATES. Specifically, this data package contains a sensitivity analysis to the carbon cushion parameter damage_Ccushion_ensemble_e1b5bd9_bf013ef_2021-09-02.h0.ensemble.sofar.nc, a sensitivity analysis to the root nitrogen stoichiometry parameter damage_Nstoich_ensemble_e1b5bd9_bf013ef_2021-09-02.h0.ensemble.sofar.nc, a sensitivity analysis to parameters controlling crown damage and recovery damage_recovery_ensemble_e1b5bd9_354f0b0_2021-09-02.h0.ensemble.sofar.nc, and a sensitivity analysis to the number of crown damage bins elm_fates_bci_*_damagebins.Eac53ccb80b-F8f994c29.2022-04-19.elm.h0.fullrun.nc. This data package also contains a high root nitrogen configuration of FATES, including both a control, and a crown damage simulation high_root_N_control_e1b5bd9_354f0b0_2021-09-02.clm2.h0.fullrun.nc and high_root_N_damage_e1b5bd9_354f0b0_2021-09-02.clm2.h0.fullrun.nc. There is an analogous low root nitrogen configuration of FATES, including a control, low_root_N_control_e1b5bd9_bf013ef_2021-09-02.clm2.h0.fullrun.nc a damage only simulation low_root_N_damageonly_e1b5bd9_bf013ef_2021-09-02.clm2.h0.fullrun.nc, a damage plus mortality simulation low_root_N_damage_mort_e1b5bd9_bf013ef_2021-09-02.clm2.h0.fullrun.nc, and a mortality only simulation low_root_N_mort_only_e1b5bd9_ef845c8_2021-09-02.clm2.h0.fullrun.nc. Finally, there is a two PFT simulation in which we test the effect of recovery on competitive dynamics, low_root_N_damage_two_pfts_stoichastic_e1b5bd9_bf013ef_2021-09-10.clm2.h0.fullrun.nc. These simulations test the effect of representing crown damage in FATES, compared with simulations that have an equivalent increase in mortality. Jupyter notebooks to analyse these files can be found at https://github.com/JessicaNeedham/Needham_etal_GCB_2022_FATES_crown_damage.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Files and scripts to support manuscript Shuman et al 2023 FATES-SPITFIRE ecosystem assembly across tropics

The dataset includes the parameter and domain files, relevant output files, and scripts to generate simulations and perform analysis with Jupyter notebooks that support the manuscript Shuman, JK et al 2023 “Dynamic ecosystem assembly and escaping the “fire-trap” in the tropics: Insights from FATES_15.0.0”. We have adapted the fire-behavior and effects module, SPITFIRE, for use with the Functionally Assembled Terrestrial Ecosystem Simulator (FATES), a size-structured vegetation demographic model. We test how climate, fire regime and fire-tolerance plant traits interact to determine the biogeography of tropical forests and grasslands. We assign different fire-tolerance strategies based on crown, leaf and bark characteristics, which are key observed fire-tolerance traits across woody plants. For these simulations, three types of vegetation compete for resources: a fire-vulnerable tree with thin bark, a vulnerable deep crown and fire-intolerant foliage; a fire-tolerant tree with thick bark, a thin crown and fire-tolerant foliage; and a fire-promoting C4 grass. We explore the model sensitivity to a critical parameter governing fuel moisture, and show that drier fuels promote increased burning, an expansion of area for grass and fire-tolerant trees and a reduction of area for fire-vulnerable trees. This conversion to lower biomass or grass areas with increased fuel drying results in increased fire burned area and its effects, which could fee back to local climate variables. Simulated size-based fire mortality for trees less than 20 cm in diameter and those with fire-vulnerable traits is higher than that for larger and/or fire-tolerant trees, in agreement with observations. Fire-disturbed forests demonstrate reasonable productivity and capture observed patterns of aboveground biomass in areas dominated by natural vegetation for the recent historical period, but have a large bias in less disturbed areas. Though the model predicts a greater extent of burned fraction than observed in areas with grass dominance, the resulting biogeography of fire-tolerant, thick-bark trees and fire-vulnerable, thin-bark trees corresponds to observations across the tropics. In areas with more than 2500 mm of precipitation, simulated fire frequency and burned area are low, with fire intensities below 150 kW m-1, consistent with observed understory fire behavior across the Amazon. Areas drier than this demonstrate fire intensities consistent with those measured in savannas and grasslands, with high values up to 4000 kW m-1. The results support a positive grass-fire feedback across the region, and suggest that forests which have existed without frequent burning may be vulnerable at higher fire intensities, which is of greater concern under intensifying climate and land use pressures. The ability of FATES to capture the connection between fire disturbance and plant fire-tolerance strategies in determining biogeography provides a useful tool for assessing the vulnerability and resilience of these critical carbon storage areas under changing conditions across the tropics.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

PARETO 0.9.0 Release

PARETO 0.9.0 Release. Highlights: New Features - Initial beneficial reuse implementation - Post-optimization timing for infrastructure buildout Bug Fixes - Address Pyomo solver bug for UI Gurobi solve - Update Toy Case Study to feasible data for hydraulics post_process - Update Jupyter notebook for fall release UI Updates - Added functionality to optimize with hydraulics options - Added new plots to KPI dashboard for water quality and hydraulics timelines

PARETO,PARETO-UI,PSE,Process Systems Engineering,P↗
DOE OSTI2024

IDAES-PSE 2.6.0 Release

The Institute for the Design of Advanced Energy Systems (IDAES) Integrated Platform is a versatile computational environment offering extensive process systems engineering (PSE) capabilities for optimizing the design and operation of complex, interacting technologies and systems. IDAES enables users to efficiently search vast, complex design spaces to discover the lowest cost solutions while supporting the full process modeling lifecycle, from conceptual design to dynamic optimization and control. The extensible, open platform empowers users to create models of novel processes and rapidly develop custom analyses, workflows, and end-user applications. IDAES-PSE 2.6.0 Release Highlights Upcoming Changes IDAES will be switching to the new Pyomo solver interface in the next release. Whilst this will hopefully be a smooth transition for most users, there are a few important changes to be aware of. The new solver interface uses a different version of the IPOPT writer (“ipopt_v2”) and thus any custom configuration options you might have set for IPOPT will not carry over and will need to be reset. By default, the new Pyomo linear presolver will be activated with ipopt_v2. Whilst are working to identify any bugs in the presolver, it is possible that some edge cases will remain. IDAES will begin deploying a new set of scaling tools and APIs over the next few releases that make use of the new solver writers. The old scaling tools and APIs will remain for backward compatibility but will begin to be deprecated. New Models, Tools and Features New Intersphinx extension automatically linking Jupyter notebook examples to project documentation New end-to-end diagnostics example demonstrated on a real problem New complementarity formulation for VLE with cubic equations of state, backward compatibility for old formulation New solver interface with presolve (ipopt_v2) in support of upcoming changes to the initialization and APIs methods, with default set to ipopt to maintain backwards compatibility; this will deprecate once all examples have been updated New forecaster and parameterized bidder methods within grid integration library Updated surrogates API and examples to support Keras 3, with backwards compatibility for older formats such as TensorFlow SavedModel (TFSM) Updated costing base dictionary to include the 2023 cost year index value Updated ProcessBlock to include information on the constructing block class Updated Flowsheet Visualizer to allow visualize() method to return value and functions Bug Fixes Fixed bug in the Modular Property Framework that would cause errors when trying to use phase-based material balances with phase equilibria. Fixed bug in Modular Properties Framework that caused errors when initializing models with non-vapor-liquid phase equilibria. Fixed typos flagged by June update to crate-ci/typos and removed DMF-related exceptions Minor corrections of units of measurement handling in power plant waste/transport costing expressions, control volume material holdup expressions, and BTX property package parameters Fixed throwing >7500 numpy deprecation warnings by replacing scalar value assignment with element extraction and item iteration calls Testing and Robustness Migrated slow tests (>10s) to integration, impacting test coverage but also yielding a nearly 30% decrease in local test runtime Pinned pint to avoid issues with older supported Python versions Pinned codecov versions to avoid tokenless upload behavior with latest version Bumped extensions to version 3.4.2 to allow pointing to non-standard install location Deprecations and Removals Python 3.8 is no longer supported. The supported Python versions are 3.9 through 3.12 The Data Management Framework (DMF) is no longer supported. Importing idaes.core.dmf will cause a deprecation warning to be displayed until the next release The SOFC Keras surrogates have been removed. The current version of the SOFC surrogate model in the examples repository is a PySMO Kriging model.

AS↗
DOE OSTI2025

python-regionalizer

A Python module and Jupyter Notebook for data regionalization (i.e., discretizing data to county or community levels).

LCA; data processing↗
DOE OSTI2021

OpenOA: An Open-Source Codebase For Operational Analysis of Wind Farms

OpenOA is an open source framework for operational data analysis of wind energy plants, implemented in the Python programming language. OpenOA provides a common data model, high level analysis workflows, and low-level convenience functions that engineers, analysts, and researchers in the wind energy industry can use to facilitate analytics workflows on operational data sets. OpenOA contains documentation, worked out examples in Jupyter notebooks, and a corresponding example dataset from the Engie Renewable’s La Haute Borne Dataset.

17 WIND ENERGY↗
DOE OSTI2021

TX$^2$: Transformer eXplainability and eXploration

The Transformer eXplainability and eXploration (Martindale & Stewart, 2021), or TX 2 software package, is a library designed for artificial intelligence researchers to better understand the performance of transformer models (Vaswani et al., 2017) used for sequence classification. The tool is capable of integrating with a trained transformer model and a dataset split into training and testing populations to produce an ipywidget (Project Jupyter Contributors, 2021) dashboard with a number of visualizations to understand model performance with an emphasis on explainability and interpretability. The TX 2 package is primarily intended to integrate into a workflow centered around Jupyter Notebooks (Kluyver et al., 2016), and currently assumes the use of PyTorch (Paszke et al., 2019) and Hugging Face transformers library (Wolf et al., 2020). The dashboard includes visualization and data exploration features to aid researchers, including an interactive UMAP embedding graph (McInnes et al., 2018) to understand classification clusters, a word salience map that can be updated as researchers alter textual entries in near real time, a set of tools to understand word frequency and importance based on the clusters in the UMAP embedding graph, and a set of traditional confusion matrix analysis tools.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

IBM Open Science Price - SWAP Gate Challenge

On November 30, 2020, IBM announced an open science price competition aimed at improving the fidelity of a SWAP gate on their 7 qubit Quantum system named ”Casablanca”. The details of the SWAP gate problem was outlined in a Jupyter notebook, which utilizes the IBM developed open source package Qiskit. The participants were allowed to work in teams with up to five people and restricted to only use open source software in solving the problem. Their results were due on April 16, 2021. Our approach to solving the SWAP gate challenge is based on the quantum optimal control techniques implemented in our open source packages Juqbox.jl and Quandary. The results from the optimal control approach is only as good as the accuracy in the description of the quantum system dynamics, characterized by a Hamiltonian model that was provided by IBM. Based on the calibrated control pulses IBM provide for the standard gate set, we developed a reverse engineering approach to calibrate our computational model. Techniques were developed to translate between Qiskit’s pulse representation and the B-spline formulation used in Juqbox and Quandary. The fidelity of the optimized pulse sequences were estimated using Qiskit’s randomized bench marking techniques.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2021

simple_RDE_model

Jupyter notebook containing a simple rotating detonation engine model to generate synthetic data for exploring data analysis techniques.

42 ENGINEERING↗
DOE OSTI2023

Testing Job Submission to FermiGrid via Dask on the Fermilab Elastic Analysis Facility

Over the course of a summer internship, Jobsub Lite task submission and Dask operations were tested in development and production EAF at Fermilab, first via terminal and subsequently via Jupyter notebook cell. Testing allowed for development of a script available to EAF users that expands and makes easier access to parallel computing resources at Fermilab and associated sites.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2023

Another Set of Python Tools for Visualizing and Manipulating Small-Angle Neutron Scattering Data: Descriptions and Examples

The GP-SANS, Bio-SANS and EQ-SANS instruments at ORNL utilize drtsans for data reduction. drtsans is built on Python, and it can be run using python scripts and Jupyter notebooks. The flexibility afforded by Python makes it possible to incorporate additional actions into the scripts used for data reduction, such as analysis and visualization. Here, a new set of tools for visualizing and manipulating SANS data that can be incorporated into the data reduction scripts for the ORNL SANS instruments, or employed during post–processing, is presented that expands the capabilities of the two previously-released tool sets.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2023

Domain Aware Deep-learning Algorithms Integrated with Scientific-computing Technologies (DADAIST)

This technical report summarized the contribution of the DADAIST project funded by the Data Model Convergence Initiative via the Laboratory Directed Research and Development (LDRD) investments at Pacific Northwest National Laboratory (PNNL). Specifically, we report the development of the NeuroMANCER (Neural Modules with Adaptive Nonlinear Constraints and Efficient Regularizations), a new open-source Scientific Machine Learning library for formulating and solving parametric constrained optimization problems, physics-informed system identification, and parametric optimal control problems. NeuroMANCER is using differentiable programming to combine modern data-driven models and optimization modeling language into a coherent algorithmic and software framework. NeuroMANCER is a Pytorch-based framework and adopts much of its philosophy focused on research and development, rapid prototyping, and streamlined deployment. Strong emphasis is given to extensibility, interoperability with the PyTorch ecosystem, and quick adaptability to custom domain problems. Neuromancer repository contains a comprehensive library of differentiable modules, including custom activation functions, matrix factorizations, deep learning architectures, neural differential equations, differential equation solvers, implicit layers such as iterative solvers, high-level API for symbolic expressions, API for modeling and control of dynamical systems, and extensive set of tutorial code examples in the form of python scripts and jupyter notebooks.

97 MATHEMATICS AND COMPUTING↗