Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Jupyter”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

PARETO 0.8.0 Release

PARETO 0.8.0 Release. Highlights: Model Updates - Applied unified sets for pipeline and trucking arcs in strategic model - Apply unified sets for pipeline and trucking arcs in operational model - Added new config argument for removal efficiency calculation method - Standardized bidirectional capacity constraint - Added dependencies removed in IDAES 2.1 - Created bounding functions & utilities - Added Hydraulics module to the strategic model - Add additional arc types to strategic model Documentation and Tutorial Updates - Improved PARETO treatment document - Introduced general tutorial and treatment module Jupyter notebooks for Strategic Model - Update docs with correct support email list address - Consolidate and deduplicate Getting Started and resources for developers - Enable Black formatting for Jupyter notebooks - Add Binder configuration files and README Bug Fixes - Fix strategic model documentation typos - Removed duplicated units from output file header UI Updates - Added view for comparing different scenarios - Added functionality for manually overriding PARETO decisions

PARETO,PARETO-UI,PSE,Process Systems Engineering,P↗
DOE OSTI2021

TX$^2$: Transformer eXplainability and eXploration

The Transformer eXplainability and eXploration (Martindale & Stewart, 2021), or TX 2 software package, is a library designed for artificial intelligence researchers to better understand the performance of transformer models (Vaswani et al., 2017) used for sequence classification. The tool is capable of integrating with a trained transformer model and a dataset split into training and testing populations to produce an ipywidget (Project Jupyter Contributors, 2021) dashboard with a number of visualizations to understand model performance with an emphasis on explainability and interpretability. The TX 2 package is primarily intended to integrate into a workflow centered around Jupyter Notebooks (Kluyver et al., 2016), and currently assumes the use of PyTorch (Paszke et al., 2019) and Hugging Face transformers library (Wolf et al., 2020). The dashboard includes visualization and data exploration features to aid researchers, including an interactive UMAP embedding graph (McInnes et al., 2018) to understand classification clusters, a word salience map that can be updated as researchers alter textual entries in near real time, a set of tools to understand word frequency and importance based on the clusters in the UMAP embedding graph, and a set of traditional confusion matrix analysis tools.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

CTT: Tools for Fine Alignment of Flash X-ray Systems

The CTT code is a compilation of the python code that was prototyped in the interest of making a method for fine alignment of Flash X-ray Systems at the lab. It contains python code for interacting with LTT and running optimization loops using LTT simulations or ray tracing with RaySpace. It Includes classes for creating phantom objects and modular geometry objects in the simulations. It allows one to generate a random phantom that has been optimized to stay with a certain cubic volume and maximize the minimum distance between any two ball phantom centers in all projections. It was primarily made to work on ball phantoms which were simulated as Teflon spheres. It allows the alignment from some nominal position to some displaced position to simulate recovery of a real geometry from the nominal geometry of the system. Fine alignment can be done with the projections in LTT while a rough alignment is faster using rayspace. If rayspace works well for the use case it should be preferred since it is considerably faster and less resource intensive in general. Some Notes about using this code is that one will have to edit the imports for the files to get to the correct LTT path, LTT GUI path and to the CTT path when importing. This was made by using anaconda with python 3 on windows 10. The LTT code was stored in a anaconda environment which seemed to help the python find the LTT although it shouldn’t be necessary if you append the path in your code using sys.path.append(r'add the path you want here’). The sys.path.append method is a quick way to give your python code access to a given folder when running it (such as LTT or CTT). Note that since the code was originally made with paths for a specific machine that a new user will have to edit those paths to make it work in all the files that use the old paths. This is somewhat tedious but will be required to run the code on a new machine. The methodology used to run and make these notebooks starts with opening a command prompt as administrator in windows 10. After this one would activate the anaconda environment in the appropriate directory using “conda activate myenv” in the command prompt. After this open a jupyter-notebook using the keyword “jupyter-notebook". After that a notebook should open in a browser. Note that some libraries used may not come with anaconda so those may have to be installed. There shouldn’t be much though since the main things used are scipy, numpy, matplotlib, and LTT. LTTQuicksetup also requires some paths that will be specific to the computer so that should be changed as well.

42 ENGINEERING↗
DOE OSTI2022

Summary of NDC Capacity Building Workshop and Regional Seismic Travel Time (RSTT) in combination with Data Sharing and Integration Training

The NNSA Seismic Cooperation Program (SCP) sponsored Stephen Myers (LLNL), Michael Begnaud (LANL), Brian Young (SNL) and Istvan Bondar (Research Center for Astronomy and Earth Sciences, Hungary) to serve as a presenters/trainers at the “NDC Capacity Building Workshop and Regional Seismic Travel Time (RSTT) in combination with Data Sharing and Integration Training” September 4-8 2022 in Muscat, Oman (See Appendix A for the agenda). The workshop and training (workshop from here forward) was organized by the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO) Provisional Technical Secretariat (PTS). The first half of the week was devoted to NDC workshop activities, and the second half was devoted to RSTT training. Fifty-five participants from 27 countries and the CTBTO-PTS attended the 5-day workshop (See Appendix B for list of participants and countries of origin). Presentations from the PTS described the International Monitoring System (IMS), International Data Centre (IDC) products, and metrics of regional data utilization. Contributed presentations from each country’s scientists included descriptions of regional and national networks, methods of data analysis, and needs for material and technical assistance. Training included an overview of the RSTT method and instruction on how to locate seismic events with the iLoc program, which utilizes RSTT travel times to reduce bias in event location estimates. Methods of seismic tomography and the need for a high-quality tomographic set, including seismological “ground truth”, were emphasized. Seismological “ground truth” or “GT” is a term that has come to mean both events with known location and events with well-characterized locations that are estimated using seismological data, typically with epicenter accuracy of 5 km or better. Notably, the instructional platform has migrated from UNIX shell scripts to Jupyter Notebooks. Jupyter Notebooks have the advantage being more visually intuitive, including display of graphics within the notebook. Each notebook includes every processing step that participants need to reproduce the entire exercise.

58 GEOSCIENCES↗
DOE OSTI2024

Supporting Greater Interactivity in the IPython Visualization Ecosystem

Interactive visualizations are invaluable tools for building intuition and supporting rapid exploration of datasets and models. Numerous libraries in Python support interactivity, and workflows that combine Jupyter and IPyWidgets in particular make it straightforward to build data analysis tools on the fly. However, the field is missing the ability to arbitrarily overlay widgets and plots on top of others to support more flexible details-on-demand techniques. This work discusses some limitations of the base IPyWidgets library, explains the benefits of IPyVuetify and how it addresses these limitations, and finally presents a new open-source solution that builds on IPyVuetify to provide easily integrated widget overlays in Jupyter.

Martindale, Nathan↗
DOE OSTI2022

CORAL: A framework for rigorous self-validated data modeling and integrative, reproducible data analysis

Abstract Background Many organizations face challenges in managing and analyzing data, especially when relevant datasets arise from multiple sources and methods. Analyzing heterogeneous datasets and additional derived data requires rigorous tracking of their interrelationships and provenance. This task has long been a Grand Challenge of data science and has more recently been formalized in the FAIR principles: that all data objects be Findable, Accessible, Interoperable, and Reusable, both for machines and for people. Adherence to these principles is necessary for proper stewardship of information, for testing regulatory compliance, for measuring the efficiency of processes, and for facilitating reuse of data-analytical frameworks. Findings We present the Contextual Ontology-based Repository Analysis Library (CORAL), a platform that greatly facilitates adherence to all 4 of the FAIR principles, including the especially difficult challenge of making heterogeneous datasets Interoperable and Reusable across all parts of a large, long-lasting organization. To achieve this, CORAL's data model requires that data generators extensively document the context for all data, and our tools maintain that context throughout the entire analysis pipeline. CORAL also features a web interface for data generators to upload and explore data, as well as a Jupyter notebook interface for data analysts, both backed by a common API. Conclusions CORAL enables organizations to build FAIR data types on the fly as they are needed, avoiding the expense of bespoke data modeling. CORAL provides a uniquely powerful platform to enable integrative cross-dataset analyses, generating deeper insights than are possible using traditional analysis tools.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

TomoPyUI : a user-friendly tool for rapid tomography alignment and reconstruction

The management and processing of synchrotron and neutron computed tomography data can be a complex, labor-intensive and unstructured process. Users devote substantial time to both manually processing their data ( i.e. organizing data/metadata, applying image filters etc. ) and waiting for the computation of iterative alignment and reconstruction algorithms to finish. In this work, we present a solution to these problems: TomoPyUI , a user interface for the well known tomography data processing package TomoPy . This highly visual Python software package guides the user through the tomography processing pipeline from data import, preprocessing, alignment and finally to 3D volume reconstruction. The TomoPyUI systematic intermediate data and metadata storage system improves organization, and the inspection and manipulation tools (built within the application) help to avoid interrupted workflows. Notably, TomoPyUI operates entirely within a Jupyter environment. Herein, we provide a summary of these key features of TomoPyUI , along with an overview of the tomography processing pipeline, a discussion of the landscape of existing tomography processing software and the purpose of TomoPyUI , and a demonstration of its capabilities for real tomography data collected at SSRL beamline 6-2c.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

Open Chemistry, JupyterLab, REST, and quantum chemistry

Quantum chemistry must evolve if it wants to fully leverage the benefits of the internet age, where the worldwide web offers a vast tapestry of tools that enable users to communicate and interact with complex data at the speed and convenience of a button press. The Open Chemistry project has developed an open-source framework that offers an end-to-end solution for producing, sharing, and visualizing quantum chemical data interactively on the web using an array of modern tools and approaches. These tools build on some of the best open-source community projects such as Jupyter for interactive online notebooks, coupled with 3D accelerated visualization, state-of-the-art computational chemistry codes including NWChem and Psi4, and emerging machine learning and data mining tools such as ChemML and ANI. They offer flexible formats to import and export data, along with approaches to compare computational and experimental data.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

BASIN-3D: A brokering framework to integrate diverse environmental data

Diverse observational and simulation datasets are needed to understand and predict complex ecosystem behavior over seasonal to decadal and century time-scales. Integration of these datasets poses a major barrier towards advancing environmental science, particularly due to differences in the structure and formats of data provided by various sources. Here, we describe BASIN-3D (Broker for Assimilation, Synthesis and Integration of eNvironmental Diverse, Distributed Datasets), a data integration framework designed to dynamically retrieve and transform heterogeneous data from different sources into a common format to provide an integrated view. BASIN-3D enables users to adopt a standardized approach for data retrieval and avoid customizations for the data type or source. We demonstrate the value of BASIN-3D with two use cases that require integration of data from regional to watershed spatial scales. The first application uses the BASIN-3D Python library to integrate time-series hydrological and meteorological data to provide standardized inputs to analytical and machine learning codes in order to predict the impacts of hydrological disturbances on large river corridors of the United States. The second application uses the BASIN-3D Django framework to integrate diverse time-series data in a mountainous watershed in East River, Colorado, United States to enable scientific researchers to explore and download data through an interactive web portal. Thus, BASIN-3D can be used to support data integration for both web-based tools, as well as data analytics using Python scripting and extensions like Jupyter notebooks. The framework is expected to be transferable to and useful for many other field and modeling studies.

Varadharajan, C↗
DOE OSTI2024

Graph neural networks for CO 2 solubility predictions in Deep Eutectic Solvents

Deep Eutectic Solvents (DESs) are a promising class of solvents for CO 2 capture. DESs are complex mixtures that can be designed to optimize CO solubility and overall capture process efficiency. However, the vast design landscape of DES mixtures makes experimental investigation prohibitive; as such, there is a need for computational models that can quickly and efficiently navigate the design space and inform data collection efforts. In this work, we propose Graph Neural Network (GNN) models for predicting CO 2 solubility for DESs; the GNN leverages a mixture graph representation that captures the molecular structure of the DES components as well as their intermolecular interactions. Here, we compare the GNN framework against alternative architectures (neural networks, graph convolution networks, and random forests) and data representations (molecular fingerprints, sigma profiles, and graphs). We show that the proposed approach offers superior predictive performance; specifically, we show that solubility can be predicted reliably directly from molecular structure (without the need of using sigma profiles as proposed in previous studies). This result is important, as obtaining sigma profiles requires expensive density functional theory computations. We also explored the ability of GNNs to predict solubility for new DES mixtures and operating conditions. We found that the model extrapolates across temperature reliably. However, we also found deficiencies in the ability of the model to predict solubility for DES mixtures, pressures, and molar ratio not included in the training sets; we show that this is due to an inherent lack of chemical diversity in datasets available in the literature. The proposed computational capabilities can thus help navigate the design space of DES and inform data collection efforts. Our models, data, and benchmarks are shared as Python code implemented in Jupyter notebooks.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

The Baghdad Atlas: A relational database of inelastic neutron-scattering (n,n ' γ) data

A relational database has been developed based on the original (n,n'γ) work carried out by A. M. Demidov et al., at the Nuclear Research Institute in Baghdad, Iraq (Demidov et al., 1978) for 105 independent measurements comprising 76 elemental samples of natural composition and 29 isotopically-enriched samples. The information from this Atlas includes: γ-ray energies and relative intensities; nuclide and level data corresponding to the residual nucleus and meta data associated with the target sample that allows for the extraction of the flux-weighted (n,n'γ) cross sections for a given transition relative to a defined value. The optimized angular-distribution-corrected fast-neutron flux-weighted partial γ-ray cross section for the production of the 846.8-keV 21+→0gs+γ-ray transition in 56Fe, determined to be $\langle$σγ$\rangle$=143(29) mb, is used for this purpose. However, different values for the adopted cross section can be readily implemented to accommodate user preference based on revised determinations of this quantity. The Atlas (n,n'γ) data has been compiled into a series of CSV-style ASCII data sets and a suite of Python scripts have been developed to build and install the database locally. The database can then be accessed directly through the SQLite engine, or using alternative methods such as the Jupyter Notebook Python-browser interface. Several examples exploiting different interaction methodologies are distributed with the complete software package.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

A methodology for decay heat characterization in molten salt reactors

Accurate decay heat prediction in molten salt reactors (MSRs) faces dual challenges: complex operational uncertainties and the need for interpretable models compatible with engineering workflows. This work presents a hybrid machine learning and segmented polynomial methodology that addresses both requirements through three key innovations. First, a modular data architecture encodes MSR-specific operational parameters (power density: 1-100 W cm -3 , humidity: 0-0.1 wt %, air ingress: 0-0.1 mol %) with uncertainty-aware temporal discretization spanning 15 orders of magnitude. Second, region-optimized machine learning models achieve 92.3 % root mean square error (RMSE) reduction over conventional polynomials while maintaining physical interpretability through automated piecewise equation generation. Third, dual front-end interfaces accelerate safety analyses — a Jupyter environment enables researchers to explore 10,000+ parameter combinations via interactive widgets, while a Streamlit web application reduces design iteration cycles through production-grade visualization tools. Operational deployment demonstrates prediction times of only a couple hundred milliseconds for 10 4 years decay profiles, enabling real-time optimization of spent fuel container designs.

42 - ENGINEERING↗
DOE OSTI2022

A FAIR-compliant parts catalogue for genome engineering and expression control in Saccharomyces cerevisiae

The synthetic biology toolkit for baker's yeast, Saccharomyces cerevisiae, includes extensive genome engineering toolkits and parts repositories. However, with the increasing complexity of engineering tasks and versatile applications of this model eukaryote, there is a continued interest to expand and diversify the rational engineering capabilities in this chassis by FAIR (findable, accessible, interoperable, and reproducible) compliance. In this study, we designed and characterised 41 synthetic guide RNA sequences to expand the CRISPR-based genome engineering capabilities for easy and efficient replacement of genomically encoded elements. Moreover, we characterize in high temporal resolution 20 native promoters and 18 terminators using fluorescein and LUDOX CL-X as references for GFP expression and OD600 measurements, respectively. Additionally, all data and reported analysis is provided in a publicly accessible jupyter notebook providing a tool for researchers with low-coding skills to further explore the generated data as well as a template for researchers to write their own scripts. We expect the data, parts, and databases associated with this study to support a FAIR-compliant resource for further advancing the engineering of yeasts.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Three-dimensional paganica fault morphology obtained from hypocenter clustering (L'Aquila 2009 seismic sequence, Central Italy)

In seismic modelling, fault planes are normally assumed to be flat due to the lack of data which can constrain fault morphology. However, incorporating 3D fault morphology is important for modelling several phenomena, for example calculating mainshock induced stress changes. Here we utilize a data-analytical method to unveil the 3D rupture morphology of faults using unsupervised clustering techniques applied to earthquake hypocenters in seismic sequences. We apply this method to the 2009 L'Aquila seismic sequence which involved a M W 6.1 mainshock on April 6th. We use a dataset of about 50,000 relocated events, mostly microearthquakes, reaching magnitude of completeness equal to 0.7. Clustering distinguishes the earthquakes as occurring in three main clusters along with other minor fault segments. We then represent the morphology of the main Paganica fault system (responsible for the largest mainshock) using splines. This method shows promise as a step toward robustly and quickly obtaining 3D rupture morphologies where earthquake sequences have been monitored. The 3D model is presented interactively online, and the processing is presented in an interactive Jupyter Notebook (https://bit.ly/2MnCFdj).

58 GEOSCIENCES↗
DOE OSTI2021

Identification and correction of temporal and spatial distortions in scanning transmission electron microscopy

Scanning transmission electron microscopy (STEM) has become the technique of choice for quantitative characterization of atomic structure of materials, where the minute displacements of atomic columns from high-symmetry positions can be used to map strain, polarization, octahedra tilts, and other physical and chemical order parameter fields. The latter can be used as inputs into mesoscopic and atomistic models, providing insight into the correlative relationships and generative physics of materials on the atomic level. However, these quantitative applications of STEM necessitate understanding the microscope induced image distortions and developing the pathways to compensate them both as part of a rapid calibration procedure for in situ imaging, and the post-experimental data analysis stage. Here, we explore the spatiotemporal structure of the microscopic distortions in STEM using multivariate analysis of the atomic trajectories in the image stacks. Based on the behavior of principal component analysis (PCA), we develop the Gaussian process (GP)-based regression method for quantification of the distortion function. The limitations of such an approach and possible strategies for implementation as a part of in-line data acquisition in STEM are discussed. Here, the analysis workflow is summarized in a Jupyter notebook that can be used to retrace the analysis and analyze the reader's data.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Cluster Analysis of Combined EDS and EBSD Data to Solve Ambiguous Phase Identifications

A common problem in analytical scanning electron microscopy (SEM) using electron backscatter diffraction (EBSD) is the differentiation of phases with distinct chemistry but the same or very similar crystal structure. X-ray energy dispersive spectroscopy (EDS) is useful to help differentiate these phases of similar crystal structures but different elemental makeups. However, open, automated, and unbiased methods of differentiating phases of similar EBSD responses based on their EDS response are lacking. This paper describes a simple data analytics-based method, using a combination of singular value decomposition and cluster analysis, to merge simultaneously acquired EDS + EBSD information and automatically determine phases from both their crystal and elemental data. I use hexagonal TiB 2 ceramic contaminated with multiple crystallographically ambiguous but chemically distinct cubic phases to illustrate the method. Code, in the form of a Python 3 Jupyter Notebook, and the necessary data to replicate the analysis are provided as Supplementary material.

47 OTHER INSTRUMENTATION↗
DOE OSTI2020

Machine Learning for Materials Scientists: An Introductory Guide toward Best Practices

This Methods/Protocols article is intended for materials scientists interested in performing machine learning-centered research. Herein, we cover broad guidelines and best practices regarding the obtaining and treatment of data, feature engineering, model training, validation, evaluation and comparison, popular repositories for materials data and benchmarking data sets, model and architecture sharing, and finally publication. In addition, we include interactive Jupyter notebooks with example Python code to demonstrate some of the concepts, workflows, and best practices discussed. Overall, the data-driven methods and machine learning workflows and considerations are presented in a simple way, allowing interested readers to more intelligently guide their machine learning research using the suggested references, best practices, and their own materials domain expertise.

36 MATERIALS SCIENCE↗
DOE OSTI2022

A FAIR and AI-ready Higgs boson decay dataset

Abstract To enable the reusability of massive scientific datasets by humans and machines, researchers aim to adhere to the principles of findability, accessibility, interoperability, and reusability (FAIR) for data and artificial intelligence (AI) models. This article provides a domain-agnostic, step-by-step assessment guide to evaluate whether or not a given dataset meets these principles. We demonstrate how to use this guide to evaluate the FAIRness of an open simulated dataset produced by the CMS Collaboration at the CERN Large Hadron Collider. This dataset consists of Higgs boson decays and quark and gluon background, and is available through the CERN Open Data Portal. We use additional available tools to assess the FAIRness of this dataset, and incorporate feedback from members of the FAIR community to validate our results. This article is accompanied by a Jupyter notebook to visualize and explore this dataset. This study marks the first in a planned series of articles that will guide scientists in the creation of FAIR AI models and datasets in high energy particle physics.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗