SEARCH · Engineering Papers

Results for “Jupyter notebook”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

All sources NASA NTRS DOE OSTI

NASA NTRSDate not supplied

What (and How) MERRA-2 Reanalysis Data are Used in Applied Sciences

The Modern Era Retrospective-analysis for Research and Applications, Version 2 (MERRA-2) is the global atmospheric data reanalysis for the satellite era produced by NASA’s Global Modeling and Assimilation Office (GMAO), using the Goddard Earth Observing System Model (GEOS)version 5.12.4. The data are officially distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). MERRA-2 data have been widely used by the Earth sciences and application community. Since MERRA-2 data were released in early 2016, the number of registered data users has grown steadily from 1,252 in 2016 to 6477 in 2020. By the end of October 2021, ~16 petabytes (over 360 million files) of data have been distributed to more than 18,900 users. Searching in Google Scholar (https://scholar.google.com/), we have found over 7,000 articles, published between January 2017 and May 2021, involving the use ofMERRA-2 data. The figure shows the numbers for various application areas in which theMERRA-2 data have been used, covering almost all of the application areas defined in NASA Applied Sciences (http://appliedsciences.nasa.gov). The largest number of articles are found in disaster research, with the subcategories ordered in flood, wildfires, hurricanes and cyclones, and other forms of severe weather. In this presentation, we will discuss the preliminary findings from a review of the selected literature that uses MERRA-2 data in applied sciences. The current analytic and interoperable data services at GES DISC are listed, such as the on-the-fly subset and analysis service, NASA Giovanni; THREDDS Data Server(TDS); and Python Jupyter notebooks. In addition, we will introduce two new services for supporting the open sciences: My Dashboard and Related Publications.

data management↗

NASA NTRSDate not supplied

GL4U: Bioinformatics training for students and educators using space omics data

Amanda Marie Saravia-butler↗

NASA NTRSDate not supplied

fmdtools Tutorial: Intro to Resilience Modelling, Simulation, and Visualization in Python With fmdtools

This workshop will cover the basics of using the fmdtools package for the simulation of hazardous scenarios for resilience simulation. The fmdtools simulation package is an open-source python toolkit for simulating the dynamic response of a system to internal and externally-driven hazardous scenarios, including faults and environmental conditions, that can be used to analyze the risks related to these hazards. Prior to the development of fmdtools, researchers had to either adapt an (often limited) propriety toolkit or develop their own design/simulation/analysis codes to develop their models of hazardous events, a significant technical burden to both (1) leveraging resilience modeling methodologies and (2) extending these methodologies with their own contributions. The fmdtools package provides a number of model constructs and simulation and analysis methods to enable the designer to focus to solely on their modeling case-study while still enabling a significant degree of model expressiveness and adaptability via Python-based model definition. This tutorial will present the setup of fmdtools and a high-level overview of its use, as well as some simple examples for understanding how to leverage its modelling, simulation, and analysis capabilities. Familiarity with jupyter notebook and basic python will be assumed.

Daniel Hulse↗

NASA NTRSDate not supplied

GL4U: Using Space Omics Data to Provide Bioinformatics Training for Students and Educators

NASA’s GeneLab project provides researchers open access to space-relevant experiment multi-omics data that can be mined to understand the effects of spaceflight on biological systems. To maximize the number of scientists who understand and utilize GeneLab data and data processing pipelines, GeneLab has created GeneLab for Colleges and Universities (GL4U). GL4U provides space biology-relevant training in bioinformatics to the next generation of scientists through direct and indirect approaches. The GeneLab team plans to host two annual data processing bootcamps, one for college-level students (direct) and one for college educators (indirect – training of trainers), in which participants learn to analyze GeneLab’s space-relevant omics data. The GL4U direct training pilot program was conducted in June 2021. During the pilot, students participated in a week-long bootcamp consisting of space biology-specific lectures and hands-on instruction using Jupyter Notebooks to analyze RNA sequence data. This pilot demonstrated the capacity of GL4U for training young scientists and encouraging data re-use. In June 2022, GL4U partnered with Jet Propulsion Laboratory’s (JPL) Planetary Protection Center of Excellence to conduct the indirect training pilot program by training educators at historically black colleges and universities (HBCUs) and minority serving institutions (MSIs). During the educator pilot, participants received materials, training, and will be provided the necessary compute resources to enable them to run the bootcamp at their home institutions or alternatively to adapt the content to implement within existing courses, thereby extending the reach of this initiative. The GL4U training program provides undergraduate students from underrepresented groups the opportunity to learn about NASA and Space Biology, and to enhance their career prospects by gaining hands-on experience analyzing omics data, a skillset that is highly applicable and marketable in the life sciences. Pre- and post-bootcamp surveys were completed by all participants and show the overwhelming success of the bootcamps.

Amanda M. Saravia-Butler↗

NASA NTRSDate not supplied

GL4U: Bioinformatics Training for Students and Educators Using Space Omics Data

Amanda Marie Saravia-butler↗

NASA NTRSDate not supplied

Characterizing Wildfires in Western US.: A Cloud-based Case Study for Interdisciplinary Research using NASA Resources

This presentation will demonstrate a case study of interdisciplinary research done in the Amazon Web Services (AWS) cloud platform, in addition to in the local machine. We conduct data analysis next to data by leveraging various cloud-based data in NASA Earthdata Cloud, which are distributed by different missions/NASA Distributed Active Archive Centers (DAACs), and cloud computing resources at NASA. For instance, we directly access multiple datasets stored in the AWS Simple Storage Service (S3) buckets using a Python Jupyter notebook through a JupyterHub interface hosted in AWS (without having to download data), and conduct data analysis next to data in the cloud. We will also show how to share the research results following Open Source policy. This case study characterizes the change in wildfire events in the western United States during the past 20 years. In particular, we focus on the wildfires in California in 2021, one of the most severe wildfire years occurring in the most recent 20 years in California. We will analyze the possible causes of wildfires, such as drought conditions and climate variability, and examine the impacts of wildfires on air quality and atmospheric composition, and on land cover. We will examine the data distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), including aerosols and meteorological data from the NASA Modern-Era Retrospective analysis for Research and Applications version 2 (MERRA-2), precipitation from the Global Precipitation Measurement (GPM) and Global Precipitation Climate Project (GPCP), and aerosol index from Ozone Monitoring Instrument (OMI). We also utilize the data distributed by the Physical Oceanography (PO) DAAC, such as Sea Surface Temperature (SST) data from the Group for High Resolution Sea Surface Temperature (GHRSST), and the data distributed by Land Processes (LP) DAAC, such as Normalized Difference Vegetation Index (NDVI).

Xiaohua Pan↗

NASA NTRSDate not supplied

Microbial Optical Data Processing: A Key Step in the Metabolic Assessment of Lunar Explorer Instrument for Space Biology Applications (LEIA) and Biosentinel’s Payload Data

The BioSensor payload platform on BioSentinel and LEIA autonomously collects optical data from microbial model organisms in liquid culture. The BioSensor is designed to monitor metabolic activity using absorbance measurements of cell density and alamarBlue, a readily available colorimetric redox indicator dye. BioSentinel, a pioneering NASA CubeSat, uses yeast to study deep space radiation. LEIA investigates radiation and lunar gravity response. The experimental setup includes 16 wells equipped with three LEDs (570, 630, and 850 nm) and their corresponding photodetectors. One well is a calibration control without biology while the rest have desiccated cultures. Autonomous rehydration initiates the experiment. Data from the BioSensor are received from the flight and ground units, enabling comparison to uncover location-based metabolic rate variations. This study presents a Python Jupyter notebook developed for efficient data processing of multiple CSV files containing date and time columns, temperature, and well illumination data. It offers a user-friendly interface while maintaining computational power, automatically recognizing and iteratively processing data files in a user-input path. A Hampel filter with a short window eliminates outlier artifacts from sensor dropout. Because absorbance is a relative measurement, conversion from raw illumination requires defining a “blank” value, so the first data points are averaged to provide the necessary denominator. A cube-root function correction mitigates undesired drift caused by air pockets during the fluidic card filling phase, maintaining optical path length consistency. Beer-Lambert's law is applied to further convert absorbance values to cell and dye form concentrations, the desired science parameters. The processed data are saved and visualized as SVG plots. Future plans include extracting specific science parameters from the processed data like growth rate and metabolic rate, and identification of features corresponding to metabolic and phenotypic shifts such as starvation, shifts from aerobic to anaerobic growth, and osmotic stresses.

Space biology↗

NASA NTRSDate not supplied

GL4U: Using Space Biology Omics Data to Provide Bioinformatics Training for Students and Educators

NASA’s GeneLab project provides researchers open access to space-relevant multi-omics data via the Open Science Data Repository (OSDR) that can be mined to understand the effects of spaceflight on biological systems. To maximize the number of scientists who understand and utilize GeneLab data and data processing pipelines, GeneLab created GeneLab for Colleges and Universities (GL4U). GL4U provides space biology-relevant training in bioinformatics to the next generation of scientists through direct (training students) and indirect (training educators) approaches. The GL4U pilot programs were conducted in June 2021 (direct training) and 2022 (indirect training). During the pilots, students and educators at Historically Black Colleges and Universities (HBCUs) and Minority Serving Institutions (MSIs) participated in a week-long (direct training) or two-week-long (indirect training) bootcamp consisting of space biology-specific lectures and hands-on instruction using Jupyter Notebooks to analyze space biology RNA sequencing data from OSDR. During the educator pilot, participants received materials, training, and the necessary compute resources to enable them to run the bootcamp at their home institutions, thereby extending the reach of this initiative. In July 2023, GeneLab is partnering with JPL to expand GL4U to include amplicon sequencing (Amp-Seq) analysis training. During the GL4U Amp-Seq bootcamp, student and educator participants will receive training on how to analyze and interpret Amp-Seq data using the NASA GeneLab data processing pipeline. All bootcamp material, including instructions for requesting compute resources, will be made publicly available on GitHub for educators to teach the GL4U content in subsequent semesters. GL4U provides undergraduate students from underrepresented groups the opportunity to learn about NASA and Space Biology, and to enhance their career prospects by gaining hands-on experience analyzing omics data, a skillset that is highly applicable and marketable in the life sciences. We present results from pre- and post-training surveys completed by all participants of the Amp-Seq bootcamp.

Amanda M Saravia-Butler↗

NASA NTRSDate not supplied

Kamodo – An Adaptable Tool to Obtain and Compare Observations and Modeling Results

What is Kamodo? -Official NASA open-source project written in Python. -Building upon the functionalization of datasets. -It is a CCMC developed and maintained software tool for access, interpolation, and visualization of space weather models and data. -It allows model developers to represent simulation results as mathematical functions which may be manipulated directly by end users. -It handles unit conversion transparency and supports interactive science discovery through jupyter notebooks with minimal coding. -All Kamodo tools are accessible through Python, and all source code is publicly available on the Kamodo NASA GitHub repositories. -Kamodo does not generate model outputs. Users need to acquire the desired model outputs before they can be functionalized by Kamodo.

Kamodo↗

NASA NTRSDate not supplied

Enabling Analysis of Air Quality Data From Tropospheric Emissions: Monitoring of POllution (Tempo) Via Cloud-Based Tools

Launched in April 2023, the Tropospheric Emissions: Monitoring of POllution (TEMPO) instrument provides high-resolution measurements of key atmospheric pollutants, such as ozone, nitrogen dioxide, and formaldehyde. Maximizing the use and utility of this new source of air quality information requires streamlining data access for a wide variety of research, public health, and other interested users. These varied applications often require the data to be structured in different ways, e.g., specific formats, array shapes, or file sizes. To enable access to TEMPO data in different forms, the NASA Atmospheric Science Data Center (ASDC), as part of the NASA Earth Science Data and Information System (ESDIS), provides a variety of cloud-based data transformation and GIS visualization tools. This presentation demonstrates methods of accessing and working with TEMPO data through these services, while highlighting aspects of the software and algorithmic workflows that perform the necessary data transformations. Examples include data subsetting, concatenation, and visualizations accessible via Jupyter notebooks and GIS software.

Daniel Kaufman↗

NASA NTRSDate not supplied

Sampling Biases of Space-Based Observations of XCO 2 Associated With South American Biomass Burning Events

We examine time series of OCO-2 xCO2 observations from 2017-2020 coincident with other remotely sensed and in-situ observations to better understand external sampling effects associated with biomass burning events. The GES DISC will publish How-To documents as Jupyter Notebooks to describe how to aggregate OCO-2 Level 2 data and create similar time series.

Kristan Morgan↗

DOE OSTI2021

DESC DC2 Data Release Note

In preparation for cosmological analyses of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST), the LSST Dark Energy Science Collaboration (LSST DESC) has created a 300 deg$^2$ simulated survey as part of an effort called Data Challenge 2 (DC2). The DC2 simulated sky survey, in six optical bands with observations following a reference LSST observing cadence, was processed with the LSST Science Pipelines (19.0.0). In this Note, we describe the public data release of the resulting object catalogs for the coadded images of five years of simulated observations along with associated truth catalogs. We include a brief description of the major features of the available data sets. To enable convenient access to the data products, we have developed a web portal connected to Globus data services. We describe how to access the data and provide example Jupyter Notebooks in Python to aid first interactions with the data. We welcome feedback and questions about the data release via a GitHub repository.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2021

Machine Learning Tools Set for Natural Gas Fuel Cell System Design

This study is focusing on leveraging the system design tools set for the next-generation solid oxide fuel cell (SOFC) based natural gas fuel cell (NGFC) system. Conventionally, system design and optimization of NGFC systems rely heavily on traditional reduced order model (ROM) techniques and designers’ experience level. For overcoming the technical barriers of system design, multiple multi-physics models and machine learning (ML) tools have been utilized to automate the conceptual design process and enhance the reliability of solutions for the NGFC system. The proposed tools set includes a physics-informed ML tool for automated ROM construction that leverages advances in deep neural networks to significantly reduce ROM prediction error for the NGFC power island compared to traditional approaches. The constructed physics-informed ML ROM can be used in system design, and optimization tools set Institute for the Design of Advanced Energy Systems (IDAES) Process Systems Engineering (PSE) framework. The tools set also provides a user-friendly graphic user interface built within Jupyter Notebooks, and the complete tools set is open-source public available.

Wang, Dewei↗

DOE OSTI2021

Science Federation Emulation Testbed: Demonstration of VFSIE Functionalities

The Virtual Federated Science Instrument Environment (VFSIE) is a digital twin of a federated infrastructure of multiple sites with science instruments and computing systems that are geographically dispersed. It emulates the federation using containers and virtual hosts (vhosts) connected over local and wide-area networks for the main purpose of supporting the development and testing of science workflows and software stacks, which will subsequently transition to the physical infrastructure. In this demo, we present VFSIE of a four sites scenario and illustrate the following use cases: (i) execution of federation stack components that expose resources to enable a science user to execute workflows, (ii) execution of remote control commands of beam-line instruments to collect measurements and position the sensor, and (iii) remote execution and access of Jupyter Notebook for tomographic image computations

Al Najjar, Anees↗

DOE OSTI2023

Virtual Engineering Software Framework for Integrated Biomass Conversion Modeling

This presentation covers the design and implementation of a software tool to systematically connect computational models of unit operations to simulate an integrated process of low-temperature conversion of biomass to fuel. This virtual engineering (VE) software was designed with the overarching goal of connecting unit models written in various programming languages and requiring different computational resources within a single, flexible framework. The models and features currently considered for the VE library include mechanistic models for pretreatment, enzymatic hydrolysis, and aerobic bioreaction; high-fidelity computational fluid dynamics (CFD) simulations for enzymatic hydrolysis and aerobic bioreaction; and the capability to perform techno-economic analyses (TEA) using Aspen Plus, a commercial software package. The CFD models require access to high-performance computing (HPC) resources, so in addition to handling multiple programming languages and interfaces, the VE software must also be capable of interacting with an HPC scheduler to submit, run, and post-process jobs. Using the Python programming language, a new VE software package has been developed that contains functionality to manage the input-output communication between various unit models, schedule simulations to run on NREL's HPC and analyze those results, and interface with existing TEA software workflows. A Jupyter-notebook GUI was also created to solicit user input and provide documentation. In cases where multiple models for a particular unit-operation exist, selection between models is accomplished through a simple checkbox, with the appropriate inputs and outputs being parsed and converted seamlessly in the background. Each operation makes use of a different programming language, but the flow of information from pretreatment to enzymatic hydrolysis to bioreaction is managed with an intuitive, centralized file-communication strategy. In this talk, the programming approach and implementation details of the notebook are presented for multiple possibilities of the conversion process, including a demonstration of the ability to manage HPC resources. Additionally, an example of a sensitivity study of treatment parameters governing the overall conversion outcome is shown which highlights the ease of defining new problems using the VE Notebook workflow and leads into a discussion of ongoing work to enable outer-loop optimization studies.

biofuel↗

DOE OSTI2020

Imperial Valley Dark Fiber Project Continuous DAS Data

The Imperial Valley Dark Fiber Project acquired Distributed Acoustic Sensing (DAS) seismic data on a ~28 km segment of dark fiber between the cities of Calipatria and Imperial in the Imperial Valley, Southern California. Dark fiber refers to unused optical fiber cables in telecommunications networks and is repurposed in this project for DAS applications. The objective, which is further detailed in the attached journal article from Ajo-Franklin et al., is to demonstrate dark fiber DAS as a tool for basin-scale geothermal exploration and monitoring. The included DAS data were recorded during two days at the beginning the project. Data is stored in the .h5 (HDF5) file format, readable using various software tools, including the 'h5read' and 'h5info' functions in Matlab. Provided here are examples of methods to read and use the data with the 'h5py' package in Python.

15 GEOTHERMAL ENERGY↗

DOE OSTI2021

The Atacama Cosmology Telescope: Summary of DR4 and DR5 Data Products and Data Access

Two recent large data releases for the Atacama Cosmology Telescope (ACT), called DR4 and DR5, are available for public access. These data include temperature and polarization maps that cover nearly half the sky at arcminute resolution in three frequency bands; lensing maps and component-separated maps covering ~ 2,100 deg 2 of sky; derived power spectra and cosmological likelihoods; a catalog of over 4,000 galaxy clusters; and supporting ancillary products including beam functions and masks. The data and products are described in a suite of ACT papers; here we provide a summary. In order to facilitate ease of access to these data we present a set of Jupyter IPython notebooks developed to introduce users to DR4, DR5, and the tools needed to analyze these data. The data products (excluding simulations) and the set of notebooks are publicly available on the NASA Legacy Archive for Microwave Background Data Analysis (LAMBDA); simulation products are available on the National Energy Research Scientific Computing Center (NERSC).

79 ASTRONOMY AND ASTROPHYSICS↗

NASA NTRS2019

GeoNEX: A Cloud Gateway for Near Real-time Processing of Geostationary Satellite Products

The emergence of a new generation of geostationary satellite sensors provides land andatmosphere monitoring capabilities similar to MODIS and VIIRS with far greater temporal resolution (5-15 minutes). However, processing such large volume, highly dynamic datasets requires computing capabilities that (1) better support data access and knowledge discovery for scientists; (2) provide resources to enable real-time processing for emergency response (wildfire, smoke, dust, etc.); and (3) provide reliable and scalable services for the broader user community. This paper presents an implementation of GeoNEX (Geostationary NASA-NOAA Earth Exchange) services that integrate scientific algorithms with Amazon Web Services (AWS) to provide near realtime monitoring (~5 minute latency) capability in a hybrid cloud-computing environment. It offers a user-friendly, manageable and extendable interface and benefits from the scalability provided by Amazon Web Services. Four use cases are presented to illustrate how to (1) search and access geostationary data; (2) configure computing infrastructure to enable near real-time processing; (3) disseminate and utilize research results, visualizations, and animations to concurrent users; and (4) use a Jupyter Notebook-like interface for data exploration and rapid prototyping. As an example of (3), the Wildfire Automated Biomass Burning Algorithm (WF_ABBA) was implemented on GOES-16 and -17 data to produce an active fire map every 5 minutes over the conterminous US. Details of the implementation strategies, architectures, and challenges of the use cases are discussed.

GeoNEX↗