SEARCH · Engineering Papers

Results for “data processing automation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

All sources NASA NTRS DOE OSTI

NASA NTRS2012

Using WorldView-2 Imagery to Track Flooding in Thailand in a Multi-Asset Sensorweb

For the flooding seasons of 2011-2012 multiple space assets were used in a "sensorweb" to track major flooding in Thailand. Worldview-2 multispectral data was used in this effort and provided extremely high spatial resolution (2m / pixel) multispectral (8 bands at 0.45-1.05 micrometer spectra) data from which mostly automated workflows derived surface water extent and volumetric water information for use by a range of NGO and national authorities. We first describe how Worldview-2 and its data was integrated into the overall flood tracking sensorweb. We next describe the use of Support Vector Machine learning techniques that were used to derive surface water extent classifiers. Then we describe the fusion of surface water extent and digital elevation map (DEM) data to derive volumetric water calculations. Finally we discuss key future work such as speeding up the workflows and automating the data registration process (the only portion of the workflow requiring human input).

surface water extent↗

NASA NTRS1996

ISS Calibration Analysis Support System

The newest generation of JPL imaging experiments, the Cassini Imaging Science Subsystem, required calibration analysis effort beyond that of its predecessor instruments. This called for streamlining the data reduction process with automation and flexibility while using software inherited from support of the Galileo Solid State Imaging Instrument.

instrument↗

NASA NTRS2012

Automated Performance Characterization of DSN System Frequency Stability Using Spacecraft Tracking Data

This software provides an automated capability to measure and qualify the frequency stability performance of the Deep Space Network (DSN) ground system, using daily spacecraft tracking data. The results help to verify if the DSN performance is meeting its specification, therefore ensuring commitments to flight missions; in particular, the radio science investigations. The rich set of data also helps the DSN Operations and Maintenance team to identify the trends and patterns, allowing them to identify the antennas of lower performance and implement corrective action in a timely manner. Unlike the traditional approach where the performance can only be obtained from special calibration sessions that are both time-consuming and require manual setup, the new method taps into the daily spacecraft tracking data. This new approach significantly increases the amount of data available for analysis, roughly by two orders of magnitude, making it possible to conduct trend analysis with good confidence. The software is built with automation in mind for end-to-end processing. From the inputs gathering to computation analysis and later data visualization of the results, all steps are done automatically, making the data production at near zero cost. This allows the limited engineering resource to focus on high-level assessment and to follow up with the exceptions/deviations. To make it possible to process the continual stream of daily incoming data without much effort, and to understand the results quickly, the processing needs to be automated and the data summarized at a high level. Special attention needs to be given to data gathering, input validation, handling anomalous conditions, computation, and presenting the results in a visual form that makes it easy to spot items of exception/ deviation so that further analysis can be directed and corrective actions followed.

Pham, Timothy T.↗

NASA NTRS2012

Automated Performance Characterization of DSN System Frequency Stability Using Spacecraft Tracking Data

This software provides an automated capability to measure and qualify the frequency stability performance of the Deep Space Network (DSN) ground system, using daily spacecraft tracking data. The results help to verify if the DSN performance is meeting its specification, therefore ensuring commitments to flight missions; in particular, the radio science investigations. The rich set of data also helps the DSN Operations and Maintenance team to identify the trends and patterns, allowing them to identify the antennas of lower performance and implement corrective action in a timely manner. Unlike the traditional approach where the performance can only be obtained from special calibration sessions that are both time-consuming and require manual setup, the new method taps into the daily spacecraft tracking data. This new approach significantly increases the amount of data available for analysis, roughly by two orders of magnitude, making it possible to conduct trend analysis with good confidence. The software is built with automation in mind for end-to-end processing. From the inputs gathering to computation analysis and later data visualization of the results, all steps are done automatically, making the data production at near zero cost. This allows the limited engineering resource to focus on high-level assessment and to follow up with the exceptions/deviations. To make it possible to process the continual stream of daily incoming data without much effort, and to understand the results quickly, the processing needs to be automated and the data summarized at a high level. Special attention needs to be given to data gathering, input validation, handling anomalous conditions, computation, and presenting the results in a visual form that makes it easy to spot items of exception/deviation so that further analysis can be directed and corrective actions followed.

Pham, Timothy T.↗

NASA NTRS2010

Seagrass Health Modeling and Prediction with NASA Science Data

Previous research has demonstrated that MODIS data products can be used as inputs into the seagrass productivity model developed by Fong and Harwell (1994). To further explore this use to predict seagrass productivity, Moderate Resolution Imaging Spectroradiometer (MODIS) custom data products, including Sea Surface Temperature, Light Attenuation, and Chlorophyll-a have been created for use as model parameter inputs. Coastal researchers can use these MODIS data products and model results in conjunction with historical and daily assessment of seagrass conditions to assess variables that affect the productivity of the seagrass beds. Current monitoring practices involve manual data collection (typically on a quarterly basis) and the data is often insufficient for evaluating the dynamic events that influence seagrass beds. As part of a NASA-funded research grant, the University of Mississippi, is working with researchers at NASA and Radiance Technologies to develop methods to deliver MODIS derived model output for the northern Gulf of Mexico (GOM) to coastal and environmental managers. The result of the project will be a data portal that provides access to MODIS data products and model results from the past 5 years, that includes an automated process to incorporate new data as it becomes available. All model parameters and final output will be available through the use National Oceanic and Atmospheric Administration?s (NOAA) Environmental Research Divisions Data Access Program (ERDDAP) tools as well as viewable using Thematic Realtime Environmental Distributed Data Services (THREDDS) and the Integrated Data Viewer (IDV). These tools provide the ability to create raster-based time sequences of model output and parameters as well as create graphs of model parameters versus time. This tool will provide researchers and coastal managers the ability to analyze the model inputs so that the factors influencing a change in seagrass productivity can be determined over time.

Robinson, Harold D.↗

DOE OSTI2022

A New Workflow of X-ray CT Image Processing and Data Analysis of Structural Features in Rock Using Open-Source Software

X-ray computed tomography (CT) images of rock specimens often contain artifacts which must be corrected before scientific analyses are performed. Here, we present a new workflow of automated image processing to utilize poor-quality X-ray CT scan images. The workflow runs on the open-source image analysis software and efficiently separates desired features from low-contrast scanned images. The new workflow is a two-step technique using contrast enhancement and automated feature segmentation to generate noise-free binary images. The results of binary images using the proposed workflow and using a conventional thresholding technique are analyzed to show the quality of the proposed method. The paper also presents a workflow of estimating the structural geometries of features in two and three dimensions. The results of the structural feature analyses and computational time were compared between the open-source (ImageJ) and commercial image analysis software (Bruker Computed Tomography Analyzer). The commercial software was more computationally efficient, but the task-specific macros in open-source software enabled the user-desired automation in image processing and data extraction of desired structural features of comparable quality.

47 OTHER INSTRUMENTATION↗

DOE OSTI2022

3DBFSVBF (3D BatFinder Smart Video BioFilter and Multi-class BatFinder Smart Video BioFilter) [SWR-22-88]

Bats are notoriously difficult to study, therefore, identifying specific behavioral trends and the precise environmental conditions at the time of collision requires a monitoring solution that can reliably collect relevant data. To date, thermal infrared video surveillance has been extensively applied to study bats and has proven to be a powerful yet cumbersome tool. Current analytical approaches are time consuming because data processing data has not been fully automated. In the past, steps have been taken to record avian and bat activity in conjunction with complicated image processing techniques that separate species from other moving objects within the field of view (i.e. clouds and portions of the wind turbine). Once the videos are collected, the post-processing does not allow real time monitoring and identification, leading to a delay in both studying the behavior of these species and determining the effectiveness of any impact reduction strategy being studied. Moreover, object identification capability is lacking, thus limiting the usefulness of video data. To resolve these issues, we are using open source 3D computer vision and machine learning techniques allowing for automatic detection of objects in real-time with the ability to correlate these objects with environmental variables and recording the flight paths of each object. The machine learning has been trained on 3D data and allows for automated real-time data collection, identification and tracking, thereby eliminating the need for long and tedious post-analysis processing of the videos. This machine learning model is an added feature to the previous BatFinder Smart Video BioFilter and increases the accuracy of that systems classification by increasing the accuracy of identifying bats (90% accuracy) and insects (69% accuracy) to a 97% accuracy. There are two object classifier machine learning models, Binary and multi-classification. Binary object classifier labeled BatFinder_Smart_Video_BioFilter.h5 distinguishes between biological objects and non-biological objects. The main goal of this object classifier is to ignore the turbine blades while detecting biological object flying withing the rotor swept area of the turbine. Non-biological objects have a probability of 0 and biological objects have a probability of 1. Multi-classifier labeled Multiclass_BatFinder_Smart_Video_BioFilter.h5 distinguishes between bats, birds, insects and non-biological.

Yarbrough, John↗

NASA NTRSDate not supplied

NASA biological and physical sciences databases: who’s the FAIRest of them all?

Conceptual models are a key part of the foundation of scientific study. Scientific data discovery and retrieval are often inaccurate and incomplete because these models are not sufficiently well-incorporated into data retrieval systems. Systems often don’t provide the necessary tools to those producing scientific data to fully and unambiguously annotate them and the result is consumers of the data cannot find them efficiently. The capability of data archives to provide these tools to link data to underlying conceptual models is one of dimensions of the recently developed “FAIR” principles (https://www.go-fair.org/fair-principles/ ), and is key to many automated processes being able to operate on these data, particularly analytics involving artificial intelligence. We used an open-source web service to measure the FAIR compliance of the three data archives operated by NASA for the biological and physical sciences: the Life Sciences Data Archive, the Physical Sciences Informatics database, and GeneLab. The service ingests references to data sets in these archives, and then executes domain-non-specific examinations of these data and metadata that test compliance to the FAIR principles. Of the 22 metrics tested, GeneLab passed 11 (50%), and PSI and LSDA each passed 7 (32%). These data were gathered using only one representative data set from each archive and we anticipate variability in results as we continue to apply these metrics to other data. A preliminary study of the failure traces for each metric suggests there is a wide range of effort and complexity in the enhancements required for each system to elevate FAIR compliance, and this is the subject of continued investigation. This information has been and will likely continue to be important information in planning these enhancements, with the goal of increased readiness of the data for automated processes.

database↗

NASA NTRS2020

Development of a Display Tool to Quality Control Weather Balloon Data for Space Launch Vehicles Using Python

Continuous atmospheric data analysis is an important factor for space launch vehicle design and operations. The balloon quality control tool was developed by NASA’s Marshall Space Flight Center (MSFC) Natural Environments Branch (NEB) for monitoring quality control processes and verifying the automated flags created on the balloon data sets analyzed. The data sets currently analyzed are comprised of high-resolution and low-resolution balloon data from NASA Kennedy Space Center (KSC), co-located on the United States Air Force’s Eastern range (ER) at the Cape Canaveral Air Force Station. The NEB was tasked to perform a quality assessment of these data sets and needed a tool to confirm the quality control (QC) flags produced from an automated process and add additional QC flags if necessary. This Graphical User Interface (GUI) was developed to visualize all of the data from these balloon sets, display any flags from the automated QC process, and add additional flags to variables if necessary. The GUI was developed in Python 3.6 utilizing different packages available such as pandas for data analysis and manipulation, NumPy for high-performance multidimensional array and tools to compute with and manipulate arrays, Matplotlib for plotting data and Tkinter to build the GUI.

Jessica K Headley↗

NASA NTRS2020

Development of a Display Tool to Quality Control Weather Balloon Data for Space Launch Vehicles

Jessica K Headley↗

DOE OSTI2023

SolarAPP+ Performance Review (2022 Data)

The Solar Automated Permit Processing Plus (SolarAPP+) platform is an online portal to facilitate and expedite rooftop solar photovoltaic (PV) permitting processes. SolarAPP+ allows PV contractors to upload system specifications, have those specifications automatically reviewed for code compliance, and receive instant approval for code-compliant systems. SolarAPP+ also provides inspection checklists to verify installation practices and adherence to approved designs. SolarAPP+ is available to authorities having jurisdiction (AHJs) at no cost. This report is part of an ongoing series of reviews of SolarAPP+ performance. Consistent with previous performance reviews, we summarize SolarAPP+ adoption trends to date and compare various metrics for PV systems permitted through SolarAPP+ versus systems permitted through conventional AHJ permitting processes. As of the end of 2022, the National Renewable Energy Laboratory (NREL) had contacted over 1,500 AHJs with significant solar permitting volume regarding SolarAPP+. Of those, 607 AHJs had at least expressed interest in the platform. 16 AHJs had begun piloting the platform and 15 of these had publicly launched the platform by the end of 2022. In 2022, 206 installers submitted 11,092 permits through the SolarAPP+ platform, including 708 permits for solar+storage systems. SolarAPP+ permits accounted for around 37% of all permits issued in participating AHJs. We compare permitting timelines through SolarAPP+ to traditional AHJ permitting processes to assess the platform's performance. Consistent with previous SolarAPP+ performance reviews, we find that permitting timelines are significantly shorter for SolarAPP+ projects. Based on median timelines, a typical SolarAPP+ project is permitted and inspected 8 business days sooner than traditional projects. We estimate that automatic SolarAPP+ permitting saved between 3,500 and 13,900 hours of AHJ staff time in 2022. Finally, we find evidence that SolarAPP+ may improve inspection outcomes, with SolarAPP+ projects failing inspections about 28% less frequently than traditional projects.

14 SOLAR ENERGY↗

DOE OSTI2024

SolarAPP+ Performance Review (2023 Data)

The Solar Automated Permit Processing Plus (SolarAPP+) platform is an online portal to facilitate and expedite rooftop solar photovoltaic (PV) and battery storage permitting processes. SolarAPP+ allows PV contractors to upload system specifications, have that information automatically reviewed for code compliance, and receive instant approval for code-compliant systems, reducing authority having jurisdiction (AHJ) staff time needed for review. SolarAPP+ also provides inspection checklists to verify installation practices and adherence to approved designs. SolarAPP+ is available to AHJs at no cost. This report is part of an ongoing series of reviews of SolarAPP+ performance. Consistent with previous performance reviews, we summarize SolarAPP+ adoption trends to date and compare various metrics for PV systems permitted through SolarAPP+ versus systems permitted through traditional AHJ permitting processes. As of the end of 2023, the National Renewable Energy Laboratory (NREL) had contacted over 1,700 AHJs with significant solar permitting volume regarding SolarAPP+. Of those, 793 AHJs had expressed interest in the platform as of the end of 2023. 161 AHJs had begun piloting the platform and 97 of these had publicly launched the platform by the end of 2023. In 2023, 668 installers submitted 18,906 permits through the SolarAPP+ platform, including 4,834 permits submitted as part of a solar plus storage program. SolarAPP+ permits accounted for around 43% of all permits issued in participating AHJs. We compare permitting timelines through SolarAPP+ to traditional AHJ permitting processes to assess the platform's performance. Consistent with previous SolarAPP+ performance reviews, we find that permitting timelines are significantly shorter for SolarAPP+ projects. Based on median timelines, a typical SolarAPP+ project is permitted and inspected 14.5 business days sooner than traditional projects. We estimate that automatic SolarAPP+ permitting saved around 7,200 hours of AHJ staff time in 2023. Finally, we estimate that SolarAPP+ eliminated over 150,000 business days in permitting-related delays in 2023.

14 SOLAR ENERGY↗

DOE OSTI2026

SolarAPP+ Performance Review (2024 Data)

The Solar Automated Permit Processing Plus (SolarAPP+) platform is an online portal to facilitate and expedite rooftop solar photovoltaic (PV) and battery storage permitting processes. SolarAPP+ allows PV contractors to upload system specifications, have that information automatically reviewed for code compliance, and receive instant approval for code-compliant systems, reducing authority having jurisdiction (AHJ) staff time needed for review. SolarAPP+ also provides inspection checklists to verify installation practices and adherence to approved designs. This report is part of an ongoing series of reviews of SolarAPP+ performance. Consistent with previous performance reviews, we summarize SolarAPP+ adoption trends to date and compare various metrics for PV systems permitted through SolarAPP+ versus systems permitted through traditional AHJ permitting processes. As of the end of 2024, 799 AHJs had expressed interest in the platform, with 264 fully adopting (215) or piloting (49) the platform. In 2024, 861 installers submitted 37,393 permits through the SolarAPP+ platform, including 27,375 permits for PV+storage systems. SolarAPP+ permits accounted for around 43% of all permits issued in all participating AHJs, and more than 60% of all permits in several participating AHJs. We compare permitting timelines through SolarAPP+ to traditional AHJ permitting processes to assess the platform's performance. Consistent with previous SolarAPP+ performance reviews, we find that permitting timelines are significantly shorter for SolarAPP+ projects. Based on median timelines, a typical SolarAPP+ project is permitted and inspected 12 business days sooner than traditional projects. We estimate that automatic SolarAPP+ permitting saved around 18,400 hours of AHJ staff time in 2024. Finally, we estimate that SolarAPP+ eliminated over 100,000 business days in permitting-related delays in 2024.

14 SOLAR ENERGY↗

DOE OSTI2022

BFSVBF (BatFinder Smart Video BioFilter) [SWR-22-87] and Multi-class BatFinder Smart Video BioFilter Keras

Bats are notoriously difficult to study, therefore, identifying specific behavioral trends and the precise environmental conditions at the time of collision requires a monitoring solution that can reliably collect relevant data. To date, thermal infrared video surveillance has been extensively applied to study bats and has proven to be a powerful yet cumbersome tool. Current analytical approaches are time consuming because data processing data has not been fully automated. In the past, steps have been taken to record avian and bat activity in conjunction with complicated image processing techniques that separate species from other moving objects within the field of view (i.e. clouds and portions of the wind turbine). Once the videos are collected, the post-processing does not allow real time monitoring and identification, leading to a delay in both studying the behavior of these species and determining the effectiveness of any impact reduction strategy being studied. Moreover, object identification capability is lacking, thus limiting the usefulness of video data. To resolve these issues, we are using open source computer vision and machine learning techniques allowing for automatic detection of objects in real-time with the ability to correlate these objects with environmental variables and recording the flight paths of each object. The code has gone through five rounds of development with images used to train the models. This advancement allows for automated real-time data collection, identification, and tracking, thereby eliminating the need for long and tedious post-analysis processing of the videos. We will discuss the two open source and publicly available machine learning models developed within this scope of this work: 1) a binary model with a 97.5% accuracy in identifying the difference between an object and an empty scene, including wind turbine and clouds; and 2) a multiple classification model with the capability of identifying the type of object detected: bats (90% accuracy), birds (83% accuracy), insects (69% accuracy) and non-biological (99% accuracy).

Yarbrough, John↗

NASA NTRS2017

An Innovative Infrastructure with a Universal Geo-Spatiotemporal Data Representation Supporting Cost-Effective Integration of Diverse Earth Science Data

The SpatioTemporal Adaptive Resolution Encoding (STARE) is a unifying scheme encoding geospatial and temporal information for organizing data on scalable computing/storage resources, minimizing expensive data transfers. STARE provides a compact representation that turns set-logic functions into integer operations, e.g. conditional sub-setting, taking into account representative spatiotemporal resolutions of the data in the datasets. STARE geo-spatiotemporally aligns data placements of diverse data on massive parallel resources to maximize performance. Automating important scientific functions (e.g. regridding) and computational functions (e.g. data placement) allows scientists to focus on domain-specific questions instead of expending their efforts and expertise on data processing. With STARE-enabled automation, SciDB (Scientific Database) plus STARE provides a database interface, reducing costly data preparation, increasing the volume and variety of interoperable data, and easing result sharing. Using SciDB plus STARE as part of an integrated analysis infrastructure dramatically eases combining diametrically different datasets.

Rilee, Michael Lee↗

NASA NTRS1971

Preprocessing techniques to reduce atmospheric and sensor variability in multispectral scanner data.

Multispectral scanner data are potentially useful in a variety of remote sensing applications. Large-area surveys of earth resources carried out by automated recognition processing of these data are particularly important. However, the practical realization of such surveys is limited by a variability in the scanner signals that results in improper recognition of the data. This paper discusses ways by which some of this variability can be removed from the data by preprocessing with resultant improvements in recognition results.

Crane, R. B.↗

NASA NTRS1997

Graphic Three-Axes Presentation of Residual Gas Analyzer Data

Residual gas analyzers (RGA) are commonly used to measure the composition of residual gases in thermal-vacuum test chambers. Measurements from RGA's are often used to identify and quantify outgassing contaminants from a test article during thermal-vacuum testing. RGA data is typically displayed as snapshots in time, showing instantaneous concentrations of ions from ionized residual gas molecules at different atomic masses. This ion concentration information can be interpreted to be representative of the composition of the residual gas in the chamber at the instant of analysis. Typically, test personnel are most interested in tracking the time history of changes in the composition of chamber residual gas to determine the relative cleanliness and the clean-up rate of the test article under vacuum. However, displays of instantaneous RGA data cannot provide test personnel with the preferred time history information. In order to gain an understanding of gas composition trends, a series of plots of individual data snapshots must be analyzed. This analysis is cumbersome and still does not provide a very satisfactory view of residual gas composition trends. A method was devised by the authors to present RCA data in a three-axis format, plotting Atomic Mass Unit (AMU), the Ionization Signal Response (ISR) as amps/torr as a function of AMU, and Time, to provide a clear graphic visualization of trends of changes in ISR with respect to time and AMU (representative of residual gas composition). This graphic visualization method provides a valuable analytical tool to interpret test article outgassing rates during thermal vacuum tests. Raw RGA data was extracted from a series of delimited ASCII files and then converted to a data array in a spreadsheet. Consequently, using the 3-D plotting functionality provided by the spreadsheet program, 3-D plots were produced. After devising the data format conversion process, the authors began developing a program to provide real-time 3-D plotting of RGA data. The intent of this program is to automate the RGA data acquisition process and to generate up-to-the-minute time history 3-D displays of stored RGA data (development of this program was not complete at the time of this writing). This paper provides a brief description of the data format conversion process and presents results from a recent test to illustrate the usefulness of this 3-D RGA data plotting technique.

Johnson, Kenneth R.↗

NASA NTRSDate not supplied

Online Rapid Analysis of Laser Heterodyne Radiometer (LHR) Data Using the Planetary Spectrum Generator (PSG)

One of the biggest challenges in developing scientific instruments is not just the build and testing of an instrument, but the method for processing the data and producing a consistent, well characterized data product that can be confidently used by the public and scientific community. Raw data products are frequently an array of numbers that are a read-out of voltages. The challenge is to convert these arrays into meaningful information as well as remove noise and interferences. Because this can be a tedious and time-consuming, the goal is to automate the process so that data ca be processed rapidly and be available in real-time for event monitoring. Here we present a rapid analysis method for Laser Heterodyne Radiometer (LHR) data that can be used to analyze data taken from a range of LHR observation modes (column, limb, etc.). This online tool uses the versatile Planetary Spectrum Generator (PSG) and allows LHR users to quickly analyze their own data using a NASA Goddard Space Flight Center (GSFC) monitored capability to ensure quality and reproducibility in the data products. Background: Development of a miniaturized LHR (mini-LHR) as a ground instrument for measuring carbon dioxide (CO 2 ) and methane (CH 4 ) in the atmospheric column started in 2009 with the commercial availability of distributive feedback lasers (DFB) in wavelengths that could measure absorption of gases in the near-IR. As new DFB lasers, detectors and hardware emerged at different wavelengths, additional gases were added to the capability. The mini-LHR was adapted into an occultation-viewing CubeSat with Lawrence Livermore National Laboratory (LLNL) for observing CO 2 , CH 4 , and H 2 O in the limb and was launched in 2019. Other versions of the LHR technology have been designed for observation of water vapor in the lunar exosphere as well as observations of trace gases in planetary atmospheres and plumes from icy moons. The PSG is an online tool developed at GSFC that can be used for synthesizing Earth and planetary spectra (atmospheres and surfaces) over a broad range of wavelengths (0.1 μm to 100 mm) for any observatory, orbiter or lander. Spectra are simulated by combining several state-of-the-art radiative transfer models, spectroscopic databases, planetary databases, as well as modern-era retrospective analysis for research and applications, version 2 (MERRA-2) data set which provides meteorological inputs such as modeled surface pressure for calculating dry-air columns.

Emily Wilson↗