Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Machine vision”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

Ground Simulation of an Autonomous Satellite Rendezvous and Tracking System Using Dual Robotic Systems

A hardware-in-the-loop ground system was developed for simulating a robotic servicer spacecraft tracking a target satellite at short range. A relative navigation sensor package "Argon" is mounted on the end-effector of a Fanuc 430 manipulator, which functions as the base platform of the robotic spacecraft servicer. Machine vision algorithms estimate the pose of the target spacecraft, mounted on a Rotopod R-2000 platform, relay the solution to a simulation of the servicer spacecraft running in "Freespace", which performs guidance, navigation and control functions, integrates dynamics, and issues motion commands to a Fanuc platform controller so that it tracks the simulated servicer spacecraft. Results will be reviewed for several satellite motion scenarios at different ranges. Key words: robotics, satellite, servicing, guidance, navigation, tracking, control, docking.

Trube, Matthew J.↗

Towards Autonomous Operations of the Robonaut 2 Humanoid Robotic Testbed

The Robonaut project has been conducting research in robotics technology on board the International Space Station (ISS) since 2012. Recently, the original upper body humanoid robot was upgraded by the addition of two climbing manipulators ("legs"), more capable processors, and new sensors, as shown in Figure 1. While Robonaut 2 (R2) has been working through checkout exercises on orbit following the upgrade, technology development on the ground has continued to advance. Through the Active Reduced Gravity Offload System (ARGOS), the Robonaut team has been able to develop technologies that will enable full operation of the robotic testbed on orbit using similar robots located at the Johnson Space Center. Once these technologies have been vetted in this way, they will be implemented and tested on the R2 unit on board the ISS. The goal of this work is to create a fully-featured robotics research platform on board the ISS to increase the technology readiness level of technologies that will aid in future exploration missions. Technology development has thus far followed two main paths, autonomous climbing and efficient tool manipulation. Central to both technologies has been the incorporation of a human robotic interaction paradigm that involves the visualization of sensory and pre-planned command data with models of the robot and its environment. Figure 2 shows screenshots of these interactive tools, built in rviz, that are used to develop and implement these technologies on R2. Robonaut 2 is designed to move along the handrails and seat track around the US lab inside the ISS. This is difficult for many reasons, namely the environment is cluttered and constrained, the robot has many degrees of freedom (DOF) it can utilize for climbing, and remote commanding for precision tasks such as grasping handrails is time-consuming and difficult. Because of this, it is important to develop the technologies needed to allow the robot to reach operator-specified positions as autonomously as possible. The most important progress in this area has been the work towards efficient path planning for high DOF, highly constrained systems. Other advances include machine vision algorithms for localizing and automatically docking with handrails, the ability of the operator to place obstacles in the robot's virtual environment, autonomous obstacle avoidance techniques, and constraint management.

Badger, Julia↗

Systems and Methods for Automated Water Detection Using Visible Sensors

Systems and methods are disclosed that include automated machine vision that can utilize images of scenes captured by a 3D imaging system configured to image light within the visible light spectrum to detect water. One embodiment includes autonomously detecting water bodies within a scene including capturing at least one 3D image of a scene using a sensor system configured to detect visible light and to measure distance from points within the scene to the sensor system, and detecting water within the scene using a processor configured to detect regions within each of the at least one 3D images that possess at least one characteristic indicative of the presence of water.

Rankin, Arturo L.↗

CMOS Active Pixel Image Sensor

A new CMOS active pixel image sensor is reported. The sensor uses a 2.0mue double-poly, double-metal foundry CMOS process and is realized as a 28 x 28 array of 40 mue x 40 mue pixels. The Sensor features TTL compatible voltages, low noise and large dynamic range, and will be useful in machine vision and smart sensor applications.

CMOS↗

Detection and Tracking of Aircraft in the Far-Field from Small Unmanned Aerial Systems

Onboard far-field aircraft detection is needed for safe non-cooperative traffic mitigation in autonomous small Unmanned Aerial System (sUAS) operations. Machine vision systems, based on standard optics and visible light detectors, possess the ideal size, weight, and power (SWaP) requirements for sUAS. This work presents the design and analysis of a novel aircraft detection and tracking pipeline based on optical sensing alone. Key contributions of the work include a refined range inequality model based on sensing and detection with FAA well-clear separation assurance distances between aircraft in mind, a detector fusion method to maximize the benefit of two image detectors, and a comparative analysis of Linear Kalman-filtering and Extended Kalman-filtering to seek optimal tracking performance. The pipeline is evaluated offline against multiple intruder platforms, using two types of flight encounters: multirotor sUAS vs. fixed-wing sUAS and multirotor sUAS vs. general aviation(GA)plane. Analysis is restricted to the rate-limiting head-on and departing collision volume cases vertically separated for safety. Results indicate that it is feasible to use the proposed optical spatial-temporal tracking algorithm to provide adequate alerting time to prevent penetration of well-clear separation volumes for both sUAS and GA aircraft.

Unmanned Aerial System↗

Towards the Development of a Multi-Agent Cognitive Networking System for the Lunar Environment

This paper details the development of a multi-agent cognitive system intended to optimize networking performance in the lunar environment. NASA’s current concept of the future of lunar communication, LunaNet [1], outlines a complex network of networks. Challenges such as scalability, interoperability and reliability must first be addressed to successfully fulfill this vision. Machine intelligence can greatly reduce the reliance on human operators and enable efficient operations for tasks such as scheduling and network management. The application of machine learning, artificial intelligence, and other automated decision-making techniques can be used to allow network nodes to intelligently sense and adapt to changes in the environment such as link disruptions, new nodes joining the network, and support for a diverse range of protocols. Cognitive networking seeks to evolve these technologies into an autonomous system with improved science data return, reliability, and scalability. In this paper, we study three main areas a means to further develop cognitive networking capabilities: networking and flight software development, analysis of wireless data for modeling and simulation, and development of algorithms for a multi-agent system.

Cognitive Networking↗

Towards the Development of a Multi-Agent Cognitive Networking System for the Lunar Environment

This paper details the development of a multi-agent cognitive system intended to optimize networking performance in the lunar environment. NASA’s current concept of the future of lunar communication, LunaNet, outlines a complex network of networks. Challenges such as scalability, interoperability and reliability must first be addressed to successfully fulfill this vision. Machine intelligence can greatly reduce the reliance on human operators and enable efficient operations for tasks such as scheduling and network management. The application of machine learning, artificial intelligence, and other automated decision-making techniques can be used to allow network nodes to intelligently sense and adapt to changes in the environment such as link disruptions, new nodes joining the network, and support for a diverse range of protocols. Cognitive networking seeks to evolve these technologies into an autonomous system with improved science data return, reliability, and scalability. In this paper, we study three main areas a means to further develop cognitive networking capabilities: networking and flight software development, analysis of wireless data for modeling and simulation, and development of algorithms for a multi-agent system.

Rachel Dudukovich↗

Towards the Development of a Multi-Agent Cognitive Networking System for the Lunar Environment

This paper details the development of a multi-agent cognitive system intended to optimize networking performance in the lunar environment. One concept of the future of lunar communication, LunaNet, outlines a complex network of networks. Challenges such as scalability, interoperability, and reliability must first be addressed to successfully fulfill this vision. Machine intelligence can greatly reduce the reliance on human operators and enable efficient operations for tasks such as scheduling and network management. Machine learning, artificial intelligence, and other automated decision-making techniques can be used to allow network nodes to intelligently sense and adapt to changes in the environment such as link disruptions, new nodes joining the network, and support for a diverse range of protocols. Cognitive networking seeks to evolve these technologies into an autonomous system with improved science data return, reliability, and scalability. In this paper, we study four main areas as a means to further develop cognitive networking capabilities: networking protocol development, analysis of wireless data for modeling and simulation, development of algorithms for a multi-agent system, and spectrum sensing technology.

cognitive networking↗

AI4MARS: A Dataset for Terrain-Aware Autonomy on Mars

Deep learning has quickly become a necessity for selfdriving vehicles on Earth. In contrast, the self-driving vehicles on Mars, including NASA’s latest rover, Perseverance, which is planned to land on Mars in February 2021, are still driven by classical machine vision systems. Deep learning capabilities, such as semantic segmentation and object recognition, would substantially benefit the safety and productivity of ongoing and future missions to the red planet. To this end, we created the first large-scale dataset, AI4Mars, for training and validating terrain classification models for Mars, consisting of ~326K semantic segmentation full image labels on 35K images from Curiosity, Opportunity, and Spirit rovers, collected through crowdsourcing. Each image was labeled by ~10 people to ensure greater quality and agreement of the crowdsourced labels. It also includes ~1.5K validation labels annotated by the rover planners and scientists from NASA’s MSL (Mars Science Laboratory) mission, which operates the Curiosity rover, and MER (Mars Exploration Rovers) mission, which operated the Spirit and Opportunity rovers. We trained a DeepLabv3 model on the AI4Mars training dataset and achieved over 96% overall classification accuracy on the test set. The dataset is made publicly available.1

Ono, Hiro↗

Benchmarking and Testing of Qualcomm Snapdragon System-on-Chip for JPL Space Applications and Missions

As some space missions become more challenging due to new environments, greater distances, or more limited size, weight, and power (SWaP) constraints, spacecraft avionics must adapt to allow the spacecraft to be more autonomous and agile---eliminating the Spacecraft-Earth-Spacecraft feedback loop whenever possible. Prime examples of such missions include Aerobots (such as Ingenuity with extremely low SWaP constraints and demanding signal/image processing during flight) and landers in possibly hostile environments (such as a Europa lander mission, with limited communication capacity, high latency, and constrained power budget). To address these challenges, JPL worked with Qualcomm to demonstrate the use of their Snapdragon 801 system-on-chip (SoC) onboard the Ingenuity Helicopter on Mars. The Qualcomm Snapdragon SoC contains various subsystems, including an ARM cluster, a Graphics processing unit, a Digital Signal Processing subsystem, a Neural Processing Engine, Image Signal Processing subsystem, among others. Since the success of Ingenuity, JPL is continuing to work with Qualcomm to address other applications of the Snapdragon SoC technology. This includes the deployment of two 855 Snapdragon development boards onboard the International Space Station (ISS) for successful in-situ benchmarking of applications in space (beyond those tested on Ingenuity). In this paper, we will examine the performance of various applications that have been identified to benefit from greater onboard computational capability. These applications include (among others): machine vision algorithms that are expected to be critical in autonomous entry-descent-and-landing scenarios and real-time Aerobot flight navigation; Hyperspectral compression algorithms; Synthetic Aperture Radar Processing along with various instrument processing algorithms. We discuss how the infusion of Qualcomm's Snapdragon SoC is capable of enabling missions that may not have been able to achieve their goals with traditional flight computing. In addition, we also show that for some algorithms, the software implementation on the Snapdragon SoC outperforms traditional FPGA implementations.

Cretu, Vlad↗

Using Fiducial Markers for Pose Estimation of an OSWEC in a Wave Tank: Preprint

In this study, we consider a novel method of sensing the motion of a wave energy converter during testing in a wave flume under the influence of incoming waves. The wave energy converter considered in our research is an oscillating surge wave energy converter, which is a hinged paddle that responds to incoming waves. Motion sensing is normally done with inertial sensors, which can hinder the motion due to suspended cables that carry power and transmit signals. Our proposed method is contactless and can be implemented economically. A camera is used to record different marker patterns affixed to the moving paddle and the motion deduced by pose estimation algorithms. Fiducial markers are commonly used for robot localization and in augmented reality. There are many types of fiducial markers, including ArUco-type markers which are accurate, fast and robust. The system consists of markers attached to the paddle element and recorded using a machine vision camera. A pose estimation algorithm is then applied to the detected markers to estimate the tilt of the paddle. In this work, we examine the challenges of image acquisition and calibration for underwater targets, compare the motion obtained by this new system with a calibrated tilt sensor and identify areas where the new system may be superior.

computer vision↗

Thermographic phosphor digital image correlation

The invention advances the ratio method of phosphor thermography using two machine vision cameras for full-field temperature measurements of a solid surface. As an example, algorithms from digital image correlation (DIC) can be used to determine the stereoscopic imaging system intrinsic and extrinsic parameters, and accurately register material points on the sample to subpixel locations in each image with 0.07 px or better accuracy. A phosphor calibration sample fabricated using aerosol deposition can be used for in situ determination of the temperature-versus-intensity ratio relationship. The full calibration methodology and several improvements on two-color phosphor thermography open the door for full-field temperature measurements in dynamic tests with deforming test specimens. In particular, thermographic phosphors can be combined with stereo digital image correlation in a novel diagnostic, TP+DIC, to measure time-resolved full-field surface strains and temperatures simultaneously, thereby enabling testing in combined thermo-mechanical environments.

Winters, Caroline↗

Who's Got the Bridge? - Towards Safe, Robust Autonomous Operations at NASA Langley's Autonomy Incubator

NASA aeronautics research has made decades of contributions to aviation. Both aircraft and air traffic management (ATM) systems in use today contain NASA-developed and NASA sponsored technologies that improve safety and efficiency. Recent innovations in robotics and autonomy for automobiles and unmanned systems point to a future with increased personal mobility and access to transportation, including aviation. Automation and autonomous operations will transform the way we move people and goods. Achieving this mobility will require safe, robust, reliable operations for both the vehicle and the airspace and challenges to this inevitable future are being addressed now in government labs, universities, and industry. These challenges are the focus of NASA Langley Research Center's Autonomy Incubator whose R&D portfolio includes mission planning, trajectory and path planning, object detection and avoidance, object classification, sensor fusion, controls, machine learning, computer vision, human-machine teaming, geo-containment, open architecture design and development, as well as the test and evaluation environment that will be critical to prove system reliability and support certification. Safe autonomous operations will be enabled via onboard sensing and perception systems in both data-rich and data-deprived environments. Applied autonomy will enable safety, efficiency and unprecedented mobility as people and goods take to the skies tomorrow just as we do on the road today.

Allen, B. Danette↗

Beyond Human Vision: Exploring Materials with Machine Intelligence

Machine intelligence has the potential to revolutionize materials science, enabling autonomous synthesis, self-driving characterization, and accelerated modeling. However, despite the promise, successful implementation of these methods in day-to-day research remains a challenge. This talk will delve into the reasons behind this, exploring how truly intelligent experiments are hindered by opaque experiment control, a lack of domain-specific models, and human-centric design. Through a focus on the characterization of next-generation microelectronics and energy storage materials, I will share insights from both successful and failed attempts to implement machine intelligence. We will then explore the next steps necessary to unlock the full potential of machine intelligence in materials science, creating a future where intelligent systems work seamlessly alongside researchers to drive innovation and discovery.

artificial intelligence↗

Machine learning-enabled computer vision for plant phenotyping: a primer on AI/ML and a case study on stomatal patterning

Abstract Artificial intelligence and machine learning (AI/ML) can be used to automatically analyze large image datasets. One valuable application of this approach is estimation of plant trait data contained within images. Here we review 39 papers that describe the development and/or application of such models for estimation of stomatal traits from epidermal micrographs. In doing so, we hope to provide plant biologists with a foundational understanding of AI/ML and summarize the current capabilities and limitations of published tools. While most models show human-level performance for stomatal density (SD) quantification at superhuman speed, they are often likely to be limited in how broadly they can be applied across phenotypic diversity associated with genetic, environmental, or developmental variation. Other models can make predictions across greater phenotypic diversity and/or additional stomatal/epidermal traits, but require significantly greater time investment to generate ground-truth data. We discuss the challenges and opportunities presented by AI/ML-enabled computer vision analysis, and make recommendations for future work to advance accelerated stomatal phenotyping.

Plant Sciences↗

YOLO11 to SAM2 pipeline for feature extraction from nuclear test films

The response to the effects of nuclear detonations is supported by models that describe the evolution of the nuclear fireball and cloud and the associated transport of active debris. Validation of those descriptions relies on data from the nuclear test operations. Video records of those events offer a rich source of information that was exploited to a limited extent in historic analyses. Computer vision and machine learning techniques are powerful tools that can be used to increase the number of measurements that can be obtained from those films. In this work, we apply computer vision techniques to automatically track the temporal evolution of the nuclear fireball. In particular, we apply You Only Look Once 11 (YOLO11) and Segment Anything Model 2 (SAM2) in combination with minimal human intervention to digitized versions of the original nuclear test films. As part of the proposed workflow, the YOLO11 model is applied to films to determine bounding boxes for the fireball within each frame. These are then used as inputs to SAM2, which uses image segmentation to determine the fireball boundaries and their temporal evolution. We assess the accuracy of our approach by using it to determine the energy released during the Trinity nuclear test and comparing the results with previous analyses based on manual measurements.

Van Exel, Kimberly [ORNL] (ORCID:0009000877463894)↗

Uncertainty quantification of fireball features extracted from nuclear test films using computer vision

Films from the US’s historic nuclear testing era comprise the only extensive collection of imagery depicting high-yield detonations. These films offer unique insights into the characteristics of flows occurring on scales that are difficult to replicate experimentally, and they are a valuable source of data for the validation of models used to describe nuclear detonations. In recent work, we implemented modern computer vision and machine learning techniques to extract features of the fireball following nuclear detonation. With a training dataset of fireball films, we fine-tuned a You Only Look Once 11 (YOLO11) model to detect and track the fireball. Applied to a video, the outer bounding box produced in each frame by YOLO11 is used as an input prompt to Meta’s Segment Anything Model 2 (SAM2), which is shown to accurately predict the boundary of the fireball over time with high resolution. These state-of-the-art computer vision foundation models exhibit impressive visual accuracy in their results but lack an output of values that robustly quantify uncertainty in scientific applications. In this paper, we develop procedures for uncertainty quantification of extracted fireball features. We outline the application of a parallel attention mechanism to calculate uncertainty ranges that complement and better pose model validation data. This higher quality fireball validation data may serve to improve prognostic models describing nuclear detonations in support of nuclear forensic and emergency response activities.

Khristy, Joel [ORNL] (ORCID:0000000209963060)↗