SEARCH · Engineering Papers
Results for “machine vision”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Estimation of Arrivals on Green at Signalized Intersections Using Stop-Bar Video Detection
Across the world, traffic congestion is increasing with alarming rapidity. Traffic signal control effectiveness, in coordinated networks, is often investigated in relation to the type of vehicle arrivals at the signalized intersections. Recently, several transportation agencies have switched from traditional loop detectors to video detection. When video cameras are accompanied by computer vision, one can extract more information about traffic “dynamics” than by using traditional inductive loop detectors. Collecting arrival times of multiple vehicles after the first arrival at the stop-bar detector might be challenging when using inductive loop detectors (since after the first arrival, detector status is always occupied). However, emerging video detection systems allow tracking of each vehicle’s entrance time in the detection zone, departure time from the detection zone, and the type of vehicle. This information can be used to estimate vehicular arrival and departure times, which then can be fed into machine learning algorithms to estimate arrivals on green (AOG). However, such research ideas have not been documented so far. Thus, this paper presents an estimation model for AOG, which was developed using multigene genetic programming. A robust experimental dataset was collected from a highly calibrated and validated microsimulation model of an 11-intersection corridor in Chattanooga, TN. The results of the model’s performance analysis showed the high accuracy of the training-, testing-, and validation datasets. The practical benefit of this model is that it can be applied to estimate arrival types at intersections where only stop-bar video detection exists.
Vision science and technology for supervised intelligent space robots
The focus of recent work in robotic vision for application in intelligent space robots such as the Extravehicular Activity (EVA) Retriever is in visual function, that is, how information about the space world is derived and then conveyed to cognition. The goal of this work in visual function is first to understand how the relevant structure of the surrounding world is evidenced by regularities among the pixels of images, then to understand how these regularities are mapped on the premises that form the primitive elements of cognition, and then to apply these understandings with the elements of visual processing (algorithms) and visual mechanism (machine organization) to intelligent space robot simulations and test beds. Since visual perception is the process of recognizing regularities in images that are known on the basis of a model of the world to be reliable related to causal structure in the environment (because perception attaches meaning to the link between a conception of the environment and the objective environment), the work involves understanding generic, generally applicable models of world structure (not merely objects) and how that structure evidences itself in images.
A Novel Machine Learning-Based Gap-Filling of Fine-Resolution Remotely Sensed Snow Cover Fraction Data By Combining Downscaling and Regression
Satellite-based remotely sensed observations of snow cover fraction (SCF) can have data gaps in spatially distributed coverage from sensor and orbital limitations. We mitigate these limitations in the example fine-resolution Moderate Resolution Imaging Spectroradiometer (MODIS) data by gap-filling using auxiliary 1-km datasets that either aid in downscaling from coarser-resolution (5 km) MODIS SCF wherever not fully covered by clouds, or else by themselves via regression wherever fully cloud covered. This study’s prototype predicts a 1-km version of the 500-m MOD10A1 SCF target. Due to noncollocatedness of spatial gaps even across input and auxiliary datasets, we consider a recent gap-agnostic advancement of partial convolution in computer vision for both training and predictive gap-filling. Partial convolution accommodates spatially consistent gaps across the input images, effectively implementing a two-dimensional masking. To overcome reduced usable data from noncollocated spatial gaps across inputs, we innovate a fully generalized three-dimensional masking in this partial convolution. This enables a valid output value at a pixel even if only a single valid input variable and its value exist in the neighborhood covered by the convolutional filter zone centered around that pixel. Thus, our gap-agnostic technique can use significantly more examples for training (∼67%) and prediction (∼100%), instead of only less than 10% for the previous partial convolution. We train an example simple three-layer legacy super-resolution convolutional neural network (SRCNN) to obtain downscaling and regression component performances that are better than baseline values of either climatology or MOD10C1 SCF as relevant. Our generalized partial convolution can enable multiple Earth science applications like downscaling, regression, classification, and segmentation that were hindered by data gaps.
Scoreboard
Emerging HPC machines have given rise to enhanced compute power that far outstrips the machine's ability to save large scale results for post-processing. To combat this, in situ data analysis techniques are slowly being adopted. With in situ data management favoring workflows composed of multiple simulations and analyses connected in transit on heterogeneous machines, scientists and engineers need a tool that enables them to create data extracts, visualizations, and interactively monitor and steer their simulations. Scoreboard Phase II is a next generation analysis software that supports composite in transit workflows on heterogeneous architectures and restores interactivity to in situ data analysis through simulation monitoring and computational steering. Scoreboard provides a simulation dashboard with graphs of metrics over time, controls for setting custom simulation steering parameters, controls for managing the set of data extracts being produced in the simulation, as well as the ability to explore data extracts, all from a web browser. Realizing the vision outlined in this project required research into making a system that integrates end to end from simulations all the way to the user. In situ tools generally suffer from complexity and excessive software dependencies. Scoreboard, by contrast, is easy to build and integrate into simulation codes and it provides first class FORTRAN support. The Scoreboard library is capable of in situ and in transit data analysis that can produce data extracts commonly needed for Computational Fluid Dynamics (CFD) analysis. Simulations can transparently stage data in transit to a Scoreboard Endpoint program, which can accept their data and produce the requested data extracts. This lets simulations return to their work while the Endpoint works on the analysis. Efficiently staging the data at scale was a topic of this research. Scoreboard provides the means to let the user manage data extracts and monitor/steer many simulations from a web browser. This area of the research focused on discovery of in transit network components to expose and control their steering parameters within an interactive browser-based user interface that includes: system topology, gathered metrics, notifications, dynamically-generated steering controls, and exploration of visualization data products.
Effects of input gradient regularization on neural networks time-series forecasting of thermal power systems
This study proposes using neural networks, specifically gated recurrent unit (GRU), long-short-term memory (LSTM), and transformer networks, to improve control strategies in a 450 MW coal-fired power plant. However, neural networks face issues of becoming overly dependent on just a few variables to make predictions, which negatively impacts control decisions that rely on the model to determine the value of all manipulated variables. The paper introduces regularization techniques, including noise injection and input gradient regularization, during the training phase. Here, the work presents novel contributions in adapting neural networks to control industrial systems and applying regularization techniques from computer vision to industrial process control. Results demonstrate the effectiveness of input gradient regularization in reducing model dependence on subsets of variables, emphasizing the balance between fidelity and controllability. Further exploration is recommended, including the development of recurrent transformers, closed-loop control testing, and a sensitivity analysis on computer models to provide further insight.
Intelligent Experiments Through Real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and Future EIC Detectors (Final Report)
The overall vision of this project was to integrate real-time artificial intelligence (AI) directly into the data acquisition and detector-control systems of nuclear physics experiments, including both fast online event selection and an autonomous detector-control feedback loop. The work carried out under the award focused on the fast online event-selection half of that vision: the efficient recording of low-momentum heavy-flavor (HF) hadron decays in proton-proton collisions at the sPHENIX experiment at the Relativistic Heavy Ion Collider (RHIC)—an observable that requires fast tracking and topological trigger selection not previously demonstrated at RHIC, and that is essential for QCD studies at future facilities such as the Electron-Ion Collider (EIC). The autonomous detector-control (GPU-based feedback) component named in the project title remained a design concept and was not implemented under this award. The Massachusetts Institute of Technology (MIT) group led the offline simulation and data processing needed to train the machine-learning (ML) models, the translation of trained models to Field-Programmable Gate Array (FPGA) firmware using the hls4ml framework, and the physics validation of heavy-flavor reconstruction. Over the award period, the team developed and hardware-tested the principal components of an AI-based heavy-flavor trigger on simulated and recorded sPHENIX tracker data: a software Bipartite Graph Attention Network (BiGAT) trigger model reaching > 95% signal efficiency at 99% background rejection; an FPGA-native hit clusterizer matching the offline clustering; smaller networks synthesized to FPGA within the required sub-10 µs latency; and an assembled decoder–clusterizer–inference firmware chain exercised on the FELIX readout board. A complete, fully integrated hardware demonstrator was not finished within the award period. This report documents the project goals, the MIT group’s contributions, the technical accomplishments, and the outlook toward applications at the future EIC ePIC detector.
Re-Design and Beat Testing of the Man-Machine Integration Design and Analysis System: MIDAS
The Man-machine Design and Analysis System (MIDAS) is a human factors design and analysis system that combines human cognitive models with 3D CAD models and rapid prototyping and simulation techniques. MIDAS allows designers to ask 'what if' types of questions early in concept exploration and development prior to actual hardware development. The system outputs predictions of operator workload, situational awareness and system performance as well as graphical visualization of the cockpit designs interacting with models of the human in a mission scenario. Recently, MIDAS was re-designed to enhance functionality and usability. The goals driving the redesign include more efficient processing, GUI interface, advances in the memory structures, implementation of external vision models and audition. These changes were detailed in an earlier paper. Two Beta test sites with diverse applications have been chosen. One Beta test site is investigating the development of a new airframe and its interaction with the air traffic management system. The second Beta test effort will investigate 3D auditory cueing in conjunction with traditional visual cueing strategies including panel-mounted and heads-up displays. The progress and lessons learned on each of these projects will be discussed.
Electric Grid Security (EGS) FY24 Annual Report
Sandia’s Electric Grid Security program advances a national vision of a secure, resilient, and affordable electric system for all users. Our achievements reflect a strategic approach combining technology development; modeling, simulation, and data analytics; and partnered demonstrations and outreach to further the adoption of advanced grid and storage technologies. Our FY24 efforts leverage the strengths of our partnerships—spanning Sandia’s core science and technology competencies as well as external technology leaders—to develop the solutions today which enable the grid of tomorrow. Key accomplishments in this report that support our strategy span our technical program areas and include: • The advancement of energy storage technologies, including creation of a national Long Duration Energy Storage Consortium; • Applications of artificial intelligence and machine learning to enhanced grid operations and planning; • Development of solid-state power conversion technologies and a new medium-voltage research lab; • New technologies to assess wildfire vulnerabilities and mitigate potential impacts; • Advanced applications of new cybersecurity technologies with industry partners; • Contributions to understanding the impacts of electromagnetic pulses and geomagnetic disturbances on grid components; and • Digital twin development for hybrid microgrids with multiple generators, storage, and loads.
Missileborne Artificial Vision System (MAVIS)
Several years ago when INTEL and China Lake designed the ETANN chip, analog VLSI appeared to be the only way to do high density neural computing. In the last five years, however, digital parallel processing chips capable of performing neural computation functions have evolved to the point of rough equality with analog chips in system level computational density. The Naval Air Warfare Center, China Lake, has developed a real time, hardware and software system designed to implement and evaluate biologically inspired retinal and cortical models. The hardware is based on the Adaptive Solutions Inc. massively parallel CNAPS system COHO boards. Each COHO board is a standard size 6U VME card featuring 256 fixed point, RISC processors running at 20 MHz in a SIMD configuration. Each COHO board has a companion board built to support a real time VSB interface to an imaging seeker, a NTSC camera, and to other COHO boards. The system is designed to have multiple SIMD machines each performing different corticomorphic functions. The system level software has been developed which allows a high level description of corticomorphic structures to be translated into the native microcode of the CNAPS chips. Corticomorphic structures are those neural structures with a form similar to that of the retina, the lateral geniculate nucleus, or the visual cortex. This real time hardware system is designed to be shrunk into a volume compatible with air launched tactical missiles. Initial versions of the software and hardware have been completed and are in the early stages of integration with a missile seeker.
Accelerating the rate of discovery: toward high-repetition-rate HED science
As high-intensity short-pulse lasers that can operate at high-repetition-rate (HRR) (>10 H z ) come online around the world, the high energy density (HED) science they enable will experience a radical paradigm shift. The >10 3 increase in shot rate over today's shot-per-hour drivers translates into dramatically faster data acquisition, more experiments, and the ability to exploit machine learning, and thus the potential to significantly accelerate the advancement of HED science. A wide range of HED experiments, from opacity investigations to secondary source generation to plasma nuclear physics, will benefit from the increased statistics, precision, and exploration of phase space. Besides increasing the rate at which scientific experiments can be performed, HRR also allows for the rapid delivery of optimal experiments supported by simulations and modeling augmented by close coupling to empirical data. To fully realize such an HRR framework, numerous subsystems must be developed and brought together, including feedback laser control loops, high-throughput targetry and diagnostics, cognitive simulation, enhanced HED codes, and advanced data analytics. This paper describes the vision for an integrated HRR laser experimental HED system and outlines some of the major considerations and challenges for realizing it.
AI foundation models for experimental fusion tasks
Artificial Intelligence (AI) foundation models, while successful in various domains of language, speech, and vision, have not been adopted in production for fusion energy experiments. This brief paper presents how AI foundation models can be used for fusion energy diagnostics, enabling, for example, visual automated logbooks to provide greater insights into chains of plasma events in a discharge, in time for between-shot analysis.
Data associated with “Different methods of estimating riverbed sediment grain size diverge at the basin scale ” (v2)
This data package is associated with the publication “Different methods of estimating riverbed sediment grain size diverge at the basin scale” published in Frontiers in Earth Science (Regier et al., 2025). The distribution of sediment grain size in streams and rivers is often quantified by the median grain size (d50), a key metric for understanding and predicting hydrologic and biogeochemical function of streams and rivers. Manual methods to measure d50 are time-consuming and ignore larger grains, while model-based methods to estimate d50 often over-generalize basin characteristics, and therefore cannot accurately represent site-scale heterogeneity. Here, we apply a machine learning-enabled photogrammetry methodology (You Only Look Once, or YOLO) for estimating d50 for grains > 2 mm based on images collected from streams and rivers throughout the Yakima River Basin (YRB). To understand how such methods may help bridge the gaps in resolution and accuracy between manual and catchment characteristics model-based d50 estimates, we compared YOLO d50 values to manual and model-based estimates across the YRB. We found distinct differences among methods for d50 averages and variability, and relationships between d50 estimates and basin characteristics. Source images can be found at https://data.ess-dive.lbl.gov/view/doi:10.15485/1892052. This data package was originally published in May 2023. It was updated August 2025 (v2; new and modified files). File and folder names were not revised to indicate changes. See the change history section in the readme for more details. In addition to the readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. This dataset is comprised of one main data folder containing (1) file-level metadata; (2) data dictionary; (3) readme; (4) and subfolders containing data, figures, and scripts. The data folder contains datasets used for the analyses in the manuscript in image, text-delimited or geospatially-referenced formats. The figures folder contains the figures from the manuscript in different formats. The scripts folder contains all of the scripts used to complete the analyses in the manuscript. All files are .csv, .rds, .dbf, .prj, .shp, .shx, .jpg, .png, .R, .Rproj, or .pdf. We acknowledge the Yakama Nation as owners and caretakers of the lands where we collected some of these data. We thank the Confederated Tribes and Bands of the Yakama Nation Tribal Council and Yakama Nation Fisheries for working with us to facilitate sample collection and optimization of data usage according to their values and worldview.
An overview of visualization and visual analytics applications in water resources management
Recent advances in information, communication, and environmental monitoring technologies have increased the availability, spatiotemporal resolution, and quality of water-related data, thereby leading to the emergence of many innovative big data applications. Among these applications, visualization and visual analytics, also known as the visual computing techniques, empower the synergy of computational methods (e.g., machine learning and statistical models) with human reasoning to improve the understanding and solution toward complex science and engineering problems. These approaches are frequently integrated with geographic information systems and cyberinfrastructure to provide new opportunities and methods for enhancing water resources management. Here, we present a comprehensive review of recent hydroinformatics applications that employ visual computing techniques to (1) support complex data-driven research problems, and (2) support the communication and decision-makings in the water resources management sector. Then, we conduct a technical review of the state-of-the-art web-based visualization technologies and libraries to share our experiences on developing shareable, adaptive, and interactive visualizations and visual interfaces for water resources management applications. We close with a vision that applies the emerging visual computing technologies and paradigms to develop the next generation of hydroinformatics applications.
Demonstrating Advanced Sensors for In-Situ Monitoring Towards Qualification of Nuclear Relevant Components
The U.S. Department of Energy’s Office of Nuclear Energy Advanced Materials and Manufacturing Technologies (AMMT) program is pursuing qualification of laser powder bed fusion (LPBF) components for nuclear applications. A major focus of this effort is the use of in situ process monitoring and machine learning–based tools to establish real-time quality assurance. The primary objective of this report is to identify and evaluate the most relevant in situ sensor systems for LPBF, and to document the deployment of these systems across platforms critical to the AMMT program. This work demonstrates how in situ monitoring can detect process anomalies, track geometry-dependent flaws, and identify limiting combinations of processing parameters—particularly those related to energy density and complex geometries (e.g., overhanging structures). To support this goal, a diverse suite of sensor modalities was evaluated across LPBF platforms, including visible and near-infrared (NIR) imaging, fringe projection profilometry, long-wavelength infrared (LWIR) thermography, and high-speed photodiode/pyrometry systems. These sensor streams were integrated with Peregrine, a machine-agnostic software platform that, among other capabilities, can generate real-time process anomaly classification. This report documents sensor deployments on multiple AMMT flagship platforms, including the Concept Laser M2 and Renishaw AM400/AM250 systems. Calibration builds with complex, flaw-prone geometries such as unsupported overhangs, stepped features, and thin walls, were used to evaluate how well Peregrine and its associated sensors could detect process anomalies and other instabilities under varied energy densities. It will be shown how Peregrine reliably identifies common process anomalies such as recoater streaking, superelevation, etc., and can be used in post-build analysis for anomaly spatial distributions throughout the build height to better understand the impact of geometry and processing parameter choice on the build. This work demonstrates measurable progress toward the vision that components can be born-qualified by establishing a real-time monitoring framework, identifying limiting process conditions, and laying the foundation for sensor fusion–enabled prediction pipelines that are scalable across platforms and applicable to nuclear-relevant components.
FY25 Electric Grid Security Annual Report
Sandia’s Electric Grid Security program advances a national vision of energy dominance and accessibility, while applying our national security -emphasis on ensuring of a secure, resilient, and affordable electric system for all users. Our achievements reflect a strategic approach combining technology development; modeling, simulation, and data analytics; and partnered demonstrations and outreach to further the adoption of advanced grid and storage technologies. Our FY25 efforts leverage the strengths of our partnerships—spanning Sandia’s core science and technology competencies as well as external technology leaders—to develop the solutions today which enable the grid of tomorrow. Key accomplishments in this report that support our strategy span our technical program areas and include: • New open-source analytical tools for systems -level planning and optimization, including significant advances to the QuESt analytical environment; • Further advancement of artificial intelligence and machine learning to enhanced grid operations and planning as we rise to the challenge of new large loads; • Development of solid-state power conversion technologies and a new medium-voltage research lab; • New technologies to assess wildfire vulnerabilities and mitigate potential impacts; • Advanced applications of new cybersecurity technologies with industry partners; • Contributions to understanding the impacts of electromagnetic pulses and geomagnetic disturbances on grid components; and • Digital twin development for hybrid microgrids with multiple generators, storage, and loads. This report indicates key areas of research and engagement and summarizes the impact of Sandia’s contributions through notable accomplishments, journal publications, patents, and technical conferences and presentations. It is provided with the hope that readers discover ways we can further team to create our modern grid and apply the outcomes of our efforts. The bulk of work described herein is funded by several offices within the U.S. Department of Energy (USDOE), including the Office of Electricity (OE); Cybersecurity, Energy Security, and Emergency Response (CESER); former offices such as the Office of Energy Efficiency and Renewable Energy (EERE), the Grid Deployment Office (GDO), the Office of Clean Energy Demonstrations (OCED), and other key programs at USDOE. As we continue to state in these annual reports, the contributors to our successes are too numerous to name here, though our team wishes to express our deep gratitude to the numerous program and project sponsors at the US Department of Energy, who often function equally as technical collaborators; our many partners in industry, academia, utilities, and other national labs; and fellow researchers and business partners at Sandia whose leadership and creativity have enabled the accomplishments described herein.
Convolutional neural networks for intra-hour solar forecasting based on sky image sequences
Accurate and timely solar forecasts play an increasingly critical role in power systems. Compared to longer forecasting timescales, very short-term solar forecasting has lagged behind in both research and practice. In this paper, we propose deep convolutional neural networks (CNNs) to provide operational intra-hour (10-minute-ahead to 60-minute-ahead) solar forecasts. We develop two CNN structures inspired by a widely-used CNN architecture. The CNNs are tailored to our solar forecasting regression tasks and rely solely on sky image sequences. Case studies based on six years of data (over 150,000 data points) demonstrate that the best CNN model has forecast skill scores of 20%-39% over the naive persistence of cloudiness benchmark, even at these very short timescales. The CNNs also have consistently superior performance when compared to shallow machine learning models with meteorological predictors, where the improvement averages around 7%. The sensitivity analyses show that the sky image length, resolution, and weather conditions have impacts on the deep learning model accuracy. In our intra-hour problem with specific setups, two sky images with a 10-minute 128 x 128 resolution yield the most accurate forecasts. Current limitations, future work, and deployment challenges and solutions are also discussed.
A Knowledge Graph Framework for Organizing Heterogeneous Datasets for Utilization in Classical and Quantum Computing: Current Challenges and Future Directions
"The escalating impact of climate change induced extreme weather events in urban, suburban, and rural environments demands a rethink of how we have been using the single event-based or use-case-based knowledge graph models. The lack of representation in interaction within environmental variables found in literature led to the development of a novel framework that reflects the true nature of the interconnectedness in our environment. We propose an Environmental Interaction Knowledge Graph (EIKG) framework. This general EIKG framework works as the basis for interconnected environmental events by knitting interrelated events such as hurricanes leading to storm surges, which lead to flood events that could cause mudslides, landslides, etc., The cascading nature of one event leading to another related event in the environment requires an adequate understanding of each event using contextual information before conducting any data-driven analytics. This vision paper showcases how the EIKG:floods, EIKG:wildfire EIKG:landslides, etc, can be derived from a base case framework of EIKG as those individual events are interconnected with some common denominator variables. As an example, the precipitation variable is used in the flood case study as well as in the wildfire case study, as excessive precipitation levels lead to floods, and lack of precipitation leads to droughts and wildfires. We identify the precipitation variable as a “common-denominator-variable” in extreme weather events that play a key role in modeling the environment leading to different extreme weather events based on the variability of that variable (varying values where low precipitation leads to drought, and high values lead to floods). We use the insights gained from EIKG to conduct classical and Quantum Machine Learning (QML) based data analysis on the research questions developed. Our preliminary study shows how the Variational Quantum Classifier (VQC) and Quantum Support Vector Classifier (QSVC) are used along with the classical machine learning models to compare the model accuracies. Our study elaborates on how a quantitative analysis uses state-of-the-art machine learning techniques that include implementing both classical and quantum machine learning models and developing the knowledge graph. The EIKG is used to organize heterogeneous datasets and integrate the relations to case-specific extreme weather events such as floods. The study uses datasets such as county-to-country residential mobility data, socioeconomic datasets from the US Census Bureau, climate and weather-related Earth Observational data from NASA, and critical infrastructure data from the Homeland Infrastructure datasets."