Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Embedded computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 919 records · Page 51

Dynamic cellular finite-element method for modelling large-scale cell migration and proliferation under the control of mechanical and biochemical cues: a study of re-epithelialization

Computational modelling of cells can reveal insight into the mechanisms of the important processes of tissue development. However, current cell models have limitations and are challenged to model detailed changes in cellular shapes and physical mechanics when thousands of migrating and interacting cells need to be modelled. In this paper we describe a novel dynamic cellular finite-element model (DyCelFEM), which accounts for changes in cellular shapes and mechanics. It also models the full range of cell motion, from movements of individual cells to collective cell migrations. The transmission of mechanical forces regulated by intercellular adhesions and their ruptures are also accounted for. Intra-cellular protein signalling networks controlling cell behaviours are embedded in individual cells. We employ DyCelFEM to examine specific effects of biochemical and mechanical cues in regulating cell migration and proliferation, and in controlling tissue patterning using a simplified re-epithelialization model of wound tissue. Our results suggest that biochemical cues are better at guiding cell migration with improved directionality and persistence, while mechanical cues are better at coordinating collective cell migration. Finally and overall, DyCelFEM can be used to study developmental processes when a large population of migrating cells under mechanical and biochemical controls experience complex changes in cell shapes and mechanics.

59 BASIC BIOLOGICAL SCIENCES↗

A robust approach to 3D neuron shape representation for quantification and classification

We consider the problem of finding an accurate representation of neuron shapes, extracting sub-cellular features, and classifying neurons based on neuron shapes. In neuroscience research, the skeleton representation is often used as a compact and abstract representation of neuron shapes. However, existing methods are limited to getting and analyzing “curve” skeletons which can only be applied for tubular shapes. This paper presents a 3D neuron morphology analysis method for more general and complex neuron shapes. First, we introduce the concept of skeleton mesh to represent general neuron shapes and propose a novel method for computing mesh representations from 3D surface point clouds. A skeleton graph is then obtained from skeleton mesh and is used to extract sub-cellular features. Finally, an unsupervised learning method is used to embed the skeleton graph for neuron classification. Extensive experiment results are provided and demonstrate the robustness of our method to analyze neuron morphology.

3D neuron morphology↗

ECP Milestone Deliverable Memo 2.3.1.04.15

The DARMA many-task framework provides asynchronous communication and load balancing functionality. This functionality is embedded in standard , modern C++ through the use of the template wrapper classes similar to futures. DARMA codes previously could not interoperate with MPI. A new C++ interface and extended semantics now allow quiescence of DARMA kernels and transfer of data ownership back into MPI, allowing 1) isolated DARMA kernels to be inserted in a larger MPI code or 2) reuse of MPI libraries like solvers within a DARMA application.

97 MATHEMATICS AND COMPUTING↗

Describing Point Defect Topology in 2D Energy Materials Through Computer Vision

Point defects such as vacancies and impurity atoms strongly impact the performance of 2D materials. Traditional efforts often rely on manual detection, a process that is time-intensive, prone to human error, and challenging to scale. Here we leverage machine learning (ML) methods to identify and quantify vacancies within 2D transition metal carbides (Ti3C2, MXenes), aiming to expedite detection while improving accuracy. MXenes exhibit valuable defect-defined electrochemical properties, but we currently lack statistical understanding of defect topology needed to fully harness these materials. Here we employ a convolutional neural network for semantic segmentation of experimental MXene images, opening an opportunity to conduct a rigorous statistical study on defect hierarchy while investigating local relaxation in the lattice. We show how the integration of ML can yield fundamental insight into point defects, providing a powerful tool that will play an increasingly crucial role in the future of materials science. ML is often not just a matter of straightforward application, and pretrained models proved ineffective in this case. Instead, we trained our own neural network (NN) and applied data augmentation techniques and fine-tuning to the training dataset. Since labeled microscopy data is often scarce, we developed training data from a previously published wide-frame MXene image, using customized Gaussian fitting to locate atomic positions. Our trained model was then applied to a large dataset of experimental images, enabling a statistical study of defect configurations across three samples prepared with different HF etchant concentrations (5%, 9.1%, and 12.5%), as shown in Fig. 1. This also allowed us to investigate local strain around vacancies, though we find that we are limited by the precision of measurements using high-angle annular dark field (HAADF) images, as shown in Fig. 2. This study demonstrates how ML enables large-scale, quantitative analysis of atomic defects - an otherwise infeasible task with traditional methods. While our NN was specialized for Ti3C2 MXenes, the pipeline we developed provides a foundation for future ML models tailored to other materials. Ultimately, we envision embedding the NN onto the microscope to give real-time feedback to the user. To make this a reality, continued work is necessary to fully understand the NN's capabilities and limitations. This study gets one step closer to our goals of automated experimentation moving away from traditional methods of manual labeling. As ML capabilities advance, we hope to continue adapting and applying these techniques in microscopy.

2D materials↗

Modeling design and control problems involving neural network surrogates

Here, we consider nonlinear optimization problems that involve surrogate models represented by neural networks. We demonstrate first how to directly embed neural network evaluation into optimization models, highlight a difficulty with this approach that can prevent convergence, and then characterize stationarity of such models. We then present two alternative formulations of these problems in the specific case of feedforward neural networks with ReLU activation: as a mixed-integer optimization problem and as a mathematical program with complementarity constraints. For the latter formulation we prove that stationarity at a point for this problem corresponds to stationarity of the embedded formulation. Each of these formulations may be solved with state-of-the-art optimization methods, and we show how to obtain good initial feasible solutions for these methods. We compare our formulations on three practical applications arising in the design and control of combustion engines, in the generation of adversarial attacks on classifier networks, and in the determination of optimal flows in an oil well network.

97 MATHEMATICS AND COMPUTING↗

Design of a Combined Proportional Integral Derivative Controller to Regulate the Temperature Inside a High-Temperature Tubular Solar Reactor

Solar fuels are proven to be promising candidates for thermochemical energy storage. However, the transient nature of solar radiation is an obstacle to maintaining a stable operational temperature inside a solar reactor. To overcome this challenge, the temperature of a solar reactor can be regulated by controlling the incoming solar radiation or the feedstock flowrate inside the reactor. In this work, a combined proportional integral derivative (PID) controller is implemented to regulate the temperature inside a high-temperature tubular solar reactor with counter-current flowing gas/particles. The control model incorporates two control systems to regulate incoming solar radiation and gas flow simultaneously. The design of the controller is based on a reduced-order numerical model of a high-temperature tubular solar reactor that is vertically oriented with an upward gas flow and downward particle flow. The reactor receives heat circumferentially through its wall over a finite segment of its length. Formulation of the heat transfer model is presented by applying the energy balance for the reactor tube and considering heat and mass transfer inside. A set of governing differential equations are solved numerically by using the finite volume method to obtain reactor wall, particles, and gas temperatures along the reactor length with various boundary conditions. Simulation results are used to tune the PID controller parameters by utilizing the Ziegler–Nichols tuning method. Both the simulation results and the controller performance are visualized on the LABVIEW platform. The controller is challenged to track different temperature setpoints with different scenarios of transient solar radiation. Here, the performance of the PID controller was compared to experimental results obtained from an industrial PID controller embedded in a 7 kW electric furnace. Results show that the combined PID controller is successful in maintaining a stable temperature inside the reactor by regulating the incoming solar radiation and the flowrate via small steady-state error and reasonable settling time and overshoot.

14 SOLAR ENERGY↗

Divisible Rigid Groups. Algebraic Closedness and Elementary Theory

A group G is said to be rigid if it contains a normal series G = G{sub 1} > G{sub 2} > … > Gm > G{sub m+1} = 1, whose quotients G{sub i}/G{sub i+1} are Abelian and, treated as right ℤ[G/G{sub i}]-modules, are torsion-free. A rigid group G is divisible if elements of the quotient G{sub i}/G{sub i+1} are divisible by nonzero elements of the ring ℤ[G/G{sub i}]. Every rigid group is embedded in a divisible one. We prove two theorems. Theorem 1 says that the following three conditions for a group G are equivalent: G is algebraically closed in the class Σ{sub m} of all m-rigid groups; G is existentially closed in the class Σ{sub m}; G is a divisible m-rigid group. Theorem 2 states that the elementary theory of a class of divisible m-rigid groups is complete.

97 MATHEMATICS AND COMPUTING↗

A multiscale multi-permeability poroplasticity model linked by recursive homogenizations and deep learning

Many geological materials, such as shale, mudstone, carbonate rock, limestone and rock salt are multi-porosity porous media in which pores of different scales may co-exist in the host matrix. When fractures propagate in these multi-porosity materials, these pores may enlarge and coalesce and therefore change the magnitude and the principal directions of the effective permeability tensors. The pore-fluid inside the cracks and the pores of host matrix may interact and exchange fluid mass, but the difference in hydraulic properties of these pores often means that a single homogenized effective permeability tensor field is insufficient to characterize the evolving hydraulic properties of these materials at smaller time scale. Furthermore, the complexity of the hydro-mechanical coupling process and the induced mechanical and hydraulic anisotropy originated from the micro-fracture and plasticity at grain scale also makes it difficult to propose, implement and validate separated macroscopic constitutive laws for numerical simulations. Here this article presents a hybrid data-driven method designed to capture the multiscale hydro-mechanical coupling effect of porous media with pores of various different sizes. At each scale, data-driven models generated from supervised machine learning are hybridized with classical constitutive laws in a directed graph that represents the numerical models. By using sub-scale simulations to generate database to train material models, an offline homogenization procedure is used to replace the up-scaling procedure to generate cohesive laws for localized physical discontinuities at both grain and specimen scales. Through a proper homogenization procedure that preserves spatial length scales, the proposed method enables field-scale simulations to gather insights from meso-scale and grain-scale micro-structural attributes. This method is proven to be much more computationally efficient than the classical DEM–FEM or FEM2 approach while at the same time more robust and flexible than the classical surrogate modeling approach. Due to the usage of bridging-scale technique, the proposed model may provide multiple opportunities to incorporate different types of simulations and experimental data across different length scales for machine learning. Numerical issues will also be discussed.

42 ENGINEERING↗

Towards physics-informed explainable machine learning and causal models for materials research

From emergent material descriptions to estimation of properties stemming from structures to optimization of process parameters for achieving best performance – all key facets of materials science and related fields have experienced tremendous growth with the introduction of data-driven models. This gradual progression goes at par with developments of machine learning workflows, from purely data-driven shallow models to those that are well-capable in encoding more complex graphs, symbolic representations, invariances, and positional embeddings. Furthermore, this perspective aims at summarizing strategic aspects of such transitions while providing insights into the requirements of bringing in explainable, interpretable predictive models, and causal learning to aid in materials design and discovery. Although the focus remains on a variety of functional materials by providing a handful of case studies, the applications of such integrated methodologies are universal to facilitate fundamental understandings of materials physics while enabling autonomous experiments.

36 MATERIALS SCIENCE↗

A biology-informed similarity metric for simulated patches of human cell membrane

Complex scientific inquiries rely increasingly upon large and autonomous multiscale simulation campaigns, which fundamentally require similarity metrics to quantify ‘sufficient’ changes among data and/or configurations. However, subject matter experts are often unable to articulate similarity precisely or in terms of well-formulated definitions, especially when new hypotheses are to be explored, making it challenging to design a meaningful metric. Furthermore, the key to practical usefulness of such metrics to enable autonomous simulations lies in in situ inference, which requires generalization to possibly substantial distributional shifts in unseen, future data. Here, we address these challenges in a cancer biology application and develop a meaningful similarity metric for ‘patches’—regions of simulated human cell membrane that express interactions between certain proteins of interest and relevant lipids. In the absence of well-defined conditions for similarity, we leverage several biology-informed notions about data and the underlying simulations to impose inductive biases on our metric learning framework, resulting in a suitable similarity metric that also generalizes well to significant distributional shifts encountered during the deployment. We combine these intuitions to organize the learned embedding space in a multiscale manner, which makes the metric robust to incomplete and even contradictory intuitions. Our approach delivers a metric that not only performs well on the conditions used for its development and other relevant criteria, but also learns key spatiotemporal relationships without ever being exposed to any such information during training.

97 MATHEMATICS AND COMPUTING↗

Beyond interpolation: Physics-inspired gating transformers for extrapolating irradiation conditions to novel nuclear fuels

The qualification of advanced nuclear fuels relies on irradiation experiments in test reactors that emulate commercial conditions. Designing these tests requires accurate prediction of key irradiation quantities, particularly heat generation rate and burnup, yet obtaining them typically involves computationally expensive multi-step simulation workflows. We propose a physics-inspired gating transformer (PIGT) that integrates an inverse-square, distance-based attenuation into the encoder representation to bias attention toward physically relevant spatial relationships while retaining data-driven flexibility. Using MiniFuel irradiation data from the High Flux Isotope Reactor at Oak Ridge National Laboratory, we benchmark against ensemble methods, feedforward and recurrent networks, convolutional models, and standard transformers. While baseline models perform well under interpolation, they exhibit a pronounced generalization gap when evaluated on fuels not included in the training set. The proposed model consistently improves extrapolative accuracy and stability, yielding the strongest performance on unseen fuel configurations. These results indicate that a lightweight physics structure embedded within attention mechanisms can substantially improve robustness, enabling more reliable surrogate predictions to accelerate the design of nuclear fuel irradiation experiments.

Fuel qualification↗

U-Surf: a global 1 km spatially continuous urban surface property dataset for kilometer-scale urban-resolving Earth system modeling

High-resolution urban climate modeling has faced substantial challenges due to the absence of a globally consistent, spatially continuous, and accurate dataset to represent the spatial heterogeneity of urban surfaces and their biophysical properties. This deficiency has long obstructed the development of urban-resolving Earth system models (ESMs) and ultra-high-resolution urban climate modeling, over large domains. Here, we present U-Surf, a first-of-its-kind 1 km resolution present-day (circa 2020) global continuous urban surface parameter dataset. Using the urban canopy model (UCM) in the Community Earth System Model as a base model for satisfying dataset requirements, U-Surf leverages the latest advances in remote sensing, machine learning, and cloud computing to provide the most relevant urban surface biophysical parameters, including radiative, morphological, and thermal properties, for UCMs at the facet and canopy level. Generated using a systematically unified workflow, U-Surf ensures internal consistency among key parameters, making it the first globally coherent urban canopy surface dataset. U-Surf significantly improves the representation of the urban land heterogeneity both within and across cities globally; provides essential, high-fidelity surface biophysical constraints to urban-resolving ESMs; enables detailed city-to-city comparisons across the globe; and supports next-generation kilometer-resolution Earth system modeling across scales. U-Surf parameters can be easily converted or adapted to various types of UCMs, such as those embedded in weather and regional climate models, as well as air quality models. The fundamental urban surface constraints provided by U-Surf can also be used as features for machine learning models and can have other broad-scale applications for socioeconomic, public health, and urban planning contexts. We expect U-Surf to advance the research frontier of urban system science, climate-sensitive urban design, and coupled human–Earth systems in the future. The dataset is publicly available at https://doi.org/10.5281/zenodo.11247598 (Cheng et al., 2024).

Cheng, Yifan [Univ. of Illinois at Urbana-Champaig↗

Structural changes and variability of the ITCZ induced by radiation–cloud–convection–circulation interactions: inferences from the Goddard Multi-scale Modeling Framework (GMMF) experiments

In this paper, we have investigated the impact of radiation–cloud–convection–circulation interaction (RC3I) on structural changes and variability of the Inter-tropical Convergence Zone (ITCZ) using the Goddard Multi-scale Modeling Framework, where cloud processes are super-parameterized, i.e., explicitly resolved with 2-D cloud resolving models embedded in each coarse grid of the host Goddard Earth Observing System-Version 5 global climate model. Experiments have been conducted under prescribed sea surface temperature conditions for 10 years (2007–2016), with and without cloud radiation feedback in the atmosphere, respectively. Diagnostic analyses separately for January and July show that RC3I leads to an enhanced and expanded Hadley Circulation characterized by (1) a quasi-uniform warming and moistening of the tropical atmosphere and a sharpening of the ITCZ with enhanced deep convection, more intense precipitation and higher clouds, (2) extended drying of the tropical marginal convective zones, and extratropical mid- to lower troposphere, and (3) a cooling of the polar regions, with increased baroclinicity and midlatitude storm track activities. Computations based on the zonal mean thermodynamic energy balance equation show that the radiative warming and cooling are strongly balanced by local adiabatic processes associated with changes in large-scale vertical motions, as well as horizontal atmospheric heat transport. In the tropics, enhanced short-wave absorption and longwave water vapor greenhouse effects by high clouds play key roles in providing strong positive feedback to the tropospheric warming. In the extratropics, increased atmospheric heat transport associated with changes in the Hadley circulation is balanced by strong longwave cooling above, and warming below due to increased high clouds. We also find a strong positive correlation between daily and pentad heavy rain in the ITCZ core, and expansion of the drier zones coupled to a contraction of the highly convective zones in the ITCZ, indicating a strong tendency RC3I-induced convective aggregation in tropical clouds i.e., wet-regions-get-wetter and contracted, and dry-areas-get-drier and expanded.

54 ENVIRONMENTAL SCIENCES↗

Adaptive Internal Model Based Current Control with Embedded Active Damping of a Three-Phase Grid-Connected Inverter with LCL Filter for PV Application

This paper presents an internal model based controller with embedded active damping for a three-phase grid-connected inverter with LCL filter. A strategy to control the grid current based on the internal model principle is presented in this paper. It is known that a third-order LCL filter apart from providing better attenuation also poses the problem of initiating unwanted resonance oscillations especially during transients. To address this problem, an active damping strategy based on virtual resistance emulation on the grid side filter is implemented. Also, to ensure effective active damping, an online parameter update strategy is implemented for the filter parameters. The proposed method provides better attenuation to oscillation damping and achieves good tracking performance without adversely affecting the bandwidth. The architecture is verified based on computer simulations via MATLAB/Simulink and PLECS domain and various important case study results are presented.

grid-connected inverter↗

Let’s Unleash the Network Judgment: A Self-Supervised Approach for Cloud Image Analysis

Accurate cloud type identification and coverage analysis are crucial in understanding the Earth’s radiative budget. Traditional computer vision methods rely on low-level visual features of clouds for estimating cloud coverage or sky conditions. Several handcrafted approaches have been proposed; however, scope for improvement still exists. Newer deep neural networks (DNNs) have demonstrated superior performance for cloud segmentation and categorization. These methods, however, need expert engineering intervention in the preprocessing steps—in the traditional methods—or human assistance in assigning cloud or clear sky labels to a pixel for training DNNs. Such human mediation imposes considerable time and labor costs. We present the application of a new self-supervised learning approach to autonomously extract relevant features from sky images captured by ground-based cameras, for the classification and segmentation of clouds. We evaluate a joint embedding architecture that uses self-knowledge distillation plus regularization. We use two datasets to demonstrate the network’s ability to classify and segment sky images—one with ~85,000 images collected from our ground-based camera and another with 400 labeled images from the WSISEG database. We find that this approach can discriminate full-sky images based on cloud coverage, diurnal variation, and cloud base height. Additionally, it semantically segments the cloud areas without labels. The approach shows competitive performance in all tested tasks, suggesting a new alternative for cloud characterization.

54 ENVIRONMENTAL SCIENCES↗

Convergent Manufacturing of Large-Scale Components for Nuclear Applications, via Additive Manufacturing and Powder Metallurgy Hot Isostatic Pressing

Powder metallurgy (PM)–hot isostatic pressing (PM-HIP) has long been recognized as a powerful route for producing fully dense, near net shape metallic components. By consolidating powders under high temperature and pressure, HIP provides isotropic properties, uniform microstructures, and scalability to complex geometries that are vital for sectors such as aerospace, energy, and nuclear power. Yet despite these advantages, the technology has remained constrained by costly trial and error canister fabrication, limitations of conventional forging, and incomplete knowledge about how the canister design influences final part properties. Additive manufacturing (AM), by contrast, thrives on design freedom and geometric flexibility but struggles with speed, scalability, and cost when applied to very large structures. The research presented in this report investigated how a convergent manufacturing approach, combining AM with PM-HIP, can merge the strengths of both technologies, leveraging AM’s flexibility for canister design and HIP’s consolidation capability to deliver reliable, large, and complex parts. The work progressed through three case studies that built on one another in scale and complexity. Small cylindrical canisters fabricated by conventional methods, laser powder bed fusion, and directed energy deposition were filled with stainless steel powders and subjected to HIP. The resulting parts demonstrated near-full density and mechanical properties on par with wrought stainless steel, showing for the first time that AM canisters can be a direct substitute for conventional ones without sacrificing quality. The next step involved a medium-scale, noncentrosymmetric T-valve, which is an enclosed, multibranch geometry that tested the limits of AM + PM-HIP integration. The T-valve achieved predictable shrinkage and uniform densification, confirming feasibility for enclosed designs. However, this study also revealed oxide inclusions and interfacial challenges at the AM + HIP boundary, underscoring the critical importance of controlling interface chemistry and employing robust, in situ strategies, such as melt pool monitoring and thermal monitoring, coupled with nondestructive evaluation techniques such as x-ray computed tomography. Finally, the effort culminated in fabricating a large-scale impeller weighing nearly 2000 lb and spanning 5 ft in diameter. Produced via multirobot wire arc AM and hot isostatic pressed to near-full density, the impeller validated industrial-scale feasibility. Predictive models closely matched experimental shrinkage, tensile properties were spatially uniform across the component, and the AM + PM-HIP interface proved mechanically sound despite the presence of oxide-decorated prior particle boundaries. This large-scale demonstration is a major milestone, showing that hybrid AM + PM‑HIP can reliably deliver components at reactor-relevant scales. Collectively, these studies charted a logical pathway: small-scale work built scientific confidence, medium-scale work highlighted opportunities and challenges, and large-scale work proved industrial impact. The overarching conclusion of this report is that AM + PM-HIP should not be seen as a replacement for forging but as a complementary pathway that provides the US with flexibility, resilience, and new options for manufacturing nuclear-grade components. Looking ahead, several directions emerge as critical to sustaining progress. Predictive modeling must become faster, more accessible, and more accurate, with digital twins and machine learning reducing reliance on trial and error. Powders and alloys must be optimized for HIP, with improved cleanliness, reduced oxides, and tailored chemistries that enhance creep, fatigue, and irradiation resistance. Interfaces between AM and HIP regions must be better engineered through coatings, machining strategies, and surface treatments to mitigate oxide formation and ensure reliable bonding to explore opportunities for HIP of targeted compositional parts, as well as multimaterial HIP cladding applications. Monitoring and nondestructive evaluation need to expand, incorporating multimodal sensors, x-ray computed tomography, and real-time data integration through platforms such as Pelican. At the same time, the pathway to industrial adoption requires techno-economic analysis, machinability studies, and qualification frameworks aligned with industry and regulatory standards. Finally, workforce and academic engagement must be strengthened. Programs that train technicians and engineers for US Navy and US Department of Energy manufacturing challenges should be paired with academic partnerships to support fundamental research, with open sharing of non-export-controlled data to accelerate innovation and build the next generation of experts. In conclusion, this report demonstrates that hybrid AM + PM-HIP is scientifically viable and strategically important. By combining the design agility of AM with the consolidation strength of HIP and embedding modeling, monitoring, and workforce development, this approach provided a transformative new capability for US manufacturing. The path forward is clear: hybrid AM + PM-HIP is not just a promising research direction but is also potentially an industrially relevant pathway that can reshape how nuclear-grade components are designed, qualified, and deployed.

36 MATERIALS SCIENCE↗

Purcell-Enhanced Emissions from Diamond Color Centers in Slow Light Photonic Crystal Waveguides

Diamond color centers are promising candidates for optically addressable quantum memories, which motivates the development of efficient photonic interfaces, often using nanophotonic cavities with narrow spectral line widths and small mode volumes. However, they require perfect spectral and spatial overlap between the cavity mode and quantum emitter, which is challenging. This is especially true for solid-state quantum emitters that are often randomly positioned and suffer from inhomogeneous broadening. Another approach to enhance light-matter interaction across large optical bandwidths and areas is using slow light waveguides. Here, in this study, we demonstrate diamond slow light photonic crystal (PhC) waveguides optically coupled to embedded silicon-vacancy (SiV) color centers. We use the recently developed thin-film diamond approach to fabricate fully suspended two-dimensional PhC waveguides. We demonstrate waveguide modes with high group indices up to 70 and observe Purcell-enhanced emissions of the SiVs. Our approach represents a practical diamond platform for robust spin-photon interfaces with color centers.

2D photonic crystal waveguide↗

Tackling the Challenges in Scene Graph Generation With Local-to-Global Interactions

In this work, we seek new insights into the underlying challenges of the scene graph generation (SGG) task. Quantitative and qualitative analysis of the visual genome (VG) dataset implies: 1) ambiguity: even if interobject relationship contains the same object (or predicate), they may not be visually or semantically similar; 2) asymmetry: despite the nature of the relationship that embodied the direction, it was not well addressed in previous studies; and 3) higher-order contexts: leveraging the identities of certain graph elements can help generate accurate scene graphs. Motivated by the analysis, we design a novel SGG framework, Local-to-global interaction networks (LOGINs). Locally, interactions extract the essence between three instances of subject, object, and background, while baking direction awareness into the network by explicitly constraining the input order of subject and object. Globally, interactions encode the contexts between every graph component (i.e., nodes and edges). Finally, Attract and Repel loss is utilized to fine-tune the distribution of predicate embeddings. By design, our framework enables predicting the scene graph in a bottom-up manner, leveraging the possible complementariness. To quantify how much LOGIN is aware of relational direction, a new diagnostic task called Bidirectional Relationship Classification (BRC) is also proposed. Overall, experimental results demonstrate that LOGIN can successfully distinguish relational direction than existing methods (in BRC task), while showing state-of-the-art results on the VG benchmark (in SGG task).

97 MATHEMATICS AND COMPUTING↗