Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Structured Natural Language”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

Dominant Controls on Preferential Flow and Their Implications for Future Soil Water Fluxes

Abstract Soil water flow, particularly preferential flow (PF), is a critical control on hydrological and biogeochemical processes, including groundwater recharge, contaminant transport, and carbon cycling. However, it remains challenging to predict PF occurrence across large environmental gradients. Here, we developed a deep learning (DL) model to estimate event‐scale soil water flow velocity and the probability of PF occurrence using high‐frequency soil moisture and precipitation data from 33 sites across the National Ecological Observatory Network. The model demonstrated high skill in predicting the binary occurrence of PF (91% F1‐score; 85% accuracy) but the performance was limited in predicting soil water velocity ( R 2 = 0.31). We found that precipitation characteristics (duration, volume, and intensity) were the most important predictors for soil water velocity. Among the non‐precipitation event variables, sand content showed relatively high predictive skill, though differences among non‐event climate variables were generally modest. Lower sand content was associated with increased predicted soil water velocity, a finding that highlights the role of soil structure in producing more non‐uniform flow, which contrasts with traditional uniform flow models. Projecting a reduced DL model under both moderate and high‐emissions future climate scenarios (2060–2099 Representative Concentration Pathways 4.5 and 8.5), we found ∼7.3% increase under RCP4.5 and ∼15% under RCP8.5 of soil water velocities compared to the historical simulation, while modeled likelihood of PF changed little. These findings suggest climate change is not making PF more frequent, but it is making existing PF pathways more efficient with important consequences for associated nutrient and contaminant transport under climate change. Plain Language Summary Water movement in soil is critical for water quality. While often modeled as a uniform flow process, in reality water moves rapidly through cracks and burrows in what is called “preferential flow” (PF), which limits natural filtration and can transport pollutants. We developed a deep learning model, trained on data from 33 U.S. sites, to predict when and how fast this PF occurs based on precipitation, soil, and climate data. The model showed that precipitation characteristics (duration, intensity, volume) were the most important predictors of PF. Lower soil sand content/higher clay content was associated with faster water flow, likely due to clay soils forming aggregates and cracks that water moves through rather than infiltrating uniformly. Further analyses based on climate projections suggest that the speed at which PF occurs will become more rapid under future climate scenarios compared to historical simulation. This highlights the need to represent PF in soil water models when assessing future water quality. Key Points The effect of precipitation peak intensity on soil water velocities declined with increasing precipitation intensity Antecedent soil moisture failed to predict preferential flow (PF), contrasting the high predictive power of sand content Climate predictions suggest that soil water velocities through PF paths will increase ∼15% by 2099

Li, Bonan↗

Conservation laws and effective hadronization models

Hadronization models based on local string-breaking dynamics are typically Markovian by construction, yet the physical ensemble of final states is shaped by global constraints that couple the entire fragmentation trajectory. Recasting hadronization as a conditioned stochastic diffusion process provides a precise mathematical resolution to this tension. In particular, this language reveals explicitly that constraints stemming from conservation laws induce non-Markovian correlations between otherwise independent fragmentation steps, and that these correlations can be absorbed exactly into a renormalization of the local dynamics through a Doob $h$-transform. We develop this formalism for a $q\bar{q}$ string in the chiral limit, where the longitudinal-transverse factorization of the Lund kernel becomes exact, enabling systematic power counting and clean ultraviolet (UV)/infrared (IR) separation. The dynamics organize naturally into a tower of effective theories distinguished by the remaining string mass, spanning a UV fixed point with scale-invariant transport coefficients, an intermediate regime where transverse phase space induces controlled running, and an IR boundary layer where non-local effects enter at leading order. The tower exhibits genuine Wilsonian structure, including $β$-functions, anomalous dimensions, and systematic matching conditions. The resulting framework achieves a clean factorization of universal microscopic fragmentation dynamics from infrared constraint effects, and opens new directions for both the theoretical analysis and practical simulation of hadronization.

Menzo, Tony [Alabama U.; Fermilab] (ORCID:00000002↗

Combining qualitative and quantitative spatial and temporal information in a hierarchical structure: Approximate reasoning for plan execution monitoring

The problem of plan generation (PG) and the problem of plan execution monitoring (PEM), including updating, queries, and resource-bounded replanning, have different reasoning and representation requirements. PEM requires the integration of qualitative and quantitative information. PEM is the receiving of data about the world in which a plan or agent is executing. The problem is to quickly determine the relevance of the data, the consistency of the data with respect to the expected effects, and if execution should continue. Only spatial and temporal aspects of the plan are addressed for relevance in this work. Current temporal reasoning systems are deficient in computational aspects or expressiveness. This work presents a hybrid qualitative and quantitative system that is fully expressive in its assertion language while offering certain computational efficiencies. In order to proceed, methods incorporating approximate reasoning using hierarchies, notions of locality, constraint expansion, and absolute parameters need be used and are shown to be useful for the anytime nature of PEM.

Hoebel, Louis J.↗

Pairwise Sequence Alignment Library

Vector extensions, such as SSE, have been part of the x86 CPU since the 1990s, with applications in graphics, signal processing, and scientific applications. Although many algorithms and applications can naturally benefit from automatic vectorization techniques, there are still many that are difficult to vectorize due to their dependence on irregular data structures, dense branch operations, or data dependencies. Sequence alignment, one of the most widely used operations in bioinformatics workflows, has a computational footprint that features complex data dependencies. The trend of widening vector registers adversely affects the state-of-the-art sequence alignment algorithm based on striped data layouts. Therefore, a novel SIMD implementation of a parallel scan-based sequence alignment algorithm that can better exploit wider SIMD units was implemented as part of the Parallel Sequence Alignment Library (parasail). Parasail features: Reference implementations of all known vectorized sequence alignment approaches. Implementations of Smith Waterman (SW), semi-global (SG), and Needleman Wunsch (NW) sequence alignment algorithms. Implementations across all modern CPU instruction sets including AVX2 and KNC. Language interfaces for C/C++ and Python.

Jeff Daily, PNNL↗

GATOR: Requirements capturing of telephony features

We are developing a natural language-based, requirements gathering system called GATOR (for the GATherer Of Requirements). GATOR assists in the development of more accurate and complete specifications of new telephony features. GATOR interacts with a feature designer who describes a new feature, set of features, or capability to be implemented. The system aids this individual in the specification process by asking for clarifications when potential ambiguities are present, by identifying potential conflicts with other existing features, and by presenting its understanding of the feature to the designer. Through user interaction with a model of the existing telephony feature set, GATOR constructs a formal representation of the new, 'to be implemented' feature. Ultimately GATOR will produce a requirements document and will maintain an internal representation of this feature to aid in future design and specification. This paper consists of three sections that describe (1) the structure of GATOR, (2) POND, GATOR's internal knowledge representation language, and (3) current research issues.

Dankel, Douglas D., II↗

Pairwise Sequence Alignment Library

Vector extensions, such as SSE, have been part of the x86 CPU since the 1990s, with applications in graphics, signal processing, and scientific applications. Although many algorithms and applications can naturally benefit from automatic vectorization techniques, there are still many that are difficult to vectorize due to their dependence on irregular data structures, dense branch operations, or data dependencies. Sequence alignment, one of the most widely used operations in bioinformatics workflows, has a computational footprint that features complex data dependencies. The trend of widening vector registers adversely affects the state-of-the-art sequence alignment algorithm based on striped data layouts. Therefore, a novel SIMD implementation of a parallel scan-based sequence alignment algorithm that can better exploit wider SIMD units was implemented as part of the Parallel Sequence Alignment Library (parasail). Parasail features: Reference implementations of all known vectorized sequence alignment approaches. Implementations of Smith Waterman (SW), semi-global (SG), and Needleman Wunsch (NW) sequence alignment algorithms. Implementations across all modern CPU instruction sets including AVX2 and KNC. Language interfaces for C/C++ and Python.

Jeff Daily, PNNL↗

Machine Learning for Predicting Team Functioning in HERA Missions

Team functioning is integral to success in future long term space exploration missions. Proactively detecting declines in team functioning can mitigate conflict and ensure mission success. This project developed a speech-based artificial intelligence (AI) system that unobtrusively predicts degradation in team functioning, including performance and cohesion, in the Human Exploration Research Analog (HERA) Campaigns 4 and 5. The AI system conducted automated analysis of the prosodic (tone of voice) and linguistic (language content) components of speech, modeling interpersonal dynamics at both the turn-taking and day-wide levels. We investigated team functioning via observing structured interactions (i.e., multi-mission space exploration vehicle-extra vehicular activity [MMSEV-EVA], team interaction battery [TIB]) and unstructured interactions before the MMSEV-EVA task. We developed machine learning models to predict team functioning (objective task accuracy, self reported team efficacy and self reported team cohesion) by analyzing OpenSmile acoustic features, linguistic descriptors extracted via the linguistic inquiry and word count (LIWC) dictionary, and semantic embeddings. In the TIB, static models using logistic regression and random forests were not able to predict task accuracy, but predicted team efficacy and cohesion during both the decision making and relational tasks to a moderate level (60-70%). Majority voting on the individual turns to predict day long team efficacy further increased accuracies (70-80%). Finally, long short-term memory (LSTM) models showed the best performance across all variables (80-91%), including task performance. In the MMSEV-EVA, static models achieved an accuracy of 60% with majority voting, which increased to 80% through the incorporation of mission day as a variable, accounting for the learning effect. A key finding across both tasks was the "team-dependent" nature of these interactions; models achieved much higher accuracy when trained on prior days of the same team's data rather than attempting to generalize across entirely different teams, with even 1-2 days of prior data per team achieving 5-15% improvement over team-independent models. In addition, the incorporation of pre-task data from the same team also improves model performance, e.g., incorporating data from the decision-making task of the TIB, which preceded the relational task, improved the prediction of team efficacy and cohesion during the latter. We compared model performance when trained on machine-generated data compared to data that had been further corrected by human annotators. Overall, models trained on human-corrected data exhibited a modest improvement in performance, particularly when acoustic features were used. We found no significant correlation between word error rate (WER) and model accuracy (r(55) = -0.08, p = 0.51), but model’s accuracy was significantly higher for medium/high quality transcription (0.74 (SD = 0.48)) compared to the low-quality group (0.64 (SD = 0.36)) (t(63)=2.82, p = 0.006). Based on these, several design recommendation emerge, that could inform Standards at NASA. Models predicting team functioning should incorporate at least one to two days of historical interaction data, include brief pre-task discussions, and explicitly model temporal learning effects, especially for longer operational tasks. Minimum quality standards for automated speech-processing pipelines are needed, given the performance gains observed with manually corrected acoustic data. Finally, systems should leverage both acoustic features and language embeddings in complementary ways, with modality choices and fusion strategies tailored to mission context, task demands, and data quality requirements.

Shrivatsa Mishra↗

Evaluating Economic Impact: An Investment Tool for Large Language Model Integration in Workweek Management

This paper explores the development and application of an investment tool designed to quantify the costs and potential savings associated with integrating large language models (LLMs) into work week management optimization (WMO) within the nuclear industry. LLMs, with their advanced natural language processing capabilities, can significantly enhance various aspects of work management, such as problem identification, prioritization, planning, scheduling, information retrieval, and information summary. Our investment tool focuses on evaluating the return on investment (ROI) for LLM applications in WMO by considering four pivotal decision factors: model selection, application, user training, and hosting options. This paper details the development and implementation of the ROI model and illustrates its application through multiple case studies, analyzing the impact of different variables, such as work time saved, number of requests, and model performance, on the computed ROI over two years. The computed ROI is also compared over different hosting solutions. Our findings indicate that ROI increases with enhanced work time savings and optimal request load but can decline with high request volumes or increased model costs. This model aids decision-makers in the nuclear industry by providing a structured approach to assessing the economic viability and potential savings from integrating LLMs into WMO processes.

97 - MATHEMATICS AND COMPUTING↗

Evaluating Economic Impact: An Investment Tool for Large Language Model Integration in Workweek Management

This paper explores the development and application of an investment tool designed to quantify the costs and potential savings associated with integrating large language models (LLMs) into work week management optimization (WMO) within the nuclear industry. LLMs, with their advanced natural language processing capabilities, can significantly enhance various aspects of work management, such as problem identification, prioritization, planning, scheduling, information retrieval, and information summary. Our investment tool focuses on evaluating the return on investment (ROI) for LLM applications in WMO by considering four pivotal decision factors: model selection, application, user training, and hosting options. This paper details the development and implementation of the ROI model and illustrates its application through multiple case studies, analyzing the impact of different variables, such as work time saved, number of requests, and model performance, on the computed ROI over two years. The computed ROI is also compared over different hosting solutions. Our findings indicate that ROI increases with enhanced work time savings and optimal request load but can decline with high request volumes or increased model costs. This model aids decision-makers in the nuclear industry by providing a structured approach to assessing the economic viability and potential savings from integrating LLMs into WMO processes.

99 - GENERAL AND MISCELLANEOUS↗

Incorporating Speech Recognition into a Natural User Interface

The Augmented/ Virtual Reality (AVR) Lab has been working to study the applicability of recent virtual and augmented reality hardware and software to KSC operations. This includes the Oculus Rift, HTC Vive, Microsoft HoloLens, and Unity game engine. My project in this lab is to integrate voice recognition and voice commands into an easy to modify system that can be added to an existing portion of a Natural User Interface (NUI). A NUI is an intuitive and simple to use interface incorporating visual, touch, and speech recognition. The inclusion of speech recognition capability will allow users to perform actions or make inquiries using only their voice. The simplicity of needing only to speak to control an on-screen object or enact some digital action means that any user can quickly become accustomed to using this system. Multiple programs were tested for use in a speech command and recognition system. Sphinx4 translates speech to text using a Hidden Markov Model (HMM) based Language Model, an Acoustic Model, and a word Dictionary running on Java. PocketSphinx had similar functionality to Sphinx4 but instead ran on C. However, neither of these programs were ideal as building a Java or C wrapper slowed performance. The most ideal speech recognition system tested was the Unity Engine Grammar Recognizer. A Context Free Grammar (CFG) structure is written in an XML file to specify the structure of phrases and words that will be recognized by Unity Grammar Recognizer. Using Speech Recognition Grammar Specification (SRGS) 1.0 makes modifying the recognized combinations of words and phrases very simple and quick to do. With SRGS 1.0, semantic information can also be added to the XML file, which allows for even more control over how spoken words and phrases are interpreted by Unity. Additionally, using a CFG with SRGS 1.0 produces a Finite State Machine (FSM) functionality limiting the potential for incorrectly heard words or phrases. The purpose of my project was to investigate options for a Speech Recognition System. To that end I attempted to integrate Sphinx4 into a user interface. Sphinx4 had great accuracy and is the only free program able to perform offline speech dictation. However it had a limited dictionary of words that could be recognized, single syllable words were almost impossible for it to hear, and since it ran on Java it could not be integrated into the Unity based NUI. PocketSphinx ran much faster than Sphinx4 which would've made it ideal as a plugin to the Unity NUI, unfortunately creating a C# wrapper for the C code made the program unusable with Unity due to the wrapper slowing code execution and class files becoming unreachable. Unity Grammar Recognizer is the ideal speech recognition interface, it is flexible in recognizing multiple variations of the same command. It is also the most accurate program in recognizing speech due to using an XML grammar to specify speech structure instead of relying solely on a Dictionary and Language model. The Unity Grammar Recognizer will be used with the NUI for these reasons as well as being written in C# which further simplifies the incorporation.

Chapa, Nicholas↗

Provenance infrastructure for multimodal x-ray experiments and reproducible analysis

Exploring relationships between material structures and desired properties for material discovery and synthesis requires mining machine-readable databases and linking experimental data with sufficiently rich annotations. There are vast gaps both in these data sources and in the connections between them. We present a provenance-based data management and analysis framework that enables capturing, persisting, and reanalyzing experimental data from the BNL National Synchrotron Light Source II (NSLS-II). Our system leverages NSLS-II Bluesky for experimental data acquisition, captures analysis parameters, and enriches the search space with results from the scientific literature. We describe in detail the challenges and benefits of data-driven approaches when comparing computational and experimental data, such as the multiple possible scenarios of interest in often ill-posed inverse problems and missing data. Our infrastructure utilizes the NSLS-II event model and includes discovery mechanisms across multiple beamlines. A text and data mining module using natural language processing classification techniques extracts relevant information from the literature that is added to data sources. We describe a provenance acquisition and analysis rerun workflow based on a directed acyclic graph that keeps track of process and data provenance and allows to replay analysis.

97 MATHEMATICS AND COMPUTING↗

OpenStudio®-MCP [SWR-26-035]

OpenStudio®-MCP is a Model Context Protocol (MCP) server that lets AI assistants perform building energy modeling through natural language. Rather than requiring users to learn the OpenStudio® SDK, EnergyPlus® scripting, or Ruby/Python automation, the server translates conversational requests into sequences of tool calls that create models, design HVAC systems, run simulations, and extract results — all within a single chat session. The server's 124 tools are organized into a skills architecture where each skill encapsulates a domain of building energy modeling (envelope, HVAC, loads, weather, simulation, results) behind typed, LLM-friendly interfaces. High-leverage operations like applying ASHRAE 90.1 baseline systems or generating standards-compliant typical buildings are exposed as single tool calls that internally wire dozens of OpenStudio® objects. Bundled measures from ComStock™ and Openstudio® -common-measures-gem are wrapped with dedicated tools and typed arguments rather than exposed through a generic measure interface, so AI models get consistent, error-resistant recipes without needing to discover measure arguments at runtime. A key design decision is structured results extraction: six SQL-based tools return surgical ~300–1,000 token responses (end-use breakdowns, envelope summaries, HVAC sizing, timeseries data) instead of requiring the AI to parse ~100K-token raw HTML reports, making iterative design exploration practical within context window limits. The codebase is designed as a reference implementation — explicit, well-commented, and modular — so that other simulation engines (EnergyPlus® standalone, TRNSYS, DOE-2) can use it as a template for building their own MCP servers.

Ball, Brian [National Laboratory of the Rockies (N↗

Structural and compositional complexities of hierarchical self-assembly: A hypergraph approach

Programmable self-assembly enables the construction of complex molecular, supramolecular, and crystalline architectures from well-designed building blocks. In this work, we introduce a hypergraph-based formalism, Blocks & Bonds (B&B), which generalizes classical chemical graph theory by incorporating directed and multicolored interactions, internal symmetries, and hierarchical organization. Within this framework, we develop the Structure Code (SC), a compact and versatile language for describing self-assembled architectures. We define a Kolmogorov-style structural complexity as the total information content of SC, obtained through its tokenization and Shannon information assignment. Complementing this encoding-based measure, we introduce a much simpler quantity, the compositional complexity, which depends only on the number and cumulative usage of block and bond types in the construction set. A central result of this work is a strong empirical correlation between the token-based structural complexity and the compositional complexity across all examined systems. Owing to this agreement, the compositional complexity emerges as the most practical and broadly applicable measure: it is easy to compute, requires no explicit encoding, and yet closely tracks the actual information content of structurally diverse architectures. Applications to molecular systems (ethylene glycol and glucose), DNA-origami lattices, and crystalline assemblies show that B&B hypergraphs provide a unified, scalable, and information-efficient representation of structural organization, naturally capturing symmetry, modularity, and stereochemistry. This framework establishes a quantitative foundation for complexity-aware classification and inverse design of programmable matter.

36 MATERIALS SCIENCE↗

Cold ion beam in a storage ring as a platform for large-scale quantum computers and simulators: Challenges and directions for research and development

The purpose of this paper is to evaluate the possibility of constructing a large-scale storage-ring-type ion-trap system capable of storing, cooling, and controlling a large number of ions as a platform for scalable quantum computing (QC) and quantum simulations. In such a trap, the ions form a crystalline beam moving along a circular path with a constant velocity determined by the frequency and intensity of the cooling lasers. In this paper, we consider a large leap forward in terms of the number of ions that serve as qubits in QC, from fewer than 100 available in state of the art linear ion-trap devices today to an order of 10 5 crystallized ions in the storage-ring setup. This new trap design unifies two different concepts: the storage rings of charged particles and the linear ion traps used for QC and mass spectrometry. In this paper, we use the language of particle accelerators to discuss the ion state and dynamics. We outline the differences between the above concepts, analyze challenges of the large ring with a revolving chain of ions, and propose goals for the research and development required to enable future quantum computers with 1000 times more qubits than available today. The challenge of creating such a large-scale quantum system while maintaining the necessary coherence of the qubits and the high fidelity of quantum logic operations is significant. Performing analog quantum simulations may be an achievable initial goal for such a device. Quantum calculations and simulations of complex quantum systems will move forward both the fundamental science and the applied research. Nuclear and particle physics, many-body quantum systems, lattice gauge theories, and nuclear structure calculations are just a few examples in which a large-scale quantum simulation system will become a very powerful tool to move forward our understanding of nature.

36 MATERIALS SCIENCE↗

Formal Modeling of Multi-Agent Systems using the Pi-Calculus and Epistemic Logic

Multi-agent systems have become important recently in computer science, especially in artificial intelligence (AI). We allow a broad sense of agent, but require at least that an agent has some measure of autonomy and interacts with other agents via some kind of agent communication language. We are concerned in this paper with formal modeling of multi-agent systems, with emphasis on communication. We propose for this purpose to use the pi-calculus, an extension of the process algebra CCS. Although the literature on the pi-calculus refers to agents, the term is used there in the sense of a process in general. It is our contention, however, that viewing agents in the AI sense as agents in the pi-calculus sense affords significant formal insight. One formalism that has been applied to agents in the AI sense is epistemic logic, the logic of knowledge. The success of epistemic logic in computer science in general has come in large part from its ability to handle concepts of knowledge that apply to groups. We maintain that the pi-calculus affords a natural yet rigorous means by which groups that are significant to epistemic logic may be identified, encapsulated, structured into hierarchies, and restructured in a principled way. This paper is organized as follows: Section 2 introduces the pi-calculus; Section 3 takes a scenario from the classical paper on agent-oriented programming [Sh93] and translates it into a very simple subset of the n-calculus; Section 4 then shows how more sophisticated features of the pi-calculus may bc brought into play; Section 5 discusses how the pi-calculus may be used to define groups for epistemic logic; and Section 6 is the conclusion.

Rorie, Toinette↗

4D scattering amplitudes and asymptotic symmetries from 2D CFT

We reformulate the scattering amplitudes of 4D at space gauge theory and gravity in the language of a 2D CFT on the celestial sphere. The resulting CFT structure exhibits an OPE constructed from 4D collinear singularities, as well as infinite-dimensional Kac-Moody and Virasoro algebras encoding the asymptotic symmetries of 4D at space. We derive these results by recasting 4D dynamics in terms of a convenient foliation of flat space into 3D Euclidean AdS and Lorentzian dS geometries. Tree-level scattering amplitudes take the form of Witten diagrams for a continuum of (A)dS modes, which are in turn equivalent to CFT correlators via the (A)dS/CFT dictionary. The Ward identities for the 2D conserved currents are dual to 4D soft theorems, while the bulk-boundary propagators of massless (A)dS modes are superpositions of the leading and subleading Weinberg soft factors of gauge theory and gravity. In general, the massless (A)dS modes are 3D Chern-Simons gauge fields describing the soft, single helicity sectors of 4D gauge theory and gravity. Consistent with the topological nature of Chern-Simons theory, Aharonov-Bohm effects record the \tracks" of hard particles in the soft radiation, leading to a simple characterization of gauge and gravitational memories. Soft particle exchanges between hard processes define the Kac-Moody level and Virasoro central charge, which are thereby related to the 4D gauge coupling and gravitational strength in units of an infrared cutoff. Lastly, we discuss a toy model for black hole horizons via a restriction to the Rindler region.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

BuildingQA: A Benchmark for Natural Language Question Answering over Building Knowledge Graphs

Graph-based representations of building metadata using ontologies like Brick are vital for smart building applications, but querying them remains a challenge for practitioners. Knowledge Graph Question Answering (KGQA) systems, meant to retrieve answers from natural language questions, traditionally require large-scale training data, making them ill-suited for the specialized and data-scarce building domain. The advent of Large Language Models (LLMs) offers a paradigm shift, enabling zero-shot natural language querying without building/domain-specific training. Yet, there is no standardized benchmark for building-specific KGQA which can guide and validate research in this area. To address this gap, our work makes three primary contributions. First, we introduce the BuildingQA Benchmark Dataset, constructed through a multi-stage process of collecting practitioner data, augmenting it with LLMs for linguistic diversity, and curating a final set of 188 questions across 4 buildings. Second, we characterize the benchmark's complexity and ambiguity, introducing a novel method to quantify its "lexical gap" and providing a four-stage diagnostic framework for analyzing how systems fail. Third, we benchmark zero-shot LLM-powered KGQA systems to establish baseline performance and analyze their failure modes. Our evaluation reveals that top-performing systems achieve a maximum F1 score of only 0.38. This result does not indicate a failure of these powerful systems, but rather underscores the unique challenges posed by our benchmark. It demonstrates a critical performance gap, showing that current methods successful on general KGs struggle with the specific lexical and structural nuances of the building domain. BuildingQA1 thus provides the benchmark dataset and foundational analysis needed to drive the development of novel, domain-aware methods required to unlock the use of semantic data in buildings.

Mulayim, Ozan Baris↗

PDF Entity Annotation Tool (PEAT)

While different text mining approaches – including the use of Artificial Intelligence (AI) and other machine based methods - continue to expand at a rapid pace, the tools used by researchers to create the labeled datasets required for training, modeling, and evaluation remain rudimentary. Labeled datasets contain the target attributes the machine is going to learn; for example, training an algorithm to delineate between images of a car or truck would generally require a set of images with a quantitative description of the underlying features of each vehicle type. Development of labeled textual data that can be used to build natural language machine learning models for scientific literature is not currently integrated into existing manual workflows used by domain experts. Published literature is rich with important information, such as different types of embedded text, plots, and tables that can all be used as inputs to train ML/natural language processing (NLP) models, when extracted and prepared in machine readable formats. Currently, both normalized data extraction of use to domain experts and extraction to support development of ML/NLP models are labor intensive and cumbersome manual processes. Automatic extraction of data and information from formats such as PDFs that are optimized for layout and human readability, not machine readability. The PDF (Portable Document Format) Entity Annotation Tool (PEAT) was developed with the goal of allowing users to annotate publications within their current print format, while also allowing those annotations to be captured in a machine-readable format. One of the main issues with traditional annotation tools is that they require transforming the PDF into plain text to facilitate the annotation process. While doing so lessens the technical challenges of annotating data, the user loses all structure and provenance that was inherent in the underlying PDF. Also, textual data extraction from PDFs can be an error prone process. Challenges include identifying sequential blocks of text and a multitude of document formats (multiple columns, font encodings, etc.). As a result of these challenges, using existing tools for development of NLP/ML models directly from PDFs is difficult because the generated outputs are not interoperable. We created a system that allows annotations to be completed on the original PDF document structure, with no plain text extraction. The result is an application that allows for easier and more accurate annotations. In addition, by including a feature that grants the user the ability to easily create a schema, we have developed a system that can be used to annotate text for different domain-centric schemas of relevance to subject matter experts. Different knowledge domains require distinct schemas and annotation tags to support machine learning.

97 MATHEMATICS AND COMPUTING↗