Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “tokenization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

An Introduction to Word Embeddings and Language Models

Language models have advanced at a phenomenal pace over the past decade. This document provides a short introduction to terminology, word embeddings (aka low-dimensional representations), and popular large-scale language models (LMs). Word embeddings are used to represent words as numerical vectors and are context-independent, meaning a word can only have a single representation (e.g., club can only be club sandwich, not golf club ). Language models can determine the probability of a given sequence of words occurring in a sentence and can provide context to distinguish between words and phrases that sound similar. LMs are context-dependent (e.g., club can be club sandwich or golf club ) and largely fall in two main classes – autoregressive and autoencoding models. Autoregressive models are pretrained on the classic language modeling task: guess the next token having read all the previous ones. Those models can be fine-tuned and achieve great results on many tasks, the most natural application is text generation. A typical example of such models is GPT, but others include GPT-2, GPT-3, CTLR, TRANSFORMER-XL, REFORMER, XLNET. Autoencoding models are pretrained by corrupting the input tokens in some way and trying to reconstruct the original sentence. They can be fine-tuned and achieve great results on many tasks such as text generation, but their most natural application is sentence classification or token classification. A typical example of such models is BERT, but others include ROBERTA, ALBERT, XML, XML-ROBERTA, FLAUBERT AND LONGFORMER.

97 MATHEMATICS AND COMPUTING↗

Scanning Program

SCAN program uses scanning algorithm to locate tokens in line of input data. Tokens can be command words, numbers, data values, labels. Using SCAN subroutines, user extracts tokens from character strings in languages with simple or complex syntax. SCAN thoroughly tested and implemented in NASA's Descent Design System for Shuttle orbiter. SCAN useful for other programs requiring input scanning. SCAN written in FORTRAN 77.

Mattison, W. C.↗

An intelligent allocation algorithm for parallel processing

The problem of allocating nodes of a program graph to processors in a parallel processing architecture is considered. The algorithm is based on critical path analysis, some allocation heuristics, and the execution granularity of nodes in a program graph. These factors, and the structure of interprocessor communication network, influence the allocation. To achieve realistic estimations of the executive durations of allocations, the algorithm considers the fact that nodes in a program graph have to communicate through varying numbers of tokens. Coarse and fine granularities have been implemented, with interprocessor token-communication duration, varying from zero up to values comparable to the execution durations of individual nodes. The effect on allocation of communication network structures is demonstrated by performing allocations for crossbar (non-blocking) and star (blocking) networks. The algorithm assumes the availability of as many processors as it needs for the optimal allocation of any program graph. Hence, the focus of allocation has been on varying token-communication durations rather than varying the number of processors. The algorithm always utilizes as many processors as necessary for the optimal allocation of any program graph, depending upon granularity and characteristics of the interprocessor communication network.

Carroll, Chester C.↗

Alternative parallel ring protocols

Communication protocols are know to influence the utilization and performance of communication network. The effect of two token ring protocols on a gigabit network with multiple ring structure is investigated. In the first protocol, a mode sends at most one message on receiving a token. In the second protocol, a mode sends all the waiting messages when a token is received. The behavior of these protocols is shown to be highly dependent on the number of rings as well as the load in the network.

Mukkamala, R.↗

Fairness problems at the media access level for high-speed networks

Most lower speed (approx. 10 Mbps) local area networks use adaptive or random access protocols like Ethernet. Others at higher speed use demand assignment like token or slotted rings. These include Cambridge ring and electronic token ring systems. Fairness issues in representatives of such protocols are discussed. In particular, Fiber Distributed Data Interface (FDDI) was selected as a demand access protocol using tokens, Carrier Sensed Multiple Access/Ring Network (CSMA/RN) a random access protocol, and Distributed Queue Dual Bus (DQDB) a demand access protocol using reservations. Fairness at the media access level was the focus, i.e., attaining access or being excessively delayed when a message is queued to be sent as a function of network location. Within that framework, the essential fairness of FDDI was observed along with severe fairness problems in DQDB and some problems for CSMA/RN. Several modifications were investigated and their ameliorative effect is shown. Finally, a unified presentation which allows comparisons of the three protocols' fairness when normalized to their capacity is given.

Maly, Kurt J.↗

Time-Extended Payoffs for Collectives of Autonomous Agents

A collective is a set of self-interested agents which try to maximize their own utilities, along with a a well-defined, time-extended world utility function which rates the performance of the entire system. In this paper, we use theory of collectives to design time-extended payoff utilities for agents that are both aligned with the world utility, and are "learnable", i.e., the agents can readily see how their behavior affects their utility. We show that in systems where each agent aims to optimize such payoff functions, coordination arises as a byproduct of the agents selfishly pursuing their own goals. A game theoretic analysis shows that such payoff functions have the net effect of aligning the Nash equilibrium, Pareto optimal solution and world utility optimum, thus eliminating undesirable behavior such as agents working at cross-purposes. We then apply collective-based payoff functions to the token collection in a gridworld problem where agents need to optimize the aggregate value of tokens collected across an episode of finite duration (i.e., an abstracted version of rovers on Mars collecting scientifically interesting rock samples, subject to power limitations). We show that, regardless of the initial token distribution, reinforcement learning agents using collective-based payoff functions significantly outperform both natural extensions of single agent algorithms and global reinforcement learning solutions based on "team games".

Tumer, Kagan↗

Computational Performance of Progressive Damage Analysis of Composite Laminates using Abaqus/Explicit with 16 to 512 CPU Cores

The computational scaling performance of progressive damage analysis using Abaqus/ Explicit is evaluated and quantified using from 16 to 512 CPU cores. Several analyses were conducted on varying numbers of cores to determine the scalability of the code on five NASA high performance computing systems. Two finite element models representative of typical models used for progressive damage analysis of composite laminates were used. The results indicate a 10 to 15 times speed up scaling from 24 to 512 cores. The run times were modestly reduced with newer generations of CPU hardware. If the number of degrees of freedom is held constant with respect to the number of cores, the model size can be increased by a factor of 20, scaling from 16 to 512 cores, with the same run time. An empirical expression was derived relating run time, the number of cores, and the number of degrees of freedom. Analysis cost was examined in terms of software tokens and hardware utilization. Using additional cores reduces token usage since the computational performance increases more rapidly than the token requirement with increasing number of cores. The in- crease in hardware cost with increasing cores was found to be modest. Overall the results show relatively good scalability of the Abaqus/Explicit code on up to 512 cores.

Bergan, A. C.↗

Neural Scaling Laws for Jet Generation

Recently observed empirical scaling laws describe the performance of foundation-type models as three independent key quantities -- dataset size, compute, and model parameters -- are modified. Extracting these scaling laws informs the training of large complex models for which the tuning of hyperparameters in traditional ways is not feasible. This work for the first time explores if scaling laws can also be observed for the task of particle jet generation -- both relevant as a pre-training objective for foundation models and as in-situ simulation by itself. We indeed replicate the key logarithmic scaling law behavior for model-size scaling. Beyond studying the next token prediction validation loss of the generative model, we also study the sliced Wasserstein distance of five physical quantities that are not immediately available to the model during training. Our study shows that this quantity is monotonically related to the next token prediction validation loss, meaning that this loss is indeed a good proxy for the physics performance. For the scaling with dataset size and compute, we observe substantially weaker scaling behavior of both the loss and the sliced Wasserstein distance. We analyze this behavior by introducing the concept of a learnable window, and argue that autoregressive next token prediction on jet constituents exhibits comparatively rapid saturation relative to language-model studies. We discuss possible origins of this behavior, including the stochastic nature of QCD radiation and differences between generative and supervised learning tasks in collider physics.

Amram, Oz [Fermilab]↗

Automating Genetic Algorithm Mutations for Molecules Using a Masked Language Model

Inspired by the evolution of biological systems, genetic algorithms have been applied to generate solutions for optimization problems in a variety of scientific and engineering disciplines. For a given problem, a suitable genome representation must be defined along with a mutation operator to generate subsequent generations. Unlike natural systems which display a variety of complex rearrangements (e.g. mobile genetic elements), mutation for genetic algorithms commonly utilizes only random point-wise changes. Furthermore, generalizing beyond point-wise mutations poses a key difficulty as useful genome rearrangements depend on the representation and problem domain. To move beyond the limitations of manually defined point-wise changes, here we propose the use of techniques from masked language models to automatically generate mutations. As a first step, common subsequences within a given population are used to generate a vocabulary. The vocabulary is then used to tokenize each genome. A masked language model is trained on the tokenized data in order to generate possible rearrangements (i.e. mutations). In order to illustrate the proposed strategy, we use string representations of molecules and use a genetic algorithm to optimize for drug-likeness and synthesizability. Finally, our results show that moving beyond random point-wise mutations accelerates genetic algorithm optimization.

59 BASIC BIOLOGICAL SCIENCES↗

EHR-BERT: A BERT-based model for effective anomaly detection in electronic health records

Objective: Physicians and clinicians rely on data contained in electronic health records (EHRs), as recorded by health information technology (HIT), to make informed decisions about their patients. The reliability of HIT systems in this regard is critical to patient safety. Consequently, better tools are needed to monitor the performance of HIT systems for potential hazards that could compromise the collected EHRs, which in turn could affect patient safety. In this paper, we propose a new framework for detecting anomalies in EHRs using sequence of clinical events. This new framework, EHR-Bidirectional Encoder Representations from Transformers (BERT), is motivated by the gaps in the existing deep-learning related methods, including high false negatives, sub-optimal accuracy, higher computational cost, and the risk of information loss. EHR-BERT is an innovative framework rooted in the BERT architecture, meticulously tailored to navigate the hurdles in the contemporary BERT method; thus, enhancing anomaly detection in EHRs for healthcare applications.Methods: The EHR-BERT framework was designed using the Sequential Masked Token Prediction (SMTP) method. This approach treats EHRs as natural language sentences and iteratively masks input tokens during both training and prediction stages. This method facilitates the learning of EHR sequence patterns in both directions for each event and identifies anomalies based on deviations from the normal execution models trained on EHR sequences.Results: Extensive experiments on large EHR datasets across various medical domains demonstrate that EHR-BERT markedly improves upon existing models. It significantly reduces the number of false positives and enhances the detection rate, thus bolstering the reliability of anomaly detection in electronic health records. This improvement is attributed to the model’s ability to minimize information loss and maximize data utilization effectively.Conclusion: EHR-BERT showcases immense potential in decreasing medical errors related to anomalous clinical events, positioning itself as an indispensable asset for enhancing patient safety and the overall standard of healthcare services. The framework effectively overcomes the drawbacks of earlier models, making it a promising solution for healthcare professionals to ensure the reliability and quality of health data.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Structural and compositional complexities of hierarchical self-assembly: A hypergraph approach

Programmable self-assembly enables the construction of complex molecular, supramolecular, and crystalline architectures from well-designed building blocks. In this work, we introduce a hypergraph-based formalism, Blocks & Bonds (B&B), which generalizes classical chemical graph theory by incorporating directed and multicolored interactions, internal symmetries, and hierarchical organization. Within this framework, we develop the Structure Code (SC), a compact and versatile language for describing self-assembled architectures. We define a Kolmogorov-style structural complexity as the total information content of SC, obtained through its tokenization and Shannon information assignment. Complementing this encoding-based measure, we introduce a much simpler quantity, the compositional complexity, which depends only on the number and cumulative usage of block and bond types in the construction set. A central result of this work is a strong empirical correlation between the token-based structural complexity and the compositional complexity across all examined systems. Owing to this agreement, the compositional complexity emerges as the most practical and broadly applicable measure: it is easy to compute, requires no explicit encoding, and yet closely tracks the actual information content of structurally diverse architectures. Applications to molecular systems (ethylene glycol and glucose), DNA-origami lattices, and crystalline assemblies show that B&B hypergraphs provide a unified, scalable, and information-efficient representation of structural organization, naturally capturing symmetry, modularity, and stereochemistry. This framework establishes a quantitative foundation for complexity-aware classification and inverse design of programmable matter.

36 MATERIALS SCIENCE↗

Learning-Based Quantum Compilation: Translating QASM to QIR with CodeBERT

We propose a learning-based approach to quantum compilation by translating OpenQASM to Quantum Intermediate Representation (QIR) using a fine-tuned CodeBERT model. Trained on 10,000 synthetic QASM-QIR pairs, the model captures code semantics while addressing QIR verbosity and the 512-token limit via a custom token compression scheme. Finetuning was performed on the Frontier supercomputer, with results showing syntactic correctness and stable validation loss reduction. Our method moves toward enabling flexible, language-modeldriven quantum software tools. It also introduces syntax error handling and the possibility of incorporating classical control constructs, addressing limitations in existing rule-based compilers like qBraid-QIR. While the current model has been validated on quantum-only circuits, we propose future evaluations on hybrid quantum-classical examples. This poster will provide architecture insights, compression examples, training loss plots, and QIR outputs. Our work highlights the potential for scalable, adaptable compilation in future quantum toolchains.

Afrose, Sharmin [ORNL]↗

OpenStudio®-MCP [SWR-26-035]

OpenStudio®-MCP is a Model Context Protocol (MCP) server that lets AI assistants perform building energy modeling through natural language. Rather than requiring users to learn the OpenStudio® SDK, EnergyPlus® scripting, or Ruby/Python automation, the server translates conversational requests into sequences of tool calls that create models, design HVAC systems, run simulations, and extract results — all within a single chat session. The server's 124 tools are organized into a skills architecture where each skill encapsulates a domain of building energy modeling (envelope, HVAC, loads, weather, simulation, results) behind typed, LLM-friendly interfaces. High-leverage operations like applying ASHRAE 90.1 baseline systems or generating standards-compliant typical buildings are exposed as single tool calls that internally wire dozens of OpenStudio® objects. Bundled measures from ComStock™ and Openstudio® -common-measures-gem are wrapped with dedicated tools and typed arguments rather than exposed through a generic measure interface, so AI models get consistent, error-resistant recipes without needing to discover measure arguments at runtime. A key design decision is structured results extraction: six SQL-based tools return surgical ~300–1,000 token responses (end-use breakdowns, envelope summaries, HVAC sizing, timeseries data) instead of requiring the AI to parse ~100K-token raw HTML reports, making iterative design exploration practical within context window limits. The codebase is designed as a reference implementation — explicit, well-commented, and modular — so that other simulation engines (EnergyPlus® standalone, TRNSYS, DOE-2) can use it as a template for building their own MCP servers.

Ball, Brian [National Laboratory of the Rockies (N↗

Sequence length scaling in vision transformers for scientific images on frontier

Vision Transformers (ViTs) are pivotal for foundational models in scientific imagery, including Earth science applications, due to their capability to process large sequence lengths. While transformers for text have inspired scaling sequence lengths in ViTs, adapting these for ViTs introduces unique challenges. We develop distributed sequence parallelism for ViTs, enabling them to handle up to 1M tokens. Our approach, leveraging DeepSpeed-Ulysses and Long-Sequence-Segmentation with model sharding, is the first to apply sequence parallelism in ViT training, achieving a 94% batch scaling efficiency on 2,048 AMD-MI250X GPUs. Evaluating sequence parallelism in ViTs, particularly in models up to 10B parameters, highlighted substantial bottlenecks. We countered these with hybrid sequence, pipeline, and flash attention strategies, to scale beyond single GPU memory limits. Our method significantly enhances climate modeling accuracy by 20% in temperature predictions, marking the first training of a vision transformer model to convergence with a sequence length of 188K tokens, using full self-attention.

Tsaris, Aristeidis (aris) [ORNL] (ORCID:0000000277↗

Automating the Analysis of Large Language Models Responses through Zero-Shot Question Answering

Recent advancements in Large Language Models (LLMs) have shown significant potential in various applications, yet their evaluation, particularly in zero-shot question answering scenarios, remains a challenging task. In this study, our objective was to explore precision metrics for Large Language Models (LLM) and design and implement a software pipeline to automatically evaluate LLMs' outputs under zero-shot question answering. Zero-shot question answering involves a model providing answers to questions about topics it hasn't seen during training. It leverages the principles of zero-shot learning by relying on semantic understanding and generalization from related knowledge. The data used was metadata from medical databases on congenital heart disease. We explored eleven LLM metrics and selected three for our evaluation: BLEU, BERTScore, and MoverScore. BLEU calculates a score based on the overlap of n-grams (contiguous sequences of n items, typically words) between the machine-generated translation and the reference translations. Higher BLEU scores indicate better correspondence between the machine-generated and human-generated translations. BERTScore is a metric used to evaluate the quality of machine-generated text by measuring the similarity of token embeddings produced by BERT (Bidirectional Encoder Representations from Transformers) between the generated text and reference text. MoverScore is a metric that quantifies the dissimilarity between the distributions of word embeddings from machine-generated text and reference text, emphasizing semantic similarity over exact token overlap. We also introduced HBKI, a composite metric summarizing these approaches. We tested five models —GPT-3, Llama-2, Gemini 1.5 Pro, Solar 10.7B, and Mixtral-8x7b. Our software pipeline, designed and implemented using Object-Oriented Programming principles, allows users to customize the selection and extraction of features for topics of interest in their own research. Our results show that MoverScore delivered the most precise evaluation of the LLM's outputs, while Mixtral-8x7b achieved the best overall performance in extracting metadata from the databases.

97 MATHEMATICS AND COMPUTING↗

An Nginx-based Content Distribution Network for HEP

With the move to HTTP/WebDAV and JSON Web Tokens as a standard protocol for transfers within the WLCG distributed storage network, a large amount of off-the-shelf technologies become viable for meeting the requirements of a Storage Element (SE). In this work, we explore the capabilities and performance of the OpenResty framework, which extends the nginx server with the LuaJIT scripting language, to recreate the feature set of a SE. We demonstrate token-authenticated HTTP read, write, and WebDAV third-party copy features, as well as a storage federation with HTTP redirect, proxy, and caching capabilities. We further explore the performance scaling in terms of throughput and requests per second.

Aarora, Aashay [UC, San Diego]↗

Masked Symbol Modeling for Demodulation of Oversampled Baseband Communication Signals in Impulsive Noise-Dominated Channels

Recent breakthroughs in natural language processing show that attention mech- anism in Transformer networks, trained via masked-token prediction, enables models to capture the semantic context of the tokens and internalize the grammar of language. While the application of Transformers to communication systems is a burgeoning field, the notion of context within physical waveforms remains under-explored. This paper addresses that gap by re-examining inter-symbol con- tribution (ISC) caused by pulse-shaping overlap. Rather than treating ISC as a nuisance, we view it as a deterministic source of contextual information embedded in oversampled complex baseband signals. We propose Masked Symbol Model- ing (MSM), a framework for the physical (PHY) layer inspired by Bidirectional Encoder Representations from Transformers methodology. In MSM, a subset of symbol-aligned samples is randomly masked, and a Transformer predicts the missing symbol identifiers using the surrounding “in-between” samples. Through this objective, the model learns the latent syntax of complex baseband waveforms. We illustrate MSM’s potential by applying it to the task of demodulating sig- nals corrupted by impulsive noise, where the model infers corrupted segments by leveraging the learned context. Our results suggest a path toward receivers that interpret, rather than merely detect communication signals, opening new avenues for context-aware PHY layer design.

Bedir, Oguz↗

Blockchain Research and Development Activities Sponsored by the U.S. Department of Energy and Utility Sector

This article provides an in-depth analysis of blockchain research in the energy sector, focusing on projects funded by the U.S. Department of Energy (DOE) and comparing them with industry-funded initiatives. A total of 110 funded activities within the U.S. power industry were successfully tracked and mapped into a newly developed categorization framework. This framework is designed to help research agencies to systematically understand their funded portfolio. Such characterization is expected to help them make effective investments, identify research gaps, measure impact, and advance technological progress to meet national goals. In line with this need, the proposed framework proposes a 2-D categorization matrix to systematically classify blockchain efforts within the energy sector.Under the proposed framework, the Energy System Domain serves as the primary classification dimension, categorizing use cases into 30 distinct applications. The second dimension, Blockchain Properties, captures the specific needs and functionalities provided by Blockchain technology. The aim was to capture blockchain’s applicability and functionality: where and why blockchain? Principles behind the selection of the viewpoint dimensions were carefully defined based on consensus obtained through the Blockchain for Optimized Security and Energy Management (BLOSEM) project. The mapped results show that activities within the Grid Automation, Coordination, and Control (31.8%), Marketplaces and Trading (25.5%), Foundational Blockchain Research (19.1%), and Supply Chain Management (17.3%) domains have been actively pursued to date. The three leading specific use case applications were identified as Transactive Energy Management for Marketplaces and Trading, Asset Management for Supply Chain Management, and Fundamental Blockchain for Foundational Blockchain Research. The Marketplaces and Trading and Retail Services Enablement domains stood out as being favored by industry by a factor greater than 2 (2.3 and 2.6, respectively), yet there seemed to be little to zero investment from DOE. Approximately 76% of the total projects prioritized Immutability, Identity Management, and Decentralization and/or Disintermediation compared to Asset Digitization and/or Tokenization, Automation, and Privacy and/or Anonymity. The greatest discrepancies between DOE and industry were in Asset Digitization and/or Tokenization and Automation. The industry efforts (36% in Asset Digitization/Tokenization and 22% in Automation) was 14 times and 2.4 times, respectively, more intensive than the DOE-sponsored efforts, indicating a significant discrepancy in industry versus government priorities. Overall, quantifying DOE-sponsored projects and industry activities through mapping provides clarity on portfolio investments and opportunities for future research.

24 POWER TRANSMISSION AND DISTRIBUTION↗