Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “tokenization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Sequence length scaling in vision transformers for scientific images on frontier

Vision Transformers (ViTs) are pivotal for foundational models in scientific imagery, including Earth science applications, due to their capability to process large sequence lengths. While transformers for text have inspired scaling sequence lengths in ViTs, adapting these for ViTs introduces unique challenges. We develop distributed sequence parallelism for ViTs, enabling them to handle up to 1M tokens. Our approach, leveraging DeepSpeed-Ulysses and Long-Sequence-Segmentation with model sharding, is the first to apply sequence parallelism in ViT training, achieving a 94% batch scaling efficiency on 2,048 AMD-MI250X GPUs. Evaluating sequence parallelism in ViTs, particularly in models up to 10B parameters, highlighted substantial bottlenecks. We countered these with hybrid sequence, pipeline, and flash attention strategies, to scale beyond single GPU memory limits. Our method significantly enhances climate modeling accuracy by 20% in temperature predictions, marking the first training of a vision transformer model to convergence with a sequence length of 188K tokens, using full self-attention.

Tsaris, Aristeidis (aris) [ORNL] (ORCID:0000000277↗
DOE OSTI2024

Automating the Analysis of Large Language Models Responses through Zero-Shot Question Answering

Recent advancements in Large Language Models (LLMs) have shown significant potential in various applications, yet their evaluation, particularly in zero-shot question answering scenarios, remains a challenging task. In this study, our objective was to explore precision metrics for Large Language Models (LLM) and design and implement a software pipeline to automatically evaluate LLMs' outputs under zero-shot question answering. Zero-shot question answering involves a model providing answers to questions about topics it hasn't seen during training. It leverages the principles of zero-shot learning by relying on semantic understanding and generalization from related knowledge. The data used was metadata from medical databases on congenital heart disease. We explored eleven LLM metrics and selected three for our evaluation: BLEU, BERTScore, and MoverScore. BLEU calculates a score based on the overlap of n-grams (contiguous sequences of n items, typically words) between the machine-generated translation and the reference translations. Higher BLEU scores indicate better correspondence between the machine-generated and human-generated translations. BERTScore is a metric used to evaluate the quality of machine-generated text by measuring the similarity of token embeddings produced by BERT (Bidirectional Encoder Representations from Transformers) between the generated text and reference text. MoverScore is a metric that quantifies the dissimilarity between the distributions of word embeddings from machine-generated text and reference text, emphasizing semantic similarity over exact token overlap. We also introduced HBKI, a composite metric summarizing these approaches. We tested five models —GPT-3, Llama-2, Gemini 1.5 Pro, Solar 10.7B, and Mixtral-8x7b. Our software pipeline, designed and implemented using Object-Oriented Programming principles, allows users to customize the selection and extraction of features for topics of interest in their own research. Our results show that MoverScore delivered the most precise evaluation of the LLM's outputs, while Mixtral-8x7b achieved the best overall performance in extracting metadata from the databases.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

An Nginx-based Content Distribution Network for HEP

With the move to HTTP/WebDAV and JSON Web Tokens as a standard protocol for transfers within the WLCG distributed storage network, a large amount of off-the-shelf technologies become viable for meeting the requirements of a Storage Element (SE). In this work, we explore the capabilities and performance of the OpenResty framework, which extends the nginx server with the LuaJIT scripting language, to recreate the feature set of a SE. We demonstrate token-authenticated HTTP read, write, and WebDAV third-party copy features, as well as a storage federation with HTTP redirect, proxy, and caching capabilities. We further explore the performance scaling in terms of throughput and requests per second.

Aarora, Aashay [UC, San Diego]↗
DOE OSTI2025

Masked Symbol Modeling for Demodulation of Oversampled Baseband Communication Signals in Impulsive Noise-Dominated Channels

Recent breakthroughs in natural language processing show that attention mech- anism in Transformer networks, trained via masked-token prediction, enables models to capture the semantic context of the tokens and internalize the grammar of language. While the application of Transformers to communication systems is a burgeoning field, the notion of context within physical waveforms remains under-explored. This paper addresses that gap by re-examining inter-symbol con- tribution (ISC) caused by pulse-shaping overlap. Rather than treating ISC as a nuisance, we view it as a deterministic source of contextual information embedded in oversampled complex baseband signals. We propose Masked Symbol Model- ing (MSM), a framework for the physical (PHY) layer inspired by Bidirectional Encoder Representations from Transformers methodology. In MSM, a subset of symbol-aligned samples is randomly masked, and a Transformer predicts the missing symbol identifiers using the surrounding “in-between” samples. Through this objective, the model learns the latent syntax of complex baseband waveforms. We illustrate MSM’s potential by applying it to the task of demodulating sig- nals corrupted by impulsive noise, where the model infers corrupted segments by leveraging the learned context. Our results suggest a path toward receivers that interpret, rather than merely detect communication signals, opening new avenues for context-aware PHY layer design.

Bedir, Oguz↗
DOE OSTI2025

Blockchain Research and Development Activities Sponsored by the U.S. Department of Energy and Utility Sector

This article provides an in-depth analysis of blockchain research in the energy sector, focusing on projects funded by the U.S. Department of Energy (DOE) and comparing them with industry-funded initiatives. A total of 110 funded activities within the U.S. power industry were successfully tracked and mapped into a newly developed categorization framework. This framework is designed to help research agencies to systematically understand their funded portfolio. Such characterization is expected to help them make effective investments, identify research gaps, measure impact, and advance technological progress to meet national goals. In line with this need, the proposed framework proposes a 2-D categorization matrix to systematically classify blockchain efforts within the energy sector.Under the proposed framework, the Energy System Domain serves as the primary classification dimension, categorizing use cases into 30 distinct applications. The second dimension, Blockchain Properties, captures the specific needs and functionalities provided by Blockchain technology. The aim was to capture blockchain’s applicability and functionality: where and why blockchain? Principles behind the selection of the viewpoint dimensions were carefully defined based on consensus obtained through the Blockchain for Optimized Security and Energy Management (BLOSEM) project. The mapped results show that activities within the Grid Automation, Coordination, and Control (31.8%), Marketplaces and Trading (25.5%), Foundational Blockchain Research (19.1%), and Supply Chain Management (17.3%) domains have been actively pursued to date. The three leading specific use case applications were identified as Transactive Energy Management for Marketplaces and Trading, Asset Management for Supply Chain Management, and Fundamental Blockchain for Foundational Blockchain Research. The Marketplaces and Trading and Retail Services Enablement domains stood out as being favored by industry by a factor greater than 2 (2.3 and 2.6, respectively), yet there seemed to be little to zero investment from DOE. Approximately 76% of the total projects prioritized Immutability, Identity Management, and Decentralization and/or Disintermediation compared to Asset Digitization and/or Tokenization, Automation, and Privacy and/or Anonymity. The greatest discrepancies between DOE and industry were in Asset Digitization and/or Tokenization and Automation. The industry efforts (36% in Asset Digitization/Tokenization and 22% in Automation) was 14 times and 2.4 times, respectively, more intensive than the DOE-sponsored efforts, indicating a significant discrepancy in industry versus government priorities. Overall, quantifying DOE-sponsored projects and industry activities through mapping provides clarity on portfolio investments and opportunities for future research.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS1985

A study of topologies and protocols for fiber optic local area network

The emergence of new applications requiring high data traffic necessitates the development of high speed local area networks. Optical fiber is selected as the transmission medium due to its inherent advantages over other possible media and the dual optical bus architecture is shown to be the most suitable topology. Asynchronous access protocols, including token, random, hybrid random/token, and virtual token schemes, are developed and analyzed. Exact expressions for insertion delay and utilization at light and heavy load are derived, and intermediate load behavior is investigated by simulation. A new tokenless adaptive scheme whose control depends only on the detection of activity on the channel is shown to outperform round-robin schemes under uneven loads and multipacket traffic and to perform optimally at light load. An approximate solution to the queueing delay for an oscillating polling scheme under chaining is obtained and results are compared with simulation. Solutions to the problem of building systems with a large number of stations are presented, including maximization of the number of optical couplers, and the use of passive star/bus topologies, bridges and gateways.

Yeh, C.↗
NASA NTRS1986

Investigation of local area networks for an obiting space station

An investigation and test of various equipment was conducted to determine its state of repair and suitability for use in a test-bed computer network (to be used to simulate a space station configuration of computers for control and monitoring). A research study was also made of various network types, including the IEEE network standards: (1) Carrier Sense Multiple Access with Collision Detection. (2) Token Passing Bus (3) Token Passing Ring, and (4) the draft ANSI standard for a High Speed Local Network, CSMA with Collision Avoidance. No firm conclusions are possible at this time as to which would be most suitable. An evaluation of the user interface of a space station database program, FREDSS (Formatted Retrieval and Entry of Data for Space Station, preliminary version), was also conducted and recommendations were made for possible improvements.

Deacetis, L. A.↗
NASA NTRS1988

L-band and SHF multiple access schemes for the MSAT system

The first generation of the Canadian Mobile Satellite (MSAT) system, planned to be operational in the early 1990s, will provide voice and data services to land, aeronautical, and maritime mobile terminals within the Canadian land mass and its territorial waters. The system will be managed by a centralized Demand Assignment Multiple Access (DAMA) control system. Users will request a communication channel by communicating with the DAMA Control System (DCS) via the appropriate signalling channels. Several access techniques for both L-band and SHF signalling channels have been investigated. For the L-band, Slotted Aloha (SA) and Reservation Aloha (RA), combined with a token scheme, are discussed here. The results of Telesat studies to date indicate that SA, when combined with token scheme, provides the most efficient access and resource management tool in a mobile propagation environment. For SHF signalling channels, slim time division multiple access (TDMA) and SA have been considered as the most suitable candidate schemes. In view of the operational environment of the SHF links, provision of a very short channel access delay and a relatively high packet success rate are highly desirable. Studies carried out generally favor slim-TDMA as the most suitable approach for SHF signalling channels.

Razi, Michael↗
NASA NTRS1988

A fiber optic tactical voice/data network based on FDDI

An asynchronous high-speed fiber optic local area network is described that supports ordinary data packet traffic simultaneously with synchronous Tl voice traffic over a common FDDI token ring channel. A voice interface module was developed that parses, buffers, and resynchronizes the voice data to the packet network. The technique is general, however, and can be applied to any deterministic class of networks, including multi-tier backbones. A conventional single token access protocol was employed at the lowest layer, with fixed packet sizes for voice and variable for data. In addition, the higher layer packet data protocols are allowed to operate independently of those for the voice thereby permitting great flexibility in reconfiguring the network. Voice call setup and switching functions were performed external to the network with PABX equipment.

Bergman, L. A.↗
NASA NTRS1989

Modeling and analysis of multiprocessor architectures

Some technologies developed for system level modeling and analysis of algorithms/architectures using an architecture design and development system are reviewed. Modeling and analysis is described with attention given to modeling constraints and analysis using constrained software graphs. An example is presented of an ADAS graph and its associated attributes, such as firing delay, token consume rate, token produce rate, firing threshold, firing condition, arc queue lengths, associated C or Ada functional model, and stochastic behavior.

Yalamanchili, S.↗
NASA NTRS1991

Compression of digital images over local area networks. Appendix 1: Item 3

Differential Pulse Code Modulation (DPCM) has been used with speech for many years. It has not been as successful for images because of poor edge performance. The only corruption in DPC is quantizer error but this corruption becomes quite large in the region of an edge because of the abrupt changes in the statistics of the signal. We introduce two improved DPCM schemes; Edge correcting DPCM and Edge Preservation Differential Coding. These two coding schemes will detect the edges and take action to correct them. In an Edge Correcting scheme, the quantizer error for an edge is encoded using a recursive quantizer with entropy coding and sent to the receiver as side information. In an Edge Preserving scheme, when the quantizer input falls in the overload region, the quantizer error is encoded and sent to the receiver repeatedly until the quantizer input falls in the inner levels. Therefore these coding schemes increase the bit rate in the region of an edge and require variable rate channels. We implement these two variable rate coding schemes on a token wing network. Timed token protocol supports two classes of messages; asynchronous and synchronous. The synchronous class provides a pre-allocated bandwidth and guaranteed response time. The remaining bandwidth is dynamically allocated to the asynchronous class. The Edge Correcting DPCM is simulated by considering the edge information under the asynchronous class. For the simulation of the Edge Preserving scheme, the amount of information sent each time is fixed, but the length of the packet or the bit rate for that packet is chosen depending on the availability capacity. The performance of the network, and the performance of the image coding algorithms, is studied.

Gorjala, Bhargavi↗
NASA NTRS1994

FDDI network test adaptor error injection circuit

An apparatus for injecting errors into a FDDI token ring network is disclosed. The error injection scheme operates by fooling a FORMAC into thinking it sent a real frame of data. This is done by using two RAM buffers. The RAM buffer normally accessed by the RBC/DPC becomes a SHADOW RAM during error injection operation. A dummy frame is loaded into the shadow RAM in order to fool the FORMAC. This data is just like the data that would be used if sending a normal frame, with the restriction that it must be shorter than the error injection data. The other buffer, the error injection RAM, contains the error injection frame. The error injection data is sent out to the media by switching a multiplexor. When the FORMAC is done transmitting the data, the multiplexor is switched back to the normal mode. Thus, the FORMAC is unaware of what happened and the token ring remains operational.

Eckenrode, Thomas↗
DOE OSTI2025

SHF: Symmetrical Hierarchical Forest with Pretrained Vision Transformer Encoder for High-Resolution Medical Segmentation

This paper presents a novel approach to addressing the long-sequence problem in high-resolution medical images for Vision Transformers (ViTs). Using smaller patches as tokens can enhance ViT performance, but quadratically increases computation and memory requirements. Therefore, the common practice for applying ViTs to high-resolution images is either to: (a) employ complex sub-quadratic attention schemes or (b) use large to medium-sized patches and rely on additional mechanisms within the model to capture the spatial hierarchy of details. We propose Symmetrical Hierarchical Forest (SHF), a lightweight approach that adaptively patches the input image to increase token information density and encode hierarchical spatial structures into the input embedding. We then apply a reverse depatching scheme to the output embeddings of the transformer encoder, eliminating the need for convolution-based decoders. Unlike previous methods that modify attention mechanisms or use a complex hierarchy of interacting models, SHF can be retrofitted to any ViT model to allow it to learn the hierarchical structure of details in high-resolution images without requiring architectural changes. Experimental results demonstrate significant gains in computational efficiency and performance: on the PAIP WSI dataset, we achieved a 3∼32×speedup or a 2.95%∼7.03% increase in accuracy (measured by Dice score) at a 64K2 resolution with the same computational budget, compared to state-of-the-art production models. On the 3D medical datasets BTCV and KiTS, training was 6×faster, with accuracy gains of 6.93% and 5.9%, respectively, compared to models without SHF.

Zhang, Enzhi [Hokkaido University, Japan]↗
DOE OSTI2025

Intern Poster

Large Language Models (LLMs) have skyrocketed in popularity after the release of ChatGPT in late 2022. Although LLMs are powerful tools, they can be subject to hallucinations, which is when an LLM (or any AI model) produces misleading/ nonsensical information. The objective is to determine if statistical methods can be used to detect hallucinations as an LLM generates its answer token by token (essentially word by word).

97 - MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Hybrid Quantum–Classical Graph Transformers for Efficient Sentiment Analysis

Quantum Machine Learning (QML) offers a promising paradigm that leverages quantum computing principles to develop efficient and expressive models for learning from complex and structured data. Recent advances in natural language processing (NLP) and artificial intelligence (AI) have demonstrated capabilities in understanding, generating, and reasoning over linguistic and multimodal information. In this work, we present the Quantum Graph Transformer (QGT), a hybrid quantum–classical architecture that extends graph transformer capabilities through quantum self-attention. The QGT models variable-length sentences as token graphs, where both the embedding encoding and the self-attention mechanisms are implemented using parameterized quantum circuits (PQCs), enabling efficient contextual learning with significantly fewer trainable parameters. We train QGT using both fully connected and 𝑘 -nearest-neighbor graph structures and evaluate it on five benchmark sentiment-classification datasets. Experimental results show that QGT consistently achieves higher or comparable accuracy to existing quantum NLP models and outperforms a Classical Graph Transformer (CGT) baseline with identical architecture, achieving 29.4 × fewer parameters while requiring 3–5 × fewer samples to reach comparable performance. These findings highlight the potential of graph-based quantum models as scalable and data-efficient architectures for natural language understanding.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2026

Peak2Patch: High-Fidelity Functional Group Identification through Attention-Based Fusion of Infrared and Mass Spectra

Identifying molecular structure based on spectroscopic readings is a key task in a variety of chemical and biological applications. Common spectroscopy techniques, such as Infrared (IR) Spectroscopy and Mass Spectrometry (MS), provide detailed information on the structure of molecular compounds but nonetheless require expert-level knowledge to decode. Machine learning has emerged as a potential solution for automating structure prediction from chemical spectra; however, current approaches generally focus on single sensor modalities, neglecting to leverage the complementary information contained within differing spectra. In this paper, we introduce Peak2Patch, a novel approach to fusion-enhanced prediction of functional groups from IR and mass spectra. First, we perform a detailed comparison of backbone networks for encoding both sparse mass spectra and dense IR spectra and demonstrate the superior performance of transformer neural networks over current state-of-the-art convolutional neural networks. Second, we evaluate three broad categories of fusion: early (raw feature), middle (deep feature), and late (decision) fusion, demonstrating the potential of a deep feature fusion-based approach. Lastly, we present Peak2Patch, our attention-based fusion scheme, which leverages cross-attention to mix features between encoded tokens of the two modalities. We validate our approach on a publicly available multimodal spectroscopic data set of 790k simulated molecules, demonstrating a large improvement in functional group prediction over both the previous state-of-the-art and our own strong single-modal baselines.

Jacobson, Philip [Sandia National Laboratories (SN↗
DOE OSTI2024

RNA language models predict mutations that improve RNA function

Structured RNA lies at the heart of many central biological processes, from gene expression to catalysis. RNA structure prediction is not yet possible due to a lack of high-quality reference data associated with organismal phenotypes that could inform RNA function. We present GARNET (Gtdb Acquired RNa with Environmental Temperatures), a new database for RNA structural and functional analysis anchored to the Genome Taxonomy Database (GTDB). GARNET links RNA sequences to experimental and predicted optimal growth temperatures of GTDB reference organisms. Using GARNET, we develop sequence- and structure-aware RNA generative models, with overlapping triplet tokenization providing optimal encoding for a GPT-like model. Leveraging hyperthermophilic RNAs in GARNET and these RNA generative models, we identify mutations in ribosomal RNA that confer increased thermostability to the Escherichia coli ribosome. The GTDB-derived data and deep learning models presented here provide a foundation for understanding the connections between RNA sequence, structure, and function.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

TransPolymer: a Transformer-based language model for polymer property predictions

Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Recently, Transformer models, equipped with self-attention mechanisms, have exhibited superior performance in natural language processing. However, such methods have not been investigated in polymer sciences. Herein, we report TransPolymer, a Transformer-based language model for polymer property prediction. Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences. Rigorous experiments on ten polymer property prediction benchmarks demonstrate the superior performance of TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on large unlabeled dataset via Masked Language Modeling. Experimental results further manifest the important role of self-attention in modeling polymer sequences. We highlight this model as a promising computational tool for promoting rational polymer design and understanding structure-property relationships from a data science view.

36 MATERIALS SCIENCE↗