Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “tokenization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Obtaining correct compile results by absorbing mismatches between data types representations

Methods and a system are provided. A method includes implementing a function, which a compiler for a first language does not have, using a compiler for a second language. The implementing step includes generating, by the compiler for the first language, a first abstract syntax tree. The implementing step further includes converting, by a converter, the first abstract syntax tree to a second abstract syntax tree of the compiler for the second language using a conversion table from data representation types in the first language to data representation types in the second language. When a compilation error occurs, the implementing step also includes generating a special node for error processing in the second abstract syntax tree and storing an error token in the special node. When unparsing, the implementing step additionally includes outputting the error token, in the form of source code written in the first language.

97 MATHEMATICS AND COMPUTING↗

SHF: Symmetrical Hierarchical Forest with Pretrained Vision Transformer Encoder for High-Resolution Medical Segmentation

This paper presents a novel approach to addressing the long-sequence problem in high-resolution medical images for Vision Transformers (ViTs). Using smaller patches as tokens can enhance ViT performance, but quadratically increases computation and memory requirements. Therefore, the common practice for applying ViTs to high-resolution images is either to: (a) employ complex sub-quadratic attention schemes or (b) use large to medium-sized patches and rely on additional mechanisms within the model to capture the spatial hierarchy of details. We propose Symmetrical Hierarchical Forest (SHF), a lightweight approach that adaptively patches the input image to increase token information density and encode hierarchical spatial structures into the input embedding. We then apply a reverse depatching scheme to the output embeddings of the transformer encoder, eliminating the need for convolution-based decoders. Unlike previous methods that modify attention mechanisms or use a complex hierarchy of interacting models, SHF can be retrofitted to any ViT model to allow it to learn the hierarchical structure of details in high-resolution images without requiring architectural changes. Experimental results demonstrate significant gains in computational efficiency and performance: on the PAIP WSI dataset, we achieved a 3∼32×speedup or a 2.95%∼7.03% increase in accuracy (measured by Dice score) at a 64K2 resolution with the same computational budget, compared to state-of-the-art production models. On the 3D medical datasets BTCV and KiTS, training was 6×faster, with accuracy gains of 6.93% and 5.9%, respectively, compared to models without SHF.

Zhang, Enzhi [Hokkaido University, Japan]↗

Intern Poster

Large Language Models (LLMs) have skyrocketed in popularity after the release of ChatGPT in late 2022. Although LLMs are powerful tools, they can be subject to hallucinations, which is when an LLM (or any AI model) produces misleading/ nonsensical information. The objective is to determine if statistical methods can be used to detect hallucinations as an LLM generates its answer token by token (essentially word by word).

97 - MATHEMATICS AND COMPUTING↗

Hybrid Quantum–Classical Graph Transformers for Efficient Sentiment Analysis

Quantum Machine Learning (QML) offers a promising paradigm that leverages quantum computing principles to develop efficient and expressive models for learning from complex and structured data. Recent advances in natural language processing (NLP) and artificial intelligence (AI) have demonstrated capabilities in understanding, generating, and reasoning over linguistic and multimodal information. In this work, we present the Quantum Graph Transformer (QGT), a hybrid quantum–classical architecture that extends graph transformer capabilities through quantum self-attention. The QGT models variable-length sentences as token graphs, where both the embedding encoding and the self-attention mechanisms are implemented using parameterized quantum circuits (PQCs), enabling efficient contextual learning with significantly fewer trainable parameters. We train QGT using both fully connected and 𝑘 -nearest-neighbor graph structures and evaluate it on five benchmark sentiment-classification datasets. Experimental results show that QGT consistently achieves higher or comparable accuracy to existing quantum NLP models and outperforms a Classical Graph Transformer (CGT) baseline with identical architecture, achieving 29.4 × fewer parameters while requiring 3–5 × fewer samples to reach comparable performance. These findings highlight the potential of graph-based quantum models as scalable and data-efficient architectures for natural language understanding.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Cardiovascular and Interventional Radiological Society of Europe Commentary on the Treatment of Chronic Cerebrospinal Venous Insufficiency

Chronic cerebrospinal venous insufficiency (CCSVI) is a putative new theory that has been suggested by some to have a direct causative relation with the symptomatology associated with multiple sclerosis (MS). The core foundation of this theory is that there is abnormal venous drainage from the brain due to outflow obstruction in the draining jugular vein and/or azygos veins. This abnormal venous drainage, which is characterised by special ultrasound criteria, called the 'venous hemodynamic insufficiency severity score' (VHISS), is said to cause intracerebral flow disturbance or outflow problems that lead to periventricular deposits. In the CCSVI theory, these deposits have a great similarity to the iron deposits seen around the veins in the legs in patients with chronic deep vein thrombosis. Zamboni, who first described this new theory, has promoted balloon dilatation to treat the outflow problems, thereby curing CCSVI and by the same token alleviating MS complaints. However, this theory does not fit into the existing bulk of scientific data concerning the pathophysiology of MS. In contrast, there is increasing worldwide acceptance of CCSVI and the associated balloon dilatation treatment, even though there is no supporting scientific evidence. Furthermore, most of the information we have comes from one source only. The treatment is called 'liberation treatment,' and the results of the treatment can be watched on YouTube. There are well-documented testimonies by MS patients who have gained improvement in their personal quality of life (QOL) after treatment. However, there are no data available from patients who underwent unsuccessful treatments with which to obtain a more balanced view. The current forum for the reporting of success in treating CCSVI and thus MS seems to be the Internet. At the CIRCE office and the MS Centre in Amsterdam, we receive approximately 10 to 20 inquiries a month about this treatment. In addition, many interventional radiologists, who are directly approached by MS patients, contact the Cardiovascular and Interventional Radiological Society of Europe (CIRSE) for advice. Worldwide, several centres are actively promoting and performing balloon dilatation, with or without stenting, for CCSVI. Thus far, no trial data are available, and there is currently no randomized controlled trial (RCT) in progress Therefore, the basis for this new treatment rests on anecdotal evidence and successful testimonies by patients on the Internet. CIRSE believes that this is not a sound basis on which to offer a new treatment, which could have possible procedure-related complications, to an often desperate patient population.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Integrating Natural Language Processing and Machine Learning Algorithms to Categorize Oncologic Response in Radiology Reports

A significant volume of medical data remains unstructured. Natural language processing (NLP) and machine learning (ML) techniques have shown to successfully extract insights from radiology reports. However, the codependent effects of NLP and ML in this context have not been well-studied. Between April 1, 2015 and November 1, 2016, 9418 cross-sectional abdomen/pelvis CT and MR examinations containing our internal structured reporting element for cancer were separated into four categories: Progression, Stable Disease, Improvement, or No Cancer. We combined each of three NLP techniques with five ML algorithms to predict the assigned label using the unstructured report text and compared the performance of each combination. The three NLP algorithms included term frequency-inverse document frequency (TF-IDF), term frequency weighting (TF), and 16-bit feature hashing. The ML algorithms included logistic regression (LR), random decision forest (RDF), one-vs-all support vector machine (SVM), one-vs-all Bayes point machine (BPM), and fully connected neural network (NN). The best-performing NLP model consisted of tokenized unigrams and bigrams with TF-IDF. Increasing N-gram length yielded little to no added benefit for most ML algorithms. With all parameters optimized, SVM had the best performance on the test dataset, with 90.6 average accuracy and F score of 0.813. The interplay between ML and NLP algorithms and their effect on interpretation accuracy is complex. The best accuracy is achieved when both algorithms are optimized concurrently.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Uncertainty in Resilience to Climate Change in India and Indian States

This study builds on an earlier analysis of resilience of India and Indian states to climate change. The previous study (Brenkert and Malone 2005) assessed current resilience; this research uses the Vulnerability-Resilience Indicators Model (VRIM) to project resilience to 2095 and to perform an uncertainty analysis on the deterministic results. Projections utilized two SRES-based scenarios, one with fast-and-high growth, one with delayed growth. A detailed comparison of two states, the Punjab and Orissa, points to the kinds of insights that can be obtained using the VRIM. The scenarios differ most significantly in the timing of the uncertainty in economic prosperity (represented by GDP per capita) as a major factor in explaining the uncertainty in the resilience index. In the fast-and-high growth scenario the states differ most markedly regarding the role of ecosystem sensitivity, land use and water availability. The uncertainty analysis shows, for example, that resilience in the Punjab might be enhanced, especially in the delayed growth scenario, if early attention is paid to the impact of ecosystems sensitivity on environmental well-being of the state. By the same token, later in the century land-use pressures might be avoided if land is managed through intensification rather than extensification of agricultural land. Thus, this methodology illustrates how a policy maker can be informed about where to focus attention on specific issues, by understanding the potential changes at a specific location and time – and, thus, what might yield desired outcomes. Model results can point to further analyses of the potential for resilience-building.

54 ENVIRONMENTAL SCIENCES↗

Deterministic Walks with Choice

This paper studies deterministic movement over toroidal grids, integrating local information, bounded memory and choice at individual nodes. The research is motivated by recent work on deterministic random walks, and applications in multi-agent systems. Several results regarding passing tokens through toroidal grids are discussed, as well as some open questions.

Cayley graphs↗

Peak2Patch: High-Fidelity Functional Group Identification through Attention-Based Fusion of Infrared and Mass Spectra

Identifying molecular structure based on spectroscopic readings is a key task in a variety of chemical and biological applications. Common spectroscopy techniques, such as Infrared (IR) Spectroscopy and Mass Spectrometry (MS), provide detailed information on the structure of molecular compounds but nonetheless require expert-level knowledge to decode. Machine learning has emerged as a potential solution for automating structure prediction from chemical spectra; however, current approaches generally focus on single sensor modalities, neglecting to leverage the complementary information contained within differing spectra. In this paper, we introduce Peak2Patch, a novel approach to fusion-enhanced prediction of functional groups from IR and mass spectra. First, we perform a detailed comparison of backbone networks for encoding both sparse mass spectra and dense IR spectra and demonstrate the superior performance of transformer neural networks over current state-of-the-art convolutional neural networks. Second, we evaluate three broad categories of fusion: early (raw feature), middle (deep feature), and late (decision) fusion, demonstrating the potential of a deep feature fusion-based approach. Lastly, we present Peak2Patch, our attention-based fusion scheme, which leverages cross-attention to mix features between encoded tokens of the two modalities. We validate our approach on a publicly available multimodal spectroscopic data set of 790k simulated molecules, demonstrating a large improvement in functional group prediction over both the previous state-of-the-art and our own strong single-modal baselines.

Jacobson, Philip [Sandia National Laboratories (SN↗

RNA language models predict mutations that improve RNA function

Structured RNA lies at the heart of many central biological processes, from gene expression to catalysis. RNA structure prediction is not yet possible due to a lack of high-quality reference data associated with organismal phenotypes that could inform RNA function. We present GARNET (Gtdb Acquired RNa with Environmental Temperatures), a new database for RNA structural and functional analysis anchored to the Genome Taxonomy Database (GTDB). GARNET links RNA sequences to experimental and predicted optimal growth temperatures of GTDB reference organisms. Using GARNET, we develop sequence- and structure-aware RNA generative models, with overlapping triplet tokenization providing optimal encoding for a GPT-like model. Leveraging hyperthermophilic RNAs in GARNET and these RNA generative models, we identify mutations in ribosomal RNA that confer increased thermostability to the Escherichia coli ribosome. The GTDB-derived data and deep learning models presented here provide a foundation for understanding the connections between RNA sequence, structure, and function.

59 BASIC BIOLOGICAL SCIENCES↗

TransPolymer: a Transformer-based language model for polymer property predictions

Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Recently, Transformer models, equipped with self-attention mechanisms, have exhibited superior performance in natural language processing. However, such methods have not been investigated in polymer sciences. Herein, we report TransPolymer, a Transformer-based language model for polymer property prediction. Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences. Rigorous experiments on ten polymer property prediction benchmarks demonstrate the superior performance of TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on large unlabeled dataset via Masked Language Modeling. Experimental results further manifest the important role of self-attention in modeling polymer sequences. We highlight this model as a promising computational tool for promoting rational polymer design and understanding structure-property relationships from a data science view.

36 MATERIALS SCIENCE↗

ATAT: Astronomical Transformer for time series and Tabular data

Context. The advent of next-generation survey instruments, such as theVera C. RubinObservatory and its Legacy Survey of Space and Time (LSST), is opening a window for new research in time-domain astronomy. The Extended LSST Astronomical Time-Series Classification Challenge (ELAsTiCC) was created to test the capacity of brokers to deal with a simulated LSST stream. Aims. Our aim is to develop a next-generation model for the classification of variable astronomical objects. We describe ATAT, the Astronomical Transformer for time series And Tabular data, a classification model conceived by the ALeRCE alert broker to classify light curves from next-generation alert streams. ATAT was tested in production during the first round of the ELAsTiCC campaigns. Methods. ATAT consists of two transformer models that encode light curves and features using novel time modulation and quantile feature tokenizer mechanisms, respectively. ATAT was trained on different combinations of light curves, metadata, and features calculated over the light curves. We compare ATAT against the current ALeRCE classifier, a balanced hierarchical random forest (BHRF) trained on human-engineered features derived from light curves and metadata. Results. When trained on light curves and metadata, ATAT achieves a macro F1 score of 82.9 ± 0.4 in 20 classes, outperforming the BHRF model trained on 429 features, which achieves a macro F1 score of 79.4 ± 0.1. Conclusions. The use of transformer multimodal architectures, combining light curves and tabular data, opens new possibilities for classifying alerts from a new generation of large etendue telescopes, such as theVera C. RubinObservatory, in real-world brokering scenarios.

Astronomy & Astrophysics↗

Third-party transfers in WLCG using HTTP

Since its earliest days, the Worldwide LHC Computational Grid (WLCG) has relied on GridFTP to transfer data between sites. The announcement that Globus is dropping support of its open source Globus Toolkit (GT), which forms the basis for several FTP client and servers, has created an opportunity to reevaluate the use of FTP. HTTP-TPC, an extension to HTTP compatible with WebDAV, has arisen as a strong contender for an alternative approach.In this paper, we describe the HTTP-TPC protocol itself, along with the current status of its support in different implementations, and the interoperability testing done within the WLCG DOMA working group’s TPC activity. This protocol also provides the first real use-case for token-based authorisation for this community. We will demonstrate the benefits of such authorisation by showing how it allows HTTP-TPC to support new technologies (such as OAuth, OpenID Connect, Macaroons and SciTokens) without changing the protocol. We will also discuss the next steps for HTTP-TPC and the plans to use the protocol for WLCG transfers.

Bockelman, Brian↗

XRootD Third Party Copy for the WLCG and HLLHC

A Third Party Copy (TPC) mechanism has existed in the pure XRootD storage environment for many years. However, using the XRootD TPC in the WLCG environment presents additional challenges due to the diversity of the storage systems involved such as EOS, dCache, DPM and ECHO, requiring that we carefully navigate the unique constraints imposed by these storage systems and their site-specific environments through customized configuration and software development. To support multitenant setups seen at many WLCG sites, X509 based authentication and authorization in XRootD was significantly improved to meet both security and functionality requirements. This paper presents architecture of the pull based TPC with optional X509 credential delegation, and how it is implemented in native XRootD and dCache. The paper discusses technical requirements, challenges, design choices and implementation details in the WLCG storage systems, as well as in FTS/gfal2. It also outlines XRootD’s plan to support newer TPC and security models such as token based authorization.

Adye, T.↗

DIRAC current, upcoming and planned capabilities and technologies

DIRAC is the interware for building and operating large scale distributed computing systems. It is adopted by multiple collaborations from various scientific domains for implementing their computing models. DIRAC provides a framework and a rich set of ready-to-use services for Workload, Data and Production Management tasks of small, medium and large scientific communities having different computing requirements. The base functionality can be easily extended by custom components supporting community specific workflows. DIRAC is at the same time an aging project, and a new DiracX project is taking shape for replacing DIRAC in the long term. This contribution will highlight DIRAC’s current, upcoming and planned capabilities and technologies, and how the transition to DiracX will take place. Examples include, but are not limited to, adoption of security tokens and interactions with Identity Provider services, integration of Clouds and High Performance Computers, interface with Rucio, improved monitoring and deployment procedures.

97 MATHEMATICS AND COMPUTING↗

dCache project status and update

The dCache project delivers an open-source, massively scalable, distributed storage system deployed internationally to satisfy today’s scientists’ ever-demanding storage requirements. Its multifaceted approach supports different use cases with the same storage, from high throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and longterm data persistence on tertiary storage. Even though dCache was initially developed for HEP experiments, today, it is used by various scientific communities, including astrophysics, biomed, and life science, each with their specific requirements. To match the needs of these new communities and keep up with the scaling demands of existing experiments, dCache is permanently evolving. With this contribution, we would like to highlight the recent developments in dCache regarding integration with CERN Tape Archive (CTA), advanced metadata handling, token-based authorization support, bulk API for QoS transitions, REST API to control interaction with the tape system, and future development directions.

Mkrtchyan, Tigran [DESY]↗

H I 21 cm ABSORPTION AND UNIFIED SCHEMES OF ACTIVE GALACTIC NUCLEI

In a recent study of z >= 0.1 active galactic nuclei (AGNs), we found that 21 cm absorption has never been detected in objects in which the ultraviolet luminosity exceeds L{sub UV} {approx} 10{sup 23} W Hz{sup -1}. In this paper, we further explore the implications that this has for the currently popular consensus that it is the orientation of the circumnuclear obscuring torus, invoked by unified schemes of AGNs, which determines whether absorption is present along our sight line. The fact that at L{sub UV} {approx}< 10{sup 23} W Hz{sup -1}, both type-1 and type-2 objects exhibit a 50% probability of detection, suggests that this is not the case and that the bias against detection of H I absorption in type-1 objects is due purely to the inclusion of the L{sub UV} {approx}> 10{sup 23} W Hz{sup -1} sources. Similarly, the ultraviolet luminosities can also explain why the presence of 21 cm absorption shows a preference for radio galaxies over quasars and the higher detection rate in compact sources, such as compact steep spectrum or gigahertz peaked spectrum sources, may also be biased by the inclusion of high-luminosity sources. Being comprised of all 21 cm searched sources at z >= 0.1, this is a necessarily heterogeneous sample, the constituents of which have been observed by various instruments. By this same token, however, the dependence on the UV luminosity may be an all encompassing effect, superseding the unified schemes model, although there is the possibility that the exclusive 21 cm non-detections at high UV luminosities could be caused by a bias toward gas-poor ellipticals. Additionally, the high UV fluxes could be sufficiently exciting/ionizing the H I above 21 cm detection thresholds, although the extent to which this is related to the neutral gas deficit in ellipticals is currently unclear. Examining the moderate UV luminosity (L{sub UV} {approx}< 10{sup 23} W Hz{sup -1}) sample further, from the profile widths and offsets from the systemic velocities, we find no discernible differences between the two AGN types. This may suggest that the bulk of the absorption generally occurs in the galactic disk, which must therefore be randomly orientated with respect to the circumnuclear torus. Furthermore, we see no difference in the reddening between the two AGN types, indicating, like the 21 cm absorption, that the orientation of the torus has little bearing on this. We also find a correlation between 21 cm line strength and the optical-near-infrared color, which suggests that the reddening is caused by dust located in the large-scale, H I absorbing disk which intervenes the sight line to the AGN.

79 ASTRONOMY AND ASTROPHYSICS↗

INTERSTELLAR SOLID HYDROGEN

We consider the possibility that solid molecular hydrogen is present in interstellar space. If so cosmic-rays and energetic photons cause ionization in the solid leading to the formation of H{sup +}{sub 6}. This ion is not produced by gas-phase reactions and its radiative transitions therefore provide a signature of solid H{sub 2} in the astrophysical context. The vibrational transitions of H{sup +}{sub 6} are yet to be observed in the laboratory, but we have characterized them in a quantum-theoretical treatment of the molecule; our calculations include anharmonic corrections, which are large. Here we report on those calculations and compare our results with astronomical data. In addition to the H{sup +}{sub 6} isotopomer, we focus on the deuterated species (HD){sup +}{sub 3} which is expected to dominate at low ionization rates as a result of isotopic condensation reactions. We can reliably predict the frequencies of the fundamental bands for five modes of vibration. For (HD){sup +}{sub 3} all of these are found to lie close to some of the strongest of the pervasive mid-infrared astronomical emission bands, making it difficult to exclude hydrogen precipitates on observational grounds. By the same token these results suggest that (HD){sup +}{sub 3} could be the carrier of the observed bands. We consider this possibility within the broader picture of interstellar medium photo-processes and we conclude that solid hydrogen may indeed be abundant in astrophysical environments.

79 ASTRONOMY AND ASTROPHYSICS↗