Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Word Embeddings”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

9 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Explaining word embeddings with perfect fidelity: a case study in predicting research impact

The best-performing approaches for scholarly document quality prediction are based on embedding models. In addition to their performance when used in classifiers, embedding models can also provide predictions even for words that were not contained in the labelled training data for the classification model, which is important in the context of the ever-evolving research terminology. Although model-agnostic explanation methods, such as Local interpretable model-agnostic explanations, can be applied to explain machine learning classifiers trained on embedding models, these produce results with questionable correspondence to the model. We introduce a new feature importance method, Self-Model Entities Rated (SMER), for logistic regression-based classification models trained on word embeddings. We show that SMER has theoretically perfect fidelity with the explained model, as the average of logits of SMER scores for individual words (SMER explanation) exactly corresponds to the logit of the prediction of the explained model. Quantitative and qualitative evaluation is performed through five diverse experiments conducted on 50,000 research articles (papers) from the CORD-19 corpus. In conclusion, through an AOPC curve analysis, we experimentally demonstrate that SMER produces better explanations than LIME, SHAP and global tree surrogates.

Coarse-grained models↗
DOE OSTI2026

Evolving language of pediatric anxiety in electronic health records

Objectives This study aimed to identify and quantify semantic drift (ie, the change in semantic meaning over time) within expert-defined anxiety-related (AR) terminology and compare it to common electronic health record (EHR) vocabulary across longitudinal pediatric clinical notes. Materials and Methods A corpus of pediatric clinical notes from 2009 to 2022 was analyzed using computational methods. Semantic drift for each term was quantified using cosine similarity between annual temporal word embeddings. Contextual meaning was examined through changes in nearest neighbors across years. The Laws of Semantic Change were applied to assess the influence of word frequency and polysemy. Vocabulary terms were categorized as AR or common EHR. Results 98% of AR terminology maintained a cosine similarity between 0.00 and 0.50, indicating moderate semantic stability, whereas 90% of common EHR terms remained between 0.00 and 0.25, showing greater contextual stability overall. Frequent terms exhibited minimal change (Frequency Coefficient = 0.04), whereas highly polysemous or abbreviated terms showed less stability (Polysemy Coefficient = 0.630). AR terminology drifted more slowly than general EHR vocabulary (Type Coefficient = −0.179), further supported by significant year–type interactions (Coef = −0.09 to −0.523). Discussion Although anxiety-related terminology demonstrates slower semantic drift than general EHR vocabulary, subtle contextual shifts still occur that may affect downstream interpretability and retrieval in automated systems. Conclusion Continuous linguistic monitoring and adaptive modeling are essential to maintain semantic fidelity and ensure the long-term reliability of clinical decision support systems as healthcare documentation evolves.

Pediatric anxiety disorders↗
DOE OSTI2025

An unsupervised machine learning based approach to identify efficient spin-orbit torque materials

Materials with large spin–orbit torque (SOT) hold considerable significance for many spintronic applications because of their potential for energy-efficient magnetization switching. Unfortunately, most of the existing materials exhibit an SOT efficiency factor that is much less than unity, requiring a large current for magnetization switching. The search for new materials that can exhibit an SOT efficiency much greater than unity is a topic of active research, and only a few such materials have been identified using conventional approaches. In this paper, we present a machine learning-based approach using a word embedding model that can identify new results by deciphering non-trivial correlations among various items in a specialized scientific text corpus. We show that such a model can be used to identify materials likely to exhibit high SOT and rank them according to their expected SOT strengths. The model captured the essential spintronics knowledge embedded in scientific abstracts within various materials science, physics, and engineering journals and identified 97 new materials to exhibit high SOT. Among them, 16 candidate materials are expected to exhibit an SOT efficiency greater than unity, and one of them has recently been confirmed with experiments with quantitative agreement with the model prediction.

Sayed, Shehrin↗
DOE OSTI2025

Visualizing Temporal Topic Embeddings with a Compass

—Dynamic topic modeling is useful at discovering the development and change in latent topics over time. However, present methodology relies on algorithms that separate document and word representations. This prevents the creation of a meaningful embedding space where changes in word usage and documents can be directly analyzed in a temporal context. This paper proposes an expansion of the compass-aligned temporal Word2Vec methodology into dynamic topic modeling. Such a method allows for the direct comparison of word and document embeddings across time in dynamic topics. This enables the creation of visualizations that incorporate temporal word embeddings within the context of documents into topic visualizations. In experiments against the current state-of-the-art, our proposed method demonstrates overall competitive performance in topic relevancy and diversity across temporal datasets of varying size. Simultaneously, it provides insightful visualizations focused on temporal word embeddings while maintaining the insights provided by global topic evolution, advancing our understanding of how topics evolve over time.

Cluster analysis↗
DOE OSTI2025

Replacing non-biomedical concepts improves embedding of biomedical concepts

Embeddings are semantically meaningful representations of words in a vector space, commonly used to enhance downstream machine learning applications. Traditional biomedical embedding techniques often replace all synonymous words representing biological or medical concepts with a unique token, ensuring consistent representation and improving embedding quality. However, the potential impact of replacing non-biomedical concept synonyms has received less attention. Embedding approaches often employ concept replacement to replace concepts that span multiple words, such as non-small-cell lung carcinoma, with a single concept identifier (e.g., D002289). Also, all synonyms of each concept are merged into the same identifier. Here, we additionally leveraged WordNet to identify and replace sets of non-biomedical synonyms with their most common representatives. This combined approach aimed to reduce embedding noise from non-biomedical terms while preserving the integrity of biomedical concept representations. We applied this method to 1,055 biomedical concept sets representing molecular signatures or medical categories and assessed the mean pairwise distance of embeddings with and without non-biomedical synonym replacement. A smaller mean pairwise distance was interpreted as greater intra-cluster coherence and higher embedding quality. Embeddings were generated using the Word2Vec algorithm applied to a corpus of 10 million PubMed abstracts. Our results demonstrate that the addition of non-biomedical synonym replacement reduced the mean intra-cluster distance by an average of 8%, suggesting that this complementary approach enhances embedding quality. Future work will assess its applicability to other embedding techniques and downstream tasks. Python code implementing this method is provided under an open-source license.

algorithms↗
DOE OSTI2025

AI-powered topic modeling: comparing LDA and BERTopic in analyzing opioid-related cardiovascular risks in women

Topic modeling is a crucial technique in natural language processing (NLP), enabling the extraction of latent themes from large text corpora. Traditional topic modeling, such as Latent Dirichlet Allocation (LDA), faces limitations in capturing the semantic relationships in the text document although it has been widely applied in text mining. BERTopic, created in 2022, leveraged advances in deep learning and can capture the contextual relationships between words. In this work, we integrated Artificial Intelligence (AI) modules to LDA and BERTopic and provided a comprehensive comparison on the analysis of prescription opioid-related cardiovascular risks in women. Opioid use can increase the risk of cardiovascular problems in women such as arrhythmia, hypotension etc. 1,837 abstracts were retrieved and downloaded from PubMed as of April 2024 using three Medical Subject Headings (MeSH) words: “opioid,” “cardiovascular,” and “women.” Machine Learning of Language Toolkit (MALLET) was employed for the implementation of LDA. BioBERT was used for document embedding in BERTopic. Eighteen was selected as the optimal topic number for MALLET and 23 for BERTopic. ChatGPT-4-Turbo was integrated to interpret and compare the results. The short descriptions created by ChatGPT for each topic from LDA and BERTopic were highly correlated, and the performance accuracies of LDA and BERTopic were similar as determined by expert manual reviews of the abstracts grouped by their predominant topics. The results of the t-SNE (t-distributed Stochastic Neighbor Embedding) plots showed that the clusters created from BERTopic were more compact and well-separated, representing improved coherence and distinctiveness between the topics. Our findings indicated that AI algorithms could augment both traditional and contemporary topic modeling techniques. In addition, BERTopic has the connection port for ChatGPT-4-Turbo or other large language models in its algorithm for automatic interpretation, while with LDA interpretation must be manually, and needs special procedures for data pre-processing and stop words exclusion. Therefore, while LDA remains valuable for large-scale text analysis with resource constraints, AI-assisted BERTopic offers significant advantages in providing the enhanced interpretability and the improved semantic coherence for extracting valuable insights from textual data.

Research & Experimental Medicine↗
DOE OSTI2024

Development of message passing-based graph convolutional networks for classifying cancer pathology reports

Abstract Background Applying graph convolutional networks (GCN) to the classification of free-form natural language texts leveraged by graph-of-words features (TextGCN) was studied and confirmed to be an effective means of describing complex natural language texts. However, the text classification models based on the TextGCN possess weaknesses in terms of memory consumption and model dissemination and distribution. In this paper, we present a fast message passing network (FastMPN), implementing a GCN with message passing architecture that provides versatility and flexibility by allowing trainable node embedding and edge weights, helping the GCN model find the better solution. We applied the FastMPN model to the task of clinical information extraction from cancer pathology reports, extracting the following six properties: main site, subsite, laterality, histology, behavior, and grade. Results We evaluated the clinical task performance of the FastMPN models in terms of micro- and macro-averaged F1 scores. A comparison was performed with the multi-task convolutional neural network (MT-CNN) model. Results show that the FastMPN model is equivalent to or better than the MT-CNN. Conclusions Our implementation revealed that our FastMPN model, which is based on the PyTorch platform, can train a large corpus (667,290 training samples) with 202,373 unique words in less than 3 minutes per epoch using one NVIDIA V100 hardware accelerator. Our experiments demonstrated that using this implementation, the clinical task performance scores of information extraction related to tumors from cancer pathology reports were highly competitive.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Signature analysis of high-throughput transcriptomics screening data for mechanistic inference and chemical grouping

Abstract High-throughput transcriptomics (HTTr) uses gene expression profiling to characterize the biological activity of chemicals in in vitro cell-based test systems. As an extension of a previous study testing 44 chemicals, HTTr was used to screen an additional 1,751 unique chemicals from the EPA’s ToxCast collection in MCF7 cells using 8 concentrations and an exposure duration of 6 h. We hypothesized that concentration-response modeling of signature scores could be used to identify putative molecular targets and cluster chemicals with similar bioactivity. Clustering and enrichment analyses were conducted based on signature catalog annotations and ToxPrint chemotypes to facilitate molecular target prediction and grouping of chemicals with similar bioactivity profiles. Enrichment analysis based on signature catalog annotation identified known mechanisms of action (MeOAs) associated with well-studied chemicals and generated putative MeOAs for other active chemicals. Chemicals with predicted MeOAs included those targeting estrogen receptor (ER), glucocorticoid receptor (GR), retinoic acid receptor (RAR), the NRF2/KEAP/ARE pathway, AP-1 activation, and others. Using reference chemicals for ER modulation, the study demonstrated that HTTr in MCF7 cells was able to stratify chemicals in terms of agonist potency, distinguish ER agonists from antagonists, and cluster chemicals with similar activities as predicted by the ToxCast ER Pathway model. Uniform manifold approximation and projection (UMAP) embedding of signature-level results identified novel ER modulators with no ToxCast ER Pathway model predictions. Finally, UMAP combined with ToxPrint chemotype enrichment was used to explore the biological activity of structurally related chemicals. The study demonstrates that HTTr can be used to inform chemical risk assessment by determining in vitro points of departure, predicting chemicals’ MeOA and grouping chemicals with similar bioactivity profiles.

Toxicology↗
DOE OSTI2025

Multimetallic Layered Composites (MMLCs) for Rapid, Economical Advanced Reactor Deployment (Final Report)

This project focused on the development of multi-metallic layered composites (MMLCs) for advanced fission reactor technologies. There are many instances where one alloy or material simply cannot meet all the demands thrown at it by a reactor system, or cannot allow it to perform as strongly as one would like. Instead of focusing all our effort on developing one perfect alloy, we seek to leverage the design principle of “separation of functionality,” used in many other arenas in design, to boost performance beyond single alloys alone. One illustrative example shows the power of this approach for molten salt-cooled reactors: A three meter tall, three meter diameter reactor vessel made of Incoloy 800 was quoted at $\$$500k in 2018. A Hastelloy N vessel was quoted at $\$$5M. An MMLC vessel, in which a layer of Hastelloy N would be weld-overlaid onto Incoloy 800, was quoted at $\$$700k, and it would achieve the same performance. The potential economic gains of leveraging this approach are therefore substantial. At a minimum, each MMLC would contain one core structural layer and one coolant-facing corrosion-resistant layer. Sometimes, MMLCs required buffer layers, as the structural and corrosion-resistant layers were metallurgically incompatible. In other words, they didn’t always play nice, thus separating layers compatible with both functioned as intermediaries to keep the composite together. However, in doing so we inevitably produce new interfaces, where new issues can arise. Therefore, this project focused on what happens at these interfaces from a combination of high temperatures, irradiation, corrosion, and time. After all, a reactor makes money when it is operating, and outages of any kind erode its economic viability. First, we set out to experimentally prove that MMLCs for at least two advanced reactor systems can be made, today, in US domestic facilities. In this respect we were successful – one MMLC (a Ni-201/Incoloy 800H composite) was successfully made and drawn into two-inch coolant piping. Others were attempted, though new issues relating to cracking in vanadium layers for one and radiation damage performance of the corrosion-resistant layer in another prevented us from moving further in those specific arenas – these are engineering problems which deserve continued focus after this project. Additional experimental work focused on long-term corrosion testing of the outermost layers of the salt-cooled and liquid lead-cooled MMLC concepts, which would then be fed into predictions of how long the MMLCs could last. Next, computational (thermodynamics and atomistic) simulation studies studied how much we expect the interfaces to “blend,” due to the mixing action of neutron irradiation. This eats into both the margin for the structural layer of each MMLC, as dilution from the corrosion-resistant layer into the structural layer would decrease the total load-bearing capacity of an MMLC of finite size. On the other hand, dilution of the corrosion-resistant layer into the structural layer further reduced the margin of corrodible material, reducing the lifetime of the MMLC or necessitating extra thickness to be imparted to the MMLC to meet its functional requirements. Work here focused on irradiation-induced segregation to predict new phases which may embrittle the MMLCs, as well as quantifying irradiation-induced mixing at each interface. The results showed that mixing is expected, but it is both steady and therefore predictable, and not lifetime-limiting for most MMLC concepts – it simply has to be accounted for in calculations of reactor performance when utilizing an MMLC. Then, full-core simulations using the experimentally-derived corrosion data, the computationally discovered irradiation-induced mixing data (partially validated by experiment), and existing, benchmarked core designs for large and small sized reactor concepts (one salt-cooled, one lead-cooled) were conducted to quantify any expansion of reactor operating envelopes achieved by utilizing these MMLCs. This new framework, called REX (Reactor Envelope Expansion), incorporates a combination of core neutronics, thermal hydraulics, and the material performance data derived from this project to see how using an MMLC expands advanced fission reactor operating envelopes. It was discovered that in some cases, MMLC utilization does indeed increase the maximum operating temperatures and cycle lengths of reactor concepts, while in other cases it does not. Finally, our tech-to-market (T2M) strategy was not necessarily to create specific embodiments of MMLCs for immediate sale (because getting into the nuclear market is incredibly slow and laden with regulation, this is a long-term goal), but rather immediate stimulation of US industry using the design approach of MMLCs derived from this project. In this respect we were successful, as one of the PhD students funded on this project co-founded Allium Engineering, Inc., which created a stainless steel / low-alloy steel MMLC to function as chloride corrosion-resistant rebar for embedding into concrete structures. Allium Engineering continues to be successful, having recently opened their first factory as of this writing.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗