Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “BERT”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

SU-E-T-221: Computational Evaluation of Proton Induced Gain in a Portable Faraday Cup

Purpose: The design of a new Portable Faraday Cup (PFC) used to calibrate proton accelerators was evaluated for energies between 50 and 220 MeV. Monte Carlo simulations performed in Geant4–10.0 were used to evaluate experimental results and reduce the relative detector error for this vacuum-less and low mass system, and invalidate current MCNP releases. Methods: The detector construction consisted of a copper conductor coated with an insulator and grounded with silver. Monte Carlo calculations in Geant4 were used to determine the net charge per proton input (gain) as a function of insulator thickness and beam energy. Kapton was chosen as the insulating material and was designed to capture backscattered electrons. Charge displacement from/into Kapton was assumed to follow a linear proportionality to the origin/terminus depth toward the outer ground layer. Kapton thicknesses ranged from 0 to 200 microns, proton energies were set to match empirical studies ranging from 70 to 250 MeV. Each setup was averaged over 1 million events using the FTFP-BERT 2.0 physics list. Results: With increasing proton energy, the gain of Cu+KA gradually converges to the limit of pure copper, with relative error between 1.52% and 0.72%. The Ag layer created a more diverging behavior, accelerating the flux of negative charge into the device and increasing relative error when compared to pure copper from 1.21% to 1.63%. Conclusion: Gain vs. beam energy signatures were acquired for each device. Further analysis reveals proportionality between insulator thickness and measured gain, albeit an inverse proportionality between beam energy and in-flux of electrons. Increased silver grounding layer thickness also decreases gain, though the relative error expands with beam energy, contrary to the Kapton layer.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Measurements of higher-order cumulants of multiplicity and net-electric charge distributions in inelastic proton-proton interactions by NA61/SHINE

This paper presents the energy dependence of multiplicity and net-electric charge fluctuations in p+p interactions at beam momenta 20, 31, 40, 80, and 158 \(\text{ Ge }\hspace{-1.00006pt}\text{ V }\!/\!c\). Results are corrected for the experimental biases and quantified with the use of cumulants and factorial cumulants. Cumulant ratios are an essential tool in the search for the critical point of strongly interacting matter in heavy ion collisions. Measurements performed in p+p interactions provide a vital baseline estimation in these studies. The measured signals are compared with the string hadronic models Epos1.99 and FTFP-BERT.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

pnnl/TransMED

Transfer Learning From Existing Diseases Via Hierarchical Multi-Modal BERT Models

Ham, Colby↗

PRIME: Protein Representation Inference for Mutation Evaluation

Protein language machine learning models built upon existing ESM-2 model developed by Evolutionary Scale (evolutionaryscale.ai) and an in-house protein language model based on the BERT model developed by Google. The code also includes model training scripts and saved checkpoints from our own training using publicly available SARS-CoV-2 protein sequences.

Gibson, Kaetlyn [Los Alamos National Lab]↗

A knowledge-informed large language model framework for U.S. nuclear power plant shutdown initiating event classification for probabilistic risk assessment

Identifying and classifying shutdown initiating events (SDIEs) is critical for developing shutdown probabilistic risk assessment for nuclear power plants. Existing computational approaches cannot achieve satisfactory performance due to the challenges of unavailable large, labeled datasets, imbalanced event types, and label noise. To address these challenges, we propose a hybrid pipeline that integrates a knowledge-informed machine learning model to prescreen non-SDIEs and a large language model (LLM) to classify SDIEs into four types. In the prescreening stage, we proposed a set of 44 SDIE text patterns that consist of the most salient keywords and phrases from six SDIE types. Text vectorization based on the SDIE patterns generates feature vectors that are highly separable by using a simple binary classifier. The second stage builds Bidirectional Encoder Representations from Transformers (BERT)-based LLM, which learns generic English language representations from self-supervised pretraining on a large dataset and adapts to SDIE classification by fine-tuning it on an SDIE dataset. The proposed approaches are evaluated on a dataset with 10,928 events using precision, recall ratio, F 1 score, and average accuracy. In conclusion, the results demonstrate that the prescreening stage can exclude more than 97% non-SDIEs, and the LLM achieves an average accuracy of 95.1% for SDIE classification.

99 - GENERAL AND MISCELLANEOUS↗

Evaluating Deception Detection Model Robustness To Linguistic Variation

With the increasing use of automated, machine learning-driven tools and the downstream impact that algorithmic judgements can have, it is critical to develop models that are robust to evolving or manipulated inputs. Evaluating the reliability of multimodal models across linguistic variations to understand model susceptibility to intentional linguistic adversarial attacks as well as natural linguistic variations is essential in this pursuit. We present extensive analysis of model robustness and susceptibility to linguistic variations in the setting of deceptive news detection, a difficult classification task that is an increasingly important problem to solve with the impact of misinformation spread online. We evaluate the effectiveness of incorporating adversarial defense strategies and measure model susceptibility to state-of-the-art adversarial attacks using two types of linguistic attacks — character and word perturbations. We consider two multiclass prediction tasks — a 3-way classification of tweets as trustworthy, propaganda, or disinformation; and a 4-way classification as clickbait, hoax, satire, or conspiracy — and compare the performance of three embeddings that have been state-of-the-art for several NLP tasks — GloVe, ELMo, and BERT — to highlight consistent trends in susceptibility, high confidence misclassifications, and high impact failures. We find that character or mixed ensemble models are the most effective defense mechanisms and that character perturbations are a more effective attack than word perturbations for deception classification.

adversarial evaluation↗

Learning to Rank in the Age of Muppets: Effectiveness–Efficiency Tradeoffs in Multi-Stage Ranking

It is well known that rerankers built on pretrained transformer models such as BERT have dramatically improved retrieval effectiveness in many tasks. However, these gains have come at substantial costs in terms of efficiency, as noted by many researchers. In this work, we show that it is possible to retain the benefits of transformer-based rerankers in a multi-stage reranking pipeline by first using feature-based learning-to-rank techniques to reduce the number of candidate documents under consideration without adversely affecting their quality in terms of recall. Applied to the MS MARCO passage and document ranking tasks, we are able to achieve the same level of effectiveness, but with up to 18 increase in efficiency. Furthermore, our techniques are orthogonal to other methods focused on accelerating transformer inference, and thus can be combined for even greater efficiency gains. A higher-level message from our work is that, even though pretrained transformers dominate the modern IR landscape, there are still important roles for “traditional” LTR techniques, and that we should not forget history.

catalysis↗

PRIMED for the Future: Purposing Raw Intake for Machine Learning-Enabled Detection (Final Report)

The COVID-19 pandemic demonstrated how a novel, elusive, and diffuse biological threat can engender uncertainty and misinformation, and it underscored the need for flexible analytical modalities agnostic to the identity of biological material. Yet even before the pandemic, recognition of the limitations of the current, list-based approach, which focuses on known pathogens and biotoxins, and of the importance of agent-agnostic biodetection was growing within the biosecurity community. In a 2018 report on “Biodefense in the Age of Synthetic Biology,” for example, the National Academy of Sciences stated that “an overreliance on the Select Agent List is a systematic weakness affecting many aspects of the United States’ current biodefense mitigation capability”. More recently, a group of biodefense researchers proposed the identification and adoption of “bioagent-agnostic signatures (BASs)” as a way of detecting and characterizing not only existing agents but also novel ones, an approach they believe will “enable a more flexible and resilient biodefense posture”. Indeed, the future of biodetection requires us to begin developing novel analytics that can identify anomalies and/or characteristics that indicate a potential threat, whether known or unknown, without looking for a specific signature that has been identified previously. To assess potential threats more rapidly, it is critical to develop agnostic artificial intelligence (AI)/machine learning (ML) systems that can be employed for real-time assessment of the nature and source of a perturbation. Such systems should be multi scale and multi-dimensional, integrating sensor data from a range of biological, chemical, and physical application spaces. Emerging deep learning (DL) models demonstrate exceptional promise for identification of discriminatory features within multi-dimensional datasets. DL models have the capacity to recognize and encode highly complex patterns in a wide range of input data modalities, including images, text, and biological/chemical/physical spectra. As such, they can execute a wide range of assessments and determinations that have traditionally required a human operator. The promise of advances in DL is apparent in the realm of human health and medicine. DL models have been validated for evaluating a variety of clinical threats to human health in a range of contexts, including infection and cancer, and they demonstrated improved performance in predicting stroke relative to human neurologists in some categories of data. Continuously evolving advances in AI/ML are expected to support more efficient evaluation of raw sequence, spectroscopy, and spectrometry data. For instance, recent advances and deployment of large language models (LLM) such as Generative Pre training Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT) have already motivated application of these models for biological function prediction. As frameworks such as LLMs become larger and more complex in their representations, their capacity to serve as pre-trained models that can be fine-tuned for biological/biodetection purposes will similarly be amplified. While existing and emerging AI/ML have found broad applicability and use cases in the clinical sciences, development for environmental evaluation and biodetection has been limited. Functionalizing such capabilities for this purpose requires an understanding of the existing technical landscape and how the respective tools and algorithms are currently being employed. This landscape awareness then allows an assessment of the current practical capabilities of existing models and the anticipated requirements and development efforts that will be needed to adapt available algorithms for biodetection applications relevant to DHS. Leveraging expertise in biodetection, ML, and operational biodetection, the effort described in this report is comprised of a systematic landscape assessment (Subtask 2.1), comparative evaluation (Subtask 2.2), and formulation of a value proposition (Subtask 2.3) for the prospect of ML-enabled, agnostic biodetection from raw, or minimally-processed, datasets.

59 BASIC BIOLOGICAL SCIENCES↗

Updates to the NuMI Flux Simulation at MicroBooNE (V.1.0)

The NuMI flux prediction being used by all experiments at Fermilab that are sensitive to the NuMI beam is based on Geant v4.9.2.03 (G4.9) and uses the default FTFP-BERT physics list with no custom cross sections. In order to correct these predictions to match world data, PPFX, the Package to Predict the FluX, is used to reweight the flux prediction. In kinematic regions of overlap between the prediction and data, mainly NA49 measurements are used to correct for various hadron production processes. Specifcally, PPFX looks at the entire ancestry chain of the neutrino, up to the incident 120 GeV proton beam, and applies corrections for various hadron interactions based on said data. These corrections are implemented as a function of the Feynman-x (x F ) and transverse momentum (p T ) of the incident hadron.

43 PARTICLE ACCELERATORS↗

Automating the Analysis of Large Language Models Responses through Zero-Shot Question Answering

Recent advancements in Large Language Models (LLMs) have shown significant potential in various applications, yet their evaluation, particularly in zero-shot question answering scenarios, remains a challenging task. In this study, our objective was to explore precision metrics for Large Language Models (LLM) and design and implement a software pipeline to automatically evaluate LLMs' outputs under zero-shot question answering. Zero-shot question answering involves a model providing answers to questions about topics it hasn't seen during training. It leverages the principles of zero-shot learning by relying on semantic understanding and generalization from related knowledge. The data used was metadata from medical databases on congenital heart disease. We explored eleven LLM metrics and selected three for our evaluation: BLEU, BERTScore, and MoverScore. BLEU calculates a score based on the overlap of n-grams (contiguous sequences of n items, typically words) between the machine-generated translation and the reference translations. Higher BLEU scores indicate better correspondence between the machine-generated and human-generated translations. BERTScore is a metric used to evaluate the quality of machine-generated text by measuring the similarity of token embeddings produced by BERT (Bidirectional Encoder Representations from Transformers) between the generated text and reference text. MoverScore is a metric that quantifies the dissimilarity between the distributions of word embeddings from machine-generated text and reference text, emphasizing semantic similarity over exact token overlap. We also introduced HBKI, a composite metric summarizing these approaches. We tested five models —GPT-3, Llama-2, Gemini 1.5 Pro, Solar 10.7B, and Mixtral-8x7b. Our software pipeline, designed and implemented using Object-Oriented Programming principles, allows users to customize the selection and extraction of features for topics of interest in their own research. Our results show that MoverScore delivered the most precise evaluation of the LLM's outputs, while Mixtral-8x7b achieved the best overall performance in extracting metadata from the databases.

97 MATHEMATICS AND COMPUTING↗

An Accuracy-Maximization Approach for Claims Classifiers in Document Content Analytics for Cybersecurity

This paper presents our research approach and findings towards maximizing the accuracy of our classifier of feature claims for cybersecurity literature analytics, and introduces the resulting model ClaimsBERT. Its architecture, after extensive evaluations of different approaches, introduces a feature map concatenated with a Bidirectional Encoder Representation from Transformers (BERT) model. We discuss deployment of this new concept and the research insights that resulted in the selection of Convolution Neural Networks for its feature mapping aspects. We also present our results showing ClaimsBERT to outperform all other evaluated approaches. This new claims classifier represents an essential processing stage within our vetting framework aiming to improve the cybersecurity of industrial control systems (ICS). Furthermore, in order to maximize the accuracy of our new ClaimsBERT classifier, we propose an approach for optimal architecture selection and determination of optimized hyperparameters, in particular the best learning rate, number of convolutions, filter sizes, activation function, the number of dense layers, as well as the number of neurons and the drop-out rate for each layer. Fine-tuning these hyperparameters within our model led to an increase in classification accuracy from 76% obtained with BertForSequenceClassification’s original model to a 97% accuracy obtained with ClaimsBERT.

Ameri, Kimia (ORCID:0000000328791871)↗

Plasma Source for Charge Control

Plasma source neutralizes electrical charge on spacecraft. When triggered by command from spacecraft potential monitor or from control system, plasma apparatus responds within about 1 s, generating charged plasma of required polarity. Discharging system to be tested on Air Force Geophysics Laboratory BERT-1 (Beam Emission Rocket Test) sounding rocket.

Aston, G.↗

Walking to Olympus: An EVA Chronology

Spacewalkers enjoy a view of Earth once reserved for Apollo, Zeus, and other denizens of Mt. Olympus. During humanity's first extravehicular activity (EVA), Alexei Leonov floated above Gibraltar, the rock ancient seafarers saw as the gateway to the great unknown Atlantic. The symbolism was clear, Leonov stepped past a new Gibraltar when he stepped into space. More than 32 years and 154 EVAs later, Jerry Linenger conducted an EVA with Vladimir Tsibliyev as part of International Space Station Phase 1. They floated together above Gibraltar. Today the symbolism has new meaning: humanity is starting to think of stepping out of Earth orbit, space travel's new Gibraltar, and perhaps obtaining a new olympian view, a close-up look at Olympus Mons on Mars. Walking to Olympus: An EVA Chronology chronicles the 154 EVAs conducted from March 1965 to April 1997. It is intended to make clear the crucial role played by EVA in the history of spaceflight, as well as to chronicle the large body of EVA "lessons learned." Russia and the U.S. define EVA differently. Russian cosmonauts are said to perform EVA any time they are in vacuum in a space suit. A U.S. astronaut must have at least his head outside his spacecraft before he is said to perform an EVA. The difference is based in differing spacecraft design philoso- phies. Russian and Soviet spacecraft have always had a specialized airlock through which the EVA cosmonaut egressed, leaving the main habitable volume of the spacecraft pressurized. The U.S. Gemini and Apollo vehicles, on the other hand, depressurized their entire habitable volume for egress. In this document, we apply the Russian definition to Russian EVAS, and the U.S. definition to U.S. EVAS. Thus, for example, Gemini 4 Command Pilot James McDivitt does not share the honor of being first American spacewalker with Ed White, even though he was suited and in vacuum when White stepped out into space. Non-EVA spaceflights are listed in the chronology to provide context and to display the large num- ber of flights in which EVA played a role. This approach also makes apparent significant EVA gaps, for example, the U.S. gap between 1985 and 1991 following the Challenger accident. This NASA History Monograph is an edited extract from an extensive EVA Chronology and Reference Book being produced by the EVA Project Office, NASA Johnson Space Center, Houston, Texas. The larger work will be published as part of the NASA Formal Series in 1998. The authors gratefully acknowledge the assistance rendered by Max Ary, Ashot Bakunts, Gert-Jan Bartelds, Frank Cepollina, Andrew Chaikin, Phillip Clark, Richard Fullerton, Steven Glenn, Linda Godwin, Jennifer Green, Greg Harris, Clifford Hess, Jeffrey Hoffman, David Homan, Steven Hopkins, Nicholas Johnson, Eric Jones, Neville Kidger, Joseph Kosmo, Alexei Lebedev, Mark Lee, James LeBlanc, Dmitri Leshchenskii, Jerry Linenger, Igor Lissov, James McBarron, Clay McCullough, Joseph McMann, Story Musgrave, Dennis Newkirk, James Oberg, Joel Powell, Lee Saegesser, Andy Salmon, Glen Swanson, Joseph Tatarewicz, Kathy Thornton, Chris Vandenberg, Charles Vick, Bert Vis, David Woods, Mike Wright, John Young, and Keith Zimmerman. Special thanks to Laurie Buchanan, John Charles, Janet Kovacevich, Joseph Loftus, Sue McDonald, Martha Munies, Colleen Rapp, and Jerry Ross. Any errors remain the responsibility of the authors.

Portree, David S. F.↗

Traveling Wave Tube (TVT) RF Power Combining Demonstration for use in the Jupiter Icy Moons Orbiter (JIMO)

The Jupiter Icy Moons Orbiter (JIMO) is set to launch between the years 2012 and 2015. It will possibly utilize a nuclear reactor power source and ion engines as it travels to the moons of Jupiter. The nuclear reactor will produce hundreds of kilowatts of power for propulsion, communication and various scientific instruments. Hence, the RF amplification devices aboard will be able to operate at a higher power level and data rate. The initial plan for the communications system is for an output of 1000 watts of RF power, a data rate of at least 10 megabits a second, and a frequency of 32 GHz. A higher data rate would be ideal to fully utilize the instruments aboard JIMO. At NASA Glenn, one of our roles in the JIMO project is to demonstrate RF power combining using multiple traveling wave tubes (TWT). In order for the power of separate TWT s to be combined, the RF output waves from each must be in-phase and have the same amplitude. Since different tubes act differently, we had to characterize each tube using a Network Analyzer. We took frequency sweeps and power sweeps to characterize each tube to ensure that they will behave similarly under the same conditions. The 200 watt Dornier tubes had been optimized to run at a lower power level (120 watts) for their extensive use in the ACTS program, so we also had to experiment with adjusting the voltage settings on several internal components (helix, anode, collector) of the tubes to reach the full 200 watt potential. from the ACTS program. Phase shifters and power attenuators were placed in the waveguide circuit at the inputs to the tubes so that adjustments could be made individually to match them exactly. A magic tee was used to route and combine the amplified electromagnetic RF waves on the tube output side. The demonstration of 200 watts of combined power was successful with efficiencies greater than 90% over a 500 MHz bandwidth. The next step will be to demonstrate the use of three amplifiers using two magic tees by adding a 200 watt Dornier tube to the Varian and Logimetrics combined setup for a total of 400 watts. After that we will use two 200 watt Dorniers for 400 watts and eventually four 200 watt Dornier tubes to demonstrate 800 watts. After demonstrating the success of power combining, we will need to verify the integrity of a modulated signal sent through the combined tubes. The purpose will be to see what effects separating and recombining will have on the modulated signal and also what effect it will have on combining efficiency. A Bit Error Rate (BER) will be determined by a Bit Error Rate Tester (BERT) by comparing the random information it transmits to what it receives back. The process began with two 100 watt tubes, a Varian and a Logimetrics, salvaged

Downey, Joseph A.↗

Natural Language Processing Analysis of Notices to Airmen for Air Traffic Management Optimization

With new emerging technologies in the field of NLP, we explore their applications to digitize and analyze heritage Air Traffic Management (ATM) documents for planning and optimizing airspace operations. Specifically, this research focuses on harvesting semi-structured or un-structured information contained in Notices to Airmen (NOTAMs). Using NLP and other advanced data analytics, we will construct a data-driven framework which facilitates finding language patterns and the use of pretrained language models for classification and extraction of useful airspace constraints and restrictions. These may lead to tools that assist airspace users in understanding the constraints more efficiently, contributing to better route planning and safer execution. This paper explores three workflows entailing different NLP tasks. First, unsupervised techniques like word embedding and topic modeling are used for pattern finding and document classification. Second, a dataset is created by extracting information from the semi-structured NOTAM format as metadata for categorizing, visualizing, and extracting key entities driving NOTAM content. Third, modern pre-built deep learning based transformer models such as BERT, RoBERTa, and XLNet are evaluated on the question answering task, an even more robust approach to information extraction, as well as their respective fine-tuning tasks. In this work we include various performance metrics for the trained models to evaluate both accuracy and precision and we show that the models can be generalized for their respective tasks. The research work developed shows promise in uncovering trends in digital NOTAMs in the NAS and also offers a new framework for digitizing and inferring insights from free-form legacy NOTAMs, that are yet to be digitized.

Natural Language Processing↗

Natural Language Processing (NLP) Analysis of NOTAMs for Air Traffic Management Optimization

With new emerging technologies in the field of NLP, we explore their applications to digitize and analyze heritage Air Traffic Management (ATM) documents for planning and optimizing airspace operations. Specifically, this research focuses on harvesting semi-structured or un-structured information contained in Notices to Airmen (NOTAMs). Using NLP and other advanced data analytics, we will construct a data-driven framework which facilitates finding language patterns and the use of pretrained language models for classification and extraction of useful airspace constraints and restrictions. These may lead to tools that assist airspace users in understanding the constraints more efficiently, contributing to better route planning and safer execution. This paper explores three workflows entailing different NLP tasks. First, unsupervised techniques like word embedding and topic modeling are used for pattern finding and document classification. Second, a dataset is created by extracting information from the semi-structured NOTAM format as metadata for categorizing, visualizing, and extracting key entities driving NOTAM content. Third, modern pre-built deep learning based transformer models such as BERT, RoBERTa, and XLNet are evaluated on the question answering task, an even more robust approach to information extraction, as well as their respective fine-tuning tasks. In this work we include various performance metrics for the trained models to evaluate both accuracy and precision and we show that the models can be generalized for their respective tasks. The research work developed shows promise in uncovering trends in digital NOTAMs in the NAS and also offers a new framework for digitizing and inferring insights from free-form legacy NOTAMs, that are yet to be digitized. Video is an mp4 download, with a play time of 9 min 35 secs.

Natural Language Processing↗

A Brief Introduction to AI/ML Applications of Air Traffic Management Data at NASA Ames

This presentation will give a brief overview of several AI/ML This presentation will give a brief overview of several AI/ML projects that NASA Ames interns are exploring in partnership with NASA Aeronautic Research Institute (NARI) and the FAA. NASA is interested in Natural Language Processing (NLP) of various legacy text and speech data within air traffic management e.g., Notices To Airmen (NOTAMs), Letters of Agreement (LoAs), Standard Operating Procedures (SOPs), and Air Traffic Control Center audio briefings. Since our focus is on applying state of the art AI/ML tools to legacy air traffic management data, we first showcase the different data sources of interest followed by a brief introduction to the techniques and language models used. We present some exciting preliminary results on each topic including both unsupervised learning techniques (e.g., clustering) and other modern language models (e.g., BERT) that help extract useful information from these data sources that are interpretable by both man and machine.

Air Traffic Management↗

TopiQAL: Topic-aware Question Answering using Scalable Domain-specific Supercomputers

We all have questions. About today's temperature, scores of our favorite baseball team, the Universe, and about vaccine for COVID-19. Life, physical, and natural scientists have been trying to find answers to various topics using scientific methods and experiments, while computer scientists have built language models as a tiny step towards automatically answering all of these questions across domains given a little bit of context. In this paper, we propose an architecture using state-of-the-art Natural Language Processing language models namely Topic Models and Bidirectional Encoder Representations from Transformers (BERT) that can transparently and automatically retrieve articles of relevance to questions across domains, and fetch answers to topical questions related to COVID-19 current and historical medical research literature. We demonstrate the benefits of using domain-specific supercomputers like Tensor Processing Units (TPUs), residing on cloud-based infrastructure, using which we could achieve significant gains in training and inference times, also with very minimal cost.

Penberthy, Scott↗