Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Large Language Models”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Autogenerating a Domain-Specific Question-Answering Data Set from a Thermoelectric Materials Database to Enable High-Performing BERT Models

We present a method for autogenerating a large domain-specific question-answering (QA) dataset from a thermoelectric materials database. We show that a small language model, BERT, once fine-tuned on this automatically generated dataset of 99,757 QA pairs about thermoelectric materials, affords better performance in the field of thermoelectric materials compared to a BERT model fine-tuned on the generic English-language QA data set, SQuAD-v2. We further show that mixing the two data sets (ours and SQuAD-v2), which have significantly different syntactic and semantic scopes, allows the BERT model to achieve even better performance. The best-performing BERT model fine-tuned on the mixed data set outperforms the models fine-tuned on the other two data sets by scoring an exact match of 67.93% and an F1 score of 72.29% when evaluated on our test data set. This has important implications as it demonstrates the ability to realize high-performing small language models, with modest computational resources, empowered by domain-specific materials data sets which can be generated according to our method.

biological databases↗
DOE OSTI2019

MPI + OpenACC: Accelerating radiation transport mini-application, minisweep, on heterogeneous systems

According to the article, architectures are rapidly evolving, and exascale machines are expected to offer billion-way concurrency. We need to rethink algorithms, languages and programming models among other components in order to migrate large scale applications and explore parallelism on these machines. Although directive-based programming models allow programmers to worry less about programming and more about science, expressing complex parallel patterns in these models can be a daunting task especially when the goal is to match the performance that the hardware platforms can offer. One such pattern is wavefront. This paper extensively studies a wavefront-based miniapplication for Denovo, a production code for nuclear reactor modeling. We parallelize the Koch–Baker–Alcouffe (KBA) parallel-wavefront sweep algorithm in the main kernel of Minisweep (the miniapplication) using CUDA 9.0, OpenMP 4.0 (SIMD) and OpenACC 2.6. Our OpenACC implementation running on NVIDIA’s next-generation Volta GPU boasts an 85.06x speedup over serial code, which is larger than CUDA’s 83.72x speedup over the same serial implementation. We also explore the scalability of our solution using MPI to decompose our simulation domain, allowing us to run on many nodes and accelerators present in state-of-the-art HPC systems. Our parallelization effort across platforms also motivated us to define an abstract parallelism model that is architecture independent, with a goal of creating software abstractions that can be used by applications employing the wavefront sweep motif.

Computer Science↗
DOE OSTI2024

Dynamic Model of Back-to-Back Converter for System Level Phasor Simulation

The power system is expected to evolve rapidly with increasing deployment of power electronic interface and conditioning systems, microgrids, and hybrid AC/DC grids. Among power electronic systems, back-to-back (BTB) converters can be a powerful interface to integrate microgrids and networked microgrids. However, in system level dynamic simulations for synchronous machine dominated system with BTB converters, a detailed electromagnetic sinusoidal models is not warranted as these simulations are focused on electromechanical transients. This paper focus on developing a simple phasor model for back- to-back converters that can be easily integrated into powerflow solvers to facilitate large-scale power system simulations. The model is implemented using C++ language and integrated into GridLAB-D software as a potential new capability. The GridLAB-D phasor domain model is validated against the electromagnetic transient (EMT) simulation of the detailed switching model. Simulation results show that the phasor model successfully captures the dominant dynamics of the converters.

Mahmood, Hisham↗
DOE OSTI2018

Abstractions and Directives for Adapting Wavefront Algorithms to Future Architectures

Architectures are rapidly evolving, and exascale machines are expected to offer billion-way concurrency. We need to rethink algorithms, languages and programming models among other components in order to migrate large scale applications and explore parallelism on these machines. Although directive-based programming models allow programmers to worry less about programming and more about science, expressing complex parallel patterns in these models can be a daunting task especially when the goal is to match the performance that the hardware platforms can offer. One such pattern is wavefront. This paper extensively studies a wavefront-based miniapplication for Denovo, a production code for nuclear reactor modeling. We parallelize the Koch-Baker-Alcouffe (KBA) parallel-wavefront sweep algorithm in the main kernel of Minisweep (the miniapplication) using CUDA, OpenMP and OpenACC. Our OpenACC implementation running on NVIDIA's next-generation Volta GPU boasts an 85.06x speedup over serial code, which is larger than CUDA's 83.72x speedup over the same serial implementation. Our experimental platform includes SummitDev, an ORNL representative architecture of the upcoming Summit supercomputer. Our parallelization effort across platforms also motivated us to define an abstract parallelism model that is architecture independent, with a goal of creating software abstractions that can be used by applications employing the wavefront sweep motif.

Searles, Robert↗
DOE OSTI2024

Hiding-in-Plain-Sight (HiPS) Attack on CLIP for Targetted Object Removal from Images

Machine learning models are known to be vulnerable to adversarial attacks, but prior works have mostly focused on single-modalities. With the rise of large multi-modal models (LMMs) like CLIP, which combine vision and language capabilities, new vulnerabilities have emerged. However, these multimodal targeted attacks aim to completely change the model's output to what the adversary wants. In many realistic scenarios, an adversary might seek to make only subtle modifications to the output, so that the changes go unnoticed by downstream models or even by humans. We introduce Hiding-in-Plain-Sight (HiPS) attacks, a novel class of adversarial attacks that subtly modifies model predictions by selectively concealing target object(s), as if the target object was absent from the scene. We propose two HiPS attack variants, HiPS-cls and HiPS-cap, and demonstrate their effectiveness in transferring to downstream image captioning models, such as CLIP-Cap, for targeted object removal from image captions.

Daw, Arka [ORNL] (ORCID:0009000633191271)↗
DOE OSTI2016

Assessment of online public opinions on large infrastructure projects: A case study of the Three Gorges Project in China

Public opinion becomes increasingly salient in the ex post evaluation stage of large infrastructure projects which have significant impacts to the environment and the society. However, traditional survey methods are inefficient in collection and assessment of the public opinion due to its large quantity and diversity. Recently, Social media platforms provide a rich data source for monitoring and assessing the public opinion on controversial infrastructure projects. This paper proposes an assessment framework to transform unstructured online public opinions on large infrastructure projects into sentimental and topical indicators for enhancing practices of ex post evaluation and public participation. The framework uses web crawlers to collect online comments related to a large infrastructure project and employs two natural language processing technologies, including sentiment analysis and topic modeling, with spatio-temporal analysis, to transform these comments into indicators for assessing online public opinion on the project. Based on the framework, we investigate the online public opinion of the Three Gorges Project on China's largest microblogging site, namely, Weibo. Assessment results present spatial-temporal distributions of post intensity and sentiment polarity, reveals major topics with different sentiments and summarizes managerial implications, for ex post evaluation of the world's largest hydropower project. The proposed assessment framework is expected to be widely applied as a methodological strategy to assess public opinion in the ex post evaluation stage of large infrastructure projects. - Highlights: • We developed a framework to assess online public opinion on large infrastructure projects with environmental impacts. • Indicators were built to assess post intensity, sentiment polarity and major topics of the public opinion. • We took the Three Gorges Project (TGP) as an example to demonstrate the effectiveness proposed framework. • We revealed spatial-temporal patterns of post intensity and sentiment polarity on the TGP. • We drew implications for a more in-depth understanding of the public opinion on large infrastructure projects.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2018

IL-Net: Using Expert Knowledge to Guide the Design of Furcated Neural Networks

Deep neural networks (DNN) excel at extracting patterns. Through representation learning and automated feature engineering on large datasets, such models have been highly successful in computer vision and natural language applications. Designing optimal network architectures from a principled or rational approach however has been less than successful, with the best successful approaches utilizing an additional machine learning algorithm to tune the network hyperparameters. However, in many technical fields, there exist established domain knowledge and understanding about the subject matter. In this work, we develop a novel furcated neural network architecture that utilizes domain knowledge as high-level design principles of the network. We demonstrate proof-of-concept by developing IL-Net, a furcated network for predicting the properties of ionic liquids, which is a class of complex multi-chemicals entities. Compared to existing state-of-the-art approaches, we show that furcated networks can improve model accuracy by approximately 20-35%, without using additional labeled data. Lastly, we distill two key design principles for furcated networks that can be adapted to other domains.

Sakloth, Khushmeen↗
DOE OSTI2018

Recurrent Neural Network Language Models for Open Vocabulary Event-Level Cyber Anomaly Detection

Automated analysis methods are crucial aids for monitoring and defending a network to protect the sensitive or confidential data it hosts. This work introduces a flexible, powerful, and unsupervised approach to detecting anomalous behavior in computer and network logs; one that largely eliminates domain-dependent feature engineering employed by existing methods. By treating system logs as threads of interleaved ``sentences'' (event log lines) to train online unsupervised neural network language models, our approach provides an adaptive model of normal network behavior. We compare the effectiveness of both standard and bidirectional recurrent neural network language models at detecting malicious activity within network log data. Extending these models, we introduce a tiered recurrent architecture, which provides context by modeling sequences of users' actions over time. Compared to isolation forests and Principal Components Analysis (PCA), two popular anomaly detection algorithms, we observe superior performance on the Los Alamos National Laboratory Cyber Security dataset. For log-line-level predictions, our best performing character-based model provides test set anomaly scores ranking red team events in the 98th percentile on average, and yields 0.97 area under the receiver operator characteristic curve, demonstrating the strong fine-grained anomaly detection performance of this approach on open vocabulary logging sources.

Cyber Analysis, Deep Learning, language analysis, ↗
DOE OSTI2022

Modelica-based modeling and simulation of district cooling systems: A case study

While equation-based object-oriented modeling language Modelica can evaluate practical energy improvements for district cooling systems, few have adopted Modelica for this type of large-scale thermo-fluid system. Further, to our best knowledge, district cooling modeling studies have yet to include hydraulics in piping networks alongside plant models featuring realistic mechanical systems and controls. These are critical details to include when looking to make energy and control improvements in many physical system installations. To fill these gaps, this study released new open-source district cooling models at the Modelica Buildings Library and leveraged these models for a real-world case study at the University of Colorado Boulder. Here, the site includes six buildings connected to a central chiller plant featuring a waterside economizer. Several energy saving strategies are pursued based on the validated model, including control setpoint optimization, equipment modification, and pump setpoint adjustments. Results indicate that a combination of the studied measures can save the campus annually 84.6 MWh of energy, 8.9% of electricity costs, 58.0 metric tons of carbon dioxide emissions, while the waterside economizer cuts down chillers’ run times by 201 days/year, reducing maintenance costs and extending chiller life.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Hardware-based Advanced Electromagnetic Transient Simulation for A Large-Scale PV Plant in Real Time Digital Simulator

Power electronics-based resources, such as high-voltage direct current (HVdc) substations, photovoltaic (PV) plants, wind plants, electric vehicle charging stations, and energy storage systems, are increasingly being integrated within the power grid. Recently, multiple reports have emphasized the necessity for high-fidelity electromagnetic transient (EMT) simulations of these large-scale power electronics-based resources to accurately understand their behavior in power grids. However, performing hardware-based EMT simulations with high-fidelity models for such large power electronics systems is challenging due to the small time-step requirements and the involvement of a large number of states. This paper presents the implementation of hardware-based high-fidelity EMT dynamic model of a large-scale PV plant, accomplished through custom model development using the specific-C language in real-time digital simulator hardware (RTDS) and software (RSCAD).

Choi, Jongchan↗
DOE OSTI2024

SEAS Communication Engine: An Extensible, Flexible Wrapper for Co-Simulation Agents

When modeling and analyzing the power grid and other large scale systems, researchers often express scenarios as optimization problems and feed them into advanced software solvers. In order to allow multiple solvers to communicate with each other and share data from different domains, the National Renewable Energy Laboratory (NREL) and associated Department of Energy (DOE) labs have developed a software framework called the Hierarchical Engine for Large-scale Infrastructure Co-Simulation (HELICS). HELICS allows cosimulation via a collection of client libraries for different languages that can be called from the appropriate optimization software. However, these client libraries do not provide a higher level of abstraction beyond reading and writing data off of the shared HELICS bus. In this paper, we describe a new software library called the SEAS Communication Engine that exposes a higher-level API for running cosimulation problems. The SEAS Engine provides a class-based abstraction on top of the Python HELICS client, in order to allow users to implement their domain-specific cosimulations without needing to interact with core HELICS primitives. This will make adoption of HELICS and cosimulation in general easier, by exposing a simpler API. In the second part of the paper, we validate our library on a collection of different simulation examples, including the canonical IEEE 13 Bus Feeder. Lastly, we demonstrate using the SEAS Engine to directly call domain-specific code written in the Julia programming language. Our hope is that this will serve as a template for easily calling software in different programming languages via the SEAS Engine, thereby avoiding code duplication and complexity.

co-simulation↗
DOE OSTI2025

genomeocean: a pretrained microbial genome foundational model (genomeoceanLLM) v1.0

We present Genomeocean, a foundational genome language model that represents the microbial genome sequences from complex environmental samples. By training on a large, diverse metagenomic dataset, Genomeocean learns species-specific sequence composition and can generate long, realistic open reading frames (ORFs). Our model employs a Byte-pair-encoding (BPE) tokenization strategy, allowing it to efficiently process large genomic datasets and generate long sequences up to 50kb. We demonstrate that fine-tuning Genomeocean can generate novel gene clusters encoding biosynthetic pathways, showcasing its ability to model both fundamental and complex biological processes. Our work establishes Genomeocean as a powerful tool for understanding microbial genome biology and paves the way for its application in a range of fields, from synthetic biology to microbiome research.

Wang, Zhong [Lawrence Berkeley National Laboratory↗
DOE OSTI2019

Model of temperature, evaporation, and productivity in elevated experimental algae raceways and comparison with commercial raceways

Elevated Experimental Raceways (EERs) have been used at many research sites in the United States to evaluate algal biomass productivity. This paper develops temperature and evaporation models for EERs and then evaluates whether the temperature profile and resultant productivity in an EER is representative of large-scale commercial raceways. Open water surface evaporation and temperature models with shading algorithms were programmed in Python and calibrated with temperature and evaporation data from the EERs in the Regional Algal Feedstock Testbed (RAFT) experiments at the University of Arizona. The final calibrated model for EERs was named the EERTEM (Elevated Experimental Raceway Temperature and Evaporation Model). The energy balance algorithms in the Biomass Assessment Tool (BAT model) were also added to the Python code and used to develop temperature simulations of standard commercial paddlewheel raceways. A comparison of BAT and EERTEM simulations indicated that EER temperature fluctuations are not representative of in ground commercial paddlewheel raceways, primarily due to the buffering effect of soil heat flux in commercial raceways. With the different temperature profiles, biomass productivities of three algae species in the EERs were compared to productivities in commercial raceways. Differences in productivity between EERTEM and BAT model temperature profiles were observed when the maximum daytime temperature of one raceway was in the optimal growth range but the temperature of the other raceway exceeded or was below the optimal growth range.

Khawam, George↗
DOE OSTI2022

Active Learning for Language Modeling

Foreign disinformation campaigns undermine national security. Various supervised language modeling techniques in NLP can help to understand and dismantle these campaigns, but they rely heavily on large, labeled (often by humans) datasets. This work provides a solution to this problem in the form of an active learning (AL) framework, which is used to generate labeled datasets and leverage human input for detecting disinformation. The developed AL framework utilizes task adaptive pretraining to fully leverage the unlabeled data and boost the performance of the classifier used for labeling. A disinformation rhetoric metric was developed to measure the presence of common rhetorical techniques used in text that are meant to deceive, for both the classifier and human to use in the task of identifying disinformation. This metric was combined with an uncertainty criterion to create a hybrid acquisition method for AL, and this hybrid method was tested alongside other acquisition functions. A sophisticated and robust stopping strategy was developed to signal the AL process should terminate, saving human time from being wasted on iterations that would not significantly benefit classifier performance.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2021

Audacity of huge: overcoming challenges of data scarcity and data quality for machine learning in computational materials discovery

Machine learning (ML)-accelerated discovery requires large amounts of high-fidelity data to reveal predictive structure–property relationships. For many properties of interest in materials discovery, the challenging nature and high cost of data generation has resulted in a data landscape that is both scarcely populated and of dubious quality. Data-driven techniques starting to overcome these limitations include the use of consensus across functionals in density functional theory, the development of new functionals or accelerated electronic structure theories, and the detection of where computationally demanding methods are most necessary. When properties cannot be reliably simulated, large experimental data sets can be used to train ML models. In the absence of manual curation, increasingly sophisticated natural language processing and automated image analysis are making it possible to learn structure–property relationships from the literature. Finally, models trained on these data sets will improve as they incorporate community feedback.

36 MATERIALS SCIENCE↗
DOE OSTI2017

Testing of Strategies for the Acceleration of the Cost Optimization

The general problem addressed in the Nuclear-Renewable Hybrid Energy System (N-R HES) project is finding the optimum economical dispatch (ED) and capacity planning solutions for the hybrid energy systems. In the present test-problem configuration, the N-R HES unit is composed of three electrical power-generating components, i.e. the Balance of Plant (BOP), the Secondary Energy Source (SES), and the Energy Storage (ES). In addition, there is an Industrial Process (IP), which is devoted to hydrogen generation. At this preliminary stage, the goal is to find the power outputs of each one of the N-R HES unit components (BOP, SES, ES) and the IP hydrogen production level that maximizes the unit profit by simultaneously satisfying individual component operational constraints. The optimization problem is meant to be solved in the Risk Analysis Virtual Environment (RAVEN) framework. The dynamic response of the N-R HES unit components is simulated by using dedicated object-oriented models written in the Modelica modeling language. Though this code coupling provides for very accurate predictions, the ensuing optimization problem is characterized by a very large number of solution variables. To ease the computational burden and to improve the path to a converged solution, a method to better estimate the initial guess for the optimization problem solution was developed. The proposed approach led to the definition of a suitable Monte Carlo-based optimization algorithm (called the preconditioner), which provides an initial guess for the optimal N-R HES power dispatch and the optimal installed capacity for each one of the unit components. The preconditioner samples a set of stochastic power scenarios for each one of the N-R HES unit components, and then for each of them the corresponding value of a suitably defined cost function is evaluated. After having simulated a sufficient number of power histories, the configuration which ensures the highest profit is selected as the optimal one. The component physical dynamics are represented through suitable ramp constraints, which considerably simplify the numerical solving. In order to test the capabilities of the proposed approach, in the present report, the dispatch problem only is tackled, i.e. a reference unit configuration is assumed, and each one of the N-R HES unit components is assumed to have a fixed installed capacity. As for the next steps, the main improvement will concern the operation strategy of the ES facility. In particular, in order to describe a more realistic battery commitment strategy, the ES operation will be regulated according to the electricity price forecasts.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2013

SIPMaP: A Tool for Modeling Irregular Parallel Computations in the Super Instruction Architecture

Performance modeling is becoming an increasingly important part of the parallel application development process, particularly for expensive computations that will be run on very high-end systems where resources are scarce. We describe a performance modeling tool SIPMaP (Super Instruction Processor Modeling and Prediction) developed for the Super-Instruction Architecture (SIA). The SIA is designed for applications where the dominant data structures are large multi-dimensional arrays and it comprises a DSL, the Super-Instruction Assembly Language (SIAL) that supports expressing algorithms in terms of blocks (tiles), and its runtime system Super Instruction Processor (SIP) that manages distribution and disk storage of the arrays. SIPMaP generates performance models from the SIAL source code. In comparison with many applications where useful performance models have been developed and reported, these programs are irregular and have other difficult to model characteristics such as extensive overlapping of communication and computation.

Computer Science↗
DOE OSTI2023

Neural Architecture Search Benchmarks: Insights and Survey

Neural Architecture Search (NAS), a promising and fast-moving research field, aims to automate the architectural design of Deep Neural Networks (DNNs) to achieve better performance on the given task and dataset. NAS methods have been very successful in discovering efficient models for various Computer Vision, Natural Language Processing, etc. The major obstacles to the advancement of NAS techniques are the demand for large computation resources and fair evaluation of various search methods. The differences in training pipeline and setting make it challenging to compare the efficiency of two NAS algorithms. A large number of NAS Benchmarks to simulate the architecture evaluation in seconds have been released over the last few years to ease the computation burden of training neural networks and can aid in the unbiased assessment of different search methods. This paper provides an extensive review of several publicly available NAS Benchmarks in the literature. We provide technical details and a deeper understanding of each benchmark and point out future directions.

97 MATHEMATICS AND COMPUTING↗