Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “New link prediction”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Evaluating Energy Efficiency Opportunities from Connected and Automated Vehicle Deployments Coupled with Shared Mobility in California

Connected and Automated Vehicles (CAVs) can be considered to be a disruptive transportation technology, with the potential to significantly improve overall transportation system efficiency; however, CAVs may increase induce vehicle miles traveled (VMT) and bring on greater energy consumption. Further, shared mobility is another disruptive transportation event that is reshaping our travel patterns. The primary goal of this project was to extensively collect data from vehicles and associated infrastructure equipped with CAV technologies from both real-world experiments and simulation studies mainly deployed in California, and develop a comprehensive framework for evaluating energy efficiency opportunities from large-scale (e.g., statewide) introduction of CAVs and a wide deployment of shared mobility systems in a variety of scenarios. To quantify the combined impact of CAV and shared mobility on travel behavior, traffic performance, and energy efficiency, a unique mesoscopic simulation-based model was developed for mobility and energy efficiency evaluation considering these disruptive transportation technologies. As a complement to existing studies on nationwide evaluation of CAVs’ energy impacts, this project was focused on data collection efforts and CAV applications under congested traffic environments that are frequently experienced on a massive scale across the major metropolitan areas in California. Extensive real-world data collection supplemented with simulation studies were conducted to cover a variety of CAV and shared mobility scenarios, particularly on scenarios less-explored in the existing research. Another key component of this project was to consider the interaction between different CAV technologies and shared mobility models, and the compound effect on energy efficiency. A comprehensive modeling suite was developed to quantify the impact of new mobility technologies on travel behavior and traffic performance. The developed modeling framework includes an energy intensity module, mode choice module and activity generation module that are integrated into an agent-based BEAM simulation platform to perform impact analysis based on a variety of scenarios. In addition, the RouteE model has been upgraded to incorporate the impact of CAVs on traffic flow, VMT and energy intensity, using micro-simulation data collected from both freeways and urban arterials. A novel fundamental influencing factor (FIF) mode choice model was developed to link CAV and shared mobility components with travel behaviors, and adapted into the BEAM-centered model framework. A statewide energy inventory was constructed under various CAV technology deployment scenarios by incorporating datasets and models for predicting vehicle market share and vehicle usage, which are tightly associated with the penetration of shared mobility systems. Based applying this modeling suite to a calibrated network in Riverside California, it was found that cooperative automated driving in general will improve mobility, but automated vehicles, even when deployed in a shared autonomous fleet, will likely bring an increase of VMT (up to 36%) due to mode shifts and deadheading. Ride-hailing vehicles typically have better energy efficiency and a higher share of electric vehicles, which helps offset the negative impact from VMT increases when estimating the system-level energy consumption. In general, simulation results show a 6% increase in energy consumption for the scenarios with an increasing shift to ride-hailing modes. The statewide analysis based on the National Household Travel Survey (NHTS) sample data is consistent with the findings from the Riverside network and validate the developed clustering-prediction modeling methodology. The outcomes from this project will help close the knowledge gap on recognizing the potential performance and energy impacts of a broad deployment of CAV and shared mobility technologies across a wide range of roadway infrastructure with varying levels of congestion. Results from this project: 1) will support policymakers in steering CAV development and deployment towards an energy favorable direction; 2) reduce uncertainties in estimating energy saving opportunities from new mobility technologies and services; 3) increase the confidence of CAV technology investors both on the infrastructure side (i.e., transportation agencies) and on the vehicle side (i.e., OEMs); and 4) expedite the deployment of energy-efficient CAV and shared mobility applications.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2024

Linking Threat Agents to Targeted Organizations: A Pipeline for Enhanced Cybersecurity Risk Metrics

In this study, we present a methodology leveraging Large Language Models (LLMs) to transform Cybersecurity Threat Intelligence (CTI) narratives into actionable insights for individual organizations. Our approach automates the extraction of machine-readable adversary SKRAM (Skills, Knowledge, Resources, Authorities, and Motivation) attributes from open-source reports, extending LLM utility beyond typical interactions. This innovation enables precise, automated assessments of cybersecurity risks posed by various adversaries. Using a chain-of-thought and multi-shot prompting strategy, our methodology advances the automation of cybersecurity feature extraction for new machine-learning models that predict the risk of adversary targeting. This approach is refined using a substantial dataset of over 150 analyst-validated threat reports and synthetic organizational data from 900 companies. Here, by bootstrapping the training data with a rule-based heuristic over synthetic data, we have developed a high-accuracy machine-learning model that allows entities to dynamically prioritize threats and defensive actions.

Cyber Threat Intelligence↗
DOE OSTI2020

Kinetic entropy-based measures of distribution function non-Maxwellianity: theory and simulations

We investigate kinetic entropy-based measures of the non-Maxwellianity of distribution functions in plasmas, i.e. entropy-based measures of the departure of a local distribution function from an associated Maxwellian distribution function with the same density, bulk flow and temperature as the local distribution. First, we consider a form previously employed by Kaufmann & Paterson (J. Geophys. Res., vol. 114, 2009, A00D04), assessing its properties and deriving equivalent forms. To provide a quantitative understanding of it, we derive analytical expressions for three common non-Maxwellian plasma distribution functions. We show that there are undesirable features of this non-Maxwellianity measure including that it can diverge in various physical limits and elucidate the reason for the divergence. We then introduce a new kinetic entropy-based non-Maxwellianity measure based on the velocity-space kinetic entropy density, which has a meaningful physical interpretation and does not diverge. We use collisionless particle-in-cell simulations of two-dimensional anti-parallel magnetic reconnection to assess the kinetic entropy-based non-Maxwellianity measures. We show that regions of non-zero non-Maxwellianity are linked to kinetic processes occurring during magnetic reconnection. We also show the simulated non-Maxwellianity agrees reasonably well with predictions for distributions resembling those calculated analytically. These results can be important for applications, as non-Maxwellianity can be used to identify regions of kinetic-scale physics or increased dissipation in plasmas.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
NASA NTRS2023

A Spaceborne Demonstration of P-Band Signals-of-Opportunity (SoOp) Reflectometry

Land-reflected signals from a geosynchronous communication satellite broadcasting in P-band (367.5 MHz) were captured in low Earth orbit (LEO) using a simple dipole antenna. A delay-Doppler map (DDM) was generated through autocorrelation. Estimates of the specular point delay were obtained from the lag of the second peak in the DDM with a bias of 239.4 m and a standard deviation of 44 m (12 m over a frozen lake) with respect to a predicted orbit model. Relative magnitudes of the first and second DDM peaks fell within the range of values predicted using dielectric models for the frozen ground and lake. Finally, retrievals of surface reflection coefficient were generated using a range of realistic values for the transmitter link budget G/T , and these also fell within the range of possible values for the antenna gain pattern. Given the lack of calibration and the large uncertainties in the receiver orbit and attitude, this agreement is sufficient to conclude a successful demonstration of the fundamental principle of single-antenna reflectometry in P-band. P-band reflectometry may offer a new approach to remote sensing of sub-canopy and root-zone soil moisture.

Delays↗
DOE OSTI2023

Predicting wind-driven spatial deposition through simulated color images using deep autoencoders

Abstract For centuries, scientists have observed nature to understand the laws that govern the physical world. The traditional process of turning observations into physical understanding is slow. Imperfect models are constructed and tested to explain relationships in data. Powerful new algorithms can enable computers to learn physics by observing images and videos. Inspired by this idea, instead of training machine learning models using physical quantities, we used images, that is, pixel information. For this work, and as a proof of concept, the physics of interest are wind-driven spatial patterns. These phenomena include features in Aeolian dunes and volcanic ash deposition, wildfire smoke, and air pollution plumes. We use computer model simulations of spatial deposition patterns to approximate images from a hypothetical imaging device whose outputs are red, green, and blue (RGB) color images with channel values ranging from 0 to 255. In this paper, we explore deep convolutional neural network-based autoencoders to exploit relationships in wind-driven spatial patterns, which commonly occur in geosciences, and reduce their dimensionality. Reducing the data dimension size with an encoder enables training deep, fully connected neural network models linking geographic and meteorological scalar input quantities to the encoded space. Once this is achieved, full spatial patterns are reconstructed using the decoder. We demonstrate this approach on images of spatial deposition from a pollution source, where the encoder compresses the dimensionality to 0.02% of the original size, and the full predictive model performance on test data achieves a normalized root mean squared error of 8%, a figure of merit in space of 94% and a precision-recall area under the curve of 0.93.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2009

Integration of Airborne Aerosol Prediction Systems and Vegetation Phenology to Track Pollen for Asthma Alerts in Public Health Decision Support Systems

The residual signal indicates that the pollen event may influence the seasonal signal to an extent that would allow detection, given accurate QA filtering and BRDF corrections. MODIS daily reflectances increased during the pollen season. The DREAM model (PREAM) was successfully modified for use with pollen and may provide 24-36 hour running pollen forecasts. Publicly available pollen forecasts are linked to general weather patterns and roughly-known species phenologies. These are too coarse for timely health interventions. PREAM addresses this key data gap so that targeting intervention measures can be determined temporally and geospatially. The New Mexico Department of Health (NMDOH) as part of its Environmental Public Health Tracking Network (EPHTN) would use PREAM a tool for alerting the public in advance of pollen bursts to intervene and reduce the health impact on asthma populations at risk.

Luvall, Jeffrey C.↗
NASA NTRS2001

Linking Science Analysis with Observation Planning: A Full Circle Data Lifecycle

A clear goal of the Virtual Observatory (VO) is to enable new science through analysis of integrated astronomical archives. An additional and powerful possibility of the VO is to link and integrate these new analyses with planning of new observations. By providing tools that can be used for observation planning in the VO, the VO will allow the data lifecycle to come full circle: from theory to observations to data and back around to new theories and new observations. The Scientist's Expert Assistant (SEA) Simulation Facility (SSF) is working to combine the ability to access existing archives with the ability to model and visualize new observations. Integrating the two will allow astronomers to better use the integrated archives of the VO to plan and predict the success of potential new observations. The full circle lifecycle enabled by SEA can allow astronomers to make substantial leaps in the quality of data and science returns on new observations. Our paper will examine the exciting potential of integrating archival analysis with new observation planning, such as performing data calibration analysis on archival images and using that analysis to predict the success of new observations, or performing dynamic signal-to-noise analysis combining historical results with modeling of new instruments or targets. We will also describe how the development of the SSF is progressing and what has been its successes and challenges.

Jones, Jeremy↗
NASA NTRS2001

Linking Science Analysis with Observation Planning: A Full Circle Data Lifecycle

A clear goal of the Virtual Observatory (VO) is to enable new science through analysis of integrated astronomical archives. An additional and powerful possibility of the VO is to link and integrate these new analyses with planning of new observations. By providing tools that can be used for observation planning in the VO, the VO will allow the data lifecycle to come full circle: from theory to observations to data and back around to new theories and new observations. The Scientist's Expert Assistant (SEA) Simulation Facility (SSF) is working to combine the ability to access existing archives with the ability to model and visualize new observations. Integrating the two will allow astronomers to better use the integrated archives of the VO to plan and predict the success of potential new observations more efficiently, The full circle lifecycle enabled by SEA can allow astronomers to make substantial leaps in the quality of data and science returns on new observations. Our paper examines the exciting potential of integrating archival analysis with new observation planning, such as performing data calibration analysis on archival images and using that analysis to predict the success of new observations, or performing dynamic signal-to-noise analysis combining historical results with modeling of new instruments or targets. We will also describe how the development of the SSF is progressing and what have been its successes and challenges.

Grosvenor, Sandy↗
NASA NTRS1988

Flight research and testing

Flight research and testing form a critical link in the aeronautic R and D chain. Brilliant concepts, elegant theories, and even sophisticated ground tests of flight vehicles are not sufficient to prove beyond doubt that an unproven aeronautical concept will actually perform as predicted. Flight research and testing provide the ultimate proof that an idea or concept performs as expected. Ever since the Wright brothers, flight research and testing have been the crucible in which aeronautical concepts have advanced and been proven to the point that engineers and companies have been willing to stake their future to produce and design new aircraft. This is still true today, as shown by the development of the experimental X-30 aerospace plane. The Dryden Flight Research Center (Ames-Dryden) continues to be involved in a number of flight research programs that require understanding and characterization of the total airplane in all the aeronautical disciplines, for example the X-29. Other programs such as the F-14 variable-sweep transition flight experiment have focused on a single concept or discipline. Ames-Dryden also continues to conduct flight and ground based experiments to improve and expand the ability to test and evaluate advanced aeronautical concepts. A review of significant aeronautical flight research programs and experiments is presented to illustrate both the progress made and the challenges to come.

Putnam, Terrill W.↗
DOE OSTI2024

Data-driven prediction of scaling and ignition of inertial confinement fusion experiments

Recent advances in inertial confinement fusion (ICF) at the National Ignition Facility (NIF), including ignition and energy gain, are enabled by a close coupling between experiments and high-fidelity simulations. Neither simulations nor experiments can fully constrain the behavior of ICF implosions on their own, meaning pre- and postshot simulation studies must incorporate experimental data to be reliable. Linking past data with simulations to make predictions for upcoming designs and quantifying the uncertainty in those predictions has been an ongoing challenge in ICF research. We have developed a data-driven approach to prediction and uncertainty quantification that combines large ensembles of simulations with Bayesian inference and deep learning. The approach builds a predictive model for the statistical distribution of key performance parameters, which is jointly informed by past experiments and physics simulations. The prediction distribution captures the impact of experimental uncertainty, expert priors, design changes, and shot-to-shot variations. We have used this new capability to predict a 10× increase in ignition probability between Hybrid-E shots driven with 2.05 MJ compared to 1.9 MJ, and validated our predictions against subsequent experiments. We describe our new Bayesian postshot and prediction capabilities, discuss their application to NIF ignition and validate the results, and finally investigate the impact of data sparsity on our prediction results.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

Dislocation nano-hydrides in nickel: Nucleation, evolution and effects on dislocation behaviors

Nano-hydrides have been predicted to precipitate at the core of edge dislocations in the Ni-H system, a mechanism that may promote hydrogen embrittlement. However, nano-hydride nucleation, growth, and effects on dislocation behavior have seldom been explored. This work combines molecular dynamics grand canonical Monte Carlo (MD-GCMC) simulations and continuum modeling to uncover a wide range of phenomena linked to dislocation nano-hydrides. Simulations reveal that nano-hydrides can be stabilized at dislocation cores with all character angles, including screw segments, due to the hydrostatic stresses around the cores of the Shockley partials. Nano-hydride nucleation takes place in these regions, and growth is dictated by the character angle θ of the perfect dislocation. The equilibrium stacking fault width d eq varies dynamically to increase the local hydrostatic stress field and facilitate the formation of the nano-hydride, forming a constriction-like feature and leading to three distinct behaviors: d eq decreases for θ > 30°, d eq remains unchanged for θ = 30°, and d eq increases for θ < 30°. Remote hydrostatic and Escaig stresses are also shown to influence the nucleation stage, implying stress concentrations such as those ahead of crack tips may facilitate nano-hydride precipitation. Moreover, we identify a new hydrogen-induced 60° dislocation reaction that emits a Shockley partial on a conjugate plane, with potential implications for twin nucleation. Testable predictions from this study are then used to reinterpret previous results from the literature. In conclusion, these findings provide a comprehensive framework to assess nano-hydride formation and evolution at dislocations in nickel and other face-centered cubic metals, with important implications to hydrogen embrittlement.

Dislocations↗
NASA NTRS2018

Thermodynamic and Dynamic Aspects of Ice Nucleation

It is known that ice nucleating particles (INP) immersed within supercooled droplets promote the formation of ice. Common theoretical models used to represent this process assume that the immersed particle lowers the work of ice nucleation without significantly affecting the dynamics of water in the vicinity of the particle. This is contrary to evidence showing that immersed surfaces significantly affect the viscosity and diffusivity of vicinal water. To study how this may affect ice formation this work introduces a model linking the ice nucleation rate to the modification of the dynamics and thermodynamics of vicinal water by immersed particles. It is shown that INP that significantly reduce the work of ice nucleation also pose strong limitations to the growth of the nascent ice germs. This leads to the onset of a new ice nucleation regime, called spinodal ice nucleation, where the dynamics of ice germ growth instead of the ice germ size determines the nucleation rate. Nucleation in this regime is characterized by an enhanced sensitivity to particle area and cooling rate. Comparison of the predicted ice nucleation rate against experimental measurements for a diverse set of species relevant to cloud formation suggests that spinodal ice nucleation may be common in nature.

Barahona, Donifan↗
NASA NTRS2005

Communications Blackout Predictions for Atmospheric Entry of Mars Science Laboratory

The Mars Science Laboratory (MSL) is expected to be a long-range, long-duration science laboratory rover on the Martian surface. MSL will provide a significant milestone that paves the way for future landed missions to Mars. NASA is studying options to launch MSL as early as 2009. MSL will be the first mission to demonstrate the new technology of 'smart landers', which include precision landing and hazard avoidance in order to -land at scientifically interesting sites that would otherwise be unreachable. There are three elements to the spacecraft; carrier (cruise stage), entry vehicle, and rover. The rover will have an X-band direct-to-Earth (DTE) link as well as a UHF proximity link. There is also a possibility of an X-band proximity link. Given the importance of collecting critical event telemetry data during atmospheric entry, it is important to understand the ability of a signal link to be maintained, especially during the period near peak convective heating. The received telemetry during entry (or played back later) will allow for the performance of the Entry-Descent-Landing technologies to be assessed. These technologies include guided entry for precision landing, hazard avoidance, a new sky-crane landing system and powered descent. MSL will undergo an entry profile that may result in a potential communications blackout caused by ionized plasma for short periods near peak heating. The vehicle will use UHF and possibly X-band during the entry phase. The purpose of this report is to quantify or bound the likelihood of any such blackout at UHF frequencies (401 MHz) and X-band frequencies (8.4 GHz). Two entry trajectory scenarios were evaluated: a stressful entry trajectory to quantify an upper-bound for any possible blackout period, and a nominal likely trajectory to quantify likelihood of blackout for such cases.

UHF↗
DOE OSTI2020

Simulations of activities, solubilities, transport properties, and nucleation rates for aqueous electrolyte solutions

This article reviews recent molecular simulation studies of "collective" properties of aqueous electrolyte solutions, specifically free energies and activity coefficients, solubilities, nucleation rates of crystals, and transport coefficients. These are important fundamental properties for biology and geoscience, but also relevant for many technological applications. Their determination from molecular-scale calculations requires large systems and long sampling times, as well as specialized sampling algorithms. As a result, such properties have not typically been taken into account during optimization of force field parameters; thus, they provide stringent tests for the transferability and range of applicability of proposed molecular models. There has been significant progress on simulation algorithms to enable the determination of these properties with good statistical uncertainties. Comparisons of simulation results to experimental data reveal deficiencies shared by many commonly used models. Moreover, there appear to exist specific tradeoffs within existing modeling frameworks, so that good prediction of some properties is linked to poor prediction for specific other properties. For example, non-polarizable models that utilize full charges on the ions generally fail to predict accurately both activity coefficients and solubilities; the concentration dependence of viscosity and diffusivity for these models is also incorrect. Scaled-charge models improve the dynamic properties and could also perform well for solubilities, but fail in the prediction of nucleation rates. Even models that do well at room temperature for some properties generally fail to capture their experimentally observed temperature dependence. Finally, the main conclusion from the present review is that qualitatively new physics will need to be incorporated in future models of electrolyte solutions to allow description of collective properties for broad ranges of concentrations, temperatures, and solvent conditions.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
NASA NTRS2016

Isotopic Ordering in Atmospheric O2 as a Tracer of Ozone Photochemistry and the Tropical Atmosphere

The distribution of isotopes within O2 molecules can be rapidly altered when they react with atomic oxygen. This mechanism is globally important: while other contributions to the global budget of O2 impart isotopic signatures, the O(3P) + O2 reaction resets all such signatures in the atmosphere on subdecadal timescales. Consequently, the isotopic distribution within O2 is determined by O3 photochemistry and the circulation patterns that control where that photochemistry occurs. The variability of isotopic ordering in O2 has not been established, however. We present new measurements of 18O18O in air (reported as delta36 values) from the surface to 33 km altitude. They confirm the basic features of the clumped-isotope budget of O2: Stratospheric air has higher delta36 values than tropospheric air (i.e., more 18O18O), reflecting colder temperatures and fast photochemical cycling of O3. Lower delta36 values in the troposphere arise from photochemistry at warmer temperatures balanced by the influx of high-delta36 air from the stratosphere. These observations agree with predictions derived from the GEOS-Chem chemical transport model, which provides additional insight. We find a link between tropical circulation patterns and regions where delta36 values are reset in the troposphere. The dynamics of these regions influences lapse rates, vertical and horizontal patterns of O2 reordering, and thus the isotopic distribution toward which O2 is driven in the troposphere. Temporal variations in delta36 values at the surface should therefore reflect changes in tropospheric temperatures, photochemistry, and circulation. Our results suggest that the tropospheric O3 burden has remained within a +/-10 percent range since 1978.

oxygen↗
DOE OSTI2020

Linking Small-Scale Flight Manoeuvers and Density Profiles to the Vertical Movement of Insects in the Nocturnal Stable Boundary Layer

Huge numbers of insects migrate over considerable distances in the stably-stratified night-time atmosphere with great consequences for ecological processes, biodiversity, ecosystem services and pest management. We used a combination of meteorological radar and lidar instrumentation at a site in Oklahoma, USA, to take a new look at the general assistance migrants receive from both vertical and horizontal airstreams during their long-distance flights. Movement in the nocturnal boundary layer (NBL) presents very different challenges for migrants compared to those prevailing in the daytime convective boundary layer, but we found that Lagrangian stochastic modelling is effective at predicting flight manoeuvers in both cases. A key feature for insect transport in the NBL is the frequent formation of a thin layer of fast-moving air – the low-level jet. Modelling suggests that insects can react rapidly to counteract vertical air movements and this mechanism explains how migrants are retained in the jet for long periods (e.g. overnight, and perhaps for several hours early in the morning). This results in movements over much longer distances than are likely in convective conditions, and is particularly significant for the reintroduction of pests to northern regions where they are seasonally absent due to low winter temperatures.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Adaptive Protection of Scientific Backbone Networks Using Machine Learning

In this article, we propose a new protection scheme for backbone networks to guarantee high service availability. The presented scheme does not require any reconfiguration immediately after the failure (i.e., it is proactive). At the same time, it does not require any reserved backup network resources either. To achieve these seemingly contradictory goals, we utilize the recent advancements in Machine Learning (ML) to implement a network intelligence that periodically re-allocates the unused capacity as protection bandwidth to meet the service availability requirements of each connection. Our goal is achieved by two components (1) predicting the traffic for the next period on each link, and (2) intelligently selecting the best fit dedicated protection scheme for the next period depending on the estimated unused (spare) bandwidth and the previous service availability violations. Note that re-allocating protection bandwidth affects neither the operational connections nor the current best practice of operators to over-provision network bandwidth to support elephant flows. Finally, we provide a case study on the real traffic from Energy Sciences Network (ESnet), a high-speed, international scientific backbone network. The key benefit of our framework is that adaptively utilizing the over-provisioned bandwidth for spare capacity is sufficient to improve the availability from three-nines to five-nines (in ESnet for the 30 examined connections). The drawback is negligible bandwidth limitations; the user perceives a minor and very temporal bandwidth limitation in less than 0.1% of the time.

42 ENGINEERING↗
DOE OSTI2024

Predictive Indicators of the Performance of Large Language Models

In several mission contexts, it is desirable to estimate the performance of large language models (LLMs) on tasks that we cannot run directly. In light of published “scaling laws” our hypothesis is that some tasks should be consistently more challenging than others based on characteristics of the task. The goal of this project was to begin quantifying how much information about LLM performance can be gained from the features of a model and a task. Two of our statistical models struggled to converge. Pass/fail test results may provide limited information for inference beyond model quality and task difficulty, but we see no evidence at this time for significant feature interaction effect sizes, arguing for simple models. Future work extending the models to capitalize on perplexity of ground truth answers is suggested. This project also introduces “Depth of Knowledge Variant Testing” as a strategy for more finely assessing language models on open domain question and answer tasks. We developed sets of questions that ask a language model to produce similar information while demonstrating increasing depth of knowledge, and also relabeled existing Q&A test questions with their depth of knowledge. Our results suggest further consideration of Bloom’s taxonomy and further refinement of prompts to properly elicit information at varying depths. In the course of this work, we set up a basic infrastructure for standardizing tasks and testing many language models on these tasks. In addition to testing the predictive quality of model features and performance across test suites, with this project we have introduced two new task features to contextualize each test question: the Dewey Classification main category of information covered, and the Bloom’s taxonomy level that corresponds to the depth of knowledge probed by the question. Splits across these and other features produced over five hundred task subtypes with distinct feature vectors, which we tested on half a dozen models.

97 MATHEMATICS AND COMPUTING↗