Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “sequence”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,225 records · Page 68

Amino acid interacting network in the receptor-binding domain of SARS-CoV-2 spike protein

The relation between amino acid (AA) sequence and biologically active conformation controls the process of polypeptide chains folding into three-dimensional (3d) protein structures. The recent achievements in the resolution achieved in cryo-electron microscopy coupled with improvements in computational methodologies have accelerated the analysis of structures and properties of proteins. However, the detailed interaction between AAs has not been fully elucidated. Herein, we present a de novo method to evaluate inter-amino acid interactions based on the concept of accurately evaluating the amino acid bond pairs (AABP). The results obtained enabled the identification of complex 3d long-range interconnected AA interacting network in proteins. The method is applied to the receptor binding domain (RBD) of the SARS-CoV-2 spike protein. We show that although nearest-neighbor AAs in the primary sequence have large AABP, other nonlocal AAs make substantial contribution to AABP with significant participation of both covalent and hydrogen bonding. Detailed analysis of AABP in RBD reveals the pivotal role they play in sequence conservation with profound implications on residue mutations and for therapeutic drug design. This approach could be easily applied to many other proteins of biomedical interest in life sciences.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Broadband cross polarization for ultra-wideline magic-angle spinning NMR

Over the past decade, there has been a sustained interest in using frequency-swept (FS) pulses for the efficient acquisition of wideline and ultra-wideline (UW) NMR powder patterns. Such experiments are typically conducted under static conditions, employing both direct- and indirect-excitation methods (i.e., WCPMG and BRAIN-CP/WCPMG, respectively). Recently, Koppe et al. demonstrated that the WCPMG pulse sequence can be used to efficiently acquire wideline and UW NMR spectra with spinning sideband (SSB) manifolds under magic-angle spinning (MAS) conditions, capitalizing on the increased signal-to-noise ratios (SNR) afforded by MAS. To date, there have been only a few instances of broadband cross-polarization (CP) experiments using FS pulses under MAS conditions and no applications to systems exhibiting wideline and/or ultra-wideline powder patterns, despite the clear advantages these experiments could offer. Herein, we demonstrate that FS pulses selectively applied to a single sideband of the S spin can be used for efficient 1 H-S polarization transfer to S = 1/2 nuclides with large anisotropic chemical shift interactions at slow to moderate MAS rates. The Hartmann–Hahn matching conditions in BRAIN-CP/WCPMG-MAS experiments bear similarity to those of standard CP sequences, yet operate over UW frequency ranges and only require low-amplitude RF pulses on the S channel. Crucial to the success of the BRAIN-CP/WCPMG-MAS experiment is careful calibration of the RF amplitude, transmitter offset, and effective frequency sweep of the FS pulse applied to the S spins at a given MAS rate. Thus, by means of numerical simulations and experimental testing, we provide recommendations for the parameterization and setup of BRAIN-CP/WCPMG-MAS experiments for their most efficient use. Results showcasing the capability of the BRAIN-CP/WCPMG-MAS pulse sequence are presented, including applications to 119 Sn, 195 Pt, and 103 Rh NMR.

Kimball, James J. [Florida State University, Talla↗

Genetic control of morphological transitions in a coacervating protein template

Nature routinely exploits liquid–liquid phase separation (LLPS) of proteins to control the assembly and mineralization of hybrid materials. Here, we show that fusion of the Car9 silica-binding peptide to an elastin-like polypeptide (ELP) yields temperature- and sequence-programmable soft matter templates for the synthesis of silicified architectures ranging in size from nanometers to micrometers. Specifically, we demonstrate unprecedented control over the diameter of silica nanoparticles (SiNP) in the 30–60 nm range with 4 nm precision, show that a single arginine residue (R4) in the Car9 sequence underpins the transition from micelles to proteinosomes, and find that substitutions in other basic residues modulate electrostatic repulsion and solvation to enable access to kinetically trapped species. These structures, which include interconnected micelles, small (∼200 nm) and large (>5 µm) vesicles, are readily visualized by SEM imaging following silicification. Molecular dynamics (MD) simulations and AlphaFold predictions reveal that mutations in positively charged residues alter interfacial packing, hydration, and conformational freedom of the silica-binding segments. Overall, our results establish sequence and thermal energy as synergistic levers for morphological control across length scales using solid-binding ELPs and establish mineralization as a powerful tool to visualize the structure of dynamic soft matter assemblies.

hierarchy↗

Test methodology for validation of multi-frequency models of renewable energy generators using small-signal perturbations

Providing trustworthy and accurate multi-frequency (or harmonic) models for renewable energy generators (REG) is an ongoing challenge for harmonic studies. There have been effective attempts to propose and design a test device to validate the harmonic models, mainly based on shunt current perturbations. However, using additional devices for perturbations is costly for converter-based test sites. This paper provides the test specifications to extend the application of the grid emulators for voltage perturbations and appropriate harmonic model validation. Besides, the effects of the sequence couplings, initial emissions, and power set-points on the test results have been overlooked in the literature. Considering these effects, this paper proposes a generic test methodology to obtain more accurate models in the sequence domain. The experimental verification of the proposed methodology is demonstrated using a 7 MVA grid emulator for testing of a 2 MVA photo-voltaic converter and a 2 MVA Type 3 wind turbine. This way, the test challenges, specifications, and recommendations are presented using the MW-scale experiments on different REGs. Furthermore, the effects of sequence couplings and initial emissions on the calculation results are investigated and compared. The proposed methodology is applicable for harmonic model validation as well as empirical modelling.

24 POWER TRANSMISSION AND DISTRIBUTION↗

A two-stage service restoration method for electric power distribution systems

Improving the reliability of power distribution systems is critically important for both utilities and customers. This calls for an efficient service restoration module within a distribution management system to support the implementation of self-healing smart grid networks. Although the emerging smart grid technologies, including distributed generators (DGs) and remote-controlled switches, enhance the self-healing capability and allow faster recovery, they still pose additional complexity to the service restoration problem, especially under cold load pickup (CLPU) conditions. Herein, a novel two-stage restoration framework is proposed to generate a restoration solutions with a sequence of control actions. The first stage generates a restoration plan that supports both the traditional service restoration using feeder reconfiguration and the grid-forming DG-assisted intentional islanding methods. The second stage generates an optimal sequence of switching operations to bring the outaged system quickly to the final restored configuration. The problem is formulated as a mixed-integer linear program that incorporates system connectivity, operating constraints, and the CLPU models. It is demonstrated that on using a multi-feeder test case, the proposed framework is effective in utilizing all available resources to quickly restore the service and generate an optimal sequence of switching actions to be used by the operator to reach the desired optimal configuration.

24 POWER TRANSMISSION AND DISTRIBUTION↗

photoD with Rubin ’s Data Preview 1: First stellar photometric distances and faint blue star deficits

Aims. We investigate the utility of Rubin’s Data Preview 1 (DP1) for estimating stellar number density profiles across the Milky Way halo. Methods. We used stellar broad-band near-UV to near-IR ugrizy photometry released in Rubin’s DP1 to estimate distance and metallicity for blue main sequence stars brighter than r = 24 in three ~1.1 sq. deg. fields at southern Galactic latitudes. Results. Compared to TRILEGAL simulations of the Galaxy’s stellar content, we found a likely deficit of blue main sequence turn-off stars with 22 < r < 24. We interpreted this discrepancy as a signature of a steeper halo number density profile at galactocentric distances 10–50 kpc than the canonical ~1/r 3 profile assumed in TRILEGAL simulations. Conclusions. This interpretation is consistent with earlier suggestions based on observations of more luminous, but much less numerous, evolved stellar populations, along with a few pencil beam surveys of blue main sequence stars in the northern sky. These results bode well for the future Galactic halo exploration with Rubin’s Legacy Survey of Space and Time (LSST).

Galaxy: fundamental parameters↗

Deep convolutional neural networks for multi-scale time-series classification and application to tokamak disruption prediction using raw, high temporal resolution diagnostic data

In this paper we discuss recent advances in deep convolutional neural networks (CNN) for sequence learning, which allow identifying long-range, multi-scale phenomena in long sequences, such as those found in fusion plasmas. We point out several benefits of these deep CNN architectures, such as not requiring experts such as physicists to hand-craft input data features, the ability to capture longer range dependencies compared to the more common sequence neural networks (recurrent neural networks like long short-term memory (LSTM) networks), and the comparative computational efficiency. We apply this neural network architecture to the popular problem of disruption prediction in fusion energy tokamaks, utilizing raw data from a single diagnostic, the Electron Cyclotron Emission imaging (ECEi) diagnostic from the DIII-D tokamak. Initial results trained on a large ECEi dataset show promise, achieving an F 1 -score of ~91% on individual time-slices using only the ECEi data. This indicates the ECEi diagnostic by itself can be sensitive to a number of pre-disruption markers useful for predicting disruptions on timescales not only for mitigation but also avoidance. Future opportunities for utilizing these deep CNN architectures with fusion data are outlined, including impact of recent upgrades to the ECEi diagnostic.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Computational evolution of high-performing unfused non-fullerene acceptors for organic solar cells

Materials optimization for organic solar cells (OSCs) is a highly active field, with many approaches using empirical experimental synthesis, computational brute force to screen a subset of chemical space, or generative machine learning methods that often require significant training sets. While these methods may find high-performing materials, they can be inefficient and time-consuming. Genetic algorithms (GAs) are an alternative approach, allowing for the “virtual synthesis” of molecules and a prediction of their “fitness” for some property, with new candidates suggested based on good characteristics of previously generated molecules. In this work, a GA is used to discover high-performing unfused non-fullerene acceptors (NFAs) based on an empirical prediction of power conversion efficiency (PCE) and provides design rules for future work. The electron-withdrawing/donating strength, as well as the sequence and symmetry, of those units are examined. The utilization of a GA over a brute-force approach resulted in speedups up to 1.8 × 10 12 . New types of units, not frequently seen in OSCs, are suggested, and in total 5426 NFAs are discovered with the GA. Of these, 1087 NFAs are predicted to have a PCE greater than 18%, which is roughly the current record efficiency. While the symmetry of the sequence showed no correlation with PCE, analysis of the sequence arrangement revealed that higher performance can be achieved with a donor core and acceptor end groups. Future NFA designs should consider this strategy as an alternative to the current A-D-A'-D-A architecture.

14 SOLAR ENERGY↗

Recognition of nonproline N-terminal residues by the Pro/N-degron pathway

Eukaryotic N-degron pathways are proteolytic systems whose unifying feature is their ability to recognize proteins containing N-terminal (Nt) degradation signals called N-degrons, and to target these proteins for degradation by the 26S proteasome or autophagy. GID4, a subunit of the GID ubiquitin ligase, is the main recognition component of the proline (Pro)/N-degron pathway. GID4 targets proteins through their Nt-Pro residue or a Pro at position 2, in the presence of specific downstream sequence motifs. Here we show that human GID4 can also recognize hydrophobic Nt-residues other than Pro. One example is the sequence Nt-IGLW, bearing Nt-Ile. Nt-IGLW binds to wild-type human GID4 with a $K_d$ of 16 μM, whereas the otherwise identical Nt-Pro–bearing sequence PGLW binds to GID4 more tightly, with a $K_d$ of 1.9 μM. Despite this difference in affinities of GID4 for Nt-IGLW vs. Nt-PGLW, we found that the GID4-mediated Pro/N-degron pathway of the yeast Saccharomyces cerevisiae can target an Nt-IGLW–bearing protein for rapid degradation. We solved crystal structures of human GID4 bound to a peptide bearing Nt-Ile or Nt-Val. We also altered specific residues of human GID4 and measured the affinities of resulting mutant GID4s for Nt-IGLW and Nt-PGLW, thereby determining relative contributions of specific GID4 residues to the GID4-mediated recognition of Nt-Pro vs. Nt-residues other than Pro. These and related results advance the understanding of targeting by the Pro/N-degron pathway and greatly expand the substrate recognition range of the GID ubiquitin ligase in both human and yeast cells.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

The F-box protein gene exo-1 is a target for reverse engineering enzyme hypersecretion in filamentous fungi

Carbohydrate active enzymes (CAZymes) are vital for the lignocellulose-based biorefinery. The development of hypersecreting fungal protein production hosts is therefore a major aim for both academia and industry. However, despite advances in our understanding of their regulation, the number of promising candidate genes for targeted strain engineering remains limited. Here, we resequenced the genome of the classical hypersecreting Neurospora crassa mutant exo-1 and identified the causative point of mutation to reside in the F-box protein–encoding gene, NCU09899. The corresponding deletion strain displayed amylase and invertase activities exceeding those of the carbon catabolite derepressed strain ?cre-1, while glucose repression was still mostly functional in ?exo-1. Surprisingly, RNA sequencing revealed that while plant cell wall degradation genes are broadly misexpressed in ?exo-1, only a small fraction of CAZyme genes and sugar transporters are up-regulated, indicating that EXO-1 affects specific regulatory factors. Aiming to elucidate the underlying mechanism of enzyme hypersecretion, we found the high secretion of amylases and invertase in ?exo-1 to be completely dependent on the transcriptional regulator COL-26. Furthermore, misregulation of COL-26, CRE-1, and cellular carbon and nitrogen metabolism was confirmed by proteomics. Finally, we successfully transferred the hypersecretion trait of the exo-1 disruption by reverse engineering into the industrially deployed fungus Myceliophthora thermophila using CRISPR-Cas9. Our identification of an important F-box protein demonstrates the strength of classical mutants combined with next-generation sequencing to uncover unanticipated candidates for engineering. These data contribute to a more complete understanding of CAZyme regulation and will facilitate targeted engineering of hypersecretion in further organisms of interest.

Gabriel, Raphael↗

Plant ecological genomics at the limits of life in the Atacama Desert

The Atacama Desert in Chile—hyperarid and with high–ultraviolet irradiance levels—is one of the harshest environments on Earth. Yet, dozens of species grow there, including Atacama-endemic plants. Herein, we establish the Talabre–Lejía transect (TLT) in the Atacama as an unparalleled natural laboratory to study plant adaptation to extreme environmental conditions. We characterized climate, soil, plant, and soil–microbe diversity at 22 sites (every 100 m of altitude) along the TLT over a 10-y period. We quantified drought, nutrient deficiencies, large diurnal temperature oscillations, and pH gradients that define three distinct vegetational belts along the altitudinal cline. Further, we deep-sequenced transcriptomes of 32 dominant plant species spanning the major plant clades, and assessed soil microbes by metabarcoding sequencing. The top-expressed genes in the 32 Atacama species are enriched in stress responses, metabolism, and energy production. Moreover, their root-associated soils are enriched in growth-promoting bacteria, including nitrogen fixers. To identify genes associated with plant adaptation to harsh environments, we compared 32 Atacama species with the 32 closest sequenced species, comprising 70 taxa and 1,686,950 proteins. To perform phylogenomic reconstruction, we concatenated 15,972 ortholog groups into a supermatrix of 8,599,764 amino acids. Using two codon-based methods, we identified 265 candidate positively selected genes (PSGs) in the Atacama plants, 64% of which are located in Pfam domains, supporting their functional relevance. For 59/184 PSGs with an Arabidopsis ortholog, we uncovered functional evidence linking them to plant resilience. As some Atacama plants are closely related to staple crops, these candidate PSGs are a “genetic goldmine” to engineer crop resilience to face climate change.

54 ENVIRONMENTAL SCIENCES↗

Structure-based design of nanobodies that inhibit seeding of Alzheimer’s patient–extracted tau fibrils

Despite much effort, antibody therapies for Alzheimer’s disease (AD) have shown limited efficacy. Challenges to the rational design of effective antibodies include the difficulty of achieving specific affinity to critical targets, poor expression, and antibody aggregation caused by buried charges and unstructured loops. To overcome these challenges, we grafted previously determined sequences of fibril-capping amyloid inhibitors onto a camel heavy chain antibody scaffold. These sequences were designed to cap fibrils of tau, known to form the neurofibrillary tangles of AD, thereby preventing fibril elongation. The nanobodies grafted with capping inhibitors blocked tau aggregation in biosensor cells seeded with postmortem brain extracts from AD and progressive supranuclear palsy (PSP) patients. The tau capping nanobody inhibitors also blocked seeding by recombinant tau oligomers. Another challenge to the design of effective antibodies is their poor blood–brain barrier (BBB) penetration. In this study, we also designed a bispecific nanobody composed of a nanobody that targets a receptor on the BBB and a tau capping nanobody inhibitor, conjoined by a flexible linker. We provide evidence that the bispecific nanobody improved BBB penetration over the tau capping inhibitor alone after intravenous administration in mice. Our results suggest that the design of synthetic antibodies that target sequences that drive protein aggregation may be a promising approach to inhibit the prion-like seeding of tau and other proteins involved in AD and related proteinopathies.

60 APPLIED LIFE SCIENCES↗

Engineered mRNA–ribosome fusions for facile biosynthesis of selenoproteins

Ribosomes are often used in synthetic biology as a tool to produce desired proteins with enhanced properties or entirely new functions. However, repurposing ribosomes for producing designer proteins is challenging due to the limited number of engineering solutions available to alter the natural activity of these enzymes. In this study, we advance ribosome engineering by describing a novel strategy based on functional fusions of ribosomal RNA (rRNA) with messenger RNA (mRNA). Specifically, we create an mRNA–ribosome fusion called RiboU, where the 16S rRNA is covalently attached to selenocysteine insertion sequence (SECIS), a regulatory RNA element found in mRNAs encoding selenoproteins. When SECIS sequences are present in natural mRNAs, they instruct ribosomes to decode UGA codons as selenocysteine (Sec, U) codons instead of interpreting them as stop codons. This enables ribosomes to insert Sec into the growing polypeptide chain at the appropriate site. Our work demonstrates that the SECIS sequence maintains its functionality even when inserted into the ribosome structure. As a result, the engineered ribosomes RiboU interpret UAG codons as Sec codons, allowing easy and site-specific insertion of Sec in a protein of interest with no further modification to the natural machinery of protein synthesis. To validate this approach, we use RiboU ribosomes to produce three functional target selenoproteins in Escherichia coli by site-specifically inserting Sec into the proteins’ active sites. Overall, our work demonstrates the feasibility of creating functional mRNA–rRNA fusions as a strategy for ribosome engineering, providing a novel tool for producing Sec-containing proteins in live bacterial cells.

59 BASIC BIOLOGICAL SCIENCES↗

Improved deep learning prediction of antigen–antibody interactions

Identifying antibodies that neutralize specific antigens is crucial for developing effective immunotherapies, but this task remains challenging for many target antigens. The rise of deep learning–based computational approaches presents a promising avenue to address this challenge. Here, we assess the performance of a deep learning approach through two benchmark tests aimed at predicting antibodies for the receptor-binding domain of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) spike protein. Three different strategies for constructing input sequence alignments are employed for predicting structural models of antigen–antibody complexes. In our initial testing set, which comprises known experimental structures, these strategies collectively yield a significant top-ranked prediction for 61% of cases and a success rate of 47%. Notably, one strategy that utilizes the sequences of known antigen binders outperforms the other two, achieving a precision of 90% in a subsequent test set of ~1,000 antibodies, balanced between true and control antibodies for the antigen, albeit with a lower recall of 25%. Our results underscore the potential of integrating deep learning methods with single B cell sequencing techniques to enhance the prediction accuracy of antigen–antibody interactions.

Science & Technology - Other Topics↗

A molecular view of peptoid-induced acceleration of calcite growth

The extensive deposits of calcium carbonate (CaCO 3 ) generated by marine organisms constitute the largest and oldest carbon dioxide (CO 2 ) reservoir. These organisms utilize macromolecules like peptides and proteins to facilitate the nucleation and growth of carbonate minerals, serving as an effective method for CO 2 sequestration. However, the precise mechanisms behind this process remain elusive. In this study, we report the use of sequence-defined peptoids, a class of peptidomimetics, to achieve the accelerated calcite step growth kinetics with the molecular level mechanistic understanding. By designing peptoids with hydrophilic and hydrophobic blocks, we systematically investigated the acceleration in step growth rate of calcite crystals using in situ atomic force microscopy (AFM), varying peptoid sequences and concentrations, CaCO 3 supersaturations, and the ratio of Ca 2+ / HCO 3 − . Mechanistic studies using NMR, three-dimensional fast force mapping (3D FFM), and isothermal titration calorimetry (ITC) were conducted to reveal the interactions of peptoids with Ca 2+ and HCO 3 − ions in solution, as well as the effect of peptoids on solvation and energetics of calcite crystal surface. Our results indicate the multiple roles of peptoid in facilitating HCO 3 − deprotonation, Ca 2+ desolvation, and the disruption of interfacial hydration layers of the calcite surface, which collectively contribute to a peptoid-induced acceleration of calcite growth. These findings provide guidelines for future design of sequence-specific biomimetic polymers as crystallization promoters, offering potential applications in environmental remediation (such as CO 2 sequestration), biomedical engineering, and energy storage where fast crystallization is preferred.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Sneaking in SpyCatcher using cell penetrating peptides for in vivo imaging

Abstract In vivo imaging of protein complexes is a powerful method for understanding the underlying biological function of these key biomolecules. Though the engineering of small, high affinity nanobodies have become more prevalent, the off-rates of these tags may result in incomplete or partial labeling of proteins in live cells. The SpyCatcher003 and SpyTag split protein system allow for irreversible, covalent binding to a short target peptide unlike nanobody-affinity based probes. However, delivering these tags into a cell without disrupting its normal function is a key challenge. Cell penetrating peptides (CPPs) are short peptide sequences that facilitate the transduction of otherwise membrane-impermeable ‘cargo’ , such as proteins, into cells. Here we report on our efforts to engineer and characterize CPP-SpyCatcher003 fusions as modular imaging probes. We selected three CPPs, CUPID, Pentratin, and pVEC, to engineer fusion protein probes for superresolution microscopy, with the aim to eliminate prior permeabilization treatments that could introduce imaging artifacts. We find that fusing the CPP sequences to SpyCatcher003 resulted in dimer and multimer formation as determined by size exclusion chromatography, dynamic light scattering, and SDS resistant dimers on SDS-PAGE gels. By isolating and labeling the monomeric forms of the engineered protein, we show these constructs retained their ability to bind SpyTag and all three CPP sequences remain membrane active, as assessed by CD spectroscopy in the presence of SDS detergent. Using fluorescence and super resolution Lattice structured illumination microscopy (Lattice SIM) imaging we show that the CPPs did not enhance uptake of SpyCatcher by E. coli, however with Caulobacter crescentus cells, we show that Penetratin, and to a lesser degree CUPID, does enhance uptake. Our results demonstrate the ability of the CPP-SpyCatcher003 to label targets within living cells, providing the groundwork for using split protein systems for targeted in vivo imaging.

59 BASIC BIOLOGICAL SCIENCES↗

ALTSched: Improved Scheduling for Time-domain Science with LSST

Abstract Telescope scheduling is the task of determining the best sequence of observations (pointings and filter choices) for a survey system. Because it is computationally intractable to optimize over all possible multiyear sequences of observations, schedulers use heuristics to pick the best observation at a given time. A greedy scheduler selects the next observation by choosing whichever one maximizes a scalar merit function, which serves as a proxy for the scientific goals of the telescope. This sort of bottom-up approach for scheduling is not guaranteed to produce a schedule for which the sum of merit over all observations is maximized. As an alternative to greedy schedulers, we introduce ALTS ched, which takes a top-down approach to scheduling. Instead of considering only the next observation, ALTS ched makes global decisions about which area of sky and which filter to observe in, and then refines these decisions into a sequence of observations taken along the meridian to minimize airmass. We implement ALTS ched for the Large Synoptic Survey Telescope (LSST) and show that it equals or outperforms the baseline greedy scheduler in essentially all quantitative performance metrics. Due to its simplicity, our implementation is considerably faster than O p S im , the simulated greedy scheduler currently used by the LSST Project: a full 10-yr survey can be simulated in 4 minutes, as opposed to tens of hours for O p S im . LSST’s hardware is fixed, so improving the scheduling algorithm is one of the only remaining ways to optimize LSST’s performance. We see ALTS ched as a prototype scheduler that gives a lower bound on the performance achievable by LSST.

79 ASTRONOMY AND ASTROPHYSICS↗

MATEY: multiscale adaptive transformer models for spatiotemporal physical systems

Accurate representation of the multiscale features in spatiotemporal physical systems using vision transformer architectures requires extremely long, computationally prohibitive token sequences. To address this issue, we propose two novel adaptive tokenization schemes that dynamically adjust patch sizes based on local features: one ensures convergent behavior to uniform patch refinement, while the other offers better computational efficiency. Moreover, we present a set of spatiotemporal attention schemes, where the temporal or axial spatial dimensions are decoupled, to evaluate their baseline computational and data efficiencies and to determine whether adaptive tokenization can improve this performance. We assess the performance of the proposed multiscale adaptive model, MATEY, in a sequence of experiments. Compared to a full spatiotemporal attention scheme or a scheme that decouples only the temporal dimension, we find that fully decoupled axial attention is less efficient and expressive, requiring more training time and model parameters to achieve the same accuracy. The experiments on the adaptive tokenization schemes show that, compared to a uniformly refined model, the proposed schemes achieve comparable or improved accuracy at a much lower cost in the tested two-dimensional settings. While the asymptotic analysis suggests the potential for favorable scaling, empirical validation at substantially longer sequence lengths remains to be performed in future work. Finally, we demonstrate in two fine-tuning tasks featuring different physics that models pretrained on PDEBench data outperform the ones trained from scratch, especially in the low data regime with frozen attention.

adaptive tokenization↗