Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data classification”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,459 records · Page 81
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Metabolic Response in Patients With Post-treatment Lyme Disease Symptoms/Syndrome

Abstract Background Post-treatment Lyme disease symptoms/syndrome (PTLDS) occurs in approximately 10% of patients with Lyme disease following antibiotic treatment. Biomarkers or specific clinical symptoms to identify patients with PTLDS do not currently exist and the PTLDS classification is based on the report of persistent, subjective symptoms for ≥6 months following antibiotic treatment for Lyme disease. Methods Untargeted liquid chromatography–mass spectrometry metabolomics was used to determine longitudinal metabolic responses and biosignatures in PTLDS and clinically cured non-PTLDS Lyme patients. Evaluation of biosignatures included (1) defining altered classes of metabolites, (2) elastic net regularization to define metabolites that most strongly defined PTLDS and non-PTLDS patients at different time points, (3) changes in the longitudinal abundance of metabolites, and (4) linear discriminant analysis to evaluate robustness in a second patient cohort. Results This study determined that observable metabolic differences exist between PTLDS and non-PTLDS patients at multiple time points. The metabolites with differential abundance included those from glycerophospholipid, bile acid, and acylcarnitine metabolism. Distinct longitudinal patterns of metabolite abundance indicated a greater metabolic variability in PTLDS versus non-PTLDS patients. Small numbers of metabolites (6 to 40) could be used to define PTLDS versus non-PTLDS patients at defined time points, and the findings were validated in a second cohort of PTLDS and non-PTLDS patients. Conclusions These data provide evidence that an objective metabolite-based measurement can distinguish patients with PTLDS and help understand the underlying biochemistry of PTLDS.

Immunology↗
DOE OSTI2022

Disaster risk and artificial intelligence: A framework to characterize conceptual synergies and future opportunities

Artificial intelligence (AI) methods have revolutionized and redefined the landscape of data analysis in business, healthcare, and technology. These methods have innovated the applied mathematics, computer science, and engineering fields and are showing considerable potential for risk science, especially in the disaster risk domain. The disaster risk field has yet to define itself as a necessary application domain for AI implementation by defining how to responsibly balance AI and disaster risk. (1) How is AI being used for disaster risk applications; and how are these applications addressing the principles and assumptions of risk science, (2) What are the benefits of AI being used for risk applications; and what are the benefits of applying risk principles and assumptions for AI-based applications, (3) What are the synergies between AI and risk science applications, and (4) What are the characteristics of effective use of fundamental risk principles and assumptions for AI-based applications? This study develops and disseminates an online survey questionnaire that leverages expertise from risk and AI professionals to identify the most important characteristics related to AI and risk, then presents a framework for gauging how AI and disaster risk can be balanced. This study is the first to develop a classification system for applying risk principles for AI-based applications. This classification contributes to understanding of AI and risk by exploring how AI can be used to manage risk, how AI methods introduce new or additional risk, and whether fundamental risk principles and assumptions are sufficient for AI-based applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Unravelling the origin of extended radio emission in narrow-line Seyfert 1 galaxies with the JVLA

Narrow-line Seyfert 1 galaxies (NLS1s) are believed to be active galactic nuclei (AGN) in the early stages of their evolution. Several dozen of them have been found to host relativistic jets, whilst the majority of NLS1s have not even been detected in radio, emphasising the heterogeneity of the class in this band. In this paper, our aim is to determine the predominant source of radio emission in a sample of 44 NLS1s, selected based on their extended kiloparsec-scale radio morphologies at 5.2 GHz. Here, we accomplish this by analysing their spatially resolved radio spectral index maps, centred at 5.2 GHz, as the spectral index carries information about the production mechanisms of the emission. In addition, we utilise several diagnostics based on mid-infrared emission to estimate the star formation activity of their host galaxies. These data are complemented by archival data to draw a more complete picture of each source. We find an extraordinary diversity among our sample. Approximately equal fractions (~10–12 sources) of our sources can be identified as AGN-dominated, composite, and host-dominated. Among the AGN-dominated sources are a few NLS1s with very extended jets, reaching distances of tens of kiloparsecs from the nucleus. One of these, J0814+5609, hosts the most extended jets found in an NLS1 so far. We also identify five NLS1s that could be classified as compact steep-spectrum sources. In addition, one source shows a possible kiloparsec-scale relic that reaches well outside the host galaxy as well as restarted nuclear activity, and one could belong to the sub-class of NLS1s that host relativistic jets that seem to be absorbed at lower radio frequencies (< 10 GHz). We further conclude that, due to the variety seen in NLS1s, simple proxies, such as the star formation diagnostics also employed in this paper and the radio loudness parameter, are not ideal tools for characterising NLS1s. We emphasise the necessity of examining NLS1s as individuals instead of making assumptions based on their classification. When these issues are properly taken into account, NLS1s offer an exceptional environment for studying the interplay between the host galaxy and several AGN-related phenomena, such as jets and outflows.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Harnessing Machine Learning and Data Fusion for Accurate Undocumented Well Identification in Satellite Images

This study utilizes satellite data to detect undocumented oil and gas wells, which pose significant environmental concerns, including greenhouse gas emissions. Three key findings emerge from the study. Firstly, the problem of imbalanced data is addressed by recommending oversampling techniques like Rotation–GaussianBlur–Solarization data augmentation (RGS), the Synthetic Minority Over-Sampling Technique (SMOTE), or ADASYN (an extension of SMOTE) over undersampling techniques. The performance of borderline SMOTE is less effective than that of the rest of the oversampling techniques, as its performance relies heavily on the quality and distribution of data near the decision boundary. Secondly, incorporating pre-trained models trained on large-scale datasets enhances the models’ generalization ability, with models trained on one county’s dataset demonstrating high overall accuracy, recall, and F1 scores that can be extended to other areas. This transferability of models allows for wider application. Lastly, including persistent homology (PH) as an additional input improves performance for in-distribution testing but may affect the model’s generalization for out-of-distribution testing. A careful consideration of PH’s impact on overall performance and generalizability is recommended. Overall, this study provides a robust approach to identifying undocumented oil and gas wells, contributing to the acceleration of a net-zero economy and supporting environmental sustainability efforts.

SMOTE↗
DOE OSTI2023

Classification of River Catchments in the Contiguous United States: Code, Dataset, Similarity Patterns, and Resulting Classes

This dataset serves as supplementary information for the paper by Ciulla F. and Varadharajan C. A Network Approach for Multiscale Catchment Classification using Traits (see reference 1). It contains environmental and physical catchment traits, such as temperatures, precipitation, land use and human interference, from 9067 sites across the contiguous United States (CONUS). The purpose of this dataset is to provide information for a better trait-based categorization of river catchments in the CONUS using networks as an analytical tool. The traits variables match the ones present in the GAGES-II dataset and the preprocessing steps are described in the Methods section (processed_dataset.csv). Additionally we include the topologies (nodes, edges and clusters, also referred as classes) of the catchment network and traits network generated by said dataset (csv and json files). A series of tables support the information carried by the network providing more detailed descriptions of cluster components (SI1.pdf). A summary of all the plots of clusters of catchments with at least 50 nodes is provided (SI2.pdf). The characteristic traits for each cluster of catchments is presented as z-score (traits_categories_zscores_per_catchment_class.csv). The link to the hydrological behavior of clusters of catchments is displayed by boxplots, each describing a particular river discharge index (SI3.pdf). Both csv and json files can be read by common text editors but the data contained into them can be better handled using programming languages like python and database oriented libraries like pandas. Pdf files can be read by any pdf reader software.[02-23-2024] Update: The code and datasets necessary to reproduce the results of the study are available as a zipped repository (code_datasets_catchments_similarity.zip).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Machine learning reveals genes impacting oxidative stress resistance across yeasts

Reactive oxygen species (ROS) are highly reactive molecules encountered by yeasts during routine metabolism and during interactions with other organisms, including host infection. Here, we characterized the variation in resistance to ROS across the ancient yeast subphylum Saccharomycotina and used machine learning (ML) to identify gene families whose sizes were predictive of ROS resistance.

AI↗
DOE OSTI2024

Extracting Vehicle Trajectories from Partially Overlapping Roadside Radar

This work presents a methodology for extracting vehicle trajectories from six partially-overlapping roadside radars through a signalized corridor. The methodology incorporates radar calibration, transformation to the Frenet space, Kalman filtering, short-term prediction, lane-classification, trajectory association, and a covariance intersection-based approach to track fusion. The resulting dataset contains 79,000 fused radar trajectories over a 26-h period, capturing diverse driving scenarios including signalized intersections, merging behavior, and a wide range of speeds. Compared to popular trajectory datasets such as NGSIM and highD, this dataset offers extended temporal coverage, a large number of vehicles, and varied driving conditions. The filtered leader–follower pairs from the dataset provide a substantial number of trajectories suitable for car-following model calibration. The framework and dataset presented in this work has the potential to be leveraged broadly in the study of advanced traffic management systems, autonomous vehicle decision-making, and traffic research.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2025

Machine Learning for Well Log Analysis in Uranium Mining

This project explores the use of Artificial Intelligence (AI) and Machine Learning (ML) techniques to automate well log analysis for uranium mining. Geophysical log data—spontaneous potential, resistivity, and gamma ray—were used to classify lithology, correlate well logs and identify roll front zonation patterns, which are critical for locating uranium ore bodies. Supervised ML algorithms such as eXtreme Gradient Boosting (XGBoost), Categorical Boosting (CatBoost), and Random Forest were trained to classify lithology with high accuracy. Gradient Boosting Machines (GBM), XGBoost, Random Forest, and Neural Networks were also used for role front zone identification. Moreover, a Fast Dynamic Time Warping (FastDTW) algorithm was employed for well log correlation. Additionally, sample lag was addressed using dynamic programming. Results demonstrate the potential of AI and ML to streamline well log analysis and enhance uranium exploration workflows.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2024

Whole-genome demography of COVID-19 virus during its pandemic period and on “panvalent” vaccine design

With over 16 million submitted genomic sequences, the SARS-CoV-2 (SC2) virus, the cause of the most recent worldwide COVID-19 pandemic, has become the most sequenced genome of all known viruses, revealing, for example, a vast number of expanding viral lineages. Since the pandemic phase appears to be over, we performed a retrospective re-examination of the demographic grouping pattern and their genomic characteristics during the entire pandemic period up to the peak of the last pandemic wave. For our study, we extracted from the NCBI only unique viral sequences and converted each sequence data to a relational vector, indicating the presence/absence of each variational event compared to a “reference” sequence. Our study revealed several genomic features that are unexpected or different from those of previous studies. For example, approximately 44,000 variants with unique sequences emerged during the pandemic period; they group into only four major viral-genomic groups and each has a set of mostly unique highly-conserved variant-genotypes (HCVGs); and a small set from the first (“ancestral”) group was inherited by the three (“descendant”) groups, suggesting that HCVGs in the next group may be predictable from the current group(s). Such a concept may be potentially important in designing “panvalent” vaccines against the current and future waves of viral infections.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2020

Radar Derived Rainfall and Rain Gauge Measurements at SRS

Over the years rainfall data for the Savannah River Site has been obtained from ground level measurements made by rain gauges. These instruments have inherent errors or biases that can impact the measured rainfall totals but are assumed as ground truth for most climatological and weather applications. With the development of weather radar technologies, various methods to derive rainfall totals from radar reflectivity values have been developed and have continued to improve. The Z-R relationship, which uses an exponential relationship to estimate rainfall rate based on radar reflectivity values, provides estimates of rainfall amounts (Z-R Level III) for locations within the radar’s detection range. The recently developed Multi-Radar Multi-Sensor (MRMS) dataset combines reflectivity-based estimates using the Z-R relationship with a network of gauges and other rainfall estimates to produce a refined set of precipitation estimates for each grid point within its domain. Comparisons were done between gauge observations and radar estimates for various SRS locations to assess whether radar derived estimates are representative of rainfall measurements at the site. Results obtained show good agreement between radar derived amounts and ground measurements, with MRMS showing stronger correlations and lower spread than Z-R Level III estimates. Outliers and errors observed appear to be related to hydrometeor classification schemes. In general, MRMS proved to provide sufficiently representative estimates of daily rainfall amounts to replace existing SRS rain gauge measurements.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

SEM petrography of dispersed organic matter in black shales: A review

Organic matter (OM)-hosted pores are important constituents of the pore system of black shales and play a crucial role in determining their methane adsorption capacity and porosity. OM-hosted pores are generally observed and described with scanning electron microscope (SEM) on Ar ion-milled surfaces. However, SEM imaging is not able to reliably distinguish OM types and relate the observed pores to specific macerals. Partly because of this inability to relate organic pores to macerals, the evolution of organic porosity during thermal maturation remains poorly understood. In this paper, we review the petrographic characteristics of dispersed organic matter (DOM) in black shales under the SEM. Organic petrographic classification of DOM developed for reflected-light microscopy is so far the most practical method when describing DOM in black shales under the SEM because this classification has information on the origin of DOM. Therefore, correlative microscopy (combination of reflected-light and electron microscopy) is the most effective method to identify both OM types and OM-hosted pores. This method, however, is not readily available to most researchers. Although identifying OM on the basis of SEM observations is a challenging task, it is achievable provided there is a good understanding of the studied shales, especially their thermal maturity and original OM composition. Therefore, the overall objective of this paper is to review petrographic characteristics of DOM in black shales under the SEM to provide some guidelines for identifying DOM from SEM observations. We also review factors that control the formation and preservation of OM-hosted pores. OM-hosted pores consist of primary and secondary organic pores. Primary organic pores are pores inherited from the biological structure of the original OM. Secondary organic pores develop during hydrocarbon generation and expulsion from oil-prone OM and are hosted by solid bitumen or pyrobitumen. The development of secondary organic pores is controlled by thermal maturity and OM type, and their preservation is subject to thermal maturity, OM content, and mineralogical composition. The presented view of the evolution of micropore and mesopore characteristics of OM with thermal maturity is based on data from the literature. The specific surface area and pore volume of OM in black shales follow a parabolic pattern with increasing thermal maturity (quantified via vitrinite reflectance, R o ). The initial increase reflects development of OM-hosted pores, and the subsequent decrease is due to denser stacking of aromatic units in the macromolecular structure of OM, with maximum values (specific surface area ~ 300 m 2 /g and pore volume ~ 0.3 cm 3 /g) reached at R o values in the 2.5-3.5% range. The contribution of OM-hosted pores to the pore characteristics of black shales depends on OM content, OM type, and thermal maturity.

03 NATURAL GAS↗
DOE OSTI2024

Machine Learning Correlation of Electron Micrographs and ToF-SIMS for the Analysis of Organic Biomarkers in Mudstone

The spatial distribution of organics in geological samples can be used to determine when and how these organics were incorporated into the host rock. Mass spectrometry (MS) imaging can rapidly collect a large amount of data, but ions produced are mixed without discrimination, resulting in complex mass spectra that can be difficult to interpret. Here, we apply unsupervised and supervised machine learning (ML) to help interpret spectra from time-of-flight-secondary ion mass spectrometry (ToF-SIMS) of an organic-carbon-rich mudstone of the Middle Jurassic of England (UK). It was previously shown that the presence of sterane molecular biomarkers in this sample can be detected via ToF-SIMS (Pasterski, M. J. et al., Astrobiology 2023, 23, 936). We use unsupervised ML on scanning electron microscopy–electron dispersive spectroscopy (SEM-EDS) measurements to define compositional categories based on differences in elemental abundances. We then test the ability of four ML algorithms─k-nearest neighbors (KNN), recursive partitioning and regressive trees (RPART), eXtreme gradient boost (XGBoost), and random forest (RF)─to classify the ToF-SIM spectra using (1) the categories assigned via SEM-EDS, (2) organic and inorganic labels assigned via SEM-EDS, and (3) the presence or absence of detectable steranes in ToF-SIMS spectra. In terms of predictive accuracy and balanced accuracy, KNN was the best performing model and RPART the worst. The feature importance, or the specific features of the ToF-SIM spectra used by the models to make classifications, cannot be determined for KNN, preventing posthoc model interpretation. Nevertheless, the feature importance extracted from the other models was useful for interpreting spectra. In conclusion, we determined that some of the organic ions used to classify biomarker containing spectra may be fragment ions derived from kerogen which is abundant in this mudstone sample.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Evaluation of KDP Estimation Algorithm Performance in Rain Using a Known-Truth Framework

Accurate estimation of specific differential phase ( K DP ) is necessary for rain rate estimation, attenuation correction, and hydrometeor classification algorithms. There are numerous published methods to process polarimetric radar observations of propagation differential phase shift (Φ DP ) and estimate K DP , but the corresponding K DP estimate uncertainty is unquantified. This study provides guidance on how commonly used K DP estimation algorithms perform in various environments. Here, we create numerous synthetic (“true”) K DP profiles, integrate over them to obtain “smoothed” Φ DP , and then add noise typical of S-band operational weather radar measurements. Each algorithm is applied to our noisy Φ DP profiles and compared to the true K DP profile such that the errors and uncertainty are quantified. The synthetic K DP profiles are Gaussian in shape, which allows systematic variations in their magnitude and width to determine how each algorithm performs in smooth, slowly changing K DP profiles, as well as steep profiles. Results demonstrate that algorithm performance is dependent on the Φ DP field received. These results are further supported by an error analysis of each algorithm for two more complicated synthetic K DP profiles. Some K DP algorithms allow users to change various tuning parameters; a subset of these tuning parameters is tested to provide guidance on how changing these parameters impacts algorithm performance. We then provide evidence that our known-truth framework provides insight into algorithm performance in observed data through two case studies.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Incidence of chronic respiratory conditions among oil spill responders: Five years of follow-up in the Deepwater Horizon Oil Spill Coast Guard Cohort study

Background: Over ten years after the Deepwater Horizon (DWH) oil spill, our understanding of long term respiratory health risks associated with oil spill response exposures is limited. Here we conducted a prospective analysis in a cohort of U.S. Coast Guard personnel with universal military healthcare. Methods: For all active duty cohort members (N = 45,193) in the DWH Oil Spill Coast Guard Cohort Study we obtained medical encounter data from October 01, 2007 to September 30, 2015 (i.e., ~2.5 years pre-spill; ~5.5 years post-spill). We used Cox Proportional Hazards regressions to calculate adjusted hazard ratios (aHR), comparing risks for incident respiratory conditions/symptoms (2010–2015) for: responders vs. non-responders; responders reporting crude oil exposure, any inhalation of crude oil vapors, and being in the vicinity of burning crude oil versus responders without those exposures. We also evaluated self-reported crude oil and oil dispersant exposures, combined. Within-responder comparisons were adjusted for age, sex, and smoking. Results: While elevated aHRs for responder/non-responder comparisons were generally weak, within-responder comparisons showed stronger risks with exposure to crude oil. Notably, for responders reporting exposure to crude oil via inhalation, there were elevated risks for all sinusitis (aHR = 1.48; 95%CI, 1.06–2.06), unspecified chronic sinusitis (aHR = 1.55; 95%CI, 1.08–2.22), chronic obstructive pulmonary disease (COPD) and other allied conditions (aHR = 1.43; 95%CI, 1.00–2.06), and dyspnea and respiratory abnormalities (aHR = 1.29; 95%CI, 1.00–1.67); there was a suggestion of elevated risk for diseases classified as asthma and reactive airway diseases (aHR = 1.18; 95%CI, 0.98–1.41), including the specific condition, asthma (aHR = 1.35; 95%CI, 0.80–2.27), the symptom, shortness of breath (aHR = 1.50; 95%CI, 0.89–2.54), and the overall classification of chronic respiratory conditions (aHR = 1.18; 95%CI, 0.98–1.43). Exposure to both crude oil and dispersant was positively associated with elevated risk for shortness of breath (HR = 2.24; 95%CI, 1.09–4.64). Conclusions: Among active duty Coast Guard personnel, oil spill clean-up exposures were associated with moderately increased risk for longer term respiratory conditions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Variation in carbon and nitrogen concentrations among peatland categories at the global scale

Peatlands account for 15 to 30% of the world’s soil carbon (C) stock and are important controls over global nitrogen (N) cycles. However, C and N concentrations are known to vary among peatlands contributing to the uncertainty of global C inventories, but there are few global studies that relate peatland classification to peat chemistry. We analyzed 436 peat cores sampled in 24 countries across six continents and measured C, N, and organic matter (OM) content at three depths down to 70 cm. Sites were distinguished between northern (387) and tropical (49) peatlands and assigned to one of six distinct broadly recognized peatland categories that vary primarily along a pH gradient. Peat C and N concentrations, OM content, and C:N ratios differed significantly among peatland categories, but few differences in chemistry with depth were found within each category. Across all peatlands C and N concentrations in the 10–20 cm layer, were 440 ± 85.1 g kg -1 and 13.9 ± 7.4 g kg -1 , with an average C:N ratio of 30.1 ± 20.8. Among peatland categories, median C concentrations were highest in bogs, poor fens and tropical swamps (446–532 g kg -1 ) and lowest in intermediate and extremely rich fens (375–414 g kg -1 ). The C:OM ratio in peat was similar across most peatland categories, except in deeper samples from ombrotrophic tropical peat swamps that were higher than other peatlands categories. Peat N concentrations and C:N ratios varied approximately two-fold among peatland categories and N concentrations tended to be higher (and C:N lower) in intermediate fens compared with other peatland types. This study reports on a unique data set and demonstrates that differences in peat C and OM concentrations among broadly classified peatland categories are predictable, which can aid future studies that use land cover assessments to refine global peatland C and N stocks.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

The Building Adapter: Automatic Mapping of Commercial Buildings for Scalable Building Analytics

This project creates new solutions for the manual metadata mapping problem: the costly process of creating a match between a building’s sensor data streams and the inputs of a building analytics engine. This goal is achieved by creating and improving techniques for metadata inference: automatically constructing new contextual information for sensing and control points based on the sensor point names and the raw time series values. The objective is to enable vendors to apply building analytics to 90% of buildings with no manual mapping, and to 10% of buildings with a 90% reduction in manual mapping. These targets are set for all types of metadata required by current analytics engines, including type, location, equipment type, and other relationships. The outcome of this project is a suite of solutions to the manual mapping problem collectively called the Building Adapter that allows vendors to apply analytics engines to new buildings at a significantly reduced cost.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2025

A hybrid neural architecture: Online attosecond x-ray characterization

The emergence of high-repetition-rate x-ray free-electron lasers (XFELs), such as SLAC’s LCLS-II, serves as our canonical example for autonomous controls that necessitate high-throughput diagnostics paired with streaming computational pipelines capable of single-shot analysis with extremely low latency. We present the deterministic characterization with an integrated parallelizable hybrid resolver architecture, a hybrid machine learning framework designed for fast, accurate analysis of XFEL diagnostics using angular streaking-based sinogram images. This architecture integrates convolutional neural networks and bidirectional long short-term memory models to denoise input, identify x-ray sub-spike features, and extract sub-spike relative delays with sub-30 attosecond temporal resolution. Deployed on low-latency hardware, it achieves over 10 kHz throughput with 168.3 μs inference latency, indicating scalability to 14 kHz with field-programmable gate array integration. By transforming regression tasks into classification problems and leveraging optimized error encoding, we achieve high precision with low-latency performance that is critical for real-time streaming event selection and experimental control feedback signals. This represents a key development in real-time control pipelines for next-generation autonomous science, generally, and high repetition-rate x-ray experiments in particular.

Accelerator Physics (physics.acc-ph)↗
DOE OSTI2021

CAN-D: A Modular Four-Step Pipeline for Comprehensively Decoding Controller Area Network Data

Controller area networks (CANs) are a broadcast protocol for real-time communication of critical vehicle subsystems. Original equipment manufacturers of passenger vehicles hold secret their mappings of CAN data to vehicle signals, and these definitions vary according to make, model, and year. Without these mappings, the wealth of real-time vehicle information hidden in the CAN packets is uninterpretable, severely impeding vehicle-related research, including CAN cybersecurity and privacy studies, aftermarket tuning, efficiency and performance monitoring, and fault diagnosis to name a few. Guided by the four-part CAN signal definition, we present CAN-D (CAN-Decoder), a modular, four-step pipeline for identifying each signal's boundaries (start bit and length), endianness (byte ordering), signedness (bit-to-integer encoding), and by leveraging diagnostic standards, augmenting a subset of the extracted signals with meaningful, physical interpretation. En route to CAN-D, we provide a comprehensive review of the CAN signal reverse engineering research. All previous methods ignore endianness and signedness, rendering them incapable of decoding many standard CAN signal definitions. Incorporating endianness grows the search space from 128 to 4.72E21 signal tokenizations and introduces a web of changing dependencies. In response, we formulate, formally analyze, and provide an efficient solution to an optimization problem, allowing identification of the optimal set of signal boundaries and byte orderings. In addition, we provide two novel, state-of-the-art signal boundary classifiers—both of which are superior to previous approaches in precision and recall in three different test scenarios—and the first signedness classification algorithm, which exhibits a $>$ 97% F-score. Altogether, CAN-D is the only solution with the potential to extract any CAN signal that is also the state of the art. In evaluation on 10 vehicles of different makes, CAN-D's average $\ell ^1$ error is five times better (81% less) than all previous methods and exhibits lower average error, even when considering only signals that meet prior methods’ assumptions. Finally, CAN-D is implemented in lightweight hardware, allowing for an on-board diagnostic (OBD-II) plugin for real-time in-vehicle CAN decoding.

42 ENGINEERING↗