Smith, Jared M. | Engineering Papers

DOE OSTI2023

Virtual supervisory control and data acquisition (SCADA) automation controller

One or more virtual supervisory control and data acquisition (SCADA) controllers are provided. The virtual SCADA controller(s) may be deployed upon a detection of a failure of a physical SCADA controller. An on-site module may detect failure including a security breach which causes the physical SCADA controller to be disconnected from a network and the virtual SCADA controller(s) to take over control. The virtual SCADA controller(s) may be on-site or off-site.

Smith, Jared M.↗

DOE OSTI2023

Independent malware detection architecture

A system and method (referred to as the system) detect malware by training a rule-based model, a functional based model, and a deep learning-based model from a memory snapshot of a malware free operating state of a monitored device. The system extracts a feature set from a second memory snapshot captured from an operating state of the monitored device and processes the feature set by the rule-based model, the functional-based model, and the deep learning-based model. The system identifies identifying instances of malware on the monitored device without processing data identifying an operating system of the monitored device, data associated with a prior identification of the malware, data identifying a source of the malware, data identifying a location of the malware on the monitored device, or any operating system specific data contained within the monitored device.

Smith, Jared M.↗

DOE OSTI2023

Beyond the Hype: An Evaluation of Commercially Available Machine-Learning-Based Malware Detectors

There is a lack of scientific testing of commercially available malware detectors, especially those that boast accurate classification of never-before-seen (i.e., zero-day) files using machine learning (ML). Consequently, efficacy of malware detectors is opaque, inhibiting end users from making informed decisions and researchers from targeting gaps in current detectors. In this paper, we present a scientific evaluation of four prominent commercial malware detection tools to assist an organization with two primary questions: To what extent do ML-based tools accurately classify previously and never-before-seen files? Is purchasing a network-level malware detector worth the cost? To investigate, we tested each tool against 3,536 total files (2,554 or 72% malicious, 982 or 28% benign) of a variety of file types, including hundreds of malicious zero-days, polyglots, and APT-style files, delivered on multiple protocols. We present statistical results on detection time and accuracy, consider complementary analysis (using multiple tools together), and provide two novel applications of the recent cost-benefit evaluation procedure of Iannacone & Bridges. Although the ML-based tools are more effective at detecting zero-day files and executables, the signature-based tool might still be an overall better option. Both network-based tools provide substantial (simulated) savings when paired with either host tool, yet both show poor detection rates on protocols other than HTTP or SMTP. Our results show that all four tools have near-perfect precision but alarmingly low recall, especially on file types other than executables and office files—37% of malware, including all polyglot files, were undetected. Priorities for researchers and takeaways for end users are given. Code for future use of the cost model is provided.

97 MATHEMATICS AND COMPUTING↗