Engineering Papers⌕ Search

Engineering topics

Patton, Robert M.

Publications and source records attributed to Patton, Robert M..

Optimizing neural networks

A system and method design and optimize neural networks. The system and method include a data store that stores a plurality of gene vectors that represent diverse and distinct neural networks and an evaluation queue stored with the plurality of gene vectors. Secondary nodes construct, train, and evaluate the neural network and automatically render a plurality of fitness values asynchronously. A primary node executes a gene amplification on a select plurality of gene vectors, a crossing-over of the amplified gene vectors, and a mutation of the crossing-over gene vectors automatically and asynchronously, which are then transmitted to the evaluation queue. The process continuously repeats itself by processing the gene vectors inserted into the evaluation queue until a fitness level is reached, a network's accuracy level plateaus, a processing time period expires, or when some stopping condition or performance metric is met or exceeded.

97 MATHEMATICS AND COMPUTING↗

System and method of content based recommendation using hypernym expansion

A system and method for document recommendation combines linguistic and term frequency approaches in order to improve overall performance and recommendation. Hypernyms are added to the source document set so that related documents can be recommended from the source document set. There are a variety of different parameters to configure that change operation of the document recommendation system.

97 MATHEMATICS AND COMPUTING↗

A Classical-Quantum Hybrid Approach for Unsupervised Probabilistic Machine Learning

For training unsupervised probabilistic machine learning models, matrix computation and sample generation are the two key steps. While GPUs excel at matrix computation, they use pseudo-random numbers to generate samples. Contrarily, Adiabatic Quantum Processors (AQP) use quantum mechanical systems to generate samples accurately and quickly, but are not suited for matrix computation. We present a Classical-Quantum Hybrid Approach for training unsupervised probabilistic machine learning models, leveraging GPUs for matrix computations and the D-Wave quantum sampling library for sample generation. We compare this approach to classical and quantum approaches across four performance metrics. Our results indicate that while the hybrid approach–which uses one AQP and one GPU–outperforms quantum and one of the classical approaches, it performs comparably to the GPU approach, and is outperformed by the CPU approach, which uses 56 high-end CPUs. Lastly, we compare sampling on AQP versus sampling library and show that AQP performs better.

Date, Prasanna↗

167-PFlops Deep Learning for Electron Microscopy: From Learning Physics to Atomic Manipulation

An artificial intelligence system called MENNDL, which used 25,200 Nvidia Volta GPUs on Oak Ridge National Laboratory’s Summit machine, automatically designed an optimal deep learning network in order to extract structural information from raw atomic-resolution microscopy data. In a few hours, MENNDL creates and evaluates millions of networks using a scalable, parallel, asynchronous genetic algorithm augmented with a support vector machine to automatically find a superior deep learning network topology and hyper-parameter set than a human expert can find in months. For the application of electron microscopy, the system furthers the goal of improving our understanding of the electron-beam-matter interactions and real-time image-based feedback, which enables a huge step beyond human capacity toward nanofabricating materials automatically. Finally, MENNDL has been scaled to the 4,200 available nodes of Summit achieving a measured 152.5 PFlops, with an estimated sustained performance of 167 PFlops when the entire machine is available.

97 MATHEMATICS AND COMPUTING↗

Exploring Flexible Communications for Streamlining DNN Ensemble Training Pipelines

Parallel training of a Deep Neural Network (DNN) ensemble on a cluster of nodes is a common practice to train multiple models in order to construct a model with a higher prediction accuracy. Existing ensemble training pipelines can perform a great deal of redundant operations, resulting in unnecessary CPU usage, or even poor pipeline performance. In order to remove these redundancies, we need pipelines with more communication flexibility than existing DNN frameworks can provide. This project investigates a series of designs to improve pipeline flexibility and adaptivity, while also increasing performance. We implement our designs using Tensorflow with Horovod, and test it using several large DNNs in a large scale GPU cluster, Titan supercomputer at Oak Ridge National Lab. Our results show that the CPU time spent during training is reduced by 2-11X. Furthermore, our implementation can achieve up to 10X speedups when CPU core limits are imposed. Our best pipeline also reduces the average power draw of the ensemble training process by 5-16% when compared to the baseline.

97 MATHEMATICS AND COMPUTING↗

Data mining for better material synthesis: The case of pulsed laser deposition of complex oxides

The pursuit of more advanced electronics, and finding solutions to energy needs often hinges upon the discovery and optimization of new functional materials. However, the discovery rate of these materials is alarmingly low. Much of the information that could drive this rate higher is scattered across tens of thousands of papers in the extant literature published over several decades but is not in an indexed form, and cannot be used in entirety without substantial effort. Many of these limitations can be circumvented if the experimentalist has access to systematized collections of prior experimental procedures and results. Here, we investigate the property-processing relationship during growth of oxide films by pulsed laser deposition. To do so, we develop an enabling software tool to (1) mine the literature of relevant papers for synthesis parameters and functional properties of previously studied materials, (2) enhance the accuracy of this mining through crowd sourcing approaches, (3) create a searchable repository that will be a community-wide resource enabling material scientists to leverage this information, and (4) provide through the Jupyter notebook platform, simple machine-learning-based analysis to learn the complex interactions between growth parameters and functional properties. Here, the results allow visualization of growth windows, trends and outliers, which can serve as a template for analyzing the distribution of growth conditions, provide starting points for related compounds and act as a feedback for first-principles calculations. Such tools will comprise an integral part of the materials design schema in the coming decade.

36 MATERIALS SCIENCE↗

Composability-Centered Convolutional Neural Network Pruning

This work studies the composability of the building blocks of structural CNN models (e.g., GoogleLeNet and Residual Networks) in the context of network pruning. We empirically validate that a network composed of pre-trained building blocks (e.g. residual blocks and Inception modules) not only gives a better initial setting for training, but also allows the training process to converge at a significantly higher accuracy in much less time. Based on that insight, we propose a composability-centered design for CNN network pruning. Experiments show that this new scheme shortens the configuration process in CNN network pruning by up to 186.8X for ResNet-50 and up to 30.2X for Inception-V3, and meanwhile, the models it finds that meet the accuracy requirement are significantly more compact than those found by default schemes.

97 MATHEMATICS AND COMPUTING↗

Do citations and readership identify seminal publications?

Here, this work presents a new approach for analysing the ability of existing research metrics to identify research which has strongly influenced future developments. More specifically, we focus on the ability of citation counts and Mendeley reader counts to distinguish between publications regarded as seminal and publications regarded as literature reviews by field experts. The main motivation behind our research is to gain a better understanding of whether and how well the existing research metrics relate to research quality. For this experiment we have created a new dataset which we call TrueImpactDataset and which contains two types of publications, seminal papers and literature reviews. Using the dataset, we conduct a set of experiments to study how citation and reader counts perform in distinguishing these publication types, following the intuition that causing a change in a field signifies research quality. Finally, our research shows that citation counts work better than a random baseline (by a margin of 10%) in distinguishing important seminal research papers from literature reviews while Mendeley reader counts do not work better than the baseline.

99 GENERAL AND MISCELLANEOUS↗

Evolving Deep Networks Using HPC

While a large number of deep learning networks have been studiedand published that produce outstanding results on natural imagedatasets, these datasets only make up a fraction of those to whichdeep learning can be applied. These datasets include text data, audiodata, and arrays of sensors that have very different characteristicsthan natural images. As these “best” networks for natural imageshave been largely discovered through experimentation and cannotbe proven optimal on some theoretical basis, there is no reasonto believe that they are the optimal network for these drasticallydifferent datasets. Hyperparameter search is thus often a very im-portant process when applying deep learning to a new problem. Inthis work we present an evolutionary approach to searching thepossible space of network hyperparameters and construction thatcan scale to 18, 000 nodes. This approach is applied to datasets ofvarying types and characteristics where we demonstrate the abilityto rapidly find best hyperparameters in order to enable practitionersto quickly iterate between idea and result.

Young, Steven↗

Audience Based View of Publication Impact

With increasing pressure to demonstrate a return on investment, scienti c user facilities and government funded research projects must strive to demonstrate positive impact of their research on oth- ers. Historically, citation counts and journal impact factor have been widely used as a single quantitative measure of impact. However, with the expansion of various modes of communication, citation counts no longer hold as much merit as a single measure of impact. This work seeks to expand on citation counts by considering the audience to which a publication draws attention. Based on commu- nication mode, there are various audiences that may be "impacted" but those audiences are not fully captured by citation count. Conse- quently, we propose that impact measures for publications should be oriented around the audience with quanti able measures based on the various communication modes.

Patton, Robert↗

A Study of Complex Deep Learning Networks on High Performance, Neuromorphic, and Quantum Computers

Current Deep Learning models use highly optimized convolutional neural networks (CNN) trained on large graphical processing units (GPU)-based computers with a fairly simple layered network topology, i.e., highly connected layers, without intra-layer connections. Complex topologies have been proposed, but are intractable to train on current systems. Building the topologies of the deep learning network requires hand tuning, and implementing the network in hardware is expensive in both cost and power.In this paper, we evaluate deep learning models using three different computing architectures to address these problems: quantum computing to train complex topologies, high performance computing (HPC) to automatically determine network topology, and neuromorphic computing for a low-power hardware implementation. Due to input size limitations of current quantum computers we use the MNIST dataset for our evaluation.The results show the possibility of using the three architectures in tandem to explore complex deep learning networks that are untrainable using a von Neumann architecture. We show that a quantum computer can find high quality values of intra-layer connections and weights, while yielding a tractable time result as the complexity of the network increases; a high performance computer can find optimal layer-based topologies; and a neuromorphic computer can represent the complex topology and weights derived from the other architectures in low power memristive hardware.This represents a new capability that is not feasible with current von Neumann architecture. It potentially enables the ability to solve very complicated problems unsolvable with current computing technologies.

97 MATHEMATICS AND COMPUTING↗

Method and system of filtering and recommending documents

Disclosed is a method and system for discovering documents using a computer and providing a small set of the most relevant documents to the attention of a human observer. Using the method, the computer obtains a seed document from the user and generates a seed document vector using term frequency-inverse corpus frequency weighting. A keyword index for a plurality of source documents can be compared with the weighted terms of the seed document vector. The comparison is then filtered to reduce the number of documents, which define an initial subset of the source documents. Initial subset vectors are generated and compared to the seed document vector to obtain a similarity value for each comparison. Based on the similarity value, the method then recommends one or more of the source documents.

97 MATHEMATICS AND COMPUTING↗

Optimizing deep learning hyper-parameters through an evolutionary algorithm. In: MLHPC '15 Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, Article No. 4

There has been a recent surge of success in utilizing Deep Learning (DL) in imaging and speech applications for its relatively automatic feature generation and, in particular for convolutional neural networks (CNNs), high accuracy classification abilities. While these models learn their parameters through data-driven methods, model selection (as architecture construction) through hyper-parameter choices remains a tedious and highly intuition driven task. To address this, Multi-node Evolutionary Neural Networks for Deep Learning (MENNDL) is proposed as a method for automating network selection on computational clusters through hyper-parameter optimization performed via genetic algorithms.

Young, Steven R.↗

Cloud computing method for dynamically scaling a process across physical machine boundaries

A cloud computing platform includes first device having a graph or tree structure with a node which receives data. The data is processed by the node or communicated to a child node for processing. A first node in the graph or tree structure determines the reconfiguration of a portion of the graph or tree structure on a second device. The reconfiguration may include moving a second node and some or all of its descendant nodes. The second and descendant nodes may be copied to the second device.

Gillen, Robert E.↗