Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “network regularization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Quantification of neural networks uncertainties with applications to SAFARI-1 axial neutron flux profiles

Deep Neural Networks (DNNs) have been widely used as a data-driven modelling tool in nuclear engineering. However, as a Machine Learning model, Artificial Neural Network (ANN) predictions are subjected to uncertainties originating from the noise in training data, incomplete coverage of the domain, and imperfect neural network architectures. In this work, we target at quantifying the prediction/approximation uncertainties of ANNs using Monte Carlo Dropout (MCD), as well as Bayesian Neural Networks (BNNs) which are solved by variational inference. With a demonstration problem in which neural networks are used to predict the assembly axial neutron flux profiles, the results have shown that the three different neural network models (regular DNNs, DNNs solved with MCD and BNNs) can produce results that agree very well among each other and with the measurement data, on cycles that are not used in the training process. Besides the excellent generalization capability, the uncertainty bands produced by MCD and BNN agree very well, and in general, they can fully envelop the noisy measurement data points. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Cluster characterization in atom probe tomography: Machine learning using multiple summary functions

In this work, we develop a machine learning-based method to characterize intracluster concentration (ρ c ), background concentration (ρ b ), clustering radius (r̄), and radius dispersity (δ r ) in simulated atom probe tomography data using multiple spatial statistics summary functions to train a Bayesian regularized neural network. Here, we build upon previous work that utilized Ripley’s K-function by incorporating additional features from nearest-neighbor spatial statistics summary functions to better characterize concentration-based metrics. The addition of nearest-neighbor based features allows for highly accurate estimates of ρ c and ρ b , both with 90% of the predictions within 4.0% of the real value; the root-mean-square errors are reduced by 81.5% and 92.8% from predictions using only K-function based features, respectively. Additionally, including these nearest-neighbor based features improves the ability to differentiate between r̄ and δ r .

36 MATERIALS SCIENCE↗

Next-Level Energy Management in Manufacturing: Facility-Level Energy Digital Twin Framework Based on Machine Learning and Automated Data Collection

This research introduces an energy prediction framework at the facility level supported by automated data collection and machine learning models. It investigates whether reducing the prediction time scale allows for applying more complex machine learning techniques and if those techniques improve the prediction accuracy. The primary advantages of this framework lie in its automation of the energy prediction process and its provision of real-time energy data suitable for use in energy dashboards or digital twins. A sitewide dataset was created by combining 15 min energy and daily production data of five shops—assembly, battery, body (electric), body (gas), and paint—from a globally recognized electric vehicle manufacturer. Various machine learning models were evaluated on daily, weekly, and monthly datasets, including, in increasingly complex order: naïve, simple linear regression, net regularized generalized linear regression, principal component regression, k-nearest neighbor, random forest, and Bayesian regularized neural network. Compared to the current state-of-the-art energy consumption prediction for the industrial facility level, this research investigates more complex models and smaller time intervals for higher accuracy. The findings revealed that the more complex monthly models require a minimum of a year and a half of data to operate, while weekly models demand a year of data to achieve improved accuracy. Daily models can operate with only six months of data but exhibit poor performance due to reduced prediction accuracy of production. Key challenges identified include access to reliable, high-quality energy and production data and the initial demand for human labor.

digital twin↗

Antarctic ice topography and 150 mb circulation from the Nimbus-6 TWERL Experiment

Many TWERLE balloons floating at an altitude of about 12.5 km, traversed the Antarctic during its 1975/6 summer. The balloons' positions were accurately determined by the Nimbus-6 satellite, which also received their sensors' data: radio altimeter, pressure and temperature. Subtracting the altimeter readings from the calculated balloon altitude above sea-level (a.s.l.) provides the Antarctic surface elevation at the balloon subtrack. The balloon altitude a.s.l. is calculated from daily synoptic maps covering latitudes south of 45 deg S. These maps are prepared using the regular radiosonde network supplemented by four times as many TWERLE balloons.

Levanon, N.↗

Satellite-tracking and Earth dynamics research programs

The major focus for operations during this period was the preliminary MERIT Campaign and its intensive tracking of LAGEOS for polar motion and Earth rotation studies. The data acquired from LAGEOS were used for other geophysical investigations, including studies of crustal dynamics, and Earth and ocean tides, and for the general development of precision orbit determination. The network performed regular tracking of several other retroreflector satellites including GEOS-1, GEOS-3, BE-C, and Starlette for refined determinations of station coordinates and Earth's gravity field and for studies of solid Earth dynamics.

Source record↗

Satellite-tracking and Earth dynamics research programs

The activities carried out by the Smithsonian Astrophysical Observatory (SAO) are described. The SAO network continued to track LAGEOS at highest priority for polar motion and Earth rotation studies, and for other geophysical investigations, including crustal dynamics, Earth and ocean tides, and the general development of precision orbit determination. The network performed regular tracking of several other retroreflector satellites including GEOS-1, GEOS-3, BE-C, and Starlette for refined determinations of station coordinates and the Earth's gravity field and for studies of solid Earth dynamics. A major program in laser upgrading continued to improve ranging accuracy and data yield. This program includes an increase in pulse repetition rate from 8 ppm to 30 ppm, a reduction in laser pulse width from 6 nsec to 2 to 3 nsec, improvements in the photoreceiver and the electronics to improve daylight ranging, and an analog pulse detection system to improve range noise and accuracy. Data processing hardware and software are discussed.

Source record↗

Satellite tracking and earth dynamics research programs

The SAO laser site in Arequipa continued routine operations throughout the reporting period except for the months of March and April when upgrading was underway. The laser in Orroral Valley was operational through March. Together with the cooperating stations in Wettzell, Grasse, Kootwikj, San Fernando, Helwan, and Metsahove the laser stations obtained a total of 37,099 quick-look observations on 978 passes of BE-C, Starlette, and LAGEOS. The Network continued to track LAGEOS at highest priority for polar motion and Earth rotation studies, and for other geophysical investigations, including crustal dynamics, Earth and ocean tides, and the general development of precision orbit determination. The Network performed regular tracking of BE-C and Starlette for refined determinations of station coordinate and the Earth's gravity field and for studies of solid earth dynamics. Monthly statistics of the passes and points are given by station and by satellite.

Source record↗

Predictive Modeling for Differential Diagnosis and Mortality Risk Assessment

The prevalence of electronic health record (EHR) systems has brought prodigious biomedical informatics opportunity. Automated machine learning methods can effectively utilize such data and have become common tools for healthcare predictive modeling. Researches in medical informatics have explored the potential of deep learning and classical models in emergent care scenarios. In particular, predicting differential diagnoses for admissions have proven useful in decreasing unnecessary lab tests and improving inpatient triage decision-making. Moreover, identification of high-risk patients for in-hospital mortality is vitally important to maximize allocation of medical resources.The Medical Information Mart for Intensive Care (MIMIC-III) database, containing de-identified critical care inpatient was used in our study. This data set captures hospital patient laboratory measurements, pharmacologic prescriptions, diagnostic data and procedure event recordings. When considering adult patients and discounting admissions with ICU length of stay less than 24 hours, there were 37,787 unique admissions and 30,414 total patients. We examined the top 25 most prevalent ICD-9 group-level disease specificities in MIMIC-III using a multi-label classification model. In-hospital mortality was modeled as binary classification with 4,155 (13%) adult patients that expired, of which 3,138 (75.5%) were in the ICU setting. The metrics AUC, F1 score, sensitivity and specificity values calculated for each disease label measured prediction performance.The usage of ICD-9 group codes reduced feature dimension from 14,567 to 942 and greatly improved distribution of patient diagnostic categories. Disease temporal patterns were captured by considering the most frequently sampled 6 vital signs and 13 laboratory values. Missing data were imputed at each time-stamp. Time-series raw hourly average values were converted into 5 summary features (mean, standard deviation, number of observations, min & max values). Patient demographic variables such as age, gender, marital status and ethnicity were also factored into the modeling. Choi et al showed that contextual embedding of medical data, diagnostic and procedural codes alone can predict future diagnoses with sensitivity as high as 0.79. We utilized an embedding technique called word2vec which allowed sparse representations of medical history to be transformed into dense word vectors. The mappings captured contextual information by treating each admission as a sentence and learning the most likely neighboring words in a sliding window fashion. Binary and multi-label classification was achieved via collapse models, which do not consider temporal information, as well as recurrent neural networks with regularization, Softmax output layer activation together with categorical cross-entropy as the loss function.

US Army collaboration↗

Integrating deep neural networks with full-waveform inversion: Reparameterization, regularization, and uncertainty quantification

Full-waveform inversion (FWI) is an accurate imaging approach for modeling the velocity structure by minimizing the misfit between recorded and predicted seismic waveforms. However, the strong nonlinearity of FWI resulting from fitting oscillatory waveforms can trap the optimization in local minima. We have adopted a neural-network-based full-waveform inversion (NNFWI) method that integrates deep neural networks with FWI by representing the velocity model with a generative neural network. Neural networks can naturally introduce spatial correlations as regularization to the generated velocity model, which suppresses noise in the gradients and mitigates local minima. Furthermore, the velocity model generated by neural networks is input to the same partial differential equation (PDE) solvers used in conventional FWI. The gradients of the neural networks and PDEs are calculated using automatic differentiation, which back propagates gradients through the acoustic PDEs and neural network layers to update the weights of the generative neural network. Experiments on 1D velocity models, the Marmousi model, and the 2004 BP model determine that NNFWI can mitigate local minima, especially for imaging high-contrast features such as salt bodies, and it significantly improves the inversion in the presence of noise. Adding dropout layers to the neural network model also allows analyzing the uncertainty of the inversion results through Monte Carlo dropout. NNFWI opens a new pathway to combine deep learning and FWI for exploiting the characteristics of deep neural networks and the high accuracy of PDE solvers. Because NNFWI does not require extra training data and optimization loops, it provides an attractive and straightforward alternative to conventional FWI.

58 GEOSCIENCES↗

Sentinel

Network intrusion detection systems (NIDS) are commonplace in network security but they frequently employ algorithms that are computational demanding requiring hardware and software with significant power requirements. Two examples of such resource-intensive algorithms used for network security are regular expression matching and broader signature pattern matching which are commonly used in deep packet inspection (DPI). Network security algorithms that have large power requirements may be a challenge for low-power internet-of-things (IoT) environments, which generally lack the power resources to implement complex security measures like computationally expensive DPI at the edge. Furthermore, IoT environments incorporating 5G standalone networks have network latency constraints beyond just power that make DPI at the edge even more difficult. Programmable logic is ideally suited for machine learning inference for DPI because of its deep instruction level parallelism and single-cycle memory access. Machine learning approaches for DPI have been explored before using the programmable logic of field programmable gate arrays (FPGA) as a potential solution for NIDS approaches that would be power-suitable for IoT. However, those previous programmable logic NIDS approaches utilize either a supervised or unsupervised learning model. Sentinel utilizes the ensemble of these two machine learning approaches known as a semi-supervised approach which has shown promise in NIDS implementations. Sentinel provides a programmable logic implementation of a semi-supervised approach for DPI which operates at much lower power and latency than a GPU implementation with negligible loss of accuracy due to quantization through a logistic regressor.

Anderson, MatthewW [Idaho National Laboratory (INL↗

HomPINNs: Homotopy physics-informed neural networks for learning multiple solutions of nonlinear elliptic differential equations

Physics-informed neural networks (PINNs) based machine learning is an emerging framework for solving nonlinear differential equations. However, due to the implicit regularity of neural network structure, PINNs can only find the flattest solution in most cases by minimizing the loss functions. In this paper, we combine PINNs with the homotopy continuation method, a classical numerical method to compute isolated roots of polynomial systems, and propose a new deep learning framework, named homotopy physics-informed neural networks (HomPINNs), for solving multiple solutions of nonlinear elliptic differential equations. The implementation of an HomPINN is a homotopy process that is composed of the training of a fully connected neural network, named the starting neural network, and training processes of several PINNs with different tracking parameters. The starting neural network is to approximate a starting function constructed by the trivial solutions, while other PINNs are to minimize the loss functions defined by boundary condition and homotopy functions, varying with different tracking parameters. These training processes are regraded as different steps of a homotopy process, and a PINN is initialized by the well-trained neural network of the previous step, while the first starting neural network is initialized using the default initialization method. Finally, several numerical examples are presented to show the efficiency of our proposed HomPINNs, including reaction-diffusion equations with a heart-shaped domain.

97 MATHEMATICS AND COMPUTING↗

Pore-Scale Simulation of Spectral Induced Polarization

A new model was developed to simulate spectral induced polarization (SIP) in porous media. The model is based on a pore-network approximation of the soil or rock, in which bonds between pores are represented by equivalent circuits. In the circuit model, electrolytic and surface conduction occur in parallel, with the electrolyte behaving as a single resistor, and the mineral surface behaving as a resistor and capacitor in series. Based on the pore geometry, fluid conductivity, and mineral surface conductivity, an impedance network is generated. SIP spectra are calculated by solving the sinusoidal steady-state electrical problem for a range of frequencies. The new model provides a tool to bridge the gap between (1) our fundamental understanding of polarization processes at the scale of a single pore or grains and (2) existing simulation capabilities for macroscale systems that do not account for pore-scale properties. The model is implemented in Matlab and can solve two-dimensional and three-dimensional (2D and 3D) networks. Examples for regular cubic lattices are presented, but the method and code are flexible with respect to pore geometries and allow for future leveraging of existing open-source capabilities for generation of pore networks and extraction of network geometries from micro imaging of core samples.

58 GEOSCIENCES↗

Observing large-scale solar surface flows with GONG: Investigation of a key element in solar activity buildup

The Global Oscillation Network Group (GONG) solar telescope network has begun regular operations, and will provide continuous Doppler images of large-scale nearly-steady motions at the solar surface, primarily those due to supergranulation. Not only the Sun's well-known magnetic network, but also flux diffusion, dispersal, and concentration at the surface appear to be controlled by supergranulation. Through such magnetoconvective interactions, magnetic stresses develop, leading to solar activity. We show a Doppler movie made from a 45.5 hr time series obtained 1995 May 9-10 using data from three of the six GONG sites (Learmonth, Tenerife, Tucson), to demonstrate the capability of this system.

Beck, John G.↗

Regular Topologies for Gigabit Wide-Area Networks: Congestion Avoidance Testbed Experiments

This document is Volume 3 of the final technical report on the work performed by SRI International (SRI) on SRI Project 8600. The document includes source listings for all software developed by SRI under this effort. Since some of our work involved the use of ST-II and the Sun Microsystems, Inc. (Sun) High-Speed Serial Interface (HSI/S) driver, we have included some of the source developed by LBL and BBN as well. In most cases, our decision to include source developed by other contractors depended on whether it was necessary to modify the original code. If we have modified the software in any way, it is included in this document. In the case of the Traffic Generator (TG), however, we have included all the ST-II software, even though BBN performed the integration, because the ST-II software is part of the standard TG release. It is important to note that all the code developed by other contractors is in the public domain, so that all software developed under this effort can be re-created from the source included here.

Denny, Barbara A.↗

Regular Topologies for Gigabit Wide-Area Networks

In general terms, this project aimed at the analysis and design of techniques for very high-speed networking. The formal objectives of the project were to: (1) Identify switch and network technologies for wide-area networks that interconnect a large number of users and can provide individual data paths at gigabit/s rates; (2) Quantitatively evaluate and compare existing and proposed architectures and protocols, identify their strength and growth potentials, and ascertain the compatibility of competing technologies; and (3) Propose new approaches to existing architectures and protocols, and identify opportunities for research to overcome deficiencies and enhance performance. The project was organized into two parts: 1. The design, analysis, and specification of techniques and protocols for very-high-speed network environments. In this part, SRI has focused on several key high-speed networking areas, including Forward Error Control (FEC) for high-speed networks in which data distortion is the result of packet loss, and the distribution of broadband, real-time traffic in multiple user sessions. 2. Congestion Avoidance Testbed Experiment (CATE). This part of the project was done within the framework of the DARTnet experimental T1 national network. The aim of the work was to advance the state of the art in benchmarking DARTnet's performance and traffic control by developing support tools for network experimentation, by designing benchmarks that allow various algorithms to be meaningfully compared, and by investigating new queueing techniques that better satisfy the needs of best-effort and reserved-resource traffic. This document is the final technical report describing the results obtained by SRI under this project. The report consists of three volumes: Volume 1 contains a technical description of the network techniques developed by SRI in the areas of FEC and multicast of real-time traffic. Volume 2 describes the work performed under CATE. Volume 3 contains the source code of all software developed under CATE.

Shacham, Nachum↗

The role of stiffness in training and generalization of ResNets

Neural ordinary differential equations (NODEs) have recently regained popularity as large-depth limits of a large class of neural networks. In particular, residual neural networks (ResNets) are equivalent to an explicit Euler discretization of an underlying NODE, where the transition from one layer to the next is one time step of the discretization. The relationship between continuous and discrete neural networks has been of particular interest. Notably, analysis from the ordinary differential equation viewpoint can potentially lead to new insights for understanding the behavior of neural networks in general. In this work, we take inspiration from differential equations to define the concept of stiffness for a ResNet via the interpretation of a ResNet as the discretization of a NODE. Here, we then examine the effects of stiffness on the ability of a ResNet to generalize, via computational studies on example problems coming from climate and chemistry models. We find that penalizing stiffness does have a unique regularizing effect, but we see no benefit to penalizing stiffness over L 2 regularization (penalization of network parameter norms) in terms of predictive performance.

97 MATHEMATICS AND COMPUTING↗

NNFDivergence

The code implements f divergence regularization for neural networks in the Python-based Pytorch framework. The methods are the main focus but the repository will also contain examples that operate on purely synthetic "toy" data or on openly available, public data from NASA.

Klein, Natalie [@lanl]↗