SEARCH · Engineering Papers

Results for “Reinforcement learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 991 records · Page 55

All sources NASA NTRS DOE OSTI

DOE OSTI2022

Cyber Resilient Design and Controls for Energy Systems

As the grid of today is evolving into a highly distributed and autonomous cyber-physical system with higher penetration of Distributed Energy Resources (DERs) and autonomous controls that are dependent on the cyber infrastructure, there are novel and increasing challenges to cyber-resilient operation of the grid. While the traditional mechanisms take a "bolt on" approach to cybersecurity and resiliency, this talk presents two novel technologies developed at NREL that "bake in" security and resilience into the design and control of the grid. An Adaptive Resilience Metrics framework is a novel mechanism for system owners and operators to understand the impact of cyber and physical system design and topology on the resilient operation, while also helping determine optimal policies in response to adverse cyber events. Along with that a zero-knowledge proof-based technique helps incorporate security into the controls layer by focusing on computational integrity rather than data integrity. By leveraging design and control properties unique to Operational Technology systems these technologies will not only help enhance cyber-resilience for the grid of today, but a highly distributed and autonomous grid of tomorrow.

cyber resilience↗

DOE OSTI2024

Reinforcement expectation in the honeybee ( Apis mellifera ): Can downshifts in reinforcement show conditioned inhibition?

When animals learn the association of a conditioned stimulus (CS) with an unconditioned stimulus (US), later presentation of the CS invokes a representation of the US. When the expected US fails to occur, theoretical accounts predict that conditioned inhibition can accrue to any other stimuli that are associated with this change in the US. Empirical work with mammals has confirmed the existence of conditioned inhibition. But the way it is manifested, the conditions that produce it, and determining whether it is the opposite of excitatory conditioning are important considerations. Invertebrates can make valuable contributions to this literature because of the well-established conditioning protocols and access to the central nervous system (CNS) for studying neural underpinnings of behavior. Nevertheless, although conditioned inhibition has been reported, it has yet to be thoroughly investigated in invertebrates. Here, we evaluate the role of the US in producing conditioned inhibition by using proboscis extension response conditioning of the honeybee (Apis mellifera). Specifically, using variations of a “feature-negative” experimental design, we use downshifts in US intensity relative to US intensity used during initial excitatory conditioning to show that an odorant in an odor–odor mixture can become a conditioned inhibitor. We argue that some alternative interpretations to conditioned inhibition are unlikely. However, we show variation across individuals in how strongly they show conditioned inhibition, with some individuals possibly revealing a different means of learning about changes in reinforcement. We discuss how the resolution of these differences is needed to fully understand whether and how conditioned inhibition is manifested in the honeybee, and whether it can be extended to investigate how it is encoded in the CNS. It is also important for extension to other insect models. In particular, work like this will be important as more is revealed of the complexity of the insect brain from connectome projects.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2023

Autonomous Cyber Defense Against Dynamic Multi-strategy Infrastructural DDoS Attacks

Dynamic Infrastructural Distributed Denial of Service (I-DDoS) attacks constantly change attack vectors to congest core backhaul links and disrupt critical network availability while evading end-system defenses. To effectively counter these highly dynamic attacks, defense mechanisms need to exhibit adaptive decision strategies for real-time mitigation. This paper presents a novel Autonomous DDoS Defense framework that employs model-based reinforcement agents. The framework continuously learns attack strategies, predicts attack actions, and dynamically determines the optimal composition of defense tactics such as filtering, limiting, and rerouting for flow diversion. Our contributions include extending the underlying formulation of the Markov Decision Process (MDP) to address simultaneous DDoS attack and defense behavior, and accounting for environmental uncertainties. We also propose a fine-grained action mitigation approach robust to classification inaccuracies in Intrusion Detection Systems (IDS). Additionally, our reinforcement learning model demonstrates resilience against evasion and deceptive attacks. Evaluation experiments using real-world and simulated DDoS traces demonstrate that our autonomous defense framework ensures the delivery of approximately 96 - 98% of benign traffic despite the diverse range of attack strategies.

Dutta, Ashutosh↗

DOE OSTI2021

Automatic identification and quantification of dense microcracks in high-performance fiber-reinforced cementitious composites through deep learning-based computer vision

Highlights: • A method is presented to detect, locate, quantify, and visualize dense microcracks in HPFRCC. • Quantification of dense microcracks is realized using deep learning method for the first time. • The presented method uses deep learning models that are trained using a realistic dataset size. • The presented method has a high computation efficiency for identifying and quantifying cracks. • The presented method provides crack width with errors up to 50 μm and a R{sup 2} value of 0.984. High-performance fiber-reinforced cementitious composites (HPFRCCs) feature high mechanical strengths, crack resistance, and durability. Under excessive loading, HPFRCCs demonstrate dense microcracks that are difficult to identify using existing methods. This study presents a computer vision method for identification, quantification, and visualization of microcracks in HPFRCCs based on deep learning. The presented method integrates multiple deep learning models and computer vision techniques in a hierarchical architecture. The crack pattern (e.g., number, width, and spacing of cracks) are automatically determined from pictures without human intervention. This study shows that the presented method achieves an accuracy of 0.992 for crack detection and an accuracy finer than 50 μm (R{sup 2} > 0.984) for quantification of crack width when deep learning models are trained using only 200 pictures of HPFRCCs and 200 pictures of conventional concrete with incorporation of data augmentation. The presented method is expected to be also applicable to other materials featuring complex cracks.

36 MATERIALS SCIENCE↗

NASA NTRS1994

Fuzzy self-learning control for magnetic servo system

It is known that an effective control system is the key condition for successful implementation of high-performance magnetic servo systems. Major issues to design such control systems are nonlinearity; unmodeled dynamics, such as secondary effects for copper resistance, stray fields, and saturation; and that disturbance rejection for the load effect reacts directly on the servo system without transmission elements. One typical approach to design control systems under these conditions is a special type of nonlinear feedback called gain scheduling. It accommodates linear regulators whose parameters are changed as a function of operating conditions in a preprogrammed way. In this paper, an on-line learning fuzzy control strategy is proposed. To inherit the wealth of linear control design, the relations between linear feedback and fuzzy logic controllers have been established. The exercise of engineering axioms of linear control design is thus transformed into tuning of appropriate fuzzy parameters. Furthermore, fuzzy logic control brings the domain of candidate control laws from linear into nonlinear, and brings new prospects into design of the local controllers. On the other hand, a self-learning scheme is utilized to automatically tune the fuzzy rule base. It is based on network learning infrastructure; statistical approximation to assign credit; animal learning method to update the reinforcement map with a fast learning rate; and temporal difference predictive scheme to optimize the control laws. Different from supervised and statistical unsupervised learning schemes, the proposed method learns on-line from past experience and information from the process and forms a rule base of an FLC system from randomly assigned initial control rules.

Tarn, J. H.↗

DOE OSTI2016

Ensuring Workforce Readiness for the Energy Industry Through 3D Simulations

In February 2009, during his joint address to Congress, President Obama challenged every American to commit to at least one year of college education and set a goal that the United States should once again have the highest proportion of college graduates in the world by the year 2020. This push by the Obama administration reflected the understanding that while a college degree was becoming a prerequisite to obtain many of the jobs in the 21. century economy; the United States still lagged behind other countries with respect to higher education attainment. Because of this, the U.S. is facing a skills gap between the credentials required for the jobs of today and tomorrow, and the number of skilled workers eligible to fill those jobs. Within the energy industry, the skills gap is compounded by two factors: an expanding domestic industry in energy generation and transmission, and an aging workforce set to retire in large numbers. In fact, a 2013 survey administered by Center for Energy Workforce Development (CEWD) estimates that 36% of the workforce may need to be replaced in the next five years, and nearly 50% will need to be replaced by the year 2023. As a 2014 report by CEWD notes, 'The message for workforce development is clear: qualified and skilled utility technicians and engineers will continue to be in high demand in all types of energy generation, transmission, and distribution jobs'. Simulations provide students the ability to practice proper procedure in a safe environment, test out multiple scenarios to determine an optimal outcome, and repeat practice to solidify learning. Industry currently applies simulation (i.e., 3D Electrical Substation Simulation) as it affords an opportunity for technicians to train in a safe, virtual environment, thus eliminating risks associated with arc flash hazards. The intent of 3D simulation is to enhance the students' understanding of the concepts learned in the course and better prepare students for field work in a related technical discipline. The 3D simulation will reinforce the concepts learned during a specific course and are intended to add real world context to the curriculum. The 3D simulation will be designed such that a student will perform specific job activities incorporating concepts learned during the course. This method of instruction ensures students have an understanding of the concepts and are able to apply the concepts in a real world context. Different scenarios will be developed to ensure a thorough understanding of the concepts with real time feedback when there is a misunderstanding of the concepts. (authors)

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1995

Emergence of relations and the essence of learning: a review of Sidman's Equivalence relations and behavior: a research story. Book review

The author reviews and comments on the book Equivalence relations and behavior: a research story by Murray Sidman. Sidman's book reports his research about equivalence relations and competencies in children with mental retardation and how it relates to behavior. Sidman used the idea of stimulus-stimulus relations among features of the environment to develop his theories about equivalence relations. Experimental work with children and animals demonstrated their ability to use equivalence relations to learn new tasks. The subject received feedback and reinforcement for specific choices made during training, then was presented with new choices during testing. Results of the tests indicate that subjects were able to establish relations and retrieve them in different situations.

NASA Discipline Space Human Factors↗

DOE OSTI2016

SU-D-BRB-05: Quantum Learning for Knowledge-Based Response-Adaptive Radiotherapy

Purpose: There is tremendous excitement in radiotherapy about applying data-driven methods to develop personalized clinical decisions for real-time response-based adaptation. However, classical statistical learning methods lack in terms of efficiency and ability to predict outcomes under conditions of uncertainty and incomplete information. Therefore, we are investigating physics-inspired machine learning approaches by utilizing quantum principles for developing a robust framework to dynamically adapt treatments to individual patient’s characteristics and optimize outcomes. Methods: We studied 88 liver SBRT patients with 35 on non-adaptive and 53 on adaptive protocols. Adaptation was based on liver function using a split-course of 3+2 fractions with a month break. The radiotherapy environment was modeled as a Markov decision process (MDP) of baseline and one month into treatment states. The patient environment was modeled by a 5-variable state represented by patient’s clinical and dosimetric covariates. For comparison of classical and quantum learning methods, decision-making to adapt at one month was considered. The MDP objective was defined by the complication-free tumor control (P{sup +}=TCPx(1-NTCP)). A simple regression model represented state-action mapping. Single bit in classical MDP and a qubit of 2-superimposed states in quantum MDP represented the decision actions. Classical decision selection was done using reinforcement Q-learning and quantum searching was performed using Grover’s algorithm, which applies uniform superposition over possible states and yields quadratic speed-up. Results: Classical/quantum MDPs suggested adaptation (probability amplitude ≥0.5) 79% of the time for splitcourses and 100% for continuous-courses. However, the classical MDP had an average adaptation probability of 0.5±0.22 while the quantum algorithm reached 0.76±0.28. In cases where adaptation failed, classical MDP yielded 0.31±0.26 average amplitude while the quantum approach averaged a more optimistic 0.57±0.4, but with high phase fluctuations. Conclusion: Our results demonstrate that quantum machine learning approaches provide a feasible and promising framework for real-time and sequential clinical decision-making in adaptive radiotherapy.

60 APPLIED LIFE SCIENCES↗

NASA NTRS2013

An Onboard ISS Virtual Reality Trainer

Prior to the retirement of the Space Shuttle, many exterior repairs on the International Space Station (ISS) were carried out by shuttle astronauts, trained on the ground and flown to the Station to perform these specific repairs. With the retirement of the shuttle, this is no longer an available option. As such, the need for ISS crew members to review scenarios while on flight, either for tasks they already trained for on the ground or for contingency operations has become a very critical issue. NASA astronauts prepare for Extra-Vehicular Activities (EVA) or Spacewalks through numerous training media, such as: self-study, part task training, underwater training in the Neutral Buoyancy Laboratory (NBL), hands-on hardware reviews and training at the Virtual Reality Laboratory (VRLab). In many situations, the time between the last session of a training and an EVA task might be 6 to 8 months. EVA tasks are critical for a mission and as time passes the crew members may lose proficiency on previously trained tasks and their options to refresh or learn a new skill while on flight are limited to reading training materials and watching videos. In addition, there is an increased need for unplanned contingency repairs to fix problems arising as the Station ages. In order to help the ISS crew members maintain EVA proficiency or train for contingency repairs during their mission, the Johnson Space Center's VRLab designed an immersive ISS Virtual Reality Trainer (VRT). The VRT incorporates a unique optical system that makes use of the already successful Dynamic On-board Ubiquitous Graphics (DOUG) software to assist crew members with procedure reviews and contingency EVAs while on board the Station. The need to train and re-train crew members for EVAs and contingency scenarios is crucial and extremely demanding. ISS crew members are now asked to perform EVA tasks for which they have not been trained and potentially have never seen before. The Virtual Reality Trainer (VRT) provides an immersive 3D environment similar to the one experienced at the VRLab crew training facility at the NASA Johnson Space Center. VRT bridges the gap by allowing crew members to experience an interactive, 3D environment to reinforce skills already learned and to explore new work sites and repair procedures outside the Station.

Miralles, Evelyn↗

DOE OSTI2022

Teaching Quantum Computing to High School Students

Quantum computing is a growing field at the intersection of physics and computer science. The goal of this article is to highlight a successfully trialled quantum computing course for high school students between the ages of 15 and 18 years old. This course was designed to bridge the gap between popular science articles and advanced undergraduate textbooks. Conceptual ideas in the text are reinforced with active learning techniques, such as interactive problem sets and simulation-based labs at various levels. The course is freely available for use and download under the Creative Commons “Attribution-NonCommercial-ShareAlike 4.0 International” license.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1993

Evolving fuzzy rules in a learning classifier system

The fuzzy classifier system (FCS) combines the ideas of fuzzy logic controllers (FLC's) and learning classifier systems (LCS's). It brings together the expressive powers of fuzzy logic as it has been applied in fuzzy controllers to express relations between continuous variables, and the ability of LCS's to evolve co-adapted sets of rules. The goal of the FCS is to develop a rule-based system capable of learning in a reinforcement regime, and that can potentially be used for process control.

Valenzuela-Rendon, Manuel↗

NASA NTRS1995

Attitude determination using an adaptive multiple model filtering Scheme

Attitude determination has been considered as a permanent topic of active research and perhaps remaining as a forever-lasting interest for spacecraft system designers. Its role is to provide a reference for controls such as pointing the directional antennas or solar panels, stabilizing the spacecraft or maneuvering the spacecraft to a new orbit. Least Square Estimation (LSE) technique was utilized to provide attitude determination for the Nimbus 6 and G. Despite its poor performance (estimation accuracy consideration), LSE was considered as an effective and practical approach to meet the urgent need and requirement back in the 70's. One reason for this poor performance associated with the LSE scheme is the lack of dynamic filtering or 'compensation'. In other words, the scheme is based totally on the measurements and no attempts were made to model the dynamic equations of motion of the spacecraft. We propose an adaptive filtering approach which employs a bank of Kalman filters to perform robust attitude estimation. The proposed approach, whose architecture is depicted, is essentially based on the latest proof on the interactive multiple model design framework to handle the unknown of the system noise characteristics or statistics. The concept fundamentally employs a bank of Kalman filter or submodel, instead of using fixed values for the system noise statistics for each submodel (per operating condition) as the traditional multiple model approach does, we use an on-line dynamic system noise identifier to 'identify' the system noise level (statistics) and update the filter noise statistics using 'live' information from the sensor model. The advanced noise identifier, whose architecture is also shown, is implemented using an advanced system identifier. To insure the robust performance for the proposed advanced system identifier, it is also further reinforced by a learning system which is implemented (in the outer loop) using neural networks to identify other unknown quantities such as spacecraft dynamics parameters, gyro biases, dynamic disturbances, or environment variations.

Lam, Quang↗

NASA NTRS2018

Initial Approach to Collect Small Unmanned Aircraft System Off-Nominal Operational Situations Data

NASA is developing the Unmanned Aircraft System Traffic Management research platform to safely integrate small unmanned aircraft operations in large-scale at low-altitudes. As a part of this effort, small unmanned aircraft system off-nominal operational situations data collection process has been developed to take lessons learned and to reinforce operational compliance. In this paper, descriptions of variables used for digital data collection and an online report form for collection of observational data from the operators (contextual data) are provided. They are used to collect off-nominal data from the Unmanned Aircraft System Traffic Management National Campaign in 2017. The digital data show that 2 out of 118 campaign operations (1.7%) encountered loss of navigation. Since the campaign aircraft used Global Positioning System for navigation, it is likely that unobstructed view of the sky at the campaign locations contributed to this small number. Also, 4 out of 47 operations (8.5%) encountered loss of communications. A relatively short distance between ground control system and aircraft, ranging from 2300 feet to 4200 feet, likely contributed to this small number. There was no data to identify the loss of communications condition, aircraft received signal strength, for the remaining 71 operations suggesting that some operators may not be monitoring unmanned aircraft communications system performance or monitoring it with different parameters. For the contextual data, due to the low number of total reports during the campaign, no significant trends emerged. This is an initial attempt to collect contextual data from small unmanned aircraft operators about off-nominal situations, and changes will be made to the future data collection to improve the amount and quality of the information.

unmanned aviation systems traffic management (UTM)↗

NASA NTRS2018

Initial Approach to Collect Small Unmanned Aircraft System Off-Nominal Operational Situations Data

Jung, Jaewoo↗

NASA NTRS1969

Periodic shock with added clock.

Periodic shock in rats maintained on variable-interval schedule of reinforcement interspersed with sequence of three different stimulus conditions

CONDITIONING (LEARNING)↗

DOE OSTI2018

Argonne Leadership Computing Facility (2018 Annual Report)

The Argonne Leadership Computing Facility enables breakthroughs in science and engineering by providing supercomputing resources and expertise to the research community. In 2018, we continued to build up programs and efforts that reinforce our overall simulation, learning, and data analysis capabilities as a facility. We also enhanced our services to better meet our users’ needs and workflow preferences.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

NASA NTRS1981

Using rewards and penalties to obtain desired subject performance

Operant conditioning procedures, specifically the use of negative reinforcement, in achieving stable learning behavior is described. The critical tracking test (CTT) a method of detecting human operator impairment was tested. A pass level is set for each subject, based on that subject's asymptotic skill level while sober. It is critical that complete training take place before the individualized pass level is set in order that the impairment can be detected. The results provide a more general basis for the application of reward/penalty structures in manual control research.

Cook, M.↗

DOE OSTI2022

A machine learning approach to determine the elastic properties of printed fiber-reinforced polymers

This work focuses on the simultaneous determination of the elastic constants and the fiber orientation state for a short fiber-reinforced polymer composite by performing a minimum of experimental tests. Here we introduce a methodology that enables the inverse determination of fiber orientation state and the in-situ polymer properties by performing tensile tests at the composite coupon level. We demonstrate the approach for the extrusion deposition additive manufacturing (EDAM) process to illustrate one application of the methodology, but the development is such that it can be applied to short fiber-reinforced polymer (SFRP) systems processed via other methods. Currently, developing composites additive manufacturing digital twins require extensive material characterization. In particular, the mechanical characterization of the orthotropic elastic properties of a composite involves extensive sample preparation and testing, therefore the elasticity tensor is generally populated using a micromechanics model. This, however, requires measuring the fiber orientation state in addition to knowing the constituent material properties. Experimentally measuring the fiber orientation state can be tedious and time consuming. Further, optical methods are limited to resolving the orientation of cylindrical fibers or cluster of non-cylindrical fibers, and computed tomography (CT) methods scan regions of volume that are much smaller than a full printed bead. Therefore, we propose a methodology, accelerated by machine learning, to identify the anisotropic mechanical properties and fiber orientation state at the same time. Early results show that inference of the fiber orientation and composite properties is possible with as few as three tensile tests. Our results show that a combination of the choice of the micromechanics model and reliable set of experiments can yield the nine elastic constants, as well as, the fiber orientation state.

36 MATERIALS SCIENCE↗