Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Actors”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Combinatorial Auction-Based Strategic Deconfliction of Federated UTM Airspace

Unmanned Aerial Vehicles (UAVs) have become commonly used to perform a wide range of commercial activities such as cinematography and medical supply delivery. Consequently, regulators have become interested in designing UAV Traffic Management systems (UTMs) to coordinate UAV traffic among a collection of UAV operators. One framework which has been recently proposed for a UTM system is a combinatorial auction. In this framework, airspace is modelled as a 4D grid of space-time cells. UAV operators bid on cells which collectively form paths for their UAVs. Ideally, an airspace auction should reveal information about the current price of flight paths to bidders, allowing bidders to identify and bid on a select number of paths instead of placing as many bids as possible in the hopes of stumbling on a cheap path. Revealing too much information, however, can allow bad actors to place bids which are intended not to win but to raise the price that a rival bidder must pay. We address these twin challenges with a new information revelation framework which provides bidders with wide-ranging pricing information while suppressing bad actors. We evaluate our framework on scenarios based on a Japan Aerospace Exploration Agency (JAXA) case study and find that it can scale to thousands of bids.

Christopher J C Leet↗
NASA NTRSDate not supplied

A Human-In-The-Loop Simulation for Urban Air Mobility in the Terminal Area

In this paper researchers propose a human-in-the-loop experiment to study human performance when tasked with tactical deconfliction in terminal area air taxi operations. The air taxi operations being considered herein are an advanced air transportation concept called Urban Air Mobility (UAM). The UAM concept aims to support not only air taxi operations, but also package delivery and emergency response among other use cases. The key innovation over current air transportation lies with the introduction of highly automated aircraft and air traffic management systems. Development of the UAM system will include transitional midterm phases where some operational services will be provided by a mixture of automation and human actors. Midterm operations present a unique challenge, since the scope of responsibility of automated systems is largely undefined, suggesting the need for direct human participation with little to inform how much human intervention is necessary. Here it is assumed that traffic management responsibilities require coordination between human actors and automated systems and focus on arrival flows for midterm operations. In the proposed human-in-the-loop simulation, virtual UAM traffic is strategically deconflicted by a Provider of Services for UAM at departure, then tactically managed by a human at the arrival facility. Generated traffic consists of UAM participants flying in UAM exclusive airspace structures, thus isolated from traditional traffic. The human operator is tasked with managing spacing of arrival traffic and executing speed adjustments as deemed necessary. Researchers propose the investigation of three levels of automation assistance: 1) no assistance; 2) spacing violation detection; 3) spacing violation detection and speed adjustment recommendations. Quantitative measures like throughput and delay are used to assess the human's capacity for accommodating airborne delays. Qualitative evaluations such as surveys and open-ended feedback are used to gain insight into human factors. These factors could introduce additional capacity constraints on traffic, independent of physical or technical constraints. Although findings for this study will not be reported as the study has not yet been executed, the authors conclude with potential outcomes informed by previous simulations in the literature and suggestions for the structure and procedures of midterm human-automation air traffic management.

UAM↗
NASA NTRSDate not supplied

Exploring the Impact of Compliance With Maneuvering Guidelines for Space Traffic Management

If the current estimate of proposed large constellations is realized, the near-Earth space environment will see more than 50,000 new satellites added to the catalog of resident space objects (RSOs) in the coming decade. This is an order of magnitude increase from the current population and poses new policy challenges as global operators seek to leverage the benefits these new satellite systems provide while also ensuring a sustainable approach to collision avoidance. Various guidelines have been proposed to date to support this effort, including the development of right of way rules to guide how a collision avoidance maneuver should be performed, and how the maneuver burden should be shared between the two satellites involved. However, it is very difficult to evaluate and compare proposed guidelines due to the complex nature of space traffic and the rapidly changing space environment. This study seeks to address this issue by utilizing the Virtual Environment for Space Traffic Analysis (VESTA), a high-fidelity simulation tool that has been developed at Georgia Tech over the past few years with the explicit purpose of evaluating the future of space traffic environment. Using this tool, a sensitivity study is performed that incorporates a likely set of future large constellations and provides metrics on the impact that a select set of proposed maneuvering guidelines would have on operators given realistic variations in spacecraft capabilities (e.g. maneuverability and propulsion capabilities), and other factors (owner-country, public vs. private, etc.). Specifically, this study compares three potential right of way rules: 1) a rule based on maneuverability proposed by the Space Safety Coalition, 2) a rule based on the geometry of the spacecraft rendezvous, and 3) a rule that equally distributes the maneuver burden between two operators, The results highlight general observations on the effectiveness and limitations of each of the proposed maneuvering guidelines. In addition to the choice of right of way rule, success of space traffic management will be significantly impacted by compliance – which operators, or how many operators, comply with the space traffic rules. The findings of this analysis have important implications for future methods that could be pursued to put in place right of way rules. For example, non-binding right of way rules may have variable levels of compliance that differ across actors. The impact of compliance by just one nation, or non-compliance by just one nation, help to demonstrate the impact of ensuring all major space actors coordinate on this effort. Overall, this analysis provides insight into the relative gains in safety (decrease in collision risk) that would likely result from more politically intense efforts to increase the number of countries implementing space traffic management rules.

conjunction assessment↗
NASA NTRSDate not supplied

A Human-In-The-Loop Simulation for Urban Air Mobility in the Terminal Area

In this presentation we propose a human-in-the-loop experiment to study the potential impact of human engagement in tactical mitigation of delay in terminal area air taxi operations. The air taxi operations being considered herein is an advanced air transportation concept called Urban Air Mobility (UAM). The UAM concept aims to support not only air taxi operations, but also package delivery and emergency response among other use cases. The key innovation over current air transportation lies with the introduction of autonomous aircraft and autonomous air traffic management systems. Development of the UAM system will include transitional midterm phases where some operational services will be provided by a mixture of automation and human actors. Midterm operations present a unique challenge, since the scope of responsibility of automated systems is largely undefined, suggesting the need for direct human participation with little to inform how much human intervention is necessary. Here we assume that traffic management responsibilities require coordination between human actors and automated systems and focus on arrival flows for midterm operations. In the proposed human-in-the-loop simulation, virtual UAM traffic is strategically deconflicted by a Provider of Services for UAM at departure, then tactically managed by a human at the arrival facility. Generated traffic consists of UAM participants flying in UAM exclusive airspace structures, thus isolated from traditional traffic. The human operator is tasked with managing spacing of arrival traffic and executing speed adjustments as deemed necessary. We propose the investigation of three levels of automation assistance: 1) no assistance; 2) spacing violation detection; 3) spacing violation detection and speed adjustment recommendations. Quantitative measures like throughput and delay are used to assess the human's capacity for accommodating airborne delays. Qualitative evaluations such as surveys and open-ended feedback are used to gain insight into human factors. These factors could introduce additional capacity constraints on traffic, independent of physical or technical constraints. Although findings for this study will not be reported as the study has not yet been executed, we conclude with potential outcomes informed by previous simulations in the literature and suggestions for the structure and procedures of midterm human-automation air traffic management.

UAM↗
NASA NTRSDate not supplied

Exploring the Effectiveness of Maneuvering Guidelines for Space Traffic Management

The number of objects in space has been increasing rapidly, and the risk of collision has grown as well. Many spacecraft operators are now receiving multiple collision warnings a day. Despite this, systems to manage space traffic have been limited: there are no broadly agreed upon guidelines or rules governing the response to predicted potential collisions. Instead, spacecraft operators generally determine whether, when, and how to respond to these warnings on a manual and ad hoc basis. Coordination between operators, if it occurs, often requires repeated communication and negotiation. Some space actors have suggested that the space community should develop right of way rules, similar to those in the ground, sea, and air domains, to guide collision response decisions. However, it is unclear whether such rules would be effective, and it’s unknown whether such rules would have an equitable impact across various spacecraft operators. To address these issues, we developed the Virtual Environment for Space Traffic Analysis (VESTA), a software tool that was used to simulate the space environment with a recent catalog of objects obtained from the U.S. Space-Track.org system. We implemented multiple potential right of way rules within this model. The analysis confirmed that the choice of right of way rule makes a meaningful difference in terms of both efficiency and distributional effects in terms of collision avoidance maneuvers. For example, our analysis shows that rules in which the less massive satellite is required to maneuver results in a more equitable distribution of maneuver responsibility among space actors and also requires less fuel mass, compared to a rule in which the more massive satellite maneuvers.

space traffic management↗
DOE OSTI2010

Assessing the Institution of the Nuclear Nonproliferation Regime

The nuclear nonproliferation regime is facing a crisis of effectiveness. During the Cold War, the regime was relatively effective in stemming the proliferation of nuclear weapons and building an institutional structure that could, under certain conditions, ensure continued success. However, in the evolving global context, the traditional approaches are becoming less appropriate. Globalization has introduced new sets of stresses on the nonproliferation regime, such as the rise of non-state actors, broadening extensity and intensity of supply chains, and the multipolarization of power. This evolving global context demands an analytical and political flexibility in order to meet future threats. Current institutional capabilities established during the Cold War are now insufficient to meet the nonproliferation regime’s current and future needs. The research was based on information gathered through interviews and reviews of the relevant literature, and two dominant themes emerged. First, that human security should be integrated into the regime to account for the rise of non-state actors and networked violence. Second, confidence in the regime’s overall effectiveness has eroded at a time where verification-based confidence is becoming more essential. The research postulates that a critical analysis of the regime that fully utilizes institutional theory, with its focus on rules, normative structures, and procedures will be essential to adapting the regime to the current global context, building mechanisms for generating trust, creating better enforcement, and providing flexibility for the future.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
DOE OSTI2019

Does geography matter? A study of technological learning and innovation in the U.S. Solar photovoltaic balance-of-systems (PV BOS) industry

Different learning channels (i.e., learning from upstream actors, local network, global codified knowledge, etc.) coexist in firms’ technological innovation processes. Although firms are exposed to globalization and convenient communication and transportation tools, researchers still argue about the role of geography in technology innovation. Geographic proximity can facilitate knowledge spillover and network-building. Localized learning and the significance of geography indicate the importance of local knowledge in innovative activities. Local knowledge tends to be more closely aligned with local firms’ technological demand and easier to understand and absorb, because they are created and applied in the same context. However, following Schumpeter, innovation researchers believe that diverse knowledge, especially knowledge from external industries and geographically distant places, is a valuable source for innovation, especially disruptive innovation. Therefore, firms must balance different learning channels, especially between localized learning and global learning. However, it is still unclear about the extent to which localized learning is important for technology innovations and which factors influence the degree of localized learning. Meanwhile, most studies in the area of localized learning usually take no account of the international dimension and the interdependent of knowledge at different geographic scales. Different learning channels (i.e., learning from upstream actors, local network, global codified knowledge, etc.) coexist in firms’ technological innovation processes. Although firms are exposed to globalization and convenient communication and transportation tools, researchers still argue about the role of geography in technology innovation. Geographic proximity can facilitate knowledge spillover and network-building. Localized learning and the significance of geography indicate the importance of local knowledge in innovative activities. Local knowledge tends to be more closely aligned with local firms’ technological demand and easier to understand and absorb, because they are created and applied in the same context. However, following Schumpeter, innovation researchers believe that diverse knowledge, especially knowledge from external industries and geographically distant places, is a valuable source for innovation, especially disruptive innovation. Therefore, firms must balance different learning channels, especially between localized learning and global learning. However, it is still unclear about the extent to which localized learning is important for technology innovations and which factors influence the degree of localized learning. Meanwhile, most studies in the area of localized learning usually take no account of the international dimension and the interdependent of knowledge at different geographic scales. To address this research gap, this paper empirically investigated firms’ learning and innovation processes in relation to the geography. The aim is to answer the following three questions: 1) how does geographic distance influence knowledge spillover; 2) how technological characteristics influence knowledge acquisition choices as regards local vs. non-local knowledge? and 3) how do local and non-local knowledge affect the quality of innovation? The subject of our empirical inquiry is photovoltaic balance-of-systems (PV BOS) components. This choice is primarily motivated by two factors: (1) large-scale deployment of distributed solar PV technologies is widely considered to be an important piece in addressing the environmental impacts of the electricity sector, making PV an important technology to study; furthermore, there is a general consensus that the biggest barrier for a wider deployment of PV is the high price of non-module components, and (2) technological aspects of non-module components of PV systems make local conditions (climate, regulation, learning networks, etc.) salient, thus affording a rich empirical setting to explore the geography-innovation-economy interactions at the local level. We find that the localization effect of technological learning in the solar PV BOS industry is at state level, the number of citations-based linkages decreases with distance, and we also find that international knowledge significantly contributes to the quality of patents, which suggests that state border might be a barrier of current knowledge spillover, and only focusing on local innovation networks is not sufficient for promoting PV BOS innovation.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2019

Solar soft cost ontology for multi-stakeholder knowledge base

Introduction: Solar soft costs can account for 60% of the cost of an installed solar system, thus cost reductions for solar hinge on reducing soft costs. Soft costs are most frequently defined as non-hardware costs. Defining something by what it isn’t makes it inherently difficult to identify what exactly it is, which in turn makes it difficult to study. Our goal is to develop an ontology that identifies 1) a consistent and defined soft cost vocabulary, 2) who creates, shares and acquires soft cost knowledge, and 3) the learning processes used to facilitate knowledge flows. Ultimately, identifying these aspects of solar soft costs will provide a pathway for public policy and market interventions to reduce soft costs. Methods: To build the solar soft cost ontology (SSCO) we built a corpus of solar soft cost literature (through keyword search), then coded the corpus for all concepts related to or influencing costs, barriers, or time. Finally, we organized the concepts into taxonomies, including soft costs and network actors, and developed the properties (i.e., attributes and relationships) that link the concepts. In phase two, we will code a set of solar installer case studies and complimentary solar sector (e.g., distributors, software providers, trade organizations) firm case studies to update the SSCO, documenting changes and additions resulting from the use of practitioner feedback. Results and Discussion: The resulting SSCO includes concepts, attributes, and relationships that provide a rich and deep knowledge base that merges academic and practitioner expertise to create a shared and exhaustive vocabulary related to soft costs, a network of actors that impact those soft costs, and the attributes and relationships that connect them. The applications of the SSCO extend to a diverse group of stakeholders, including researchers, policy makers, and practitioners, engaged in reducing solar soft costs.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2020

Closing the Loops on Solar Photovoltaics Modules: An Agent-Based Modeling Approach for the Study of Circular Economy Strategies

Solar photovoltaics (PV) installed capacity have grown exponentially since the early 2000s (average annual growth rate of 50%). With 4,700 GW projected installed capacity by 2050, the volume of PV panels waste is also expected to become substantial. Though renewables are a sine qua non to the establishment of a truly circular economy (CE), the issue arising from their end-of-life management needs to be resolved. The management of PV end-of-life also represent a singular opportunity to create value, from recovered valuable, rare or critical materials (e.g. silver, tellurium, indium). Moreover, the PV case illustrates some of the current barriers to CE; for instance, the need for a common definition of waste (PV are defined as e-waste in the European Union but as general waste in the United States) and potential loss of innovations' advantages (PV average efficiency has continuously grown the past 10 years). In this study, an agent-based modeling (ABM) approach is proposed to simulate PV end-of-life management in the United States. The model explores how the decisions of the various actors involved in handling PV waste affect the quantities of PV that are reused, recycled, or land-filled. In the model, manufacturers, residential, and nonresidential PV owners, installers, and recyclers are represented by agents while governmental policies and regulations are treated as exogenous variables. PV-market data are used to define agents' characteristics (e.g., installed PV capacities or producing costs). Agents' decision rules draw on the literature related to industrial symbiosis (IS) and peoples' waste behaviors as they appear as the most widely used model to implement CE principles at a meso level. Specifically, the concepts of mutual trust between IS actors and knowledge about the IS philosophy are yielded to define the agents' decision process. The primary outputs of the ABM are the costs and volume of waste associated with each end-of-life pathway. Preliminaries results indicate recycling of residential PV is affected by the cost, perceived difficulty of recycling behaviors, and social norms. Moreover, the type of network connecting agents and the initial recycling rate strongly influence results. The model also highlights the crucial role of recycling behavior adoption: in case of early failure of PV, recycled volumes increase by about 40%. This represents an additional 5.7 billion USD from potential recovered silver. Finally, although results for the PV case are presented here, the developed ABM aims at being a general tool for the study of CE strategies and the CE transition. Furthermore, steps of this research include extending the model to encompass circular economy strategies (e.g., design for recycling or lifetime extension) and validating its general architecture from various case studies.

14 SOLAR ENERGY↗
DOE OSTI2021

Prioritizing Countries by Concern Regarding Access to Weapons of Mass Destruction Materials

This article presents a global prioritization methodology that evaluates the relative risks of non-state actor acquisition of materials that could be used in chemical, biological, radiological, nuclear and high explosive Weapons of Mass Destruction (WMD) from the country’s relevant infrastructure. Prioritization is based on three domains: 1. Assessing relative scale of materials in each country, 2. The country’s corresponding security posture, and 3. The presence of threat actors. The output is a list of countries prioritized from greatest risk to least. Rather than providing an overall 1 to N ranking, however, the results are placed into tiers based upon their natural groupings within the three domains. The countries in the highest tiers are flagged as potential US national security concern; those scoring in the middle and at the bottom are flagged as posing lower US national security concern. A systematic approach assesses each country by leveraging many disciplines, such as risk and decision analysis, as well as expert judgement. A quantitative value model based on Multi-Attribute Value Theory (MAVT) organizes the objectives scoring criteria into a value tree using lessons learned from previous studies, published literature, and expert judgement. The article presents the prioritization categories and corresponding value model scoring criteria to include measurement type, weight, range, and value preference. Country names and data are notional in order to share the details on the underlying methodology and model without identification of actual security risks. A deliberative process addresses factors external to the model and scrutinizes inputs, methodology, model, and results.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
DOE OSTI2022

A Distributed Trust Model Simulator for Energy Grid of Things Distributed Energy Resource Management System

The evolution of networks into more distributed, self-reliant nodes has mitigated single-point failures that plagued traditional centralized networks. Applied to power grids, distributed systems can increase the integrity and availability of grid services while also offering a power management solution. However, while distributed networks provide scalability, security, and sustainability compared to centralized networks, their distributed nature makes them harder for anomaly detection and prevention. Incorporating a Distributed Trust Model (DTM) System into an Energy Grid of Things Distributed Energy Resource Management System (EGOT DERMS) allows grid participants to be characterized and their communication to be analyzed for possible attacks. A Trust Model simulator is needed to evaluate and improve the DTM System.Trustworthiness is calculated using a Trust Model. While many trust models exist, most only consider 2-3 matrices to evaluate trust. The TM proposed in this thesis uses a Metric Vector of Trust (MVoT) monitoring 17 parameters when assessing trust. Moreover, unlike standard trust models, the proposed trust model establishes a method to test the trust between various actors within the network and probe the trust model itself. Using a Trust Model Simulator, MVoT calaculations, initial values, and parameters are fine-tuned to achieve high-confidence message classifications and minimize false positives. The DTM System and Trust Mode Simulation Suite allow for distributed trust evaluation with a real-time classification of EGOT DERMS actors, providing additional security for distributed systems.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2012

Design and field test of collaborative tools in the service of an innovative organization

This paper presents the design process of collaborative tools, based on ICT, aiming at supporting the tasks of the team that manages an outage of an energy production plant for maintenance activities. The design process follows an iterative and multidisciplinary approach, based on a collective tasks modeling of the outage management team in the light of Socio Organizational and Human (SOH) field studies, and on the state of the art of ICT. Field test of the collaborative tools designed plays a great place in this approach, allowing taking into account the operational world but involves also some risks which must be managed. To implement tools on all the production plants, we build an 'operational concept' with a level of description which authorizes the evolution of tools and allows some local adaptations. The field tests provide lessons on the ICT topics. For examples: the status of the remote access tools, the potential of use of a given information input by an actor for several individual and collective purposes, the actors perception of the tools meaning, and the requirements for supporting the implementation of change. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory

Risk-sensitive reinforcement learning (RL) has garnered significant attention in recent years due to the growing interest in deploying RL agents in real-world scenarios. A critical aspect of risk awareness involves modelling highly rare risk events (rewards) that could potentially lead to catastrophic outcomes. These infrequent occurrences present a formidable challenge for data-driven methods aiming to capture such risky events accurately. While risk-aware RL techniques do exist, they suffer from high variance estimation due to the inherent data scarcity. Our work proposes to enhance the resilience of RL agents when faced with very rare and risky events by focusing on refining the predictions of the extreme values predicted by the state-action value distribution. To achieve this, we formulate the extreme values of the state-action value function distribution as parameterized distributions, drawing inspiration from the principles of extreme value theory (EVT). We propose an extreme value theory based actor-critic approach, namely, Extreme Valued Actor-Critic (EVAC) which effectively addresses the issue of infrequent occurrence by leveraging EVT-based parameterization. Importantly, we theoretically demonstrate the advantages of employing these parameterized distributions in contrast to other risk-averse algorithms. Our evaluations show that the proposed method outperforms other risk averse RL algorithms on a diverse range of benchmark tasks, each encompassing distinct risk scenarios.

Wang, Yu↗
DOE OSTI2023

Off-policy deep reinforcement learning with automatic entropy adjustment for adaptive online grid emergency control

Electric overloading conditions and contingencies put modern power systems at risk of voltage collapse and blackouts. Load shedding is crucial to maintain voltage stability for grid emergency control. However, the rule- or model-based schemes rely on accurate dynamic system models and face considerable challenges in adapting to various operating conditions and uncertain event occurrences. Here, to address these issues, this paper proposes a novel deep reinforcement learning (DRL)-based voltage stability control algorithm with automatic entropy adjustment (AEA) for grid emergency control. Various dynamic network components for complex system operations are modeled to construct the DRL environment. An off-policy soft actor-critic architecture is developed to maximize the expected reward and policy entropy simultaneously. The AEA mechanism is proposed to facilitate the policy maximum entropy procedure, and the proposed method can automatically provide effective discrete and continuous actions against various fault scenarios. Our approach accomplishes high sampling efficiency, scalability, and auto-adaptivity of the control policies under high uncertainties. Comparative studies with the existing DRL-based control methods in IEEE benchmarks indicate salient performance improvement of the proposed method for dynamic system emergency control.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Network Reconfiguration for Enhanced Operational Resilience Using Reinforcement Learning

This paper proposes a reinforcement learning-based approach for distribution network reconfiguration(DNR) to enhance the resilience of the electric power supply. Resilience enhancements usually require solving large-scale stochastic optimization problems that are computationally expensive and sometimes infeasible. The exceptional performance of reinforcement learning techniques has encouraged their adoption in various power system control studies, specifically resilience-based real-time applications. In this paper, a single agent framework is developed using an Actor-Critic algorithm (ACA) to determine statuses of tie-switches in a distribution feeder impacted by an extreme weather event. The proposed approach provides a fast-acting control algorithm that reconfigures the feeder topology to reduce or even avoid load shedding. The problem is formulated as a discrete Markov decision process in such a way that a system state captures the system topology and its operational characteristics. An action is made to open or close a specific set of tie-switches after which a reward is calculated to evaluate the practicality and advantage of that action. The iterative Markov process is used to train the proposed ACA under diverse failure scenarios and is demonstrated on the 33-node distribution feeder system. Results show the capability of the proposed ACA to determine proper switching action of tie-switches with accuracy exceeding 93%.

actor critic↗
DOE OSTI2020

Deep Reinforcement Learning-Based Edge Caching in Wireless Networks

With the purpose to offload data traffic in wireless networks, content caching techniques have recently been studied intensively. Using these techniques and caching a portion of the popular files at the local content servers, the users can be served with less delay. Most of the content replacement policies are based on the content popularity, that depends on the users' preferences. In practice, such information varies over time. Therefore, an approach to determine the file popularity patterns must be incorporated into caching policies. In this context, we study content caching at the wireless network edge using a deep reinforcement learning framework with Wolpertinger architecture. In particular, we propose deep actor-critic reinforcement learning based policies for both centralized and decentralized content caching. For centralized edge caching, we aim at maximizing the cache hit rate. In decentralized edge caching, we consider both the cache hit rate and transmission delay as performance metrics. These proposed frameworks are assumed to neither have any prior information on the file popularities nor know the potential variations in such information. Via simulation results, the superiority of the proposed frameworks is verified by comparing them with other policies, including least frequently used (LFU), least recently used (LRU), and first-in-first-out (FIFO) policies.

42 ENGINEERING↗
DOE OSTI2021

Distributional Deep Reinforcement Learning-Based Emergency Frequency Control

Emergency frequency control is one of the most critical approaches to maintain power system stability after major disturbances. With the increasing number of grid-connected renewable energy sources, existing model-based methods of frequency control are facing up with challenges of computational speed and scalability for large-scale systems. In this paper, the emergency frequency control problem is formulated as a Markov Decision Process (MDP) and solved through a novel Distributional Deep Reinforcement Learning (DDRL) method, namely the distributional soft actor critic (DSAC) method. Compared with other RL methods that only estimate the mean value, the proposed DSAC model estimates the distribution of value function over returns. This advancement can lead to more insights and knowledge for the agent, with the benefit of a much faster and more stable learning process, and the improved frequency control performance. Here, the simulation results on IEEE 39-bus and IEEE 118-bus systems demonstrate the effectiveness and robustness of proposed models, as well as the advantage compared to other state-of-the-art DRL algorithms.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Neural network approaches for parameterized optimal control

Here, we consider numerical approaches for deterministic, finite-dimensional optimal control problems whose dynamics depend on unknown or uncertain parameters. We seek to amortize the solution over a set of relevant parameters in an offline stage to enable rapid decision-making and be able to react to changes in the parameter in the online stage. To tackle the curse of dimensionality arising when the state and/or parameter are high-dimensional, we represent the policy using neural networks. We compare two training paradigms: First, our model-based approach leverages the dynamics and definition of the objective function to learn the value function of the parameterized optimal control problem and obtain the policy using a feedback form. Second, we use actor-critic reinforcement learning to approximate the policy in a data-driven way. Using an example involving a two-dimensional convection-diffusion equation, which features high-dimensional state and parameter spaces, we investigate the accuracy and efficiency of both training paradigms. While both paradigms lead to a reasonable approximation of the policy, the model-based approach is more accurate and considerably reduces the number of PDE solves.

97 MATHEMATICS AND COMPUTING↗