Results for “reporting” — Page 24

NASA NTRS2008

2008 NASA Range Safety Annual Report

Welcome to the 2008 edition of the NASA Range Safety Annual Report. Funded by NASA Headquarters, this report provides a NASA Range Safety overview for current and potential range users. This year, along with full length articles concerning various subject areas, we have provided updates to standard subjects with links back to the 2007 original article. Additionally, we present summaries from the various NASA Range Safety Program activities that took place throughout the year, as well as information on several special projects that may have a profound impact on the way we will do business in the future. The sections include a program overview and 2008 highlights of Range Safety Training; Range Safety Policy; Independent Assessments and Common Risk Analysis Tools Development; Support to Program Operations at all ranges conducting NASA launch operations; a continuing overview of emerging Range Safety-related technologies; Special Interests Items that include recent changes in the ELV Payload Safety Program and the VAS explosive siting study; and status reports from all of the NASA Centers that have Range Safety responsibilities. As is the case each year, contributors to this report are too numerous to mention, but we thank individuals from the NASA Centers, the Department of Defense, and civilian organizations for their contributions. We have made a great effort to include the most current information available. We recommend that this report be used only for guidance and that the validity and accuracy of all articles be verified for updates. This is the third year we have utilized this web-based format for the annual report. We continually receive positive feedback on the web-based edition, and we hope you enjoy this year's product as well. It has been a very busy and productive year on many fronts as you will note as you review this report. Thank you to everyone who contributed to make this year a successful one, and I look forward to working with all of you in the years to come.

Lamoreaux, Richard W.↗

NASA NTRS2011

NASA Aviation Safety Reporting System (ASRS)

The NASA Aviation Safety Reporting System (ASRS) collects, analyzes, and distributes de-identified safety information provided through confidentially submitted reports from frontline aviation personnel. Since its inception in 1976, the ASRS has collected over 900,000 reports and has never breached the identity of the people sharing their information about events or safety issues. From this volume of data, the ASRS has released over 5,500 aviation safety alerts concerning potential hazards and safety concerns. The ASRS processes these reports, evaluates the information, and provides de-identified report information through the online ASRS Database at http://asrs.arc.nasa.gov. The NASA ASRS is also a founding member of the International Confidential Aviation Safety Systems (ICASS) group which is a collection of other national aviation reporting systems throughout the world. The ASRS model has also been replicated for application to improving safety in railroad, medical, fire fighting, and other domains. This presentation \vill discuss confidential, voluntary, and non-punitive reporting systems and their advantages in providing information for safety improvements.

Connell, Linda↗

NASA NTRS2013

International VLBI Service for Geodesy and Astrometry 2012 Annual Report

This volume of reports is the 2012 Annual Report of the International VLBI Service for Geodesy and Astrometry (IVS). The individual reports were contributed by VLBI groups in the international geodetic and astrometric community who constitute the permanent components of IVS. The IVS 2012 Annual Report documents the work of the IVS components for the calendar year 2012, our fourteenth year of existence. The reports describe changes, activities, and progress ofthe IVS. Many thanks to all IVS components who contributed to this Annual Report. With the exception of the first section and parts of the last section (described below), the contents of this Annual Report also appear on the IVS Web site athttp:ivscc.gsfc.nasa.gov/publications/ar2012

VLBI↗

NASA NTRS2014

International VLBI Service for Geodesy and Astrometry 2013 Annual Report

This volume of reports is the 2013 Annual Report of the International VLBI Service for Geodesy and Astrometry (IVS). The individual reports were contributed by VLBI groups in the international geodetic and astrometric community who constitute the permanent components of IVS. The IVS 2013 Annual Report documents the work of the IVS components for the calendar year 2013, our fifteenth year of existence. The reports describe changes, activities, and progress of the IVS. Many thanks to all IVS components who contributed to this Annual Report. With the exception of the first section and the last section, the contents of this Annual Report also appear on the IVS Web site at http://ivscc.gsfc.nasa.gov/publications/ar2013.

VLBI↗

NASA NTRS2017

NASA Aviation Safety Reporting System (ASRS)

The NASA Aviation Safety Reporting System (ASRS) collects, analyzes, and distributes de-identified safety information provided through confidentially submitted reports from frontline aviation personnel. Since its inception in 1976, the ASRS has collected over 1.4 million reports and has never breached the identity of the people sharing their information about events or safety issues. From this volume of data, the ASRS has released over 6,000 aviation safety alerts concerning potential hazards and safety concerns. The ASRS processes these reports, evaluates the information, and provides selected de-identified report information through the online ASRS Database at http:asrs.arc.nasa.gov. The NASA ASRS is also a founding member of the International Confidential Aviation Safety Systems (ICASS) group which is a collection of other national aviation reporting systems throughout the world. The ASRS model has also been replicated for application to improving safety in railroad, medical, fire fighting, and other domains. This presentation will discuss confidential, voluntary, and non-punitive reporting systems and their advantages in providing information for safety improvements.

Connell, Linda J.↗

NASA NTRS2016

NASA Aviation Safety Reporting System (ASRS)

The NASA Aviation Safety Reporting System (ASRS) collects, analyzes, and distributes de-identified safety information provided through confidentially submitted reports from frontline aviation personnel. Since its inception in 1976, the ASRS has collected over 1.4 M reports and has never breached the identity of the people sharing their information about events or safety issues. From this volume of data, the ASRS has released over 6,000 aviation safety alerts concerning potential hazards and safety concerns. The ASRS processes these reports, evaluates the information, and provides selected de-identified report information through the online ASRS Database at http:asrs.arc.nasa.gov. The NASA ASRS is also a founding member of the International Confidential Aviation Safety Systems (ICASS) group which is a collection of other national aviation reporting systems throughout the world. The ASRS model has also been replicated for application to improving safety in railroad, medical, fire fighting, and other domains. This presentation will discuss confidential, voluntary, and non-punitive reporting systems and their advantages in providing information for safety improvements.

Connell, Linda↗

NASA NTRS2020

International VLBI Service for Geodesy and Astrometry 2017+2018 Biennial Report

Since its creation in 1999, the International VLBI Service for Geodesy and Astrometry (IVS) has documented its progress and current status in the form of annual or biennial reports. The first sixteen years were recorded in the form of annual reports, while starting with the years 2015+2016 the rhythm was changed to two years. Hence, the 2017+2018 Biennial Report constitutes the second installment of the two-yearly reporting scheme, documenting the work of the IVS components for the calendar years 2017 and 2018. As a testament to its usefulness, the general structure of the reporting has remained stable over the years: the individual components of the IVS contributed short reports describing their numerous activities, progress, and future plans. Without the continued input from the VLBI groups of the international geodetic and astrometric community, this publication could not be compiled and the IVS itself would not be able to flourish. So, once again many thanks to all IVS components who contributed to this Biennial Report.

Armstrong, Kyla L.↗

NASA NTRS2020

Monitoring Water-Related Ecosystems with Earth Observation Data in Support of Sustainable Development Goal (SDG) 6 Reporting

Lack of national data on water-related ecosystems is a major challenge to achieving the Sustainable Development Goal (SDG) 6 targets by 2030. Monitoring surface water extent, wetlands, and water quality from space can be an important asset for many countries in support of SDG 6reporting. We demonstrate the potential for Earth observation (EO) data to support country reporting for SDG Indicator 6.6.1, ‘Change in the extent of water-related ecosystems over time’ and identify important considerations for countries using these data for SDG reporting. The spatial extent of water-related ecosystems, and the partial quality of water within these ecosystems is investigated for seven countries. Data from the Moderate Resolution Imaging Spectroradiometer (MODIS) and Landsat 5, 7, and 8 with Shuttle Radar Topography Mission (SRTM) are used to measure surface water extent at 250 m and 30 m spatial resolution, respectively, in Cambodia, Jamaica, Peru, the Philippines, Senegal, Uganda, and Zambia. The extent of mangroves is mapped at 30 m spatial resolution using Landsat 8 Operational Land Imager (OLI), Sentinel-1, and SRTM data for Jamaica, Peru, and Senegal. Using Landsat 8 and Sentinel 2A imagery, total suspended solids and chlorophyll-a are mapped overtime for a select number of large surface water bodies in Peru, Senegal, and Zambia. All of the EO datasets used are of global coverage and publicly available at no cost. The temporal consistency and long time-series of many of the datasets enable replicability over time, making reporting of change from baseline values consistent and systematic. We find that statistical comparisons between different surface water data products can help provide some degree of confidence for countries during their validation process and highlight the need for accuracy assessments when using EO-based land change data for SDG reporting. We also raise concern that EO data in the context of SDG Indicator 6.6.1reporting may be more challenging for some countries, such as small island nations, than others to use in assessing the extent of water-related ecosystems due to scale limitations and climate variability. Country-driven validation of the EO data products remains a priority to ensure successful data integration in support of SDG Indicator 6.6.1 reporting. Multi-country studies such as this one can be valuable tools for helping to guide the evolution of SDG monitoring methodologies and provide a useful resource for countries reporting on water-related ecosystems. The EO data analyses and statistical methods used in this study can be easily replicated for country-driven validation of EO data products in the future.

Raha Hakimdavar↗

NASA NTRSDate not supplied

An Approach to Identifying Aspects of Positive Pilot Behavior within the Aviation Safety Reporting System

The National Airspace System (NAS) is constantly evolving as air traffic continues to ramp up to pre-pandemic numbers and projected to grow to unprecedented levels in the coming years. As well as increasing demand to the current system, emerging operations such as Unmanned Autonomous Systems are also expected to add to complexity in the airspace. To address these issues, the industry and government agencies supporting the NAS will need to rely upon additional automation and new technologies to address future operational requirements, while continuing to be a world-leading safe transportation system. As these new technologies are implemented, the system continues to rely on human pilots and controllers in the loop to monitor the system and intervene in situations the automation cannot handle. The goal of proactively addressing safety is of foremost concern to ensure passenger confidence. The industry has implemented various Safety Monitoring Systems to identify safety risks and proactively address them before they result in a serious incident or accident. One such program is the Aviation Safety Reporting System (ASRS). ASRS is a long-established system where pilots and controllers voluntarily and anonymously report safety incidents they experienced and observed during line operations by providing rich text narratives describing the events, the environment, and conditions leading to the safety event of concern. These narratives provide insight and context around events of interest and can be used to identify emerging problems. They can trigger investigations within Flight Operational Quality Assurance or Flight Data Monitoring programs. However, this process typically focuses on the adverse events and the unsafe aspects of the operations surrounding the reported or detected events. This perspective of investigating factors that went wrong around an adverse event is commonly referred to as Safety I. Alternatively, characterizing successful actions that operators perform every day under varying conditions that keep the system within safe operating bounds is a concept referred to as Safety II. The benefit of the Safety II view is that the scope is much larger than that of Safety I since a vast majority of the operations result in successful flights. Many of the successful techniques used to manage operational threats are not documented in standard operating procedures or taught during training. They are typically acquired over time by working with experienced pilots during line operations or in many cases after experiencing a problem for the first time and reacting to it in situ, drawing from years of experience to manage the threat. In an attempt to quantify these positive actions, we are proposing an approach to extracting key behaviors within ASRS reports that can support the Safety II concept. Our analysis assumes that ASRS reports contain some descriptions of corrective actions that operators performed to prevent a situation from leading to an accident. Leveraging recent advances in Natural Language Process modeling, we have developed an approach to extract positive sentiment from reports, embed these positive statements in a vector space where they can be numerically analyzed, and clustering these statements into similar contextual categories. From these contextualized categories we can attempt to summarized and distilled aspects of the positive behavior. The goal is to identify categories of behavior that describe consistent operator techniques that supports the Safety II concept. With this information, airlines may enable learning from these positive actions, or address procedures that need to be changed to avoid having pilots implement a workaround. These insights can provide a lens into what is “going right” in the operations that may otherwise not be known widely within the community. It is envisioned that this approach can be extended to other narrative programs such as Line Operation Safety Audit or Learning Improvement Team reports where similar observed behavior can be analyzed to extract positive actions and inform the overall operations.

NLP↗

NASA NTRSDate not supplied

An Approach to Identifying Aspects of Positive Pilot Behavior within the Aviation Safety Reporting System

The National Airspace System (NAS) is constantly evolving as air traffic continues to ramp up to pre-pandemic numbers and projected to grow to unprecedented levels in the coming years. As well as increasing demand to the current system, emerging operations such as Unmanned Autonomous Systems are also expected to add to complexity in the airspace. To address these issues, the industry and government agencies supporting the NAS will need to rely upon additional automation and new technologies to address future operational requirements, while continuing to be a world-leading safe transportation system. As these new technologies are implemented, the system continues to rely on human pilots and controllers in the loop to monitor the system and intervene in situations the automation cannot handle. The goal of proactively addressing safety is of foremost concern to ensure passenger confidence. The industry has implemented various Safety Monitoring Systems to identify safety risks and proactively address them before they result in a serious incident or accident. One such program is the Aviation Safety Reporting System (ASRS). ASRS is a long-established system where pilots and controllers voluntarily and anonymously report safety incidents they experienced and observed during line operations by providing rich text narratives describing the events, the environment, and conditions leading to the safety event of concern. These narratives provide insight and context around events of interest and can be used to identify emerging problems. They can trigger investigations within Flight Operational Quality Assurance or Flight Data Monitoring programs. However, this process typically focuses on the adverse events and the unsafe aspects of the operations surrounding the reported or detected events. This perspective of investigating factors that went wrong around an adverse event is commonly referred to as Safety I. Alternatively, characterizing successful actions that operators perform every day under varying conditions that keep the system within safe operating bounds is a concept referred to as Safety II. The benefit of the Safety II view is that the scope is much larger than that of Safety I since a vast majority of the operations result in successful flights. Many of the successful techniques used to manage operational threats are not documented in standard operating procedures or taught during training. They are typically acquired over time by working with experienced pilots during line operations or in many cases after experiencing a problem for the first time and reacting to it in situ, drawing from years of experience to manage the threat. In an attempt to quantify these positive actions, we are proposing an approach to extracting key behaviors within ASRS reports that can support the Safety II concept. Our analysis assumes that ASRS reports contain some descriptions of corrective actions that operators performed to prevent a situation from leading to an accident. Leveraging recent advances in Natural Language Process modeling, we have developed an approach to extract positive sentiment from reports, embed these positive statements in a vector space where they can be numerically analyzed, and clustering these statements into similar contextual categories. From these contextualized categories we can attempt to summarized and distilled aspects of the positive behavior. The goal is to identify categories of behavior that describe consistent operator techniques that supports the Safety II concept. With this information, airlines may enable learning from these positive actions, or address procedures that need to be changed to avoid having pilots implement a workaround. These insights can provide a lens into what is “going right” in the operations that may otherwise not be known widely within the community. It is envisioned that this approach can be extended to other narrative programs such as Line Operation Safety Audit or Learning Improvement Team reports where similar observed behavior can be analyzed to extract positive actions and inform the overall operations.

NLP↗

NASA NTRSDate not supplied

An Approach to Identifying Aspects of Positive Pilot Behavior within the Aviation Safety Reporting System

The National Airspace System (NAS) is constantly evolving as air traffic continues to ramp up to pre-pandemic numbers and projected to grow to unprecedented levels in the coming years. As well as increasing demand to the current system, emerging operations such as Unmanned Autonomous Systems are also expected to add to complexity in the airspace. To address these issues, the industry and government agencies supporting the NAS will need to rely upon additional automation and new technologies to address future operational requirements, while continuing to be a world-leading safe transportation system. As these new technologies are implemented, the system continues to rely on human pilots and controllers in the loop to monitor the system and intervene in situations the automation cannot handle. The goal of proactively addressing safety is of foremost concern to ensure passenger confidence. The industry has implemented various Safety Monitoring Systems to identify safety risks and proactively address them before they result in a serious incident or accident. One such program is the Aviation Safety Reporting System (ASRS). ASRS is a long-established system where pilots and controllers voluntarily and anonymously report safety incidents they experienced and observed during line operations by providing rich text narratives describing the events, the environment, and conditions leading to the safety event of concern. These narratives provide insight and context around events of interest and can be used to identify emerging problems. They can trigger investigations within Flight Operational Quality Assurance or Flight Data Monitoring programs. However, this process typically focuses on the adverse events and the unsafe aspects of the operations surrounding the reported or detected events. This perspective of investigating factors that went wrong around an adverse event is commonly referred to as Safety I. Alternatively, characterizing successful actions that operators perform every day under varying conditions that keep the system within safe operating bounds is a concept referred to as Safety II. The benefit of the Safety II view is that the scope is much larger than that of Safety I since a vast majority of the operations result in successful flights. Many of the successful techniques used to manage operational threats are not documented in standard operating procedures or taught during training. They are typically acquired over time by working with experienced pilots during line operations or in many cases after experiencing a problem for the first time and reacting to it in situ, drawing from years of experience to manage the threat. In an attempt to quantify these positive actions, we are proposing an approach to extracting key behaviors within ASRS reports that can support the Safety II concept. Our analysis assumes that ASRS reports contain some descriptions of corrective actions that operators performed to prevent a situation from leading to an accident. Leveraging recent advances in Natural Language Process modeling, we have developed an approach to extract positive sentiment from reports, embed these positive statements in a vector space where they can be numerically analyzed, and clustering these statements into similar contextual categories. From these contextualized categories we can attempt to summarized and distilled aspects of the positive behavior. The goal is to identify categories of behavior that describe consistent operator techniques that supports the Safety II concept. With this information, airlines may enable learning from these positive actions, or address procedures that need to be changed to avoid having pilots implement a workaround. These insights can provide a lens into what is “going right” in the operations that may otherwise not be known widely within the community. It is envisioned that this approach can be extended to other narrative programs such as Line Operation Safety Audit or Learning Improvement Team reports where similar observed behavior can be analyzed to extract positive actions and inform the overall operations.

NLP↗

NASA NTRS2023

International VLBI Service for Geodesy and Astrometry 2021+2022 Biennial Report

Since its creation in 1999, the International VLBI Service for Geodesy and Astrometry (IVS) has documented its progress and current status in the form of annual or biennial reports. The first sixteen years were recorded in the form of annual reports. Starting in 2015/2016 the cadence was changed to two years; hence, this Biennial Report marks eight years of using the two-yearly reporting scheme. The general structure of the reporting has remained stable over the years: the individual components of the IVS contributed short reports describing their numerous activities, progress, and future plans. Without the continued input from the VLBI groups of the international geodetic and astrometric community, this publication could not be compiled and the IVS itself would not be able to flourish. So, once again many thanks to all IVS components who contributed to this Biennial Report

Space Geodesy↗

DOE OSTI2023

Design and License Application Development for TRISO-X (Final Scientific Technical Report)

This is the final progress report submitted by X-energy, LLC (XE) to the Department of Energy in support of cooperative agreement DE-NE0008745. This report provides a high-level summary of the work performed during the entire period of performance, running from August 24, 2018 – August 22, 2022. This span of time covers the original 3-year award and a one year no-cost extension. There were five tasks within this project: (1) project management, (2) systems engineering and studies, (3) TRISO-X Facility design, (4) facility license application development, and (5) support to application review. Detailed reporting during execution of the project was provided by a total of 16 quarterly reports, voluntary monthly update presentations, and annual summary presentations. Technical work products include 103 X-energy technical reports, 38 subcontractor (Centrus technical reports), 75 Nuclear Criticality Safety Evaluations/Calculation reports, and 272 miscellaneous design documents (e.g., Engineering Service Orders, Engineering Component Specifications, Procedures, Guidelines/Policies, Design and Interface Requirements documents, and Drawings). All 29 of the X Energy milestones/deliverables were met early or on time and are archived in the DOE Office of Nuclear Energy’s Program Information Control System: Nuclear Energy under Fiscal Year 2018, Work Breakdown Structure F.OA – Industry FOA FY 2018 Awards, F.03 – X-Energy TRISO-X Project. All other work products are available to DOE upon request.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2024

Bridging the time scale in exascale computing of chemical systems (Final Technical Report)

This report summarizes the work carried out with support of the United States Department of Energy under Award DE-SC0019441. The theme of this project was to develop and apply methods that allowed for the acceleration of atomistic calculations, particularly in challenging areas such as multiphase systems, electrified interfaces, uncertainty estimation, and applications requiring chemical accuracy, which tend to be applications where simulation time is severely bottlenecked by the computational time requirements. Much of the focus was on the application of emerging machine-learning methodologies, although a wide range of methodologies were employed. This report has two major sections. The first focuses on the methodological advances themselves. Within this part, we report a number of major advances, a few examples of which are described here. We report the first machine-learning scheme for the acceleration of electronically grand-canonical calculations (that is, those applicable to electrochemistry). We report new methods of performing transfer learning, in which physics-based priors can be used to provide predictions, often with uncertainty estimates, of images well outside of training sets; we also offer ways to fine-tune these transfer-learning models. We provide a new systematic means to generate and apply minimal training data sets to very large (10,000’s of atoms) systems, with only small training sets appropriate for electronic structure. We developed new methodologies to integrate surface vibrations into surface adsorption calculations. We made advances to the applicability of diffusion Monte Carlo methods to allow (learned) force prediction, finite-size error correction, and force-free means of searching for transition states. We integrated machine-learned atomistic predictions into mechanism generation codes. Additionally, we released new software including AmpTorch, a modernized version of our original atomistic machine-learning code Amp. The second part of this report focuses on the scientific applications that accompanied, and were often enabled by, the methodological advances described earlier. A few examples follow, but full details are in the individual chapters of the report. For example, we developed a general theory of phonon-induced friction on molecular adsorbates. We showed fundamentally how solvent influences the adsorption and desorption process and how it differs from the processes typically involved at the solid–gas interface, making aqueous-phase and electrocatalysis different from traditional thermocatalysis. We examined how metal–insulator and magnetic transitions can be probed, and accelerated exciton dynamics via Frenkel Hamiltonian parameters. We showed that the nearsighted force-training approach, developed within this project, can predict both the stability and reactivity of large nanoparticles, and can also lead to insights on catalyst coverage on binding energies and entropies. These applied studies, which generally integrated with our method development, allowed us to push forward the theoretical understanding of several reaction classes.

08 HYDROGEN↗

DOE OSTI2025

Mechanical Solutions Scan Report

Power lines, poles, and towers are the backbone of the United States (U.S.) electric-power grid. These transmission and distribution networks route electricity from generator to loads. The characteristics of these routes are rapidly changing -- trending towards decentralized renewable generation, electric heating, vehicle charging, and large data-center loads. Coupled with aging infrastructure and the increased frequency of extreme weather events, there is concern about the future reliability and transmission capacity of conductors and adjacent components. This scan report seeks to provide an overview of mechanical solutions to challenges caused by extreme weather events associated with components of transmission and distribution infrastructure, including conductor heat sag, ice accumulation, wind, and wildfire. Many options could increase transmission capacity or reliability, and these are at various stages of technological readiness. Some have only been lab tested, while some have been widely deployed in the U.S. or overseas for decades. The solution categories and providers featured in this report are intended to be comprehensive at the time of publication and to serve as a reference for decision-makers concerned about transmission and distribution reliability. There are two other categories of large, complex solutions, which are not covered in this report: replacing existing conductors with advanced conductors and implementing digital grid enhancing technologies. A separate scan report titled “Advanced Conductor Scan Report,” which discusses advanced carbon-core conductors, was published by the Idaho National Laboratory (INL) in 2023. Information on digital technologies, such as dynamic line ratings, power-flow controllers, and other power electronics and communications-based devices, can be found on the Grid- Enhancing Technologies landing page. Mechanical grid-enhancing technologies, or solutions covered in this report, often do not require full equipment replacement and do not rely on digital components. Mechanical technologies are overlooked because they may be older, simpler, or seemingly “more obvious” than digital or carbon-core technologies. However, it is wise to consider mechanical solutions in a thorough evaluation of grid enhancing technology solutions.

24 - POWER TRANSMISSION AND DISTRIBUTION↗

NASA NTRSDate not supplied

A Survey Protocol to Assess Meaningfulness and Usefulness of Automated Topic Finding in the NASA Aviation Safety Reporting System

Context: The NASA Aviation Safety Reporting System (ASRS) is a voluntary confidential aviation safety reporting system. The ASRS receives reports from pilots, air traffic controllers, flight attendants and other involved in aviation operations. The reports are de-identified and coded by ASRS expert safety analysts and a short descriptive synopsis is written to describe the safety issue. The de-identified reports are then disseminated to the aviation community in a number of ways including entry into an online database, Safety Alert Bulletins and For Your Information Notices, and the CALLBACK newsletter. Key to these publications are the timely processing (de-identification, coding and summarization) of new reports, which is currently done by ASRS expert safety analysts. Thus, we believe topic modelling could decrease effort in ASRS, if topics are comprehensible. Aim: We propose a methodology to evaluate whether automated topic finding using topic modelling provides meaningful and useful topics. Method: We extend the total error survey methodology to evaluate user topic comprehension of machine learning outputs. To accomplish this we performed a literature review to identify existing methods and define a construct for topic comprehension, utilizing existing ASRS synopsis writing practices to more precisely define meaningfulness and usefulness. Results: A survey protocol was created that addresses the limitations of other survey protocols found in the literature review, which we found lacking in rationale and clear protocol definition. Conclusion: The surveying of user understanding in machine learning outputs presents challenges due to the explosion of parameters to control for and the lack of systematic approach presented in the literature. More reproducible work and survey protocols are needed in the literature and our work is one step towards that direction.

topic finding↗

DOE OSTI2024

Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations

One of the most promising applications of machine learning in computational physics is to accelerate the solution of partial differential equations (PDEs). The key objective of machine-learning-based PDE solvers is to output a sufficiently accurate solution faster than standard numerical methods, which are used as a baseline comparison. Here, we first perform a systematic review of the ML-for-PDE-solving literature. Out of all of the articles that report using ML to solve a fluid-related PDE and claim to outperform a standard numerical method, we determine that 79% (60/76) make a comparison with a weak baseline. Second, we find evidence that reporting biases are widespread, especially outcome reporting and publication biases. We conclude that ML-for-PDE-solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to under-reporting of negative results. To a large extent, these issues seem to be caused by factors similar to those of past reproducibility crises: researcher degrees of freedom and a bias towards positive results. We call for bottom-up cultural changes to minimize biased reporting as well as top-down structural reforms to reduce perverse incentives for doing so.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

Background: The importance of infectious disease epidemic forecasting and prediction research is underscored by decades of communicable disease outbreaks, including COVID-19. Unlike other fields of medical research, such as clinical trials and systematic reviews, no reporting guidelines exist for reporting epidemic forecasting and prediction research despite their utility. We therefore developed the EPIFORGE checklist, a guideline for standardized reporting of epidemic forecasting research. Methods and findings: We developed this checklist using a best-practice process for development of reporting guidelines, involving a Delphi process and broad consultation with an international panel of infectious disease modelers and model end users. The objectives of these guidelines are to improve the consistency, reproducibility, comparability, and quality of epidemic forecasting reporting. The guidelines are not designed to advise scientists on how to perform epidemic forecasting and prediction research, but rather to serve as a standard for reporting critical methodological details of such studies. Conclusions: These guidelines have been submitted to the EQUATOR network, in addition to hosting by other dedicated webpages to facilitate feedback and journal endorsement.

60 APPLIED LIFE SCIENCES↗