Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “automated verification”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

486 records · Page 27
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

Software Development and Testing Support for the Avionics Systems Telemetry Tool Suite

The Customer Avionics Interface Development and Analysis (CAIDA) team provides modeling and simulation software for the verification of the Launch Control System (LCS). In late 2015, CAIDA began development of the Data Exchange Message (DEM) Generator (DEMGen) and development of the DEM Modifier (DEMMod) in early 2017. DEMGen is able to mimic the telemetry streams normally sent by different simulators allowing increased user control and more telemetry instances. DEMMod takes in a telemetry stream and allows the modification of entire packets or simple measurement values. Together, these tools provide the capabilities to simulate and test complex launch scenarios to ensure LCS is fully prepared for any anomalous behavior during an actual launch. In early 2018, CAIDA began a new project called the CAIDA Advanced Telemetry Tool (CATT), which combined the code and documentation of DEMGen and DEMMod. CATT now allows the integration of new tools into a singular program providing compact and easy access. In the fall of 2018, the author worked with another intern, Antonio Negron, to develop several new CATT features along with proper documentation and unit testing. In the following spring, the author continued that work by integrating the features into CATT, solidifying the documentation, and expanding the testing to cover more cases. The outcome of both semesters includes an inspection engine for automated packet analysis, a packet-masking feature, and a multi-measurement sequencing feature.

Thomas, Taylor Walter↗
NASA NTRS2023

On-Board Deployment Event Verification for GOES-R Spacecraft

As is common with many spacecraft designs, the GOES-R vehicles require a series of deployment events to transition from the launch configuration to the operational configuration. Rather than implementing additional sensors to verify various deployments, the GOES-R program developed an alternate approach that uses existing gyro rate sensing. The approach includes two pieces: the first is a new onboard shock detection capability to confirm initiation of individual deployment events, and the second is a ground-based dynamics verification step to confirm completion of deployment events. We first present the algorithm that detects shock events for various deployment devices along with the parameter tuning performed during ground tests. We then show inflight performance of the shock detection algorithm. While shock detection is useful for observing initiation of deployment events, completion of some deployment events cannot be determined by shock detection alone. For these events, such as solar panel latch-up and deployable boom extension, the program developed dynamics models for the deployment transient responses. High-rate gyro data were recorded for these events, which allow the ground team to verify that the appendages were fully deployed. We show the predictive models for these events and corresponding flight results.

GeoXO↗
DOE OSTI2020

CO 2 Injection Monitoring with a Scalable, Automated Sparse Seismic Array

Conference presentation at American Institute of Chemical Engineers (AIChE) Annual Meeting, Virtual, November 16–20, 2020. The scalable, automated sparse seismic array (SASSA) method has been ideated as a cost-effective alternative or complementary to conventional 3D seismic methods. Based on a sparse array of receivers deployed strategically on Earth’s surface and one or more fixed seismic sources, the SASSA method can be used to study specific portions of the subsurface to detect the movement of CO 2 in a reservoir with lower environmental footprint and shorter turnaround time to deliver results than conventional 3D seismic methods. Energy & Environmental Research Center (EERC) carried out seismic monitoring of an oilfield undergoing CO 2 enhanced oil recovery in Montana, USA, using the SASSA method.

02 PETROLEUM↗
NASA NTRSDate not supplied

Recent Improvements and Verification of A Full Body Model in Opensim

BACKGROUND: The dynamic feasibility [1,2] criterion, that the subject’s Center of Pressure (COP) be located within the Base of Support (BOS) which outlines the feet, has aided in assessing the stability of human motion recorded on earth while performing the recorded tasks in lunar gravity or as countermeasures exercises on a vibration isolation and stabilization system in microgravity. The convex hull of the BOS on the platform under the subject’s feet was estimated using virtual markers on the feet of the scaled subject model. The COP was calculated using the ground reaction forces and moments determined from motion capture data with biomechanical modeling tools [3]. Occasionally, large-amplitude oscillatory spikes or “artifacts” were observed in the subject’s linear and angular momentum derivatives, affecting some COP data derived from motion capture. The purpose of this investigation was to assess and improve the accuracy of model scaling and BOS estimation as well as to determine the efficacy of model adjustments in mitigating artifacts influencing motion capture-derived ground reaction force and COP results. METHODS: To aid evaluation of proposed process and model updates, motion capture data were collected for two subjects during unit test and range of motion trials, lunar tasks, and countermeasures exercise motions. Markers were added to the full body Plug-in Gait marker set [4] during data collection. New markers were placed on the front, back, sides, and top of the head to improve scaling using distances between marker pairs. Medial elbow markers were added to stabilize the upper arm during OpenSim Inverse Kinematics (IK) [5]. Finally, markers were added on the outer edge of the heels and on the outside edges of the first and last toes on each foot. These additional foot markers were made available to test new automated foot scaling techniques and to calculate the error between the subject’s estimated and recorded BOS. The modified unscaled OpenSim Full Body Rajagopal Model [6,7] was adjusted using some previously investigated techniques [8] to mitigate rapid shifts in joint angles occurring during IK, as these were found to cause the spike artifacts observed in subsequent stages of analysis. Since OpenSim models use Euler angles and rotation sequences, the arm axes of rotation were adjusted, and the pelvis order of rotation was changed to minimize the likelihood of encountering “gimbal lock” during common human motion. The model clavicle, arm, elbow, wrist, pelvis, and ankle angle limits were adjusted to better accommodate the full human range of motion seen in exercise and lunar data. The shoulder joint center was calculated using a “pivoting” algorithm [9], and both shoulder joint center and upper arm markers were included during IK to provide additional shoulder stability on a case-by-case basis. The quality of IK results was assessed by three criteria: minimizing error between recorded motion capture markers and model markers, checking for reasonable rates of change in joint angles between fames (i.e., no IK artifacts), and ensuring the absence of spikes in the inertial forces and angular momentum derivatives calculated using a custom OpenSim plugin [10]. RESULTS: The additional markers placed on the subject during data collection allowed the head and feet to be scaled more accurately using distances between new marker pairs. Scaling with BOS markers placed on the subject and removing the limit on subtalar angle resulted in more accurate BOS determination. Unrealistically large changes in joint angles between frames could be reduced by including clavicle, sternum, and medial elbow markers during IK. In cases with large arm ranges of motion, results could be further improved by running IK using medial elbow and virtual shoulder joint center markers. Model adjustments significantly improved the IK results affecting COP calculation and increased the accuracy of BOS estimation.

C A Bell↗
DOE OSTI2025

Identifying synoptic controls on boundary layer thermodynamic and cloud properties in a regional forecast model

Although most of our understanding of boundary layer cloudiness is based on idealized, subtropical, barotropic marine environments, boundary layer clouds exist across a range of conditions. In this study, we use the Naval Research Laboratory's Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS) and an automated cold-front-relative analysis framework to explore the boundary layer structure associated with low clouds across a transect through the cold front of a midlatitude synoptic cyclone. The model credibly captures boundary layer structure in line with conceptual models of midlatitude cyclones and ground-based observations at Graciosa Island in the Azores. The warm sector is conditionally unstable, with clouds that are too shallow and with too little liquid water, compared to cloud property retrievals from satellite and surface-based instruments. The cold-frontal region exhibits convection associated with weak stability and ascent. Northwest of the cold front, the boundary layer is well mixed, deeper, and capped by a strong inversion maintained by large-scale subsidence. Simulated clouds in frontal and post-frontal regions are mostly too thick, with too much liquid water and too little cloud base drizzle. The post-frontal clouds are associated with grid-scale updrafts, which appear to be the model's attempt to represent mesoscale organization of cellular convection typically observed in the cold sector of midlatitude cyclones. The deep, well-mixed post-frontal boundary layers and cloudiness are maintained by strong surface fluxes, as in cold air outbreaks. Our analysis framework serves as a unique approach to model verification, and our results offer insights into differences in boundary layer cloud behavior between subtropical and synoptic cold-sector regimes.

Eissner, Jordan M. [Univ. of Kansas, Lawrence, KS ↗
NASA NTRS2019

Bridging the Gap Between Requirements and Model Analysis: Evaluation on Cyber-Physical Challenge Problems

We present a framework for introducing high-level requirement specifications in the automated analysis of dataflow models. By integrating the FRET requirements elicitation tool with the CoCoSim analyzer, our framework enables the analysis of hierarchical Simulink models against requirements written in a restricted English language. More precisely, we support: automatic extraction of Simulink model information and association of high-level requirements with target model signals and components; translation of temporal logic formulas into synchronous dataflow CoCoSpec specifications as well as Simulink monitors; and interpretation of counterexamples produced by the analysis both at the requirement and model level. The features provided by our framework are generic and can be used to integrate other requirements elicitation and Simulink/Lustre analysis tools. We report on the lessons learned from the application of our approach to the Lockheed Martin Cyber-Physical, aerospace-inspired challenge problems. For the analysis, we used the Kind2, Zustre, and Simulink Design Verifier (SLDV) tools.

Simulink↗
NASA NTRS2019

Advancing Aircraft Operations in a Net-Centric Environment with the Incorporation of Increasingly Autonomous Systems and Human Teaming

NextGen has begun the modernization of the nation’s air transportation system, with goals to improve system safety, increase operation efficiency and capacity, provide enhanced predictability, resilience and robustness. With these improvements, NextGen is poised to handle significant increases in air traffic operations, more than twice the number recorded in 2016, by 2025.1 NextGen is evolving toward collaborative decision-making across many agents, including automation, by use of a Net-Centric architecture, which in itself creates a very complex environment in which the navigation and operation of aircraft are to take place. An intricate environment such as this, coupled with the expected upsurge of air traffic operations generates concern respecting the ability of the human-agent to both fly and manage aircraft within. Therefore, it is both necessary and practical to begin the process of increasingly autonomous systems within the cockpit that will act independently to assist the human-agent achieve the overall goal of NextGen. However, the straightforward technological development and implementation of intelligent machines into the cockpit is only part of what is necessary to maintain, at minimum, or improve human-agent functionality, as desired, while operating in NextGen. The full integration of Increasingly Autonomous Systems (IAS) within the cockpit can only be accomplished when the IAS works in concert with the human, formulating trust between the two, thereby establishing a team atmosphere. Imperative to cockpit implementation is ensuring the proper performance of the IAS by the development team and the human-agent with which it will be paired when given a specific piloting, navigation, or observational task. Described in this paper are the steps taken, at NASA Langley Research Center, during the second and third phases of the development of an IAS, the Traffic Data Manager (TDM), its verification and validation by human-agents, and the foundational development of Human Autonomy Teaming (HAT) between the two.

Houston, Vincent E.↗
NASA NTRS2022

Flight Rule Design, Implementation, Verification, and Validation for the Psyche Mission

NASA Jet Propulsion Lab (JPL)’s upcoming mission Psyche will begin its journey to the asteroid (16) Psyche in late 2022 in an effort to better understand its origins and, in turn, better understand our own. Operating the spacecraft safely will require the dedicated efforts of a small team that understands the spacecraft’s operational constraints, as well as a set of powerful spacecraft models designed to catch command errors that can pose risks to mission success. One of the responsibilities of the operations team is to ensure adherence to a set of Flight Rules written by spacecraft and instrument experts that are designed to mitigate these risks. Psyche’s innovations in Flight Rule design principles and advancements in the tools and processes used to implement and check Flight Rules are discussed. A comparison of Psyche’s approach to Flight Rules to other JPL missions will provide lessons learned for future missions that must perform constraint checking during operations. Flight Rule development faces several major challenges. First, flight rule developers must work with Subject Matter Experts (SME) to write the rules in a way that captures the intent of the constraint in a straightforward, enforceable manner. Second, software implementers must correctly interpret flight rules into code so that it meets the original intent of the SME. Finally, a means must be provided for SMEs to validate flight rule implementations without requiring them to understand the underlying software. Innovative processes intended to efficiently close the loop between stakeholders and software developers are described, such as the use of test-driven development to provide stakeholders with easy-to-review implementations. New guidelines for flight rule writing, designed to address these challenges, are described for future missions to adopt and build upon. Psyche Mission System has a variety of new and heritage tools that improve in the Flight Rule validation and checking process. Psyche developed a powerful, new tool called RandSEQ and made significant improvements to Octopusjam, two valuable tools that aid the development of Flight Rule unit tests. Advancements in the models and processes for performing sequence validation with SEQuence GENerator (SEQGEN), the primary, high-heritage tool used for automated flight rule checks on Psyche, are described. The development of new software and the advancements to existing software put Psyche at the forefront of Flight Rule technology. All missions must perform detailed constraint checking, so a comparison of Psyche’s approach to some of these items to the approaches taken by other missions such as Dawn, M2020, and Europa Clipper is done, specifically to examine SME-developer communication, tools used, and development process. Lessons learned from this comparison will be provided.

Weise, Tim↗
NASA NTRS2006

Proceedings of the 2004 High Spatial Resolution Commercial Imagery Workshop

Topics covered include: NASA Applied Sciences Program; USGS Land Remote Sensing: Overview; QuickBird System Status and Product Overview; ORBIMAGE Overview; IKONOS 2004 Calibration and Validation Status; OrbView-3 Spatial Characterization; On-Orbit Modulation Transfer Function (MTF) Measurement of QuickBird; Spatial Resolution Characterization for QuickBird Image Products 2003-2004 Season; Image Quality Evaluation of QuickBird Super Resolution and Revisit of IKONOS: Civil and Commercial Application Project (CCAP); On-Orbit System MTF Measurement; QuickBird Post Launch Geopositional Characterization Update; OrbView-3 Geometric Calibration and Geopositional Accuracy; Geopositional Statistical Methods; QuickBird and OrbView-3 Geopositional Accuracy Assessment; Initial On-Orbit Spatial Resolution Characterization of OrbView-3 Panchromatic Images; Laboratory Measurement of Bidirectional Reflectance of Radiometric Tarps; Stennis Space Center Verification and Validation Capabilities; Joint Agency Commercial Imagery Evaluation (JACIE) Team; Adjacency Effects in High Resolution Imagery; Effect of Pulse Width vs. GSD on MTF Estimation; Camera and Sensor Calibration at the USGS; QuickBird Geometric Verification; Comparison of MODTRAN to Heritage-based Results in Vicarious Calibration at University of Arizona; Using Remotely Sensed Imagery to Determine Impervious Surface in Sioux Falls, South Dakota; Estimating Sub-Pixel Proportions of Sagebrush with a Regression Tree; How Do YOU Use the National Land Cover Dataset?; The National Map Hazards Data Distribution System; Recording a Troubled World; What Does This-Have to Do with This?; When Can a Picture Save a Thousand Homes?; InSAR Studies of Alaska Volcanoes; Earth Observing-1 (EO-1) Data Products; Improving Access to the USGS Aerial Film Collections: High Resolution Scanners; Improving Access to the USGS Aerial Film Collections: Phoenix Digitizing System Product Distribution; System and Product Characterization: Issues Approach; Innovative Approaches to Analysis of Lidar Data for the National Map; Changes in Imperviousness near Military Installations; Geopositional Accuracy Evaluations of QuickBird and OrbView-3: Civil and Commercial Applications Project (CCAP); Geometric Accuracy Assessment: OrbView ORTHO Products; QuickBird Radiometric Calibration Update; OrbView-3 Radiometric Calibration; QuickBird Radiometric Characterization; NASA Radiometric Characterization; Establishing and Verifying the Traceability of Remote-Sensing Measurements to International Standards; QuickBird Applications; Airport Mapping and Perpetual Monitoring Using IKONOS; OrbView-3 Relative Accuracy Results and Impacts on Exploitation and Accuracy Improvement; Using Remotely Sensed Imagery to Determine Impervious Surface in Sioux Falls, South Dakota; Applying High-Resolution Satellite Imagery and Remotely Sensed Data to Local Government Applications: Sioux Falls, South Dakota; Automatic Co-Registration of QuickBird Data for Change Detection Applications; Developing Coastal Surface Roughness Maps Using ASTER and QuickBird Data Sources; Automated, Near-Real Time Cloud and Cloud Shadow Detection in High Resolution VNIR Imagery; Science Applications of High Resolution Imagery at the USGS EROS Data Center; Draft Plan for Characterizing Commercial Data Products in Support of Earth Science Research; Atmospheric Correction Prototype Algorithm for High Spatial Resolution Multispectral Earth Observing Imaging Systems; Determining Regional Arctic Tundra Carbon Exchange: A Bottom-Up Approach; Using IKONOS Imagery to Assess Impervious Surface Area, Riparian Buffers and Stream Health in the Mid-Atlantic Region; Commercial Remote Sensing Space Policy Civil Implementation Update; USGS Commercial Remote Sensing Data Contracts (CRSDC); and Commercial Remote Sensing Space Policy (CRSSP): Civil Near-Term Requirements Collection Update.

Source record↗
NASA NTRS2006

Reentry-Vehicle Shape Optimization Using a Cartesian Adjoint Method and CAD Geometry

A DJOINT solutions of the governing flow equations are becoming increasingly important for the development of efficient analysis and optimization algorithms. A well-known use of the adjoint method is gradient-based shape. Given an objective function that defines some measure of performance, such as the lift and drag functionals, its gradient is computed at a cost that is essentially independent of the number of design variables (e.g., geometric parameters that control the shape). Classic aerodynamic applications of gradient-based optimization include the design of cruise configurations for transonic and supersonic flow, as well as the design of high-lift systems. are perhaps the most promising approach for addressing the issues of flow solution automation for aerodynamic design problems. In these methods, the discretization of the wetted surface is decoupled from that of the volume mesh. This not only enables fast and robust mesh generation for geometry of arbitrary complexity, but also facilitates access to geometry modeling and manipulation using parametric computer-aided design (CAD). In previous work on Cartesian adjoint solvers, Melvin et al. developed an adjoint formulation for the TRANAIR code, which is based on the full-potential equation with viscous corrections. More recently, Dadone and Grossman presented an adjoint formulation for the two-dimensional Euler equations using a ghost-cell method to enforce the wall boundary conditions. In Refs. 18 and 19, we presented an accurate and efficient algorithm for the solution of the adjoint Euler equations discretized on Cartesian meshes with embedded, cut-cell boundaries. Novel aspects of the algorithm were the computation of surface shape sensitivities for triangulations based on parametric-CAD models and the linearization of the coupling between the surface triangulation and the cut-cells. The accuracy of the gradient computation was verified using several three-dimensional test cases, which included design variables such as the free stream parameters and the planform shape of an isolated wing. The objective of the present work is to extend our adjoint formulation to problems involving general shape changes. Factors under consideration include the computation of mesh sensitivities that provide a reliable approximation of the objective function gradient, as well as the computation of surface shape sensitivities based on a direct-CAD interface. We present detailed gradient verification studies and then focus on a shape optimization problem for an Apollo-like reentry vehicle. The goal of the optimization is to enhance the lift-to-drag ratio of the capsule by modifying the shape of its heat-shield in conjunction with a center-of-gravity (c.g.) offset. This multipoint and multi-objective optimization problem is used to demonstrate the overall effectiveness of the Cartesian adjoint method for addressing the issues of complex aerodynamic design.

Nemec, Marian↗
DOE OSTI2026

Locating Undocumented Wells Using Historical Oil and Gas Exploration Maps: A Case Study in Osage County, Oklahoma

Undocumented oil and gas wells lack reliable information about their locations and characteristics, making them difficult to identify. These wells can result in unanticipated delays and costs in the development of nearby surface and subsurface resources, and, if improperly plugged, can cause contamination. This study leverages historical petroleum exploration maps to locate such wells, focusing on Osage County, Oklahoma. Two sets of early 20th century oil and gas exploration maps by the United States Geological Survey were georeferenced and analyzed using a computer vision model to detect well symbols. The locations of detected wells were compared to the location of known wells in the database from the Bureau of Indian Affairs Osage Agency to identify potential undocumented wells. The analysis yielded over 500 potential undocumented wells, with dry holes constituting the largest fraction. Field verification confirmed the presence of some undocumented wells. Comparison with prior work revealed limited overlap, underscoring the complementary value of historical oil and gas maps for locating undocumented wells. This approach demonstrates the utility of integrating historical cartographic resources with modern geospatial and machine learning techniques to improve the identification and management of undocumented wells.

Energy - Petroleum↗
NASA NTRS2022

Flight Software Dictionary Development for the Mars2020 Rover

The Mars2020 project, developed and operated by the Jet Propulsion Laboratory (JPL), successfully landed the Perseverance rover and its flying companion Ingenuity on the surface of Mars on February 18th 2021. Perseverance combines heritage and cutting-edge flight software and hardware to accomplish crucial mission requirements related to Martian surface sampling. The design, development, and operation of NASA’s large strategic science missions require the ability to communicate spacecraft capabilities to hundreds of engineers across multiple disciplines. The interaction between flight and ground software development, Verification and Validation (V&V), Assembly, Test, and Launch Operations (ATLO), and management each demand quick understanding of unique slices of information for each discipline. This information includes the current capabilities of the flight system as well as future capabilities and their status as they are developed and tested. Despite the fundamental and critical nature of this information, the flight software dictionaries used to track it are a stumbling block for many projects. These dictionaries provide the cornerstone for the interpretation of data sent from the spacecraft, allowing for quick comprehension by engineers on the ground. During both spacecraft development and operations, flight software dictionary management includes significant challenges due to the large number of interfacing systems and the subtle yet distinct needs of each.The engineering of flight software dictionaries for Mars2020 had numerous challenges, most-notably: parallel dictionary development to support simultaneous separate flight software build campaigns for each mission phase (cruise and surface), managing requests for operations-enabling information without perturbing the heritage interface with the rover, and the introduction of new tools by the dictionary stakeholders that forced the dictionary team to innovate and redesign the heritage tool chain. These challenges generated guiding principles for the dictionary development effort: emphasize coding best practices and unit testing in the dictionary code development tool chain, use institutionally provided COTS (commercial-off-the-shelf) tools whenever possible, and maintain the heritage flight-ground interface all while advancing operations-enabling information via a loosely coupled interface.Throughout development and operations, the Mars2020 dictionary toolchain included IBM DOORS Next Generation, GitHub, Microsoft Excel, Docker, Jenkins, and a significant custom-built Python codebase. Significant interfaces included JPL’s command and control software, heritage flight software team tools and processes, and the many cloud-based ground tools developed for the mission.This paper will discuss the requirements for the Mars2020 dictionary development, the development team’s response to those requirements, lessons learned throughout the process, steps taken towards automated deliveries and continuous integration of stakeholder inputs, potential toolchain improvements for Mars2020, and key takeaways that could be applied to future missions.

Pyrzak, Guy↗
NASA NTRSDate not supplied

NASA's Moon to Mars Autonomous Habitat Status

NASA is developing a strategy for sending humans to the Mars vicinity, known broadly as the Moon to Mars (M2M) Campaign. A critical part of this campaign is the development of in-space and surface habitation systems capable of substantially extending human presence beyond Low Earth Orbit (LEO). Mars missions feature an in-space transit habitat capable of supporting crews of four on ~850-1200-day missions, including transit to and from Mars and time in Mars orbit. Surface and transit habitats are complex elements which must keep crewmembers healthy and productive in deep-space environments with limited resources, long rescue times in contingency situations, and communication delays; all within constrained mass, volume, and power budgets. These habitats provide crew both living and workspace as well as most of the resources needed to support crew life. For deep space habitats, automation needs to be employed due to latency and for significant amounts of time when the habitats are uncrewed. Automation of systems is possible in space applications, but there are limitations. Outside of the Earth’s (or any) magnetosphere, radiation environments are harsh to both the physical hardware and the software components. Radiation (charged particles and ionizing electromagnetic waves) degrades and damages the hardware and causes single event upsets (SEUs) in software. If the hardware is damaged, data can be lost, or control actions not made. For software, SEUs cause algorithms to result in different solutions, or incorrect commands to be sent out. This means that algorithms and hardware used for deep space systems are different than what is used on Earth. Radiation-tolerant hardware is generations behind the current state-of-the-art hardware. Recent NASA missions, such as James Webb Space Telescope, continue to rely on older technologies such as the RAD750 processor, and the most advanced processors are still single core and less than 1.5 GHz. There have been attempts to use higher performance processors, but these often take multiple mitigation steps to handle the radiation environments, which limits the processing power and/or throughput. Current techniques for radiation mitigation have been redundancies, voting, physical separation of hardware, encasing materials, under-clocking hardware, and more. Some radiation mitigation techniques do provide benefits such as having a redundant system to improve the probability that a system will be available when needed. Autonomous software systems will have fewer interactions with humans on deep space missions and therefore need to be able to handle more off-nominal conditions. Microgravity also complicates the autonomous aspects of the mission because autonomous systems are usually built from known deterministic states, but microgravity causes physical objects to shift and move changing the location an autonomous system placed the object. Not only does the software need to be reliable and deterministic, losing resources due to a software error is not only costly but detrimental to reputation. The combination of having lower performance hardware and having to be able to verify and deterministically run software and an ever-changing environment makes deep space autonomous systems more complicated. Multiple gaps have been identified including verification of autonomous software algorithms (including artificial intelligence and machine learning), higher performance processors (graphics and general purpose), high speed networks (onboard and transmissions), memory, power distribution, data security, and variations from these. These gaps need to be closed for more advanced systems to be deployed and reduce the size, weight, and power impacts on the habitats.

Scott B. Tashakkor↗
DOE OSTI2022

Verification and Validation of the Stakeholder Tool for Assessing Radioactive Transportation (START) – 22323

The U.S. Department of Energy (DOE) Office of Integrated Waste Management is planning for the eventual transportation, storage, and disposal of spent nuclear fuel (SNF) and high-level radioactive waste (HLW) from nuclear power plant and DOE sites. The Stakeholder Tool for Assessing Radioactive Transportation (START) is a web-based, geospatial decision-support tool developed for evaluating routing options and other aspects of transporting SNF and HLW, covering rail, truck, barge, and intermodal infrastructure and operations in the continental United States. The verification and validation (V&V) process is intended to independently assess START to provide confidence in the ability of START to accurately provide intended results. The V&V process checks the START tool using a variety of methods, ranging from independent hand calculations to comparison of START performance and results to those of other codes. The V&V activity was conducted independently from the START development team with opportunities to provide feedback and collaborate throughout the process. The V&V analyzed attributes of transportation routes produced by START, including route distance and both population and population density captured within buffer zones around routes. Population in the buffer zone, population density in the buffer zone, and route distance were all identified as crucial outputs of the START code and were subject to V&V tasks. Some of the improvements identified through the V&V process were standardizing the underlying population data in START, changing the projection of the population raster data, and changes to the methodology used for population density to improve its applicability for expected users. This collaboration also led to suggested improvements to some of the underlying shape file segments within START. A formal series of test routes went through the V&V process from START version 3.2.1 following the implementation of the recommended improvements to START. The V&V of the buffer zone population and distance reported in START were achieved by utilizing a custom workflow developed in QGIS. This platform was chosen because it is fully independent of START. The buffer zone population and route distances in START showed excellent agreement to independent V&V test results. Over 200 route test cases were run; in all cases, the percent difference in the population within the buffer zone was less than +/- 5%, and the majority of cases were below +/- 1%. These cases were also duplicated in ArcMap, which is Environmental Systems Research Institute’s (ESRI) GIS desktop application. The ArcMap results also show good agreement; however, they cannot be considered independent of START because START also uses ESRI software. The V&V tasks are currently addressing population density and estimated radiation dose in START as well as implementing further automation of the V&V process to allow larger suites of test routes to be run more easily and quickly by less experienced GIS users. These improvements to the V&V process will make it a valuable tool for the development team and the stakeholders so that each new iteration of START can easily be checked for quality and consistency. This paper and presentation will describe the V&V process, findings and implications, and ongoing V&V activities.

START, DOE, Transporation, dose, GIS↗
NASA NTRS2015

Multipurpose Pressure Vessel Scanner and Photon Doppler Velocimetry

Critical flight hardware typically undergoes a series of nondestructive evaluation methods to screen for defects before it is integrated into the flight system. Conventionally, pressure vessels have been inspected for flaws using a technique known as fluorescent dye penetrant, which is biased to inspector interpretation. An alternate method known as eddy current is automated and can detect small cracks better than dye penetrant. A new multipurpose pressure vessel scanner has been developed to perform internal and external eddy current scanning, laser profilometry, and thickness mapping on pressure vessels. Before this system can be implemented throughout industry, a probability of detection (POD) study needs to be performed to validate the system’s eddy current crack/flaw capabilities. The POD sample set will consist of 6 flight-like metal pressure vessel liners with defects of known size. Preparation for the POD includes sample set fabrication, system operation, procedure development, and eddy current settings optimization. For this, collaborating with subject matter experts was required. This technical paper details the preparation activities leading up to the POD study currently scheduled for winter 2015/2016. Once validated, this system will be a proven innovation for increasing the safety and reliability of necessary flight hardware.Additionally, testing of frangible joint requires Photon Doppler Velocimetry (PDV) and Digital Image Correlation instrumentation. There is often noise associated with PDV data, which necessitates a frequency modulation (FM) signal-to-noise pre-test. Generally, FM radio works by varying the carrier frequency and mixing it with a fixed frequency source, creating a beat frequency which is represented by audio frequency that can be heard between about 20 to 20,000 Hz. Similarly, PDV reflects a shifted frequency (a phenomenon known as the Doppler Effect) from a moving source and mixes it with a fixed source frequency, which results in a beat frequency. However, for PDV, discerning the signal from the noise is difficult without a moving source to induce the modulation. A rotating wheel is currently being used as the moving source but its configuration is impractical and has cumbersome placement inside the current frangible joint test cell. As a way to combat this problem and verify a satisfactory signal-to-noise ratio, a reflective moving crystal piezo will be used to modulate a beat frequency, and an absorptive target will be used to block the signal in order to determine any back reflection coming from the probe and discern the true signal-to noise ratio. The piezo will be mounted and inserted onto the test table on an extendable telescopic antenna grounded by a magnetic base in the test zone. This piezo configuration will be more compatible within the test zone and allow for easy removal of the disk following acceptable signal verification and prior to frangible joint tests.Additionally, topics of what was learned and smaller tasks given at White Sands Test Facility (WSTF) will be discussed. All statements in this paper are newly gained knowledge of what I have learned, observed, and have done while at WSTF.

Ellis, Tayera↗
NASA NTRS2015

Multipurpose Pressure Vessel Scanner and Photon Doppler Velocimetry

Critical flight hardware typically undergoes a series of nondestructive evaluation methods to screen for defects before it is integrated into the flight system. Conventionally, pressure vessels have been inspected for flaws using a technique known as fluorescent dye penetrant, which is biased to inspector interpretation. An alternate method known as eddy current is automated and can detect small cracks better than dye penetrant. A new multipurpose pressure vessel scanner has been developed to perform internal and external eddy current scanning, laser profilometry, and thickness mapping on pressure vessels. Before this system can be implemented throughout industry, a probability of detection (POD) study needs to be performed to validate the system's eddy current crack/flaw capabilities. The POD sample set will consist of 6 flight-like metal pressure vessel liners with defects of known size. Preparation for the POD includes sample set fabrication, system operation, procedure development, and eddy current settings optimization. For this, collaborating with subject matter experts was required. This technical paper details the preparation activities leading up to the POD study currently scheduled for winter 2015/2016. Once validated, this system will be a proven innovation for increasing the safety and reliability of necessary flight hardware. Additionally, testing of frangible joint requires Photon Doppler Velocimetry (PDV) and Digital Image Correlation instrumentation. There is often noise associated with PDV data, which necessitates a frequency modulation (FM) signal-to-noise pre-test. Generally, FM radio works by varying the carrier frequency and mixing it with a fixed frequency source, creating a beat frequency which is represented by audio frequency that can be heard between about 20 to 20,000 Hz. Similarly, PDV reflects a shifted frequency (a phenomenon known as the Doppler Effect) from a moving source and mixes it with a fixed source frequency, which results in a beat frequency. However, for PDV, discerning the signal from the noise is difficult without a moving source to induce the modulation. A rotating wheel is currently being used as the moving source but its configuration is impractical and has cumbersome placement inside the current frangible joint test cell. As a way to combat this problem and verify a satisfactory signal-to-noise ratio, a reflective moving crystal piezo will be used to modulate a beat frequency, and an absorptive target will be used to block the signal in order to determine any back reflection coming from the probe and discern the true signal-to-noise ratio. The piezo will be mounted and inserted onto the test table on an extendable telescopic antenna grounded by a magnetic base in the test zone. This piezo configuration will be more compatible within the test zone and allow for easy removal of the disk following acceptable signal verification and prior to frangible joint tests.

Ellis, Tayera↗
NASA NTRS2008

Certifying Auto-Generated Flight Code

Model-based design and automated code generation are being used increasingly at NASA. Many NASA projects now use MathWorks Simulink and Real-Time Workshop for at least some of their modeling and code development. However, there are substantial obstacles to more widespread adoption of code generators in safety-critical domains. Since code generators are typically not qualified, there is no guarantee that their output is correct, and consequently the generated code still needs to be fully tested and certified. Moreover, the regeneration of code can require complete recertification, which offsets many of the advantages of using a generator. Indeed, manual review of autocode can be more challenging than for hand-written code. Since the direct V&V of code generators is too laborious and complicated due to their complex (and often proprietary) nature, we have developed a generator plug-in to support the certification of the auto-generated code. Specifically, the AutoCert tool supports certification by formally verifying that the generated code is free of different safety violations, by constructing an independently verifiable certificate, and by explaining its analysis in a textual form suitable for code reviews. The generated documentation also contains substantial tracing information, allowing users to trace between model, code, documentation, and V&V artifacts. This enables missions to obtain assurance about the safety and reliability of the code without excessive manual V&V effort and, as a consequence, eases the acceptance of code generators in safety-critical contexts. The generation of explicit certificates and textual reports is particularly well-suited to supporting independent V&V. The primary contribution of this approach is the combination of human-friendly documentation with formal analysis. The key technical idea is to exploit the idiomatic nature of auto-generated code in order to automatically infer logical annotations. The annotation inference algorithm itself is generic, and parametrized with respect to a library of coding patterns that depend on the safety policies and the code generator. The patterns characterize the notions of definitions and uses that are specific to the given safety property. For example, for initialization safety, definitions correspond to variable initializations while uses are statements which read a variable, whereas for array bounds safety, definitions are the array declarations, while uses are statements which access an array variable. The inferred annotations are thus highly dependent on the actual program and the properties being proven. The annotations, themselves, need not be trusted, but are crucial to obtain the automatic formal verification of the safety properties without requiring access to the internals of the code generator. The approach has been applied to both in-house and commercial code generators, but is independent of the particular generator used. It is currently being adapted to flight code generated using MathWorks Real-Time Workshop, an automatic code generator that translates from Simulink/Stateflow models into embedded C code.

Denney, Ewen↗
DOE OSTI2023

The Uniform Methods Project: Smart Thermostat Evaluation Protocol

A smart thermostat is an internet-connected device that controls home heating, ventilation, and air-conditioning (HVAC) equipment and can automatically adjust temperature set points to optimize performance and achieve energy savings. Smart thermostat features often include two way communication, occupancy detection (such as geofencing and occupancy sensors), schedule learning, and seasonal optimization algorithms. Smart thermostats can control most conventional HVAC systems, including central air conditioners, heat pumps, and forced air furnaces. Several types of residential utility programs offer smart thermostats as replacements measures. Working with smart thermostat vendors, utilities can offer separate optimization programs to produce energy savings beyond those achieved by installing a smart thermostat. From an evaluation perspective, smart thermostat programs have several noteworthy features. First, the energy savings from a smart thermostat may change over the life of the device. As a smart thermostat is connected to the internet, original equipment manufacturers can update the thermostat software to improve the thermostat's energy efficiency. Likewise, users can adjust the thermostat settings and schedules over time in response to changes in weather, thermal comfort, energy prices, or preferences for energy efficiency. Additionally, many thermostat manufacturers offer seasonal optimization programs that recommend changes or make minor, automated adjustments to the thermostat settings to improve energy efficiency. These opt-in programs are now standard offerings for many smart thermostat manufacturers and provided at no additional cost to users. The potential for software updates and continuous optimization and the evolving nature of user interactions mean future energy savings may differ from first-year savings and the energy savings of smart thermostats may need to be evaluated more than once. Second, smart thermostats often have small unit energy savings relative to a home's total energy consumption, especially in comparison to whole- home retrofit programs. This can make it difficult to detect the smart thermostat savings in billing or advanced metering infrastructure (AMI) meter consumption data. For example, as cooling loads in many regions average about 20% of annual electricity consumption, smart thermostat savings of 10% of cooling energy use would equate to a 2% reduction in home electricity consumption. Evaluators should use regression analysis of whole-home billing consumption or advanced metering infrastructure (AMI) meter consumption data to evaluate smart thermostat savings because, as explained at greater length below , these data are usually available to evaluators and regression can control for the impacts of weather and other potentially confounding factors on a home's energy consumption. Finally, as with other energy efficiency programs, participation in smart thermostat programs is self-selective. As discussed at greater length below , smart thermostat participants tend to be, among other things, younger, higher-income, and more likely to adopt electric vehicles (EVs) and internet connected devices than nonparticipants. These differences are often unobservable to the evaluator and correlated with a home's energy consumption, creating the potential for bias in estimating savings. Due to the small unit savings of thermostats, errors and biases from self-selection that may not be very consequential when evaluating a whole- home retrofits (e.g., ±2% of home electricity consumption) can have a major impact when evaluating the savings and cost-effectiveness of smart thermostat programs. A percentage point change in the estimated savings could affect the cost-effectiveness of a program. This means it is important for evaluators to assess and to minimize the potential for error from selection bias in estimating smart thermostat program savings. The Uniform Methods Project provides model protocols for determining energy savings and demand reductions that result from specific energy efficiency measures implemented through state and utility programs. In most cases, the measure protocols are based on a particular option identified by the International Performance Verification and Measurement Protocol ; however, this work provides a more detailed approach to implementing that option. Each chapter is written by technical experts in collaboration with their peers, reviewed by industry experts, and subject to public review and comment. The UMP protocols can be used by utilities, program administrators, public utility commissions, evaluators, and other stakeholders for both program planning and evaluation.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗