Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data cleaning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Compression of Solar Spectroscopic Observations: a Case Study of MgII k Spectral Line Profiles Observed by NASA’s IRIS Satellite

In this study we extract the deep features and investigate the compression of the MgII k spectral line profiles observed in quiet Sun regions by NASA’s IRIS satellite. The data set of line profiles used for the analysis was obtained on April 20th, 2020, at the center of the solar disc, and contains almost 300,000 individual MgII k line profiles after data cleaning. The data are separated into train and test subsets. The train subset was used to train the autoencoder of the varying embedding layer size. The early stopping criterion was implemented on the test subset to prevent the model from overfitting. Our results indicate that it is possible to compress the spectral line profiles more than 27 times (which corresponds to the reduction of the data dimensionality from 110 to 4) while having a 4DN average reconstruction error, which is comparable to the variations in the line continuum. The mean squared error and the reconstruction error of even statistical moments sharply decrease when the dimensionality of the embedding layer increases from 1 to 4 and almost stop decreasing for higher numbers. The observed occasional improvements in training for values higher than 4 indicate that a better compact embedding may potentially be obtained if other training strategies and longer training times are used. The features learned for the critical four-dimensional case can be interpreted. In particular, three of these four features mainly control the line width, line asymmetry, and line dip formation respectively. The presented results are the first attempt to obtain a compact embedding for spectroscopic line profiles and confirm the value of this approach, in particular for feature extraction, data compression, and denoising.

SMD↗
NASA NTRS2000

DMSP SSJ4 Data Restoration, Classification, and On-Line Data Access

Compress and clean raw data file for permanent storage We have identified various error conditions/types and developed algorithms to get rid of these errors/noises, including the more complicated noise in the newer data sets. (status = 100% complete). Internet access of compacted raw data. It is now possible to access the raw data via our web site, http://www.jhuapl.edu/Aurora/index.html. The software to read and plot the compacted raw data is also available from the same web site. The users can now download the raw data, read, plot, or manipulate the data as they wish on their own computer. The users are able to access the cleaned data sets. Internet access of the color spectrograms. This task has also been completed. It is now possible to access the spectrograms from the web site mentioned above. Improve the particle precipitation region classification. The algorithm for doing this task has been developed and implemented. As a result, the accuracies improved. Now the web site routinely distributes the results of applying the new algorithm to the cleaned data set. Mark the classification region on the spectrograms. The software to mark the classification region in the spectrograms has been completed. This is also available from our web site.

Wing, Simon↗
NASA NTRS1978

User's guide to the Nimbus-4 backscatter ultraviolet experiment data sets

The first year's data from the Nimbus 4 backscatter ultraviolet (BUV) experiment have been archived in the National Space Science Data Center (NSSDC). Backscattered radiances in the ultraviolet measured by the satellite were used to compute the global total ozone for the period April 1970 - April 1971. The data sets now in the NSSDC are the results obtained by the Ozone Processing Team, which has processed the data with the purpose of determining the best quality of the data. There are four basic sets of data available in the NSSDC representing various stages in processing. The primary data base contains organized and cleaned data in telemetry units. The radiance data has had most of the engineering calibrations performed. The detailed total ozone data is the result of computations to obtain the total ozone; the Compressed Total Ozone data is a convenient condensation of the detailed total ozone. Product data sets are also included.

Lowrey, B. E.↗
NASA NTRS1992

A P-band SAR interference filter

The synthetic aperture radar (SAR) interference filter is an adaptive filter designed to reduce the effects of interference while minimizing the introduction of undesirable side effects. The author examines the adaptive spectral filter and the improvement in processed SAR imagery using this filter for Jet Propulsion Laboratory Airborne SAR (JPL AIRSAR) data. The quality of these improvements is determined through several data fidelity criteria, such as point-target impulse response, equivalent number of looks, SNR, and polarization signatures. These parameters are used to characterize two data sets, both before and after filtering. The first data set consists of data with the interference present in the original signal, and the second set consists of clean data which has been coherently injected with interference acquired from another scene.

Taylor, Victor B.↗
NASA NTRS2024

QSF18 Nonresponse Follow-up Reminders Survey Data Supplemental File

This minimal data set contains anonymized study subject identifier (PARTICIPANT_ID) and non-response follow up type (group) from the single events surveys conducted during the Quiet Supersonic Flights 2018 risk reduction study in Galveston, Texas, in November 2018. Nonresponse follow up groups and procedures are defined and discussed in Page et al. 2020, Section 6.2 (NASA/CR-2020-220589/Volume I). The data cleaning conventions are consistent with the assumptions of Lee et al. in the treatment of the single events survey data (Lee, Rathsam, Wilson (2020). Journal of the Acoustical Society of America. 147, doi: 10.1121/10.0001021). Filename: reminder_groups.csv Dimensions: 371 rows by 2 columns. Variables: PARTICIPANT_ID, group PARTICPANT_ID: numeric (integer, six digits) group: character string taking one of four values ('Email - No Reminder'; 'Email - Reminder'; 'Text - No Reminder'; 'Text - Reminder').

sample survey↗
NASA NTRS2011

Image Segmentation, Registration, Compression, and Matching

A novel computational framework was developed of a 2D affine invariant matching exploiting a parameter space. Named as affine invariant parameter space (AIPS), the technique can be applied to many image-processing and computer-vision problems, including image registration, template matching, and object tracking from image sequence. The AIPS is formed by the parameters in an affine combination of a set of feature points in the image plane. In cases where the entire image can be assumed to have undergone a single affine transformation, the new AIPS match metric and matching framework becomes very effective (compared with the state-of-the-art methods at the time of this reporting). No knowledge about scaling or any other transformation parameters need to be known a priori to apply the AIPS framework. An automated suite of software tools has been created to provide accurate image segmentation (for data cleaning) and high-quality 2D image and 3D surface registration (for fusing multi-resolution terrain, image, and map data). These tools are capable of supporting existing GIS toolkits already in the marketplace, and will also be usable in a stand-alone fashion. The toolkit applies novel algorithmic approaches for image segmentation, feature extraction, and registration of 2D imagery and 3D surface data, which supports first-pass, batched, fully automatic feature extraction (for segmentation), and registration. A hierarchical and adaptive approach is taken for achieving automatic feature extraction, segmentation, and registration. Surface registration is the process of aligning two (or more) data sets to a common coordinate system, during which the transformation between their different coordinate systems is determined. Also developed here are a novel, volumetric surface modeling and compression technique that provide both quality-guaranteed mesh surface approximations and compaction of the model sizes by efficiently coding the geometry and connectivity/topology components of the generated models. The highly efficient triangular mesh compression compacts the connectivity information at the rate of 1.5-4 bits per vertex (on average for triangle meshes), while reducing the 3D geometry by 40-50 percent. Finally, taking into consideration the characteristics of 3D terrain data, and using the innovative, regularized binary decomposition mesh modeling, a multistage, pattern-drive modeling, and compression technique has been developed to provide an effective framework for compressing digital elevation model (DEM) surfaces, high-resolution aerial imagery, and other types of NASA data.

Yadegar, Jacob↗
NASA NTRSDate not supplied

Vacuum Sealable Container (VSC) and Astronaut Lunar Drill (ALD) for Artemis

Introduction: NASA’s Artemis Program is under development to send first woman and next man to the Moon. Artemis will utilize a suite of new technology for Lunar exploration, including new space vehicles, new space suits, and new Astronaut Tools. Honeybee Robotics has been working with NASA JSC to develop a new Vacuum Sealable Container (VSC) and new Astronaut Lunar Drill (ALD) for the upcoming Artemis missions. Vacuum Sealable Container: Sample return continues to be the “Holy Grail” of space exploration, allowing for the analysis of materials using Earth-based laboratories instead of needing to miniaturize and ruggedize instrumentation for space. The Apollo missions to the Moon had several kinds of Sealable Containers which brought back Lunar samples for analysis [1]. These samples are still being analyzed, fifty years later. The VSC requirements are different from that for Apollo containers and as such, new development was required. One major difference between Artemis samples and those from Apollo is the desire to bring back volatiles which may be part of lunar regolith. The VSC is designed to withstand a high-pressure differential caused by sublimating volatiles. Because of the new, stricter sealing requirements, additional features have been added to the VSC. For example, the seal on the container is required to be more robust, thus required more force to actuate, and the seal must be locked in place with a secondary mechanism. Astronaut Lunar Drill: The ALD is designed to be a multi-functional platform for Lunar sample acquisition. The drill builds on lessons learned from the Apollo Lunar Surface Drill (ALSD), as well as Honeybee’s long history of mechanized sample acquisition devices for space [2]. The main functionality of the ALD is Deep Core Regolith Drilling. Additional functionality includes Surface Rock Coring (SRC), and GeoTech Tools (GTT). The ALD is a rotary-percussive drill designed with deep drilling in mind. The ALD is currently designed to have decoupled rotary and percussion subsystems to allow for maximum battery life and reduced fatigue on the crewmember. Honeybee drill technology will automatically engage the percussion when needed to drill at maximum efficiency. The mechanized drill stand helps improve drilling efficiency; the system utilizes advanced drilling algorithms which only require the crewmember to hold a single switch. Additionally, the stand aids in extraction of deep cores, something which was a problem on Apollo. The SRC functionality of the ALD utilizes Honeybee’s Eccentric Tube Core Breakoff technology to collect and retain rock core samples. This technology has also been infused into the Perseverance rover mission. The ALD is removable from the stand to allow crewmembers to collect samples from large boulders. Bringing back rock cores samples instead of full rocks allows for a wider variety of samples to be returned to Earth for study and puts them in a uniform form-factor for effective sealing and analysis. SRC bits will utilize the power of the drill’s percussion system to drill hard Lunar rocks and expedite sample acquisition. The mechanized stand on the ALD allows for additional attachments for taking geotechnical measurements with a Static Cone Penetrometer (SCP) and a Shear Vane (SV). With the stand, the ALD can take SCP measurements with the touch of a button, storing data for return to Earth. SV measurements utilize the ALD’s Rotary motor to spin the vanes in a controlled manner, getting clean data untampered by human error. References: [1] Bar Cohen and Zacny (2009), Drilling in Extreme Environments - Penetration and Sampling on Earth and Other Planets, Wiley. [2] Bar-Cohen and Zacny, Advances in Terrestrial and Extraterrestrial Drilling, CRC Press. [3] Myrick (2003), Core Break-off Mechanism. US Patent No. 6,550,549 Acknowledgements: This work has been supported by NASA via SBIR Phase 3.

Artemis↗
NASA NTRS1986

Life Sciences Research Facility automation requirements and concepts for the Space Station

An evaluation is made of the methods and preliminary results of a study on prospects for the automation of the NASA Space Station's Life Sciences Research Facility. In order to remain within current Space Station resource allocations, approximately 85 percent of planned life science experiment tasks must be automated; these tasks encompass specimen care and feeding, cage and instrument cleaning, data acquisition and control, sample analysis, waste management, instrument calibration, materials inventory and management, and janitorial work. Task automation will free crews for specimen manipulation, tissue sampling, data interpretation and communication with ground controllers, and experiment management.

Rasmussen, Daryl N.↗
NASA NTRS1993

The HYDICE instrument design and its application to planetary instruments

The Hyperspectral Digital Imagery Collection Experiment (HYDICE) instrument represents a significant advance in the state of the art in hyperspectral sensors. It combines a higher signal-to-noise ratio (SNR) and significantly better spatial and spectral resolution and radio metric accuracy than systems flying on aircraft today. The need for 'clean' data, i.e., data free of sampling artifacts and excessive spatial or spectral noise, is a key driver behind the difficult combination of performance requirements laid out for HYDICE. Most of these involve the sensor optics and detector. This paper presents an optimized approach to those requirements, one that comprises push broom scanning, a single, mechanically cooled focal plane, a double-pass prism spectrometer, and an easily fabricated yet wide-field telescope. Central to the approach is a detector array that covers the entire spectrum from 0.4 to 2.5 microns. Among the major benefits conferred by such a design are optical and mechanical simplicity, low polarization sensitivity, and coverage of the entire spectrum without suffering the spectral gaps caused by beam splitters. The overall system minimizes interfaces to the C-141 aircraft on which it will be flown, can be calibrated on the ground and in flight to accuracies better than those required, and is designed for simple, push-button operation. Only unprocessed data are recorded during flight. A ground data processing station provides quick-look, calibration correction, and archiving capabilities, with a throughput better than the requirements. Overall performance of the system is expected to provide the solid database required to evaluate the potential of hyperspectral imagery in a wide variety of applications. HYDICE can be regarded as a test bed for future planetary instruments. The ability to spectrally image a wide field of view over multiple spectral octaves offers obvious advantages and is expected to maximize science return for the required cost and weight.

Basedow, R.↗
NASA NTRS1996

Feedback shift register sequences versus uniformly distributed random sequences for correlation chromatography

Two alternative input sequences are commonly employed in correlation chromatography (CC). They are sequences derived according to the algorithm of the feedback shift register (i.e., pseudo random binary sequences (PRBS)) and sequences derived by using the uniform random binary sequences (URBS). These two sequences are compared. By applying the "cleaning" data processing technique to the correlograms that result from these sequences, we show that when the PRBS is used the S/N of the correlogram is much higher than the one resulting from using URBS.

NASA Discipline Exobiology↗
NASA NTRS2011

Wire Test Grip Fixture

Wire-testing issues, such as the gripping strains imposed on the wire, play a critical role in obtaining clean data. In a standard test frame fitted with flat wedge grips, the gripping action alone creates stresses on the wire specimen that cause the wire to fail at the grip location. A new test frame, which is outfitted with a vacuum chamber, negated the use of any conventional commercially available wire test fixtures, as only 7 in. (17.8 cm) existed between the grip faces. An innovative grip fixture was designed to test thin gauge wire for a variety of applications in an existing Instron test frame outfitted with a vacuum chamber.

Burke, Christopher S.↗
NASA NTRS1987

Variable-Sweep Transition Flight Experiment (VSTFE): Stability code development and clean-up glove data analysis

The primary objective of the Variable Sweep Transition Flight Experiment (VSTFE) was to establish an improved swept wing transition criterion. The development of the Unified Stability System gave a way of quickly examining disturbance growth for a wide variety of laminar boundary layers. The disturbance growth traces shown are too scattered to define a transition criteria to replace the F-111 data band, which has been used successfully to design NLF gloves. Still, a careful review of the clean-up glove data may yield cases for which the transition location is known more accurately. Liquid crystal photographs of the clean-up glove show much spanwise variation in the transition front for some conditions, and this further complicates the analyses. Several high quality cases are needed in which the transition front is well defined and at a relatively constant chordwise station.

Rozendaal, Roger A.↗
NASA NTRS2010

De-Trending Techniques: Methods for Cleaning Questionable Shock Data

Not all zero shifted acceleration data can De-trended using this technique. DC shifts, improper AC coupling, Circuit noise/EMI/EMR, Equivalent RC circuit gain response/Circuit saturation(Slew Rate Limited), fixture grounding and wiring losses can all contribute to bad shock data being recorded. Some data that is zero-shifted or exhibit large instantaneous velocity shifts is inherently bad and a retest is warranted. Clean Acceleration-Time history data can be bad upon examining the Velocity & Displacement profiles. Laser Vibrometers provide a high level of accuracy for pyrotechnic shock testing. Engineering judgment and experience will determine the validity of Shock data.

Grillo, Vincent J.↗
NASA NTRSDate not supplied

Digital Twin Technology for Aviation

As technology progresses, so have the tools for data visualization. This project presents a digital twin model of the San Francisco airport displaying a 10-minute window of historical flight data, visualizing the trajectory data of airplanes and vehicles in three dimensions. Multiple different cameras where implemented to fully utilize the 3D visualization. This is a 100:1 feet scale model created in Autodesk Maya, using the airport center as the origin and recalculating all coordinates accordingly featuring the airport, some surrounding buildings, and the bay. For this project, six different models of airplanes were modeled at a 50:1 feet scale with texturing to mimic real-world aircraft models along with certain airlines. The animation is driven through archived data captured from NASA’s Sherlock Open-Data Portal, cleaned of noisy data points, processed into useable data formats, and implemented into a Maya ASCII file of animation paths with the corresponding previously-stated airplane models attached all using Java based conversion program.

Aleksander Schade↗
NASA NTRS2005

An XMM-Newton Search for Crab-like Supernova Remnants

The primary goals of the study are to search for evidence of non-thermal emission that would suggest the presence of a pulsar in this compact SNR. We have performed the reduction of the EPIC data for this observation, cleaning the data to remove time intervals of enhanced particle background, and have created maps in several energy bands, and on a variety of smoothing scales. We find no evidence for emission from the SNR. Given the small angular size of the SNR, we conclude that rather than being a young remnant, it is actually fairly old, but distant. At its current stage of evolution, the remnant shell has apparently entered the radiative phase, wherein the shell temperature has cooled sufficiently to be either below X-ray-emitting temperatures or at temperatures easily absorbed the foreground interstellar material. We have thus concluded that this SNR is not a viable candidate for a young ejecta-rich or pulsar-driven SNR.

Mushotzky, Richard↗
NASA NTRS1986

An experimental study of the aerodynamics of a NACA 0012 airfoil with a simulated glaze ice accretion

An experimental study was conducted in the Ohio State University subsonic wind tunnel to measure the detailed aerodynamic characteristics of an airfoil with a simulated glaze ice accretion. A NACA 0012 model with interchangeable leading edges and pressure taps every one percent chord was used. Surface pressure and wake data were taken on the airfoil clean, with forced transition and with a simulated glaze ice shape. Lift and drag penalties due to the ice shape were found and the surface pressure clearly showed that large separation bubbles were present. Both total pressure and split-film probes were used to measure velocity profiles, both for the clean model and for the model with a simulated ice accretion. A large region of flow separation was seen in the velocity profiles and was correlated to the pressure measurements. Clean airfoil data were found to compare well to existing airfoil analysis methods.

Bragg, M. B.↗
NASA NTRS2020

ICESat-2 Tracking App for Public Engagement

Information regarding the predicted ground tracks of Earth observing satellites is typically difficult to find and understand for the general public. This paper will describe and demonstrate an iOS mobile application for tracking NASA’s ICESat-2 (Ice, Cloud, and land Elevation Satellite 2), making it easier for users to see exactly when the satellite will be passing over any location in the world. ICESat-2, launched in 2018, uses green lasers to track elevation changes in polar ice and indirectly measure trees, land, and water, providing a precise height map of our planet. Most satellite tracking mobile apps display information about where a specific satellite is at that given moment and when it will be passing over the user’s location in the near future. ICESat-2 the app differs by allowing users to search for data about the satellite’s future flybys relative to a specific search location and radius that they get to choose. The search results include points up to three months into the future. By supplying user-centric results, users are provided relevant data in an easy-to- access manner. Additionally, this data is very clear to visualize in-app through maps, pins, and ground track lines. This becomes an incredibly useful tool not only to plan an observation of the satellite as it passes, but also for knowing when elevation data for a specific area will be available. In addition, this flyby information helps students and citizen scientists take more valuable tree height measurements to better validate the elevation data collected by the satellite. To achieve this outreach product, three key components were developed. These include a Python script to simplify the raw ground track data, a Node.js server that actively takes requests, and the iOS app itself. From initial beta tests, it has been noted that providing users with a map for visual awareness of search radius and result coordinates gives them a more comprehensive understanding of the data. Overall, by cleaning the raw data and making it available in a much more accessible and easier to understand way, ICESat-2 the app has exceeded expectations in providing educational and exciting data for all.

Harbeck, Kaitlin↗
NASA NTRS2023

Easy, Scalable Subsetting of GEDI Point Clouds

The GEDI Subsetter, a Python tool developed for NASA’s Multi-mission Algorithm and Analysis Platform (MAAP), optimizes the accessibility and visualization of GEDI point clouds by enabling users to efficiently subset data in a convenient, scalable manner. Complex science data often requires users to learn new software skills and handle many large files. Handling and cleaning large data sets is tedious and error-prone. These challenges significantly impede analysis. One of the goals of NASA's MAAP is to provide a platform that lowers the barrier to conducting research and analysis at scale. When a group of MAAP users wanted to conduct above-ground biomass estimation using GEDI data, we found that their existing workflow for leveraging GEDI data suffered from the barriers mentioned above. Furthermore, their workflow did not scale easily beyond a small number of granules. We found that existing tools related to GEDI data retrieval and subsetting were too limiting, so the GEDI Subsetter was written to support MAAP users’ needs. Being able to run many subsetting jobs simultaneously in the MAAP, and parallelizing the code itself, has led to significant speed improvements in obtaining relevant data, reducing subsetting time from hours to minutes. MAAP users can now more quickly and easily obtain only the data relevant to their research, by choosing which GEDI collection they want to work with (L1A, L2A, L2B, or L4A), and how they want to subset it, by specifying an area of interest, a temporal range, and relevant attributes. This has significantly reduced the feedback loop for users, allowing them to much more quickly subset GEDI data and begin their analysis. Although the GEDI Subsetter originally targeted users of the MAAP, it is generalized such that it can also be used outside of the MAAP and includes a command-line interface for convenience. Furthermore, with minor modifications, it should be possible to use it with non-GEDI data as the general pattern should be applicable to other sparse/track-based sensors.

Charles Daniels↗