Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “raw data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Data Preservation, Information Preservation, and Lifecyle of Information Management at NASA GES DISC

Data lifecycle management awareness is common today; planners are more likely to consider lifecycle issues at mission start. NASA remote sensing missions are typically subject to life cycle management plans of the Distributed Active Archive Center (DAAC), and NASA invests in these national centers for the long-term safeguarding and benefit of future generations. As stewards of older missions, it is incumbent upon us to ensure that a comprehensive enough set of information is being preserved to prevent the risk for information loss. This risk is greater when the original data experts have moved on or are no longer available. Preservation of items like documentation related to processing algorithms, pre-flight calibration data, or input-output configuration parameters used in product generation, are examples of digital artifacts that are sometimes not fully preserved. This is the grey area of information preservation; the importance of these items is not always clear and requires careful consideration. Missing important metadata about intermediate steps used to derive a product could lead to serious challenges in the reproducibility of results or conclusions. Organizations are rapidly recognizing that the focus of life-cycle preservation needs to be enlarged from the strict raw data to the more encompassing arena of information lifecycle management. By understanding what constitutes information, and the complexities involved, we are better equipped to deliver longer lasting value about the original data and derived knowledge (information) from them. The NASA Earth Science Data Preservation Content Specification is an attempt to define the content necessary for long-term preservation. It requires new lifecycle infrastructure approach along with content repositories to accommodate artifacts other than just raw data. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) setup an open-source Preservation System capable of long-term archive of digital content to augment its raw data holding. This repository is being used for such missions as HIRDLS, UARS, TOMS, OMI, among others. We will provide a status of this implementation; report on challenges, lessons learned, and detail our plans for future evolution to include other missions and services.

data management↗

Estimating Total Electron Content Using 1,000+ GPS Receivers

A computer program uses data from more than 1,000 Global Positioning System (GPS) receivers in an Internet-accessible global network to generate daily estimates of the global distribution of vertical total electron content (VTEC) of the ionosphere. This program supersedes an older program capable of processing readings from only about 200 GPS receivers. This program downloads the data via the Internet, then processes the data in three stages. In the first stage, raw data from a global subnetwork of about 200 receivers are preprocessed, station by station, in a Kalman-filter-based least-squares estimation scheme that estimates satellite and receiver differential biases for these receivers and for satellites. In the second stage, an observation equation that incorporates the results from the first stage and the raw data from the remaining 800 receivers is solved to obtain the differential biases for these receivers. The only remaining error sources for which an account cannot be given are multipath and receiver noise contributions. The third stage is a postprocessing stage in which all the processed data are combined and used to generate new data products, including receiver differential biases and global and regional VTEC maps and animations.

Komjathy, Attila↗

Compression of a Data Stream by Selection among a Set of Compression Tools

A stream of raw data is compressed prior to transmissio in a communication channel by a system which includes modules for choosing a current segment of the raw data stream for processing and defining a set of operators for representing data segments by a mathematical operation and parameters thereof. The system performs a competitive evaluation of different tools comprising different combinations of one or more of the operators and the parameters threrof with respect to the current data segment in order to determine relative abilities among the different tools to reduce the number of bits required to represent the current data segment. The system then selects a tool and a set of parameters thereof found in the competitive evaluation to have a superior ability relative to others of the different tools to reduce a number of bits required to represent the current data segment.

Arlid, Bertelrud↗

SEASAT synthetic-aperture radar data user's manual

The SEASAT Synthetic-Aperture Radar (SAR) system, the data processors, the extent of the image data set, and the means by which a user obtains this data are described and the data quality is evaluated. The user is alerted to some potential problems with the existing volume of SEASAT SAR image data, and allows him to modify his use of that data accordingly. Secondly, the manual focuses on the ultimate focuses on the ultimate capabilities of the raw data set and evaluates the potential of this data for processing into accurately located, amplitude-calibrated imagery of high resolution. This allows the user to decide whether his needs require special-purpose data processing of the SAR raw data.

Pravdo, S. H.↗

Optimizing a Small RNAseq Analysis Pipeline for NASA GeneLab Using Open-Source Tools and Libraries

Small RNA sequencing (small RNAseq) is a powerful tool for studying the regulation of gene expression in various organisms. Small RNAseq has been leveraged in space biology research to study how expression of small RNAs, e.g. micro RNAs (miRNAs), small interfering RNAs (siRNAs), and piwi-interacting RNAs (piRNAs), change upon exposure to the space environment. NASA GeneLab currently hosts small RNAseq raw data derived from space-relevant experiments on the Open Science Data Repository (OSDR). To maximize the accessibility of these data to the scientific community, in addition to hosting raw data, which is only interpretable by bioinformaticians, GeneLab plans to process all small RNAseq datasets and make those processed data available to the scientific community via the OSDR. In this study, we present the development of the GeneLab standardized pipeline for processing small RNAseq datasets. Using human, plant, and synthetic small RNAseq datasets, we interrogate various open-source software and publicly available databases to evaluate their accuracy and reproducibility in each step of the pipeline. For quality control and adapter detection and trimming, we evaluated TrimGalore!, FASTX, SeqKit, and DNApi methods to optimize alignment to reference genomes. We compared BWA, Bowtie, and Bowtie2 to determine the optimal alignment tool. For each alignment tool we also assessed various reference databases, including Ensembl reference genomes and different types of small RNA reference databases, including genome, hairpin, and miRNA references from the miRbase and MirGeneDB databases. To quantify the aligned data, we compared SAMtools, HTSeq, and RSEM for counting alignment events from each alignment tool used. Finally, we evaluated various tools, including DESeq2 and EdgeR, for data normalization and subsequent differential expression analysis. We will present the results from our comparative analyses for each pipeline step and propose a consensus pipeline for processing small RNAseq data derived from various organisms exposed to the space environment.

SmallRNAseq, NASA GeneLab, quality control, adapte↗

Laser Disdrometer / Standardized Data

This dataset contains raw data from the laser disdrometer at Nantucket. The disdrometer measures droplet size distribution, hydrometer type, and precipitation rate.

17 WIND ENERGY↗

RHOD Site - Surface Meteorological Station / Processed Data

This dataset contains raw data from the WFIP3 RHOD site supplementary meteorological sensors: T/RH and barometer; 1-sec average. The supplementary meteorological suite was added to PNNL Surface Flux Station to provide independent measurements of air temperature, relative humidity, and atmospheric pressure.

17 WIND ENERGY↗

Intelligent Systems Technologies and Utilization of Earth Observation Data

The addition of raw data and derived geophysical parameters from several Earth observing satellites over the last decade to the data held by NASA data centers has created a data rich environment for the Earth science research and applications communities. The data products are being distributed to a large and diverse community of users. Due to advances in computational hardware, networks and communications, information management and software technologies, significant progress has been made in the last decade in archiving and providing data to users. However, to realize the full potential of the growing data archives, further progress is necessary in the transformation of data into information, and information into knowledge that can be used in particular applications. Sponsored by NASA s Intelligent Systems Project within the Computing, Information and Communication Technology (CICT) Program, a conceptual architecture study has been conducted to examine ideas to improve data utilization through the addition of intelligence into the archives in the context of an overall knowledge building system (KBS). Potential Intelligent Archive concepts include: 1) Mining archived data holdings to improve metadata to facilitate data access and usability; 2) Building intelligence about transformations on data, information, knowledge, and accompanying services; 3) Recognizing the value of results, indexing and formatting them for easy access; 4) Interacting as a cooperative node in a web of distributed systems to perform knowledge building; and 5) Being aware of other nodes in the KBS, participating in open systems interfaces and protocols for virtualization, and achieving collaborative interoperability.

Ramapriyan, H. K.↗

Methodology to Define Delivery Accuracy Under Current Day ATC Operations

In order to enable arrival management concepts and solutions in a NextGen environment, ground- based sequencing and scheduling functions have been developed to support metering operations in the National Airspace System. These sequencing and scheduling algorithms as well as tools are designed to aid air traffic controllers in developing an overall arrival strategy. The ground systems being developed will support the management of aircraft to their Scheduled Times of Arrival (STAs) at flow-constrained meter points. This paper presents a methodology for determining the undelayed delivery accuracy for current day air traffic control operations. This new method analyzes the undelayed delivery accuracy at meter points in order to understand changes of desired flow rates as well as enabling definition of metrics that will allow near-future ground automation tools to successfully achieve desired separation at the meter points. This enables aircraft to meet their STAs while performing high precision arrivals. The research presents a possible implementation that would allow delivery performance of current tools to be estimated and delivery accuracy requirements for future tools to be defined, which allows analysis of Estimated Time of Arrival (ETA) accuracy for Time-Based Flow Management (TBFM) and the FAA's Traffic Management Advisor (TMA). TMA is a deployed system that generates scheduled time-of-arrival constraints for en- route air traffic controllers in the US. This new method of automated analysis provides a repeatable evaluation of the delay metrics for current day traffic, new releases of TMA, implementation of different tools, and across different airspace environments. This method utilizes a wide set of data from the Operational TMA-TBFM Repository (OTTR) system, which processes raw data collected by the FAA from operational TMA systems at all ARTCCs in the nation. The OTTR system generates daily reports concerning ATC status, intent and actions. Due to its availability, ease of use, and vast collection of data across several airspaces it was determined that the OTTR data set would be the best method to utilize moving forward with this analysis. The particular variables needed for further analysis were determined along with the necessary OTTR reports, by working closely with the repository team additional analysis reports were developed that provided key ETA and STA information at the freeze horizon. One major benefit of the OTTR data is that using the correct reports the data across several airports could be analyzed over large periods of time. The OTTR data processes the TBFM data daily and is stored in various formats across several airspaces. This allowed us to develop our own parsing methods and raw data processing that would not rely on other computationally expensive tools that perform more in depth analysis of similar sets of data. The majority of this work consisted of the development of the ability to filter flights to create a subset of flights that could be considered undelayed, which is defined as a flight at the freeze horizon with an ETA and STA difference that was minimal or close to zero. This was a broad method that allowed the consideration of a large data set which consisted of all the traffic across a two month period in 2013, the hottest and coldest months, arriving into four airports: George Bush Intercontinental, Denver International, Los Angeles International, and Phoenix Sky Harbor.

delivery accuracy↗

Machine Learning-Based Extreme Data Reduction for Prompt Supernova Pointing at DUNE

One of the goals of the Deep Underground Neutrino Experiment (DUNE) is to use the massive underground liquid argon time projection chamber (LArTPC) detectors at its far site for multimessenger astronomy (MMA), in the detection of neutrinos from core-collapse supernovae (SNe). Its current baseline trigger strategy detects activity in the detector that is consistent with supernova (SN) neutrinos and saves the raw data for further offline analysis but provides no prompt pointing information crucial for optical follow-ups by other observatories. This approach is based on the assumption that prompt pointing determination using raw data is computationally prohibitive. In this article, we demonstrate a proof-of-concept based on applying extreme data reduction on the buffered SN data in the DUNE data acquisition (DAQ) system’s front-end computers using a machine learning (ML) workflow. This reduces the data by ~5 orders of magnitude, allowing a full track reconstruction to be carried out quickly on a single server. The total time to perform the ML-based data reduction and the full track reconstruction is less than the time to transfer the SN data back to Fermilab or a high-performance computing (HPC) center. This shows that prompt processing of raw SN data is possible and, in fact, trivial once the data have been reduced to reject radiological backgrounds, paving the way to a high-quality SN pointing trigger that is based on fully reconstructed data instead of trigger primitives (TPs).

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Data for Photosynthetic responses to temperature across the tropics: a meta-analytic approach

This data package contains raw data, data output, and code of photosynthetic temperature response curves from tropical forests. Data are used in "Photosynthetic responses to temperature across the tropics: a meta-analytic approach". The research investigates how photosynthetic optimum temperatures and shapes of photosynthetic temperature response curves varies across tropical forest climates and growth conditions. Growth climate variables include mean annual temperature, max and max temperature, diurnal temperature range, and aridity index. Meta-analysis combines 18 datasets with representation from Africa, Oceana, North American, and South America. Growth conditions analyzed considers deciduousness, successional status, light conditions, and whether plants are grown in situ or ex situ. All files, except for the raw datafile (Tropical_MetaAnalysis_Master_3.8.22_Edit.csv), have been processed using R code, which is provided.

54 ENVIRONMENTAL SCIENCES↗

Towards an Introspective Dynamic Model of Globally Distributed Computing Infrastructures

Large-scale scientific collaborations like ATLAS, Belle II, CMS, DUNE, and others involve hundreds of research institutes and thousands of researchers spread across the globe. These experiments generate petabytes of data, with volumes soon expected to reach exabytes. Consequently, there is a growing need for computation, including structured data processing from raw data to consumer-ready derived data, extensive Monte Carlo simulation campaigns, and a wide range of end-user analysis. To manage these computational and storage demands, centralized workflow and data management systems are implemented. However, decisions regarding data placement and payload allocation are often made disjointly and via heuristic means. A significant obstacle in adopting more effective heuristic or AI-driven solutions is the absence of a quick and reliable introspective dynamic model to evaluate and refine alternative approaches. In this study, we aim to develop such an interactive system using real-world data. By examining job execution records from the PanDA workflow management system, we have pinpointed key performance indicators such as queuing time, error rate, and the extent of remote data access. The dataset includes five months of activity. Additionally, we are creating a generative AI model to simulate time series of payloads, which incorporate visible features like category, event count, and submitting group, as well as hidden features like the total computational load—derived from existing PanDA records and computing site capabilities. These hidden features, which are not visible to job allocators, whether heuristic or AI-driven, influence factors such as queuing times and data movement.

kilic, Ozgur Ozan [Brookhaven National Laboratory ↗

Design concepts for an on-board coherent optical image processor

On-board spacecraft image data processing systems for transmitting processed data rather than raw data are discussed. A brief history of the development of the optical data processing techniques is presented along with the conceptual design of a coherent optical system with a noncoherent image input.

Husain-Abidi, A. S.↗

SEC sensor parametric test and evaluation system

This system provides the necessary automated hardware required to carry out, in conjunction with the existing 70 mm SEC television camera, the sensor evaluation tests which are described in detail. The Parametric Test Set (PTS) was completed and is used in a semiautomatic data acquisition and control mode to test the development of the 70 mm SEC sensor, WX 32193. Data analysis of raw data is performed on the Princeton IBM 360-91 computer.

Source record↗

Evaluation of RDBMS packages for use in astronomy

Tabular data sets arise in many areas of astronomical data analysis, from raw data (such as photon event lists) to final results (such as source catalogs). The Starlink catalog access and reporting package, SCAR, was originally developed to handle IRAS data and it has been the principal relational DBMS in the Starlink software collection for several years. But SCAR has many limitations and is VMS-specific, while Starlink is in transition from VMS to Unix. Rather than attempt a major re-write of SCAR for Unix, it seemed more sensible to see whether any existing database packages are suitable for general astronomical use. The authors first drew up a list of desirable properties for such a system and then used these criteria to evaluate a number of packages, both free ones and those commercially available. It is already clear that most commercial DBMS packages are not very well suited to the requirements; for example, most cannot carry out efficiently even fairly basic operations such as joining two catalogs on an approximate match of celestial positions. This paper reports the results of the evaluation exercise and notes the problems in using a standard DBMS package to process scientific data. In parallel with this the authors have started to develop a simple database engine that can handle tabular data in a range of common formats including simple direct-access files (such as SCAR and Exosat DBMS tables) and FITS tables (both ASCII and binary).

Page, C. G.↗

Harnessing the Risk-Related Data Supply Chain: An Information Architecture Approach to Enriching Human System Research and Operations Knowledge

NASA's Human Research Program (HRP) and Space Life Sciences Directorate (SLSD), not unlike many NASA organizations today, struggle with the inherent inefficiencies caused by dependencies on heterogeneous data systems and silos of data and information spread across decentralized discipline domains. The capture of operational and research-based data/information (both in-flight and ground-based) in disparate IT systems impedes the extent to which that data/information can be efficiently and securely shared, analyzed, and enriched into knowledge that directly and more rapidly supports HRP's research-focused human system risk mitigation efforts and SLSD s operationally oriented risk management efforts. As a result, an integrated effort is underway to more fully understand and document how specific sets of risk-related data/information are generated and used and in what IT systems that data/information currently resides. By mapping the risk-related data flow from raw data to useable information and knowledge (think of it as the data supply chain), HRP and SLSD are building an information architecture plan to leverage their existing, shared IT infrastructure. In addition, it is important to create a centralized structured tool to represent risks including attributes such as likelihood, consequence, contributing factors, and the evidence supporting the information in all these fields. Representing the risks in this way enables reasoning about the risks, e.g. revisiting a risk assessment when a mitigation strategy is unavailable, updating a risk assessment when new information becomes available, etc. Such a system also provides a concise way to communicate the risks both within the organization as well as with collaborators. Understanding and, hence, harnessing the human system risk-related data supply chain enhances both organizations' abilities to securely collect, integrate, and share data assets that improve human system research and operations.

Buquo, Lynn↗

Harnessing the Risk-Related Data Supply Chain: An Information Architecture Approach to Enriching Human System Research and Operations Knowledge

An Information Architecture facilitates the understanding and, hence, harnessing of the human system risk-related data supply chain which enhances the ability to securely collect, integrate, and share data assets that improve human system research and operations. By mapping the risk-related data flow from raw data to useable information and knowledge (think of it as a data supply chain), the Human Research Program (HRP) and Space Life Science Directorate (SLSD) are building an information architecture plan to leverage their existing, and often shared, IT infrastructure.

Buquo, Lynn E.↗

Software for Generating Strip Maps from SAR Data

Jurassicprok is a computer program that generates strip-map digital elevation models and other data products from raw data acquired by an airborne synthetic-aperture radar (SAR) system. This software can process data from a variety of airborne SAR systems but is designed especially for the GeoSAR system, which is a dual-frequency (P- and X-band), single-pass interferometric SAR system for measuring elevation both at the bare ground surface and top of the vegetation canopy. Jurassicprok is a modified version of software developed previously for airborne-interferometric- SAR applications. The modifications were made to accommodate P-band interferometric processing, remove approximations that are not generally valid, and reduce processor-induced mapping errors to the centimeter level. Major additions and other improvements over the prior software include the following: a) A new, highly efficient multi-stage-modified wave-domain processing algorithm for accurately motion compensating ultra-wideband data; b) Adaptive regridding algorithms based on estimated noise and actual measured topography to reduce noise while maintaining spatial resolution; c) Exact expressions for height determination from interferogram data; d) Fully calibrated volumetric correlation data based on rigorous removal of geometric and signal-to-noise decorrelation terms; e) Strip range-Doppler image output in user-specified Doppler coordinates; f) An improved phase-unwrapping and absolute-phase-determination algorithm; g) A more flexible user interface with many additional processing options; h) Increased interferogram filtering options; and i) Ability to use disk space instead of random- access memory for some processing steps.

Hensley, Scott↗