Results for “Data Science” — Page 48

DOE OSTI2017

The Modern Research Data Portal: A Design Pattern for Networked, Data-Intensive Science

Here we describe best practices for providing convenient, high-speed, secure access to large data via research data portals. We capture these best practices in a new design pattern, the Modern Research Data Portal, that disaggregates the traditional monolithic web-based data portal to achieve orders-of-magnitude increases in data transfer performance, support new deployment architectures that decouple control logic from data storage, and reduce development and operations costs. We introduce the design pattern; explain how it leverages high-performance Science DMZs and cloud-based data management services; review representative examples at research laboratories and universities, including both experimental facilities and supercomputer sites; describe how to leverage Python APIs for authentication, authorization, data transfer, and data sharing; and use coding examples to demonstrate how these APIs can be used to implement a range of research data portal capabilities. Sample code at a companion web site, https://docs.globus.org/mrdp, provides application skeletons that readers can adapt to realize their own research data portals.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

NASA NTRSDate not supplied

Elevating the Quality of Space Omics Sequencing Data: Innovations and Methodologies from NASA GeneLab Sample Processing Laboratory

NASA’s GeneLab, part of the NASA Open Science Data Repository, is a space-related database that hosts a diverse range of transcriptomics, proteomics, epigenomics and genomics data. The NASA GeneLab Sample Processing Laboratory (SPL) generates omics data from biological experiments conducted aboard the International Space Station, Space Shuttle and space related ground experiments, this omics data then hosted on the GeneLab repository. Samples generated such experiments pose numerous technical challenges such as small experimental sample size, variance in dissection times, limited tissue preservation methods, prolonged storage time, and more. GeneLab SPL team had developed specialized expertise in nucleic acid extraction, library preparation and sequencing of such biological samples via extensive training and years of experience. In order to ensure data accuracy and consistency across experiments, SPL has developed standardized protocols for each species and tissue type. These protocols in conjunction with quality control metrics and data standards are crucial in generating of high-quality data. SPL protocols and standards have been developed in collaboration with the scientific community and had been made publicly available on the GeneLab portal, guaranteeing comparability of datasets across spaceflight experiments. To ensure reliability of data generation, SPL leverages cutting-edge innovations in laboratory automation for sample processing. By leveraging these state-of-the-art platforms, SPL achieves high levels of data reproducibility while significantly minimizing sources of bias and variability, especially across experiments with large numbers of samples. Over the past few years, the space biology investigator community has accessed SPL-generated data from the Open Science Data Repository for a myriad of data re-analysis and re-use studies. We observe a trend that in-house SPL-generated data consistently outperforms outsourced sequencing data in terms of technical standards, quality control metrics, timeliness of data delivery, and sequencing and reagent efficiency. Superior data generation has and will continue to enable discoveries in disease, diagnostic tools, and the biological effects of long duration spaceflight.

GeneLab↗

NASA NTRSDate not supplied

Elevating the Quality of Space Omics Sequencing Data: Innovations and Methodologies from NASA GeneLab Sample Processing Laboratory

NASA’s GeneLab, part of the NASA Open Science Data Repository, is a space-related database that hosts a diverse range of transcriptomics, proteomics, epigenomics and genomics data. The NASA GeneLab Sample Processing Laboratory (SPL) generates omics data from biological experiments conducted aboard the International Space Station, Space Shuttle and space related ground experiments, this omics data then hosted on the GeneLab repository. Samples generated such experiments pose numerous technical challenges such as small experimental sample size, variance in dissection times, limited tissue preservation methods, prolonged storage time, and more. GeneLab SPL team had developed specialized expertise in nucleic acid extraction, library preparation and sequencing of such biological samples via extensive training and years of experience. In order to ensure data accuracy and consistency across experiments, SPL has developed standardized protocols for each species and tissue type. These protocols in conjunction with quality control metrics and data standards are crucial in generating of high-quality data. SPL protocols and standards have been developed in collaboration with the scientific community and had been made publicly available on the GeneLab portal, guaranteeing comparability of datasets across spaceflight experiments. To ensure reliability of data generation, SPL leverages cutting-edge innovations in laboratory automation for sample processing. By leveraging these state-of-the-art platforms, SPL achieves high levels of data reproducibility while significantly minimizing sources of bias and variability, especially across experiments with large numbers of samples. Over the past few years, the space biology investigator community has accessed SPL-generated data from the Open Science Data Repository for a myriad of data re-analysis and re-use studies. We observe a trend that in-house SPL-generated data consistently outperforms outsourced sequencing data in terms of technical standards, quality control metrics, timeliness of data delivery, and sequencing and reagent efficiency. Superior data generation has and will continue to enable discoveries in disease, diagnostic tools, and the biological effects of long duration spaceflight.

GeneLab↗

DOE OSTI2021

Bioinformatic teaching resources - for educators, by educators - using KBase, a free, user-friendly, open source platform

Over the past year, biology educators and staff at the Department of Energy Systems Biology Knowledgebase (KBase) initiated a collaborative effort to develop a curriculum for bioinformatics education. KBase is a free and easily accessible data science platform that integrates many bioinformatics resources into a graphical user interface built upon reproducible analysis notebooks. KBase held conversations with college and high school instructors to understand how KBase could potentially support their educational goals. These conversations morphed into a working group of biological and data science instructors that adapted the KBase platform to their curriculum needs, specifically around concepts in Genomics, Metagenomics, Pangenomics, and Phylogenetics. The KBase Educators Working Group developed modular, adaptable, and customizable instructional units. Each instructional module contains teaching resources, publicly available data, analysis tools, and markdown capability to tailor instructions and learning goals for each class. The online user interface enables students to conduct hands-on data science research and analyses without requiring programming skills or their own computational resources (these are provided by KBase). Alongside these resources, KBase continues to work with instructors, supporting the development of additional curriculum modules. For anyone new to the platform, KBase, and the growing KBase Educators Organization, provides a community network, accompanied by community-sourced guidelines, instructional templates, and peer support to use KBase within a classroom whether virtual or in-person.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

Sample Identifiers and Metadata to Support Data Management and Reuse in Multidisciplinary Ecosystem Sciences

Physical samples are foundational entities for research across biological, Earth, and environmental sciences. Data generated from sample-based analyses are not only the basis of individual studies, but can also be integrated with other data to answer new and broader-scale questions. Ecosystem studies increasingly rely on multidisciplinary team-science to study climate and environmental changes. While there are widely adopted conventions within certain domains to describe sample data, these have gaps when applied in a multidisciplinary context. In this study, we reviewed existing practices for identifying, characterizing, and linking related environmental samples. We then tested practicalities of assigning persistent identifiers to samples, with standardized metadata, in a pilot field test involving eight United States Department of Energy projects. Participants collected a variety of sample types, with analyses conducted across multiple facilities. We address terminology gaps for multidisciplinary research and make recommendations for assigning identifiers and metadata that supports sample tracking, integration, and reuse. Furthermore, our goal is to provide a practical approach to sample management, geared towards ecosystem scientists who contribute and reuse sample data.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS2014

Data Albums: An Event Driven Search, Aggregation and Curation Tool for Earth Science

One of the largest continuing challenges in any Earth science investigation is the discovery and access of useful science content from the increasingly large volumes of Earth science data and related information available. Approaches used in Earth science research such as case study analysis and climatology studies involve gathering discovering and gathering diverse data sets and information to support the research goals. Research based on case studies involves a detailed description of specific weather events using data from different sources, to characterize physical processes in play for a specific event. Climatology-based research tends to focus on the representativeness of a given event, by studying the characteristics and distribution of a large number of events. This allows researchers to generalize characteristics such as spatio-temporal distribution, intensity, annual cycle, duration, etc. To gather relevant data and information for case studies and climatology analysis is both tedious and time consuming. Current Earth science data systems are designed with the assumption that researchers access data primarily by instrument or geophysical parameter. Those who know exactly the datasets of interest can obtain the specific files they need using these systems. However, in cases where researchers are interested in studying a significant event, they have to manually assemble a variety of datasets relevant to it by searching the different distributed data systems. In these cases, a search process needs to be organized around the event rather than observing instruments. In addition, the existing data systems assume users have sufficient knowledge regarding the domain vocabulary to be able to effectively utilize their catalogs. These systems do not support new or interdisciplinary researchers who may be unfamiliar with the domain terminology. This paper presents a specialized search, aggregation and curation tool for Earth science to address these existing challenges. The search tool automatically creates curated "Data Albums", aggregated collections of information related to a specific science topic or event, containing links to relevant data files (granules) from different instruments; tools and services for visualization and analysis; and information about the event contained in news reports, images or videos to supplement research analysis. Curation in the tool is driven via an ontology based relevancy ranking algorithm to filter out non-relevant information and data.

Ramachandran, Rahul↗

NASA NTRS1991

Radio science ground data system for the Voyager-Neptune encounter, part 1

The Voyager radio science experiments at Neptune required the creation of a ground data system array that includes a Deep Space Network complex, the Parkes Radio Observatory, and the Usuda deep space tracking station. The performance requirements were based on experience with the previous Voyager encounters, as well as the scientific goals at Neptune. The requirements were stricter than those of the Uranus encounter because of the need to avoid the phase-stability problems experienced during that encounter and because the spacecraft flyby was faster and closer to the planet than previous encounters. The primary requirement on the instrument was to recover the phase and amplitude of the S- and X-band (2.3 and 8.4 GHz) signals under the dynamic conditions encountered during the occultations. The primary receiver type for the measurements was open loop with high phase-noise and frequency stability performance. The receiver filter bandwidth was predetermined based on the spacecraft's trajectory and frequency uncertainties.

Kursinski, E. R.↗

NASA NTRS2018

A Case Study Comparing Citizen Science Aurora Data with Global Auroral Boundaries Derived from Satellite Imagery and Empirical Models

Aurorasaurus is a citizen science project that offers a new, global data source consisting of ground-based reports of the aurora. For this case study, aurora data collected during the 17-18 March 2015 geomagnetic storm are examined to identify their conjunctions with Defense Meteorological Satellite Program (DMSP) satellite passes over the high latitude auroral regions. This unique set of aurora data can provide ground-truth validation of existing auroral precipitation models. Particularly, the solar wind driven, Oval Variation, Assessment, Tracking, Intensity, and Online Nowcasting (OVATION) Prime 2013 (OP-13) model and a Kp-dependent model of Zhang-Paxton (Z-P) are utilized for our boundary validation efforts. These two similar models are compared for the first time. Global equatorward auroral boundaries are derived from the OP 13 model and the DMSP Special Sensor Ultraviolet Spectrographic Imager (SSUSI) far ultraviolet (FUV) data using the Z-P model at a fixed flux level of 0.2 erg cm(exp -2)s(exp -1). These boundaries are then compared with citizen science reports as well as with each other. Even though there are some large differences between the global boundaries for a few cases, the average difference is about 1.5 deg in geomagnetic latitude, with OP-13 being equatorward of Z-P model. When these boundaries are compared with each other as a function of local time, no clear overall trend as a function of local time was observed. It is also found that the ground based reports are more consistent with the predictions of the OP-13 model.

Kosar, Burcu C.↗

NASA NTRS1992

User Scientific Data Systems: Experience Report

This paper presents an abbreviated history of NASA science data management system development over the past ten years by selecting two case studies, each representative of a distinct era of science data management systems.

Scientific Data↗

NASA NTRS1978

The DSS radio science subsystem: Data handling of Very Long Baseline Interferometry (VLBI) data

The key characteristics functional requirements, and operation of the radio science subsystem are discussed as they pertain to usage of deep radio sources by the DSN VLBI system.

Berman, A. L.↗

NASA NTRS1991

The 1990 annual statistics and highlights report

The National Space Science Data Center (NSSDC) has archived over 6 terabytes of space and Earth science data accumulated over nearly 25 years. It now expects these holdings to nearly double every two years. The science user community needs rapid access to this archival data and information about data. The NSSDC has been set on course to provide just that. Five years ago the NSSDC came on line, becoming easily reachable for thousands of scientists around the world through electronic networks it managed and other international electronic networks to which it connected. Since that time, the data center has developed and implemented over 15 interactive systems, operational nearly 24 hours per day, and is reachable through DECnet, TCP/IP, X25, and BITnet communication protocols. The NSSDC is a clearinghouse for the science user to find data needed through the Master Directory system whether it is at the NSSDC or deposited in over 50 other archives and data management facilities around the world. Over 13,000 users accessed the NSSDC electronic systems, during the past year. Thousands of requests for data have been satisfied, resulting in the NSSDC's sending out a volume of data last year that nearly exceeded a quarter of its holdings. This document reports on some of the highlights and distribution statistics for most of the basic NSSDC operational services for fiscal year 1990. It is intended to be the first of a series of annual reports on how well NSSDC is doing in supporting the space and Earth science user communities.

Green, James L.↗

NASA NTRSDate not supplied

Introduction to Air Traffic Management

The presentation introduces students and faculty to air traffic management with focus on air traffic data for data-science. Starting with the common attributes of transportation systems — highway transportation, air transportation and data transportation, the initial set of slides discuss the purpose of data-science in air traffic management, reasons why air traffic management is challenging, and the multidisciplinary nature of air traffic management research. The history of flight from 1903 — Wright Flyer — to 1987 — formation of the National Air Traffic Controllers Association — is briefly discussed. The national airspace system is described in terms of airports in the U. S., air traffic control facilities (flight service stations, terminal, enroute and system command center), airspace geometry (sectors, airways and navaids), governing regulations and directives, airspace classification (Class A through G), special use airspace, visual flight rules and instrument flight rules. The contents of a flight-plan are described. Weather briefing is discussed. The surveillance equipment used for surface, terminal area and enroute are described, and the aircraft states obtained using the surveillance data are listed. Airline operations control functions — schedule development, flight planning, resource scheduling and flight following — are noted. Next, the roles and responsibilities of air traffic controllers and traffic flow managers are discussed. Separation standards and conflict resolution techniques are outlined. Finally, traffic flow management techniques are reviewed with an illustrative example.

Air Traffic Management↗

NASA NTRSDate not supplied

Introduction to Air Traffic Management

The presentation introduces students and faculty to air traffic management with focus on air traffic data for data-science. Starting with the common attributes of transportation systems — highway transportation, air transportation and data transportation, the initial set of slides discuss the purpose of data-science in air traffic management, reasons why air traffic management is challenging, and the multidisciplinary nature of air traffic management research. The history of flight from 1903 — Wright Flyer — to 1987 — formation of the National Air Traffic Controllers Association — is briefly discussed. The national airspace system is described in terms of airports in the U. S., air traffic control facilities (flight service stations, terminal, enroute and system command center), airspace geometry (sectors, airways and navaids), governing regulations and directives, airspace classification (Class A through G), special use airspace, visual flight rules and instrument flight rules. The contents of a flight-plan are described. Weather briefing is discussed. The surveillance equipment used for surface, terminal area and enroute are described, and the aircraft states obtained using the surveillance data are listed. Airline operations control functions — schedule development, flight planning, resource scheduling and flight following — are noted. Next, the roles and responsibilities of air traffic controllers and traffic flow managers are discussed. Separation standards and conflict resolution techniques are outlined. Finally, traffic flow management techniques are reviewed with an illustrative example.

Air Traffic Management↗

NASA NTRSDate not supplied

Search Enhancements using Natural Language Processing Techniques

NASA Goddard Earth Sciences Data and Information Services Center (GESDISC) is one of the 12 NASA Science Mission Directorate Data Centers. The main goal of GESDISC is to provide earth science data, information, and services to the earth science data community. Consequently, data discovery is at the center of our mission and our search engine is the primary tool for our users to interact, find, and access our data. Existing search approaches are largely focused on hard-matching of keywords in the search query with dataset metadata. Here we propose to expand the search by introducing a complementary natural language processing (NLP) search. At the heart of our proposed NLP search, we trained a joint embedding using scientific text corpus and a curated set of dataset metadata. The embedding learns the association between words in our dataset metadata and those of the scientific text corpus. This enables us to go beyond simple hard-matching of a query and data set metadata and have a notion of “similarity” between the search query and the datasets. We further integrated our NLP search into the Elastic Search (ES) framework leveraging similarity search capabilities offered through the “dense_vector” field type. Our preliminary evaluations show that our proposed NLP search has the potential to be utilized to complement the existing search engine and serve as a base for a dataset recommendation system.

Armin Mehrabian↗

NASA NTRS2004

Processing Spacecraft Data Without Confusion

Producing multiple versions of the same data product for the same time frame with the same remotely sensed inputs can be a recipe for disaster. Yet, amidst the commotion of satellite launch and early operations (LEO), such data processing is needed. After LEO, the situation gets worse. Processing newly arriving data ("forward processing") is augmented with reprocessing and algorithm development, comparison, evaluation, and testing -- often happening all at the same time. The problem can be analyzed in three main parts -- maintaining multiple versions of algorithms and data so that end-product users are not overwhelmed,allocating computer resources efficiently, and simplifying production operations so that va st amounts of data can be processed with minimal staff and fewer errors. OMIDAPS provides a framework for execution of algorithms that transform lower level data acquired by OMI on NASA's Aura satellite into higher level science data products. In contrast to traditional science data processing systems, we address all parts of the problem with an innovative approach allowing multiple data processing to run within a single physical system. The data products, imports, exports, and execution planning are all segregated into distinct "ArchiveSets." This paper describes reasons for multiple concurrent productions on a typical satellite data processing project using OMI as an example. It describes the virtual data processing system concept and its advantages over separate physical processing strings. It explores the specific implementation of the virtual systems within OMIDAPS and discusses some of the implications of our approach and describes how virtual processing is used to accomplish the overall mission of OMI data processing.

Tilmes, Curt↗

NASA NTRS2016

Soil Moisture Active Passive Mission L4_SM Data Product Assessment (Version 2 Validated Release)

During the post-launch SMAP calibration and validation (Cal/Val) phase there are two objectives for each science data product team: 1) calibrate, verify, and improve the performance of the science algorithm, and 2) validate the accuracy of the science data product as specified in the science requirements and according to the Cal/Val schedule. This report provides an assessment of the SMAP Level 4 Surface and Root Zone Soil Moisture Passive (L4_SM) product specifically for the product's public Version 2 validated release scheduled for 29 April 2016. The assessment of the Version 2 L4_SM data product includes comparisons of SMAP L4_SM soil moisture estimates with in situ soil moisture observations from core validation sites and sparse networks. The assessment further includes a global evaluation of the internal diagnostics from the ensemble-based data assimilation system that is used to generate the L4_SM product. This evaluation focuses on the statistics of the observation-minus-forecast (O-F) residuals and the analysis increments. Together, the core validation site comparisons and the statistics of the assimilation diagnostics are considered primary validation methodologies for the L4_SM product. Comparisons against in situ measurements from regional-scale sparse networks are considered a secondary validation methodology because such in situ measurements are subject to up-scaling errors from the point-scale to the grid cell scale of the data product. Based on the limited set of core validation sites, the wide geographic range of the sparse network sites, and the global assessment of the assimilation diagnostics, the assessment presented here meets the criteria established by the Committee on Earth Observing Satellites for Stage 2 validation and supports the validated release of the data. An analysis of the time average surface and root zone soil moisture shows that the global pattern of arid and humid regions are captured by the L4_SM estimates. Results from the core validation site comparisons indicate that "Version 2" of the L4_SM data product meets the self-imposed L4_SM accuracy requirement, which is formulated in terms of the ubRMSE: the RMSE (Root Mean Square Error) after removal of the long-term mean difference. The overall ubRMSE of the 3-hourly L4_SM surface soil moisture at the 9 km scale is 0.035 cubic meters per cubic meter requirement. The corresponding ubRMSE for L4_SM root zone soil moisture is 0.024 cubic meters per cubic meter requirement. Both of these metrics are comfortably below the 0.04 cubic meters per cubic meter requirement. The L4_SM estimates are an improvement over estimates from a model-only SMAP Nature Run version 4 (NRv4), which demonstrates the beneficial impact of the SMAP brightness temperature data. L4_SM surface soil moisture estimates are consistently more skillful than NRv4 estimates, although not by a statistically significant margin. The lack of statistical significance is not surprising given the limited data record available to date. Root zone soil moisture estimates from L4_SM and NRv4 have similar skill. Results from comparisons of the L4_SM product to in situ measurements from nearly 400 sparse network sites corroborate the core validation site results. The instantaneous soil moisture and soil temperature analysis increments are within a reasonable range and result in spatially smooth soil moisture analyses. The O-F residuals exhibit only small biases on the order of 1-3 degrees Kelvin between the (re-scaled) SMAP brightness temperature observations and the L4_SM model forecast, which indicates that the assimilation system is largely unbiased. The spatially averaged time series standard deviation of the O-F residuals is 5.9 degrees Kelvin, which reduces to 4.0 degrees Kelvin for the observation-minus-analysis (O-A) residuals, reflecting the impact of the SMAP observations on the L4_SM system. Averaged globally, the time series standard deviation of the normalized O-F residuals is close to unity, which would suggest that the magnitude of the modeled errors approximately reflects that of the actual errors. The assessment report also notes several limitations of the "Version 2" L4_SM data product and science algorithm calibration that will be addressed in future releases. Regionally, the time series standard deviation of the normalized O-F residuals deviates considerably from unity, which indicates that the L4_SM assimilation algorithm either over- or under-estimates the actual errors that are present in the system. Planned improvements include revised land model parameters, revised error parameters for the land model and the assimilated SMAP observations, and revised surface meteorological forcing data for the operational period and underlying climatological data. Moreover, a refined analysis of the impact of SMAP observations will be facilitated by the construction of additional variants of the model-only reference data. Nevertheless, the “Version 2” validated release of the L4_SM product is sufficiently mature and of adequate quality for distribution to and use by the larger science and application communities.

SMAP L4_SM↗

DOE OSTI2016

Cosmology constraints from shear peak statistics in Dark Energy Survey Science Verification data

Shear peak statistics has gained a lot of attention recently as a practical alternative to the two point statistics for constraining cosmological parameters. We perform a shear peak statistics analysis of the Dark Energy Survey (DES) Science Verification (SV) data, using weak gravitational lensing measurements from a 139 deg$^2$ field. We measure the abundance of peaks identified in aperture mass maps, as a function of their signal-to-noise ratio, in the signal-to-noise range $0<\mathcal S / \mathcal N<4$. To predict the peak counts as a function of cosmological parameters we use a suite of $N$-body simulations spanning 158 models with varying $\Omega_{\rm m}$ and $\sigma_8$, fixing $w = -1$, $\Omega_{\rm b} = 0.04$, $h = 0.7$ and $n_s=1$, to which we have applied the DES SV mask and redshift distribution. In our fiducial analysis we measure $\sigma_{8}(\Omega_{\rm m}/0.3)^{0.6}=0.77 \pm 0.07$, after marginalising over the shear multiplicative bias and the error on the mean redshift of the galaxy sample. We introduce models of intrinsic alignments, blending, and source contamination by cluster members. These models indicate that peaks with $\mathcal S / \mathcal N>4$ would require significant corrections, which is why we do not include them in our analysis. We compare our results to the cosmological constraints from the two point analysis on the SV field and find them to be in good agreement in both the central value and its uncertainty. As a result, we discuss prospects for future peak statistics analysis with upcoming DES data.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

NASA NTRSDate not supplied

AI Curation Methods for NASA Scientific Data

The NASA Open Science Data Repository (OSDR) serves as a central hub for sharing and accessing NASA's vast collection of scientific data, supporting researchers across diverse fields. To enhance the efficiency, accuracy, and accessibility of this data, we are leveraging advanced artificial intelligence (AI) techniques as part of the AI for Curation project. By integrating large language models (LLMs) into our data curation workflow, we aim to streamline the entire process—from data submission to user interaction. This initiative focuses on improving key areas, including data ingestion, curation, and user engagement with curated datasets, impacting multiple domains and a wide user base. First, we are developing tools that can automatically parse data in various formats, using LLMs to convert unstructured data into structured, standardized formats. This reduces the manual effort required for curation, allowing curators to focus on more critical scientific analyses. Additionally, AI and machine learning (ML) models are being implemented to automate data validation and verification, ensuring the highest standards of data quality and reliability. Finally, we are creating a conversational AI agent to interact with the curated scientific studies in OSDR, helping users easily navigate the repository and access relevant data. By enhancing data discoverability and accessibility, these advancements will foster new research opportunities and promote the principles of open science.

Walter Alvarado↗