Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data science”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 865 records · Page 48
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

A proximal trust-region method for nonsmooth optimization with inexact function and gradient evaluations

Many applications require minimizing the sum of smooth and nonsmooth functions. For example, basis pursuit denoising problems in data science require minimizing a measure of data misfit plus an $\ell^1$-regularizer. Similar problems arise in the optimal control of partial differential equations (PDEs) when sparsity of the control is desired. Here, we develop a novel trust-region method to minimize the sum of a smooth nonconvex function and a nonsmooth convex function. Our method is unique in that it permits and systematically controls the use of inexact objective function and derivative evaluations. When using a quadratic Taylor model for the trust-region subproblem, our algorithm is an inexact, matrix-free proximal Newton-type method that permits indefinite Hessians. We prove global convergence of our method in Hilbert space and demonstrate its efficacy on three examples from data science and PDE-constrained optimization.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2012

Update on Apollo Data Restoration by the NSSDC and the PDS Lunar Data Node

The Lunar Data Node (LDN) , under the auspices of the Geosciences Node of the Planetary Data System (PDS) and the National Space Science Data Center (NSSDC), is continuing its efforts to recover and restore Apollo science data. The data being restored are in large part archived with NSSDC on older media, but unarchived data are also being recovered from other sources. They are typically on 7- or 9-track magnetic tapes, often in obsolete formats, or held on microfilm, microfiche, or paper documents. The goal of the LDN is to restore these data from their current form, which is difficult for most researchers to access, into common digital formats with all necessary supporting data (metadata) and archive the data sets with PDS. Restoration involves reading the data from the original media, deciphering the data formats to produce readable digital data and converting the data into usable tabular formats. Each set of values in the table must then be understood in terms of the quantity measured and the units used. Information on instrument properties, operational history, and calibrations is gathered and added to the data set, along with pertinent references, contacts, and other ancillary documentation. The data set then undergoes a peer review and the final validated product is archived with PDS. Although much of this effort has concentrated on data archived at NSSDC in the 1970's, we have also recovered data and information that were never sent to NSSDC. These data, retrieved from various outside sources, include raw and reduced Gamma-Ray Spectrometer data from Apollos 15 and 16, information on the Apollo 17 Lunar Ejecta And Meteorites experiment, Dust Detector data from Apollos 11, 12, 14, and I5, raw telemetry tapes from the Apollo ALSEPs, and Weekly Status Reports for all the Apollo missions. These data are currently being read or organized, and supporting data is being gathered. We are still looking for the calibrated heat flow data from Apollos 15 and 17 for the period 1975-1977, any assistance or information on these data would be welcome. NSSDC has recently been tasked to release its hard-copy archive, comprising photography, microfilm, and microfiche. The details are still being discussed, but we are concentrating on recovering the valuable lunar data from these materials while they are still readily accessible. We have identified the most critical of these data and written a LASER proposal to fund their restoration. Included in this effort are data from the Apollo 15 and 16 Mass Spectrometers and the Apollo 17 Par-UV Spectrometer and ancillary information on the Apollo 17 Surface Electrical Properties Experiment.

Williams, David R.↗
NASA NTRS2003

Exploring New Methods of Displaying Bit-Level Quality and Other Flags for MODIS Data

The NASA Distributed Active Archive Center (DAAC) at the National Snow and Ice Data Center (NSIDC) archives and distributes snow and sea ice products derived from the MODerate resolution Imaging Spectroradiometer (MODIS) on board NASA's Terra and Aqua satellites. All MODIS standard products are in the Earth Observing System version of the Hierarchal Data Format (HDF-EOS). The MODIS science team has packed a wealth of information into each HDF-EOS file. In addition to the science data arrays containing the geophysical product, there are often pixel-level Quality Assurance arrays which are important for understanding and interpreting the science data. Currently, researchers are limited in their ability to access and decode information stored as individual bits in many of the MODIS science products. Commercial and public domain utilities give users access, in varying degrees, to the elements inside MODIS HDF-EOS files. However, when attempting to visualize the data, users are confronted with the fact that many of the elements actually represent eight different 1-bit arrays packed into a single byte array. This project addressed the need for researchers to access bit-level information inside MODIS data files. In an previous NASA-funded project (ESDIS Prototype ID 50.0) we developed a visualization tool tailored to polar gridded HDF-EOS data set. This tool,called the Polar researchers to access, geolocate, visualize, and subset data that originate from different sources and have different spatial resolutions but which are placed on a common polar grid. The bit-level visualization function developed under this project was added to PHDIS, resulting in a versatile tool that serves a variety of needs. We call this the EOS Imaging Tool.

Khalsa, Siri Jodha Singh↗
NASA NTRS2018

Analysis and Review of NASA Earth Science Metadata: How Automation Plays a Role

The Analysis and Review of the Common Metadata Repository (CMR ARC) Team reviews all EOSDIS metadata. The team’s objective is to achieve consistency, correctness, and completeness for all metadata records in the CMR, as well as improve the discoverability of NASA's Earth Science data within the CMR framework. This work is currently being completed at Marshall Space Flight Center. CMR makes a single discovery point possible for NASA's Earth Science data users. The CMR team, in collaboration with three other core metadata teams, contributes to the stewardship of NASA's Earth Science data through a process of continual curation and the ongoing development of the Unified Metadata Model (UMM). A key tool now used in the curation process, referred to as the NASA CMR Dashboard, is an online curation dashboard developed in collaboration with software development company, Element 84. This tool facilitates the review of Earth Science metadata records and subsequent stakeholder collaboration on the resolution of identified issues. A key capability of the new tool is a suite of automated compliance checks written in Python 3.6 that verify the integrity of various metadata elements across multiple standards.

Staton, Patrick↗
DOE OSTI2018

Aurorasaurus database of real-time, crowd-sourced aurora data for space weather research

This work documents the details of Aurorasaurus citizen science data for the period spanning 2015 and 2016 as well as its routine data filtering protocols. Aurorasaurus citizen science data is a collection of auroral sightings submitted to the project via its website or apps and mined from social media. It is a robust data set and particularly abundant during strong geomagnetic storms when auroral precipitation models have the highest uncertainty. This data is offered to the scientific community for use through an open-access database in its raw and scientific formats, each of which is described in detail in this technical report. Furthermore, by demonstrating its scientific utility, we aim to encourage its integration into auroral research.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Science-Driven Data Management for Multi-Tiered Storage (Final Report)

Scientific discovery at the exascale will not be possible without significant new research in the management, storage and retrieval over the long lifespan of the extreme amounts of data that will be produced. Our thesis is that adding application level knowledge about data to guide the actions of the storage system provides substantial benefits to the organization, storage, and access to extreme scale data, resulting in improved productivity for computational science. In this project we will demonstrate novel techniques to facilitate efficient mapping of data objects, even partitioning individual variables, from the user space onto multiple storage tiers, and enable application-guided data reductions and transformations to address capacity and bandwidth bottlenecks. Our goal is to address the associated Input/ Output (I/O) and storage challenges in the context of current and emerging storage landscapes, and expedite insights into mission critical scientific processes.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
NASA NTRS1991

The moderate resolution imaging spectrometer (MODIS) science and data system requirements

The Moderate Resolution Imaging Spectrometer (MODIS) has been designated as a facility instrument on the first NASA polar orbiting platform as part of the Earth Observing System (EOS) and is scheduled for launch in the late 1990s. The near-global daily coverage of MODIS, combined with its continuous operation, broad spectral coverage, and relatively high spatial resolution, makes it central to the objectives of EOS. The development, implementation, production, and validation of the core MODIS data products define a set of functional, performance, and operational requirements on the data system that operate between the sensor measurements and the data products supplied to the user community. The science requirements guiding the processing of MODIS data are reviewed, and the aspects of an operations concept for the production of data products from MODIS for use by the scientific community are discussed.

Ardanuy, Philip E.↗
DOE OSTI2018

BigData Express: Toward Schedulable, Predictable, and High-Performance Data Transfer

Big Data has emerged as a driving force for scientific discoveries. Large scientific instruments (e.g., colliders, and telescopes) generate exponentially increasing volumes of data. To enable scientific discovery, science data must be collected, indexed, archived, shared, and analyzed, typically in a widely distributed, highly collaborative manner. Data transfer is now an essential function for science discoveries, particularly within big data environments. Although significant improvements have been made in the area of bulk data transfer, the currently available data transfer tools and services can not successfully address the high-performance and time-constraint challenges of data transfer required by extreme-scale science applications for the following reasons: disjoint end-to-end data transfer loops, cross-interference between data transfers, and existing data transfer tools and services are oblivious to user requirements (deadline and QoS requirements). Fermilab has been working on the BigData Express project to address these problems. BigData Express seeks to provide a schedulable, predictable, and high-performance data transfer service for big data science. The BigData Express software is being deployed and evaluated at multiple research institutions, which include UMD, StarLight, FNAL, KISTI, KSTAR, SURFnet, Ciena, and other sites. Meanwhile, the BigData Express research team is collaborating with the StarLight International/National Communications Exchange Facility to deploy BigData Express at various research platforms, including Pacific Research Platform, National Research Platform, and Global Research Platform. It is envisioned that we are working toward building a high-performance data transfer federation for big data science.

Lu, Qiming↗
DOE OSTI2020

LDM-151: Data Management Science Pipelines Design

The LSST Science Requirements Document (the LSST SRD) specifies a set of data product guidelines, designed to support science goals envisioned to be enabled by the LSST observing program. Following these guidelines, the details of these data products have been described in the LSST Data Products Definition Document (DPDD), and captured in a formal flow-down from the SRD via the LSST System Requirements (LSR), Observatory System Specifications (OSS), to the Data Management System Requirements (DMSR). The LSST Data Management subsystem's responsibilities include the design, implementation, deployment and execution of software pipelines necessary to generate these data products. This document describes the design of the scientific aspects of those pipelines.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2018

Aurorasaurus Database of Real-Time, Crowd-Sourced Aurora Data for Space Weather Research

This technical report documents the details of Aurorasaurus citizen science data for the period spanning 2015 and 2016 as well as its routine data filtering protocols. Aurorasaurus citizen science data is a collection of auroral sightings submitted to the project via its website or apps and mined from social media. It is a robust data set and particularly abundant during strong geomagnetic storms when auroral precipitation models have the highest uncertainty. These data are offered to the scientific community for use through an openaccess database in its raw and scientific formats, each of which is described in detail in this technical report. Furthermore, by demonstrating its scientific utility, we aim to encourage its integration into auroral research.

Citizen science↗
NASA NTRS2012

Enabling Earth Science Through Cloud Computing

Cloud Computing holds tremendous potential for missions across the National Aeronautics and Space Administration. Several flight missions are already benefiting from an investment in cloud computing for mission critical pipelines and services through faster processing time, higher availability, and drastically lower costs available on cloud systems. However, these processes do not currently extend to general scientific algorithms relevant to earth science missions. The members of the Airborne Cloud Computing Environment task at the Jet Propulsion Laboratory have worked closely with the Carbon in Arctic Reservoirs Vulnerability Experiment (CARVE) mission to integrate cloud computing into their science data processing pipeline. This paper details the efforts involved in deploying a science data system for the CARVE mission, evaluating and integrating cloud computing solutions with the system and porting their science algorithms for execution in a cloud environment.

science data system↗
NASA NTRS1993

Radioastron flight operations

Radioastron is a space-based very-long-baseline interferometry (VLBI) mission to be operational in the mid-90's. The spacecraft and space radio telescope (SRT) will be designed, manufactured, and launched by the Russians. The United States is constructing a DSN subnet to be used in conjunction with a Russian subnet for Radioastron SRT science data acquisition, phase link, and spacecraft and science payload health monitoring. Command and control will be performed from a Russian tracking facility. In addition to the flight element, the network of ground radio telescopes which will be performing co-observations with the space telescope are essential to the mission. Observatories in 39 locations around the world are expected to participate in the mission. Some aspects of the mission that have helped shaped the flight operations concept are: separate radio channels will be provided for spacecraft operations and for phase link and science data acquisition; 80-90 percent of the spacecraft operational time will be spent in an autonomous mode; and, mission scheduling must take into account not only spacecraft and science payload constraints, but tracking station and ground observatory availability as well. This paper will describe the flight operations system design for translating the Radioastron science program into spacecraft executed events. Planning for in-orbit checkout and contingency response will also be discussed.

Altunin, V. I.↗
DOE OSTI2016

Cosmology from cosmic shear with Dark Energy Survey Science Verification data

We present the first constraints on cosmology from the Dark Energy Survey (DES), using weak lensing measurements from the preliminary Science Verification (SV) data. We use 139 square degrees of SV data, which is less than 3% of the full DES survey area. Using cosmic shear 2-point measurements over three redshift bins we find σ 8 (m=0.3) 0.5 = 0:81 ± 0:06 (68% confidence), after marginalising over 7 systematics parameters and 3 other cosmological parameters. Furthermore, we examine the robustness of our results to the choice of data vector and systematics assumed, and find them to be stable. About 20% of our error bar comes from marginalising over shear and photometric redshift calibration uncertainties. The current state-of-the-art cosmic shear measurements from CFHTLenS are mildly discrepant with the cosmological constraints from Planck CMB data. Our results are consistent with both datasets. Our uncertainties are ~30% larger than those from CFHTLenS when we carry out a comparable analysis of the two datasets, which we attribute largely to the lower number density of our shear catalogue. We investigate constraints on dark energy and find that, with this small fraction of the full survey, the DES SV constraints make negligible impact on the Planck constraints. The moderate disagreement between the CFHTLenS and Planck values of σ 8 (Ω m =0.3) 0.5 is present regardless of the value of w.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2014

Data Preservation, Information Preservation, and Lifecyle of Information Management at NASA GES DISC

Data lifecycle management awareness is common today; planners are more likely to consider lifecycle issues at mission start. NASA remote sensing missions are typically subject to life cycle management plans of the Distributed Active Archive Center (DAAC), and NASA invests in these national centers for the long-term safeguarding and benefit of future generations. As stewards of older missions, it is incumbent upon us to ensure that a comprehensive enough set of information is being preserved to prevent the risk for information loss. This risk is greater when the original data experts have moved on or are no longer available. Preservation of items like documentation related to processing algorithms, pre-flight calibration data, or input-output configuration parameters used in product generation, are examples of digital artifacts that are sometimes not fully preserved. This is the grey area of information preservation; the importance of these items is not always clear and requires careful consideration. Missing important metadata about intermediate steps used to derive a product could lead to serious challenges in the reproducibility of results or conclusions. Organizations are rapidly recognizing that the focus of life-cycle preservation needs to be enlarged from the strict raw data to the more encompassing arena of information lifecycle management. By understanding what constitutes information, and the complexities involved, we are better equipped to deliver longer lasting value about the original data and derived knowledge (information) from them. The NASA Earth Science Data Preservation Content Specification is an attempt to define the content necessary for long-term preservation. It requires new lifecycle infrastructure approach along with content repositories to accommodate artifacts other than just raw data. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) setup an open-source Preservation System capable of long-term archive of digital content to augment its raw data holding. This repository is being used for such missions as HIRDLS, UARS, TOMS, OMI, among others. We will provide a status of this implementation; report on challenges, lessons learned, and detail our plans for future evolution to include other missions and services.

data management↗
DOE OSTI2023

DSI Python API Demo May 2023 [Slides]

Data Science Infrastructure (DSI) is developing searchable databases and workflows for simulation and experimental data derived from ASC clients. Short term goal: Make data easily accessible through metadata indexing and querying while respecting data permissions. Longer term goal: Use this data for data science activities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

BigData Express: Toward Predictable, Schedulable, and High-Performance Data Transfer [Slides]

Big data has emerged as a driving force for scientific discoveries. Large scientific instruments (e.g., colliders, and telescopes) generate exponentially increasing volumes of data. To enable scientific discovery, science data must be collected, indexed, archived, shared, and analyzed, typically in a widely distributed and highly collaborative manner. Data transfer is now an essential function for science discoveries, particularly within big data environments. To support data transfer for big data science, there is a need for predictable, high performance, scalable, end-to-end, programmable networks that enable science applications to use network resources most efficiently. Software-Defined Networking (SDN) is a novel network architecture and technological approach that separates the network control and data planes.

97 MATHEMATICS AND COMPUTING↗