Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “science data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 703 records · Page 39
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

The Dark Energy Survey Image Processing Pipeline

The Dark Energy Survey (DES) is a five-year optical imaging campaign with the goal of understanding the origin of cosmic acceleration. DES performs a ~5000 deg 2 survey of the southern sky in five optical bands (g, r, i, z, Y) to a depth of ~24th magnitude. Contemporaneously, DES performs a deep, time-domain survey in four optical bands (g, r, i, z) over ~27 deg 2 . DES exposures are processed nightly with an evolving data reduction pipeline and evaluated for image quality to determine if they need to be retaken. Difference imaging and transient source detection are also performed in the time domain component nightly. On a bi-annual basis, DES exposures are reprocessed with a refined pipeline and coadded to maximize imaging depth. In this paper, we describe the DES image processing pipeline in support of DES science, as a reference for users of archival DES data, and as a guide for future astronomical surveys.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2016

Accelerating Science Impact through Big Data Workflow Management and Supercomputing

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. ATLAS, one of the largest collaborations ever assembled in the the history of science, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. To manage the workflow for all data processing on hundreds of data centers the PanDA (Production and Distributed Analysis)Workload Management System is used. An ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF), is realizing within BigPanDA and megaPanDA projects. These projects are now exploring how PanDA might be used for managing computing jobs that run on supercomputers including OLCF’s Titan and NRC-KI HPC2. The main idea is to reuse, as much as possible, existing components of the PanDA system that are already deployed on the LHC Grid for analysis of physics data. The next generation of PanDA will allow many data-intensive sciences employing a variety of computing platforms to benefit from ATLAS experience and proven tools in highly scalable processing.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1985

NASA's Pilot Land Data System development program

The NASA Pilot Land Data System (PLDS) project is intended to enhance the effectiveness of data processing capabilities used by researchers applying remote sensing data in land science research. Two sites in the centerminous U.S. have been selected as study areas scanned by Landsat, Nimbus and GOES instruments. The data will be analyzed by teams of researchers representing different fields of expertise. The PLDS program will explore data management, networking and communications, system access capabilities, land analysis software, special processes and overall systems engineerng. The data will be processed by researchers working interactively through remote supermicrocomputer workstations using a variety of operating systems and on-site software capabilities.

Price, R. D.↗
NASA NTRS1988

Data processing assessment for the Lunar Geoscience Observer imaging spectrometer

On the Lunar Geoscience Observer project, a Visible and Infrared Mapping Spectrometer instrument has been proposed. This instrument will have science data input rates in the hundreds of kilobits per second (kbps) and an average telemetry output data rate of 4 kbps. Techniques that can be used to reduce the throughput of the instrument are editing, summing and averaging, data compression, data preprocessing, pattern recognition and snapshot data taking. Due to instrument limitations in the buffer memory size and processing speeds, a careful selection of the available techniques must be made.

Irigoyen, R. E.↗
NASA NTRS1972

Processing on-board optical data for planetary approach navigation.

The Mariner 9 spacecraft's science television camera provided an optical navigation experiment with TV pictures containing images of Mars' natural satellites against star backgrounds. Required TV image data and spacecraft engineering data were extracted from the spacecraft telemetry stream in this near-real time experiment designed to validate the navigation content of spacecraft-based optical data. This paper discusses the computer programs developed to prepare optical data for use in a navigation filter. Ground and in-flight calibration allowed pointing knowledge of better than 6 arc seconds. System performance during Mars approach was excellent, leading to extremely accurate trajectory estimates. The experiment provides a basis for the design of equivalent systems for future missions.

Acton, C. H., Jr.↗
NASA NTRS1987

Computer Sciences and Data Systems, volume 1

Topics addressed include: software engineering; university grants; institutes; concurrent processing; sparse distributed memory; distributed operating systems; intelligent data management processes; expert system for image analysis; fault tolerant software; and architecture research.

Source record↗
NASA NTRS2007

Mars Global Reference Atmospheric Model (Mars-GRAM 2005) Applications for Mars Science Laboratory Mission Site Selection Processes

The new Mars-GRAM auxiliary profile capability, using data from TES observations, mesoscale model output, or other sources, allows a potentially higher fidelity representation of the atmosphere, and a more accurate way of estimating inherent uncertainty in atmospheric density and winds. Figure 3 indicates that, with nominal value rpscale=1, Mars-GRAM perturbations would tend to overestimate observed or mesoscale-modeled variability. To better represent TES and mesoscale model density perturbations, rpscale values as low as about 0.4 could be used. Some trajectory model implementations of Mars-GRAM allow the user to dynamically change rpscale and rwscale values with altitude. Figure 4 shows that an mscale value of about 1.2 would better replicate wind standard deviations from MRAMS or MMM5 simulations at the Gale, Terby, or Melas sites. By adjusting the rpscale and rwscale values in Mars-GRAM based on figures such as Figure 3 and 4, we can provide more accurate end-to-end simulations for EDL at the candidate MSL landing sites.

Justh, H. L.↗
NASA NTRS2017

AIRS-Only Product in Giovanni for Exploring Up-to-Date AIRS Observation and Comparing with AIRS+AMSU Product

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) has been the home of processing, archiving, and distribution services for the Atmospheric Infrared Sounder (AIRS) mission since its launch in 2002 for global observations of the atmospheric state. Giovanni, a Web-based application developed by the GES DISC, provides a simple and intuitive way to visualize, analyze, and access vast amounts of Earth science remote sensing data without having to download the data. Most important AIRS variables, including temperature and humidity profiles, outgoing longwave radiation, cloud properties, and trace gases, are available in Giovanni. AIRS is an instrument suite comprised of a hyperspectral infrared instrument (AIRS) and two multichannel microwave instruments, the Advanced Microwave Sounding Unit (AMSU) and the Humidity Sounder for Brazil (HSB). As HSB ceased operation in the very early stages of the AIRS mission, the AIRS project operates two parallel retrieval algorithms: one using both IR and MW measurements (AIRS+AMSU) and the other using only IR measurements (AIRS-only), which covers most of the mission duration. The AIRS+AMSU product is better quality, and the variables in Giovanni are from this product. However, generation of the AIRS+AMSU product has been suspended since the AMSU instrument anomaly occurred in late September 2016. To continue exploring up-to-date AIRS observations, the same set of variables from the AIRS-only product have been added to Giovanni by the GES DSIC. This will also support comparison of AIRS-only with AIRS+AMSU retrievals. In this presentation, we demonstrate the visualization of the AIRS-only product and plots/statistics of comparison with AIRS+AMSU product using Giovanni.

AIRS↗
NASA NTRSDate not supplied

Why Near Real-time/Low Latency is Important to Monitor the Changing World

An essential factor for remote sensing data products in impacting decision making is latency, or the time between earth observation and data are available to users. In many applications areas, latency plays an important or even decisive role where low latency earth observations help people to make timely, data-based decisions. Within the open and free NASA resources, NASA’s Land, Atmosphere Near real-time Capability for Earth Observing System (EOS) (LANCE) supports users interested in monitoring a wide variety of natural and man-made phenomena using near real-time (NRT) data products that are made available much quicker than routine processing allows. The combination of all available LANCE satellite products provides global coverage at multiple times per day, which makes it possible to meet user needs in various areas of applications including water resources, agriculture, air quality, wildland fire and many other disasters monitoring and management. As one of the prime users of LANCE, NASA’s Earth Science Applied Sciences Program promotes the use of LANCE data products to demonstrate applications in decision making and facilitates the end-user feedback to the science team to improve data products. One of the most critical applications that we expect data to be processed as close to the user as possible is wildland file response and management. User feedback indicates that LANCE NRT fire data products within 3 hours latency would meet the needs of the wildland fire community. Other examples of earth science application areas for which low latency is particularly important include detecting volcanic eruptions, early warning of disasters, tracking extreme weather events, and monitoring air quality.

Tian Yao↗
NASA NTRS2011

The NASA Decadal Survey Aerosol, Cloud, Ecosystems Mission

In 2007, the National Academy of Sciences delivered a Decadal Survey (Earth Science and Applications from Space: National Imperatives for the Next Decade and Beyond) for NASA, NOAA, and USGS, which is a prioritization of future satellite Earth observations. The recommendations included 15 missions (13 for NASA, two for NOAA), which were prioritized into three groups or tiers. One of the second tier missions is the Aerosol, Cloud, (ocean) Ecosystems (ACE) mission, which focuses on climate forcing, cloud and aerosol properties and interactions, and ocean ecology, carbon cycle science, and fluxes. The baseline instruments recommended for ACE are a cloud radar, an aerosol/cloud lidar, an aerosol/cloud polarimeter, and an ocean radiometer. The instrumental heritage for these measurements are derived from the Cloudsat, CALIPSO, Glory, SeaWiFS and Aqua (MODIS) missions. In 2008, NASA HQ, lead by Hal Maring and Paula Bontempi, organized an interdisciplinary science working group to help formulate the ACE mission by refining the science objectives and approaches, identifying measurement (satellite and field) and mission (e.g., orbit, data processing) requirements, technology requirements, and mission costs. Originally, the disciplines included the cloud, aerosol, and ocean biogeochemistry communities. Subsequently, an ocean-aerosol interaction science working group was formed to ensure the mission addresses the broadest range of science questions possible given the baseline measurements, The ACE mission is a unique opportunity for ocean scientists to work closely with the aerosol and cloud communities. The science working groups are collaborating on science objectives and are defining joint field studies and modeling activities. The presentation will outline the present status of the ACE mission, the science questions each discipline has defined, the measurement requirements identified to date, the current ACE schedule, and future opportunities for broader community participation.

McClain, Charles R.↗
DOE OSTI2017

Processing of Ice Cloud In Situ Data Collected by Bulk Water, Scattering, and Imaging Probes: Fundamentals, Uncertainties, and Efforts toward Consistency

Abstract In situ observations of cloud properties made by airborne probes play a critical role in ice cloud research through their role in process studies, parameterization development, and evaluation of simulations and remote sensing retrievals. To determine how cloud properties vary with environmental conditions, in situ data collected during different field projects processed by different groups must be used. However, because of the diverse algorithms and codes that are used to process measurements, it can be challenging to compare the results. Therefore it is vital to understand both the limitations of specific probes and uncertainties introduced by processing algorithms. Since there is currently no universally accepted framework regarding how in situ measurements should be processed, there is a need for a general reference that describes the most commonly applied algorithms along with their strengths and weaknesses. Methods used to process data from bulk water probes, single-particle light-scattering spectrometers and cloud-imaging probes are reviewed herein, with emphasis on measurements of the ice phase. Particular attention is paid to how uncertainties, caveats, and assumptions in processing algorithms affect derived products since there is currently no consensus on the optimal way of analyzing data. Recommendations for improving the analysis and interpretation of in situ data include the following: establishment of a common reference library of individual processing algorithms, better documentation of assumptions used in these algorithms, development and maintenance of sustainable community software for processing in situ observations, and more studies that compare different algorithms with the same benchmark datasets.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1990

High rate information systems - Architectural trends in support of the interdisciplinary investigator

Data systems requirements in the Earth Observing System (EOS) Space Station Freedom (SSF) eras indicate increasing data volume, increased discipline interplay, higher complexity and broader data integration and interpretation. A response to the needs of the interdisciplinary investigator is proposed, considering the increasing complexity and rising costs of scientific investigation. The EOS Data Information System, conceived to be a widely distributed system with reliable communication links between central processing and the science user community, is described. Details are provided on information architecture, system models, intelligent data management of large complex databases, and standards for archiving ancillary data, using a research library, a laboratory and collaboration services.

Handley, Thomas H., Jr.↗
DOE OSTI2017

Skills and Knowledge for Data-Intensive Environmental Research

The scale and magnitude of complex and pressing environmental issues lend urgency to the need for integrative and reproducible analysis and synthesis, facilitated by data-intensive research approaches. However, the recent pace of technological change has been such that appropriate skills to accomplish data-intensive research are lacking among environmental scientists, who more than ever need greater access to training and mentorship in computational skills. Here, we provide a roadmap for raising data competencies of current and next-generation environmental researchers by describing the concepts and skills needed for effectively engaging with the heterogeneous, distributed, and rapidly growing volumes of available data. We articulate five key skills: (1) data management and processing, (2) analysis, (3) software skills for science, (4) visualization, and (5) communication methods for collaboration and dissemination. We provide an overview of the current suite of training initiatives available to environmental scientists and models for closing the skill-transfer gap.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRSDate not supplied

Joint ESA-NASA Multi-Mission Algorithm and Analysis Platform (MAAP)

The scientific community is faced with a need for greatly improved data sharing, analysis, visualization and advanced collaboration based firmly on open science principles. Recent and upcoming launches of new satellite missions with more complex and voluminous data, as well as the ever more urgent need to better understand the global carbon budget and related ecological processes, provided the immediate rational for the ESA-NASA Multi-mission Algorithm and Analysis Platform (MAAP). This highly collaborative joint project of ESA and NASA established a framework between ESA and NASA to share data, science algorithms and compute resources in order to foster and accelerate scientific research conducted by ESA and NASA EO data users. Presented to the public in October 2021, the current version of MAAP provides a common cloud-based platform with computing capabilities co-located with the data, a collaborative coding and analysis environment, and a set of interoperable tools and algorithms developed to support the estimation and visualization of global above-ground biomass. Data from the Global Ecosystem Dynamics Investigation (GEDI) mission on the International Space Station and the Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2) have been instrumental in the first products of MAAP including the first comprehensive map of Boreal above-ground Biomass and a current Global Biomass Harmonization Activity, but the platform is also being specifically designed to support the forthcoming ESA Biomass mission and incorporate data from the upcoming NASA-ISRO SAR (NISAR) mission. While these missions and the corresponding research which includes airborne, field, and calibration/validation data collection and analyses, provide a wealth of data and information relating to global biomass estimation, they also present data storing, processing and sharing challenges. The NISAR mission alone will produce about 80TB/day. These large data volumes present a challenge that would otherwise place accessibility limits on the scientific community and impact scientific progress. Other challenges being addressed by MAAP include: 1) Enabling researchers to easily discover, process, visualize and analyze large volumes of data from both agencies; 2) Providing a wide variety of data in the same coordinate reference frame to enable comparison, analysis, data evaluation, and data generation; 3) Providing a version-controlled science algorithm development environment that supports tools, co-located data and processing resources; and 4) Addressing intellectual property and sharing challenges related to collaborative algorithm development and sharing of data and algorithms. MAAP products can be explored on the MAAP Dashboard at https://earthdata.nasa.gov/maap-biomass or the joint platform entrance at scimaap.net. MAAP also can be accessed through individual NASA (https://maap-project.org) and ESA (https://esa-maap.org/) landing pages.

cloud computing↗
NASA NTRS2013

System for Contributing and Discovering Derived Mission and Science Data

A system was developed to provide a new mechanism for members of the mission community to create and contribute new science data to the rest of the community. Mission tools have allowed members of the mission community to share first order data (data that is created by the mission s process in command and control of the spacecraft or the data that is captured by the craft itself, like images, science results, etc.). However, second and higher order data (data that is created after the fact by scientists and other members of the mission) was previously not widely disseminated, nor did it make its way into the mission planning process.

Wallick, Michael N.↗
DOE OSTI2019

Data automation at light sources

Rapidly growing data volumes at light sources demand increasingly automated data collection, distribution, and analysis processes, in order to enable new scientific discoveries while not overwhelming finite human capabilities. We present here the case for automating and outsourcing light source science using cloud-hosted data automation and enrichment services, institutional computing resources, and high- performance computing facilities to provide cost-effective, scalable, and reliable implementations of such processes. We discuss three specific services that accomplish these goals for data distribution, automation, and transformation. In the first, Globus cloud-hosted data automation services are used to implement data capture, distribution, and analysis workflows for Advanced Photon Source and Advanced Light Source beamlines, leveraging institutional storage and computing. In the second, such services are combined with cloud-hosted data indexing and institutional storage to create a collaborative data publication, indexing, and discovery service, the Materials Data Facility (MDF), built to support a host of informatics applications in materials science. The third integrates components of the previous two projects with machine learning capabilities provided by the Data and Learning Hub for science (DLHub) to enable on-demand access to machine learning models from light source data capture and analysis workflows, and provides simplified interfaces to train new models on data from sources such as MDF on leadership scale computing resources. We draw conclusions about best practices for building next-generation data automation systems for future light sources.

Blaiszik, Ben↗
NASA NTRS1994

Telecommunications issues of intelligent database management for ground processing systems in the EOS era

Future NASA earth science missions, including the Earth Observing System (EOS), will be generating vast amounts of data that must be processed and stored at various locations around the world. Here we present a stepwise-refinement of the intelligent database management (IDM) of the distributed active archive center (DAAC - one of seven regionally-located EOSDIS archive sites) architecture, to showcase the telecommunications issues involved. We develop this architecture into a general overall design. We show that the current evolution of protocols is sufficient to support IDM at Gbps rates over large distances. We also show that network design can accommodate a flexible data ingestion storage pipeline and a user extraction and visualization engine, without interference between the two.

Touch, Joseph D.↗
DOE OSTI2026

RTN-117: Image Calibration and Instrument Signal Removal for the First Year of the LSST

The NSF-DOE Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) requires calibration products that provide uniform, stable, and accurate photometric and astrometric performance across the 3.2 gigapixel focal plane and throughout the 10-year survey. This paper details the algorithms and workflows used to produce instrument calibrations and remove instrumental artifacts for Data Preview 2 (DP2)---the first end-to-end processing demonstration using on-sky data with the LSST Camera (LSSTCam). We describe the verification, acceptance, and certification framework used to assess calibration quality and quantify residual systematics. We show baseline metrics on calibrated science images to evaluate the robustness of the calibration and instrument signature removal (ISR) data processing pipelines for DP2. Finally, we summarize the known limitations observed in DP2 production and outline expected algorithmic improvements for the first public LSST data release (Data Release~1, DR1).

79 ASTRONOMY AND ASTROPHYSICS↗