Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “science data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

The Dark Energy Survey Image Processing Pipeline

The Dark Energy Survey (DES) is a five-year optical imaging campaign with the goal of understanding the origin of cosmic acceleration. DES performs a ~5000 deg 2 survey of the southern sky in five optical bands (g, r, i, z, Y) to a depth of ~24th magnitude. Contemporaneously, DES performs a deep, time-domain survey in four optical bands (g, r, i, z) over ~27 deg 2 . DES exposures are processed nightly with an evolving data reduction pipeline and evaluated for image quality to determine if they need to be retaken. Difference imaging and transient source detection are also performed in the time domain component nightly. On a bi-annual basis, DES exposures are reprocessed with a refined pipeline and coadded to maximize imaging depth. In this paper, we describe the DES image processing pipeline in support of DES science, as a reference for users of archival DES data, and as a guide for future astronomical surveys.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2016

Accelerating Science Impact through Big Data Workflow Management and Supercomputing

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. ATLAS, one of the largest collaborations ever assembled in the the history of science, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. To manage the workflow for all data processing on hundreds of data centers the PanDA (Production and Distributed Analysis)Workload Management System is used. An ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF), is realizing within BigPanDA and megaPanDA projects. These projects are now exploring how PanDA might be used for managing computing jobs that run on supercomputers including OLCF’s Titan and NRC-KI HPC2. The main idea is to reuse, as much as possible, existing components of the PanDA system that are already deployed on the LHC Grid for analysis of physics data. The next generation of PanDA will allow many data-intensive sciences employing a variety of computing platforms to benefit from ATLAS experience and proven tools in highly scalable processing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Processing of Ice Cloud In Situ Data Collected by Bulk Water, Scattering, and Imaging Probes: Fundamentals, Uncertainties, and Efforts toward Consistency

Abstract In situ observations of cloud properties made by airborne probes play a critical role in ice cloud research through their role in process studies, parameterization development, and evaluation of simulations and remote sensing retrievals. To determine how cloud properties vary with environmental conditions, in situ data collected during different field projects processed by different groups must be used. However, because of the diverse algorithms and codes that are used to process measurements, it can be challenging to compare the results. Therefore it is vital to understand both the limitations of specific probes and uncertainties introduced by processing algorithms. Since there is currently no universally accepted framework regarding how in situ measurements should be processed, there is a need for a general reference that describes the most commonly applied algorithms along with their strengths and weaknesses. Methods used to process data from bulk water probes, single-particle light-scattering spectrometers and cloud-imaging probes are reviewed herein, with emphasis on measurements of the ice phase. Particular attention is paid to how uncertainties, caveats, and assumptions in processing algorithms affect derived products since there is currently no consensus on the optimal way of analyzing data. Recommendations for improving the analysis and interpretation of in situ data include the following: establishment of a common reference library of individual processing algorithms, better documentation of assumptions used in these algorithms, development and maintenance of sustainable community software for processing in situ observations, and more studies that compare different algorithms with the same benchmark datasets.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

Skills and Knowledge for Data-Intensive Environmental Research

The scale and magnitude of complex and pressing environmental issues lend urgency to the need for integrative and reproducible analysis and synthesis, facilitated by data-intensive research approaches. However, the recent pace of technological change has been such that appropriate skills to accomplish data-intensive research are lacking among environmental scientists, who more than ever need greater access to training and mentorship in computational skills. Here, we provide a roadmap for raising data competencies of current and next-generation environmental researchers by describing the concepts and skills needed for effectively engaging with the heterogeneous, distributed, and rapidly growing volumes of available data. We articulate five key skills: (1) data management and processing, (2) analysis, (3) software skills for science, (4) visualization, and (5) communication methods for collaboration and dissemination. We provide an overview of the current suite of training initiatives available to environmental scientists and models for closing the skill-transfer gap.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Data automation at light sources

Rapidly growing data volumes at light sources demand increasingly automated data collection, distribution, and analysis processes, in order to enable new scientific discoveries while not overwhelming finite human capabilities. We present here the case for automating and outsourcing light source science using cloud-hosted data automation and enrichment services, institutional computing resources, and high- performance computing facilities to provide cost-effective, scalable, and reliable implementations of such processes. We discuss three specific services that accomplish these goals for data distribution, automation, and transformation. In the first, Globus cloud-hosted data automation services are used to implement data capture, distribution, and analysis workflows for Advanced Photon Source and Advanced Light Source beamlines, leveraging institutional storage and computing. In the second, such services are combined with cloud-hosted data indexing and institutional storage to create a collaborative data publication, indexing, and discovery service, the Materials Data Facility (MDF), built to support a host of informatics applications in materials science. The third integrates components of the previous two projects with machine learning capabilities provided by the Data and Learning Hub for science (DLHub) to enable on-demand access to machine learning models from light source data capture and analysis workflows, and provides simplified interfaces to train new models on data from sources such as MDF on leadership scale computing resources. We draw conclusions about best practices for building next-generation data automation systems for future light sources.

Blaiszik, Ben↗
DOE OSTI2026

RTN-117: Image Calibration and Instrument Signal Removal for the First Year of the LSST

The NSF-DOE Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) requires calibration products that provide uniform, stable, and accurate photometric and astrometric performance across the 3.2 gigapixel focal plane and throughout the 10-year survey. This paper details the algorithms and workflows used to produce instrument calibrations and remove instrumental artifacts for Data Preview 2 (DP2)---the first end-to-end processing demonstration using on-sky data with the LSST Camera (LSSTCam). We describe the verification, acceptance, and certification framework used to assess calibration quality and quantify residual systematics. We show baseline metrics on calibrated science images to evaluate the robustness of the calibration and instrument signature removal (ISR) data processing pipelines for DP2. Finally, we summarize the known limitations observed in DP2 production and outline expected algorithmic improvements for the first public LSST data release (Data Release~1, DR1).

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2016

Scattering of a laser beam on a wet blood smear and measurement of red cell size distribution

We report an automated laser system that allows the red cell size distribution to be measured. Experiments are performed on laser light scattering by a suspension of oriented red blood cells (a wet blood smear). Based on an analysis of the angular distribution of light intensity in the diffraction pattern, we have restored the red cell size distribution. The average diameter of a red blood cell is determined with an error of less than 1%, and the spread of red blood cells in size – with an error of about 20%. We discuss the problems of photometry and processing of diffraction patterns, preparing blood samples and data processing algorithms, including methods for solving the inverse scattering problem. (laser biophotonics)

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2021

MIGHTEE: total intensity radio continuum imaging and the COSMOS/XMM-LSS Early Science fields

ABSTRACT MIGHTEE is a galaxy evolution survey using simultaneous radio continuum, spectropolarimetry, and spectral line observations from the South African MeerKAT telescope. When complete, the survey will image ∼20 deg2 over the COSMOS, E-CDFS, ELAIS-S1, and XMM-Newton Large Scale Structure field (XMM-LSS) extragalactic deep fields with a central frequency of 1284 MHz. These were selected based on the extensive multiwavelength data sets from numerous existing and forthcoming observational campaigns. Here, we describe and validate the data processing strategy for the total intensity continuum aspect of MIGHTEE, using a single deep pointing in COSMOS (1.6 deg2) and a three-pointing mosaic in XMM-LSS (3.5 deg2). The processing includes the correction of direction-dependent effects, and results in thermal noise levels below 2 $\mathrm{\mu }$Jy beam−1 in both fields, limited in the central regions by classical confusion at ∼8 arcsec angular resolution, and meeting the survey specifications. We also produce images at ∼5 arcsec resolution that are ∼3 times shallower. The resulting image products form the basis of the Early Science continuum data release for MIGHTEE. From these images we extract catalogues containing 9896 and 20 274 radio components in COSMOS and XMM-LSS, respectively. We also process a close-packed mosaic of 14 additional pointings in COSMOS and use these in conjunction with the Early Science pointing to investigate methods for primary beam correction of broad-band radio images, an analysis that is of relevance to all full-band MeerKAT continuum observations, and wide-field interferometric imaging in general. A public release of the MIGHTEE Early Science continuum data products accompanies this article.

Heywood, I. (ORCID:0000000168645057)↗
DOE OSTI2023

Artificial Intelligence in Nuclear Physics

Artificial Intelligence (AI) and Machine Learning (ML) are rapidly developing fields providing data-driven algorithms to predict, classify, and make decisions based on data. Nuclear Physics Research is data-driven and AI/ML techniques have been implemented for experiment and accelerator control, in theoretical applications, and in data processing and analysis. These algorithms open possibilities for automation, thereby augmenting human capabilities. Additionally, Open Science is enabled by simultaneous analyses of multiple data sources, leading to scientific knowledge. This talk will summarize current applications of AI/ML in nuclear physics, as well as accelerator applications, and will cover upcoming initiatives and research in AI/ML.

Jeske, Torri↗
DOE OSTI2016

Blending Education and Polymer Science: Semiautomated Creation of a Thermodynamic Property Database

Structured databases of chemical and physical properties play a central role in the everyday research activities of scientists and engineers. In materials science, researchers and engineers turn to these databases to quickly query, compare, and aggregate various properties, thereby allowing for the development or application of new materials. The vast majority of these databases have been generated manually, through decades of labor-intensive harvesting of information from the literature, yet while there are many examples of commonly used databases, a significant number of important properties remain locked within the tables, figures, and text of publications. The question addressed in our work is whether and to what extent the process of data collection can be automated. Students of the physical sciences and engineering are often confronted with the challenge of finding and applying property data from the literature, and a central aspect of their education is to develop the critical skills needed to identify such data and discern their meaning or validity. To address shortcomings associated with automated information extraction while simultaneously preparing the next generation of scientists for their future endeavors, we developed a novel course-based approach in which students develop skills in polymer chemistry and physics and apply their knowledge by assisting with the semiautomated creation of a thermodynamic property database.

Collaborative/Cooperative Learning↗
DOE OSTI2020

Atomic kinetics of laboratory photoionized plasmas relevant to astrophysics. Final report

The major goal of this project was to study the atomic and radiation physics of photoionized plasmas driven by a broadband x-ray flux and its impact on plasma x-ray heating and ionization, emissivity and opacity, and radiation-hydrodynamics. This problem is relevant to the physics of accretion disks surrounding black holes, x-ray binaries, and active galactic nuclei in astrophysics. In addition, it is also important for understanding non-equilibrium laboratory plasmas driven by a distribution of photons. To this end, and motivated by previous and ongoing gas-cell experiments performed by the PI at the Z facility of Sandia National Laboratories, we developed a new experimental platform at the 1MA Zebra pulsed power accelerator of the University of Nevada, Reno based on using a gas jet produced by a supersonic nozzle. The gas jet is turned into a photoionized plasma driven by the x- ray flux from a wire-array z-pinch. Establishing this new experimental platform at Zebra was a major goal of this project. In addition, we have also dedicated effort to continue complementary experiments at the Z facility using the gas-cell photoionized plasma set up with support from the Fundamental Science Program at Z as well as to perform data processing and analysis, and theory and modeling simulations, including electron kinetics and radiation-hydrodynamics. Since we seek to test and establish what physics models are needed to describe photoionized plasma guided by experimental observation, these complementary experimental efforts were central to achieving the goals of this project.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

NANT Site - ASSIST / Processed Data

This dataset contains processed summary data from the ASSIST operated by NOAA Physical Science Laboratory on Nantucket Island for WFIP3.

17 WIND ENERGY↗
DOE OSTI2017

HEP Analysis Ecosystem Workshop Report

Over the past 20 years the HEP community has developed and gravitated around an analysis ecosystem centered on ROOT. ROOT and its ecosystem both dominate HEP analysis and impact the full event processing chain, providing foundation libraries, I/O services etc. that have prevalence in the field. The analysis tools landscape is however evolving in ways that can have a durable impact on the analysis ecosystem and a strong influence on the analysis and core software landscape a decade from now, a timescale currently in an intensive planning round with the HEP Software Foundation(HSF) Community White Paper process. Data intensive analysis is growing in importance in other sciences and in the wider world. Powerful tools and new development initiatives, both within our field and in the wider open source community, have emerged. Creative developers have an ever more powerful open source toolkit available and are applying it towards innovations that leverage both open source and the ROOT ecosystem. ROOT itself is approaching a major re-engineering with ROOT 7, leveraging the powerful evolution of C++, with a major overhaul in its interfaces as seen both by its users and by satellite tools in the ROOT ecosystem.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

ARM Aerosol Measurement Science Group 2019 Strategic Planning Workshop Report

This report summarizes the results of a U.S. Department of Energy Atmospheric Radiation Measurement (ARM) user facility workshop held in November 2019 to advance a science-based strategy for ARM’s aerosol measurement program. This was the second such workshop since the Aerosol Measurement Science Group (AMSG) was chartered in 2015 to enhance coordination of ARM observations of aerosols and atmospheric trace gases with the needs of ARM users. The results presented here reflect the AMSG’s focus in recent years on science-based strategies that will contribute to the increased use of ARM data to fulfill its mission of improving process representations and predictability in climate models. Sessions held during the workshop range from interfacing with models through aerosol sampling strategies to calibration protocols and data products. The strategies set forth here were also developed to be directly relevant to ARM’s updated Decadal Vision. The AMSG workshops have been designed to recommend actions that will enable ARM to evolve and continue to meet its mission. To that end, the AMSG will develop an actionable plan from the recommendations outlined here. Some are well defined and can reasonably be accomplished in the short term. Others are less definite or of a larger scope that calls for a longer-term implementation. Further discussion will be required to develop and prioritize actionable items related to such areas. Task teams comprising the appropriate expertise and perspective from the AMSG and other members of the community will be formed to achieve this outcome. Some particular topics are recognized as high priority, so plans are underway to develop task teams and to hold follow-on discussions to address them. Four areas currently being considered for short, focused discussion are 1) aerosol measurements on the North Slope of Alaska, 2) improving data usability for modeling, 3) strategies for advancing remote sensing, vertical profiling, and distributed measurements of aerosols, and 4) aerosol sampling strategies at existing ARM sites to provide intensive modes of operation to promote data usage for process and modeling studies.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

ARM Aerosol Measurement Science Group 2019 Strategic Planning Workshop Report

This report summarizes the results of a U.S. Department of Energy Atmospheric Radiation Measurement (ARM) user facility workshop held in November 2019 to advance a science-based strategy for ARM’s aerosol measurement program. This was the second such workshop since the Aerosol Measurement Science Group (AMSG) was chartered in 2015 to enhance coordination of ARM observations of aerosols and atmospheric trace gases with the needs of ARM users. The results presented here reflect the AMSG’s focus in recent years on science-based strategies that will contribute to the increased use of ARM data to fulfill its mission of improving process representations and predictability in climate models. Sessions held during the workshop range from interfacing with models through aerosol sampling strategies to calibration protocols and data products. The AMSG workshops have been designed to recommend actions that will enable ARM to evolve and continue to meet its mission. To that end, the AMSG will develop an actionable plan from the recommendations outlined here. Some are well defined and can reasonably be accomplished in the short term. Others are less definite or of a larger scope that calls for a longer-term implementation. Further discussion will be required to develop and prioritize actionable items related to such areas. Task teams comprising the appropriate expertise and perspective from the AMSG and other members of the community will be formed to achieve this outcome. Some particular topics are recognized as high priority, so plans are underway to develop task teams and to hold follow-on discussions to address them. Four areas currently being considered for short, focused discussion are 1) aerosol measurements on the North Slope of Alaska, 2) improving data usability for modeling, 3) strategies for advancing remote sensing, vertical profiling, and distributed measurements of aerosols, and 4) aerosol sampling strategies at existing ARM sites to provide intensive modes of operation to promote data usage for process and modeling studies.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Pattern-matching indexing of Laue and monochromatic serial crystallography data for applications in materials science

Serial crystallography data can be challenging to index, as each frame is processed individually, rather than being processed as a whole like in conventional X-ray single-crystal crystallography. An algorithm has been developed to index still diffraction patterns arising from small-unit-cell samples. The algorithm is based on the matching of reciprocal-lattice vector pairs, as developed for Laue microdiffraction data indexing, combined with three-dimensional pattern matching using a nearest-neighbors approach. As a result, large-bandpass data (e.g. 5–24 keV energy range) and monochromatic data can be processed, the main requirement being prior knowledge of the unit cell. Furthermore, angles calculated in the vicinity of a few theoretical and experimental reciprocal-lattice vectors are compared, and only vectors with the highest number of common angles are selected as candidates to obtain the orientation matrix. Global matching on the entire pattern is then checked. Four indexing options are available, two for the ranking of the theoretical reciprocal-lattice vectors and two for reducing the number of possible candidates. Furthermore, the algorithm has been used to index several data sets collected under different experimental conditions on a series of model samples. Additionally, knowing the crystallographic structure of the sample and using this information to rank the theoretical reflections based on the structure factors helps the indexing of large-bandpass data for the largest-unit-cell samples. For small-bandpass data, shortening the candidate list to determine the orientation matrix should be based on matching pairs of reciprocal-lattice vectors instead of triplet matching.

36 MATERIALS SCIENCE↗
DOE OSTI2019

High-Resolution Mass Spectrometry Quantification: Impact of Differences in Data Processing of Centroid and Continuum Data

High-resolution mass spectrometry (HRMS) in full scan mode acquires all ions present in the sample of interest offering a lot of qualitative information. This, in combination with the improved performance of the new generation HRMS systems, triggers more (bio) analysts to switch from triple quad MS systems to HRMS for quantitative analysis. Quantitative processing of HRMS data is performed based on narrow mass extraction windows rather than on nominal mass product ion chromatograms (SRM or MRM). Optimal processing of HRMS data requires different considerations and software tools and can have an impact on data processing and final results. The selection of centroid versus continuum/profile data for processing, selection of the optimal narrow mass extraction window, using theoretical versus measured accurate mass for the extraction of the ion chromatograms as well as differences in calculations and data handling residing in the different vendor software packages are tackled in the presented manuscript. These differences are illustrated on HRMS data acquired for the same plasma samples on three different platforms, i.e., a Sciex QToF, a Waters QToF, and a Thermo Orbitrap system, and processed in four different software packages, i.e., Sciex Analyst® TF, Waters Masslynx, Waters Unifi, and Thermo Xcalibur. The impact of these differences on quantitative HRMS performance was evaluated on calibration curves of eight small molecule compounds in plasma using four different ways of processing. Simple guidelines are provided for the selection of an optimal mass extraction window for continuum and centroided data. .

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2017

A Full Mesh ATCA-based General Purpose Data Processing Board (Pulsar II)

The Pulsar II is a custom ATCA full mesh enabled FPGA-based processor board which has been designed with the goal of creating a scalable architecture abundant in flexible, non-blocking, high bandwidth interconnections. The design has been motivated by silicon-based tracking trigger needs for LHC experiments. In this technical memo we describe the Pulsar II hardware and its performance, such as the performance test results with full mesh backplanes from different vendors, how the backplane is used for the development of low-latency time-multiplexed data transfer schemes and how the inter-shelf and intra-shelf synchronization works.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗