Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,261 records · Page 70

Particle-Tracking Proton Computed Tomography—Data Acquisition, Preprocessing, and Preconditioning

Proton CT (pCT) is a promising new imaging technique that can reconstruct relative stopping power (RSP) more accurately than x-ray CT in each cubic millimeter voxel of the patient. This, in turn, will result in better proton range accuracy and, therefore, smaller planned tumor volumes (PTV). The hardware description and some reconstructed images have previously been reported. In a series of two contributions, we focus on presenting the software algorithms that convert pCT detector data to the final reconstructed pCT images for application in proton treatment planning. There were several options on how to accomplish this, and we will describe our solutions at each stage of the data processing chain. In the first paper of this series, we present the data acquisition with the pCT tracking and energy-range detectors and how the data are preprocessed, including the conversion to the well-formatted track information from tracking data and water-equivalent path length from the data of a calibrated multi-stage energy-range detector. These preprocessed data are then used for the initial image formation with an FDK cone-beam CT algorithm. The output of data acquisition, preprocessing, and FDK reconstruction is presented along with illustrative imaging results for two phantoms, including a pediatric head phantom. The second paper in this series will demonstrate the use of iterative solvers in conjunction with the superiorization methodology to further improve the images resulting from the upfront FDK image reconstruction and the implementation of these algorithms on a hybrid CPU/GPU computer cluster.

42 ENGINEERING↗

Characterization and operation optimization of large aperture optical interferometers using binary pseudorandom array test standards

Recently, a technique for calibration of the Modulation Transfer Function (MTF) of a broad variety of metrology instrumentation has been established. The technique is based on test samples structured according to binary pseudorandom (BPR) one-dimensional sequences and two-dimensional arrays. The inherent power spectral density of BPR gratings and arrays, has a deterministic white-noise-like character that allows a direct determination of the MTF with a uniform sensitivity over the entire spatial frequency range and field-of-view of an instrument. As such, the BPR samples satisfy the characteristics of a test standard: Functionality, ease of specification and fabrication, reproducibility, and low sensitivity to manufacturing error. Here we discuss our recent developments working with support of the U.S. Department of Energy on industrialization of the technique. The goal is to develop affordable BPR test samples, application procedures, and data processing software, suitable for thorough characterization of optical interferometers and microscopes, as well as x-ray, electron (scanning and transmission), and atomic force microscopes. We report on the development of BPR array test samples optimized for advanced characterization (including the instrumental MTF and aberrations) and operation optimization of large aperture optical interferometers. We describe the sample fabrication process and tests to verify the compliance to desired surface topography. The data acquisition and analysis procedures for application of the technique for precise focusing of Fizeau interferometer are discussed in detail.

Yashchuk, VV↗

pnnl/deimos_gui

The DEIMoS Graphical User Interface was created as an interface for DEIMoS: Data Extraction for Integrated Multidimensional Spectrometry. DEIMoS is a Python package to process data from mass spectrometry instrument developed at PNNL.

Oostrom, Marjolein↗

Soil microbial function and fungal community data along an ectomycorrhizal basal area gradient, Vermont, May 2019

Mycorrhizal fungi can affect soil organic matter cycling through several mechanisms including priming, nutrient competition, and direct enzyme production. Differences in nutrient foraging strategies between ectomycorrhizal (EcM) and arbuscular mycorrhizal (AM) fungi produce divergent belowground dynamics: where EcM can take up organic nitrogen and directly break down SOM by producing enzymes, AM fungi are limited to scavenging mineral N. EcM-associated tree species also have leaf litter with relatively higher ratios of carbon to nitrogen (C:N), and belowground saprotrophic communities more dominated by fungi. Consequently, free-living microbes in EcM-dominated soils should experience nitrogen limitation, with subsequent increases in enzyme production and decreased carbon use efficiency (CUE). However, the relative importance of the effects of substrate quality and fungal community composition on enzyme production and CUE are unclear. To assess this distinction, we sampled the organic horizon and 10 cm of the mineral horizon in northern temperate forest soils along a gradient of EcM dominance (see SCxTS_DBH.csv for full tree species information). We characterized fungal community composition by measuring EcM relative abundances from extracted fungal DNA (To run raw ASV data through R code, use SCxTS_ASV_taxonomy.csv, and R code ITS_processing.Rmd; to see processed data with EcM relative abundance, use file SCxTS_sum_afitch_Feb18.20.csv) and the fungal to bacterial ratios (see data file SCxTS_plfa.csv) from phospholipid fatty acid analysis. Soil microbial functions were measured as potential activities of five hydrolytic and two oxidative enzymes ( To run raw data through R code us SCxTS_metadata.csv, hydrolytic_oxidative_raw.csv, MUBraw.csv, Enzyme_soilweight.csv, and incubationtime.csv, and R code SCxTS_enzymes.Rmd; to use Enzyme data ready to run through statistics code, use file Enzyme.data.csv) and microbial carbon use efficiency (see file SCxTS_CUE.csv and for full calculations file 18O_CUEcalcs_SCxTS_2019.xlsx). We assessed soil substrate quality as the soil carbon:nitrogen ratio (See file Organic_CN.csv). Statistical analysis can be run using R code from file SCxTS_analysis.Rmd.

54 ENVIRONMENTAL SCIENCES↗

Panorama 360 (Final Report)

This is the final technical report for the DOE-funded Panorama 360 project. Panorama 360 provided a resource for the collection, analysis, and sharing of performance data about end-to-end scientific workflows executing on DOE facilities. The work focused on workflows that include experimental data generation at DOE facilities. The main activities of Panorama 360 include the development of: 1. A distributed repository that stores different types of workflow execution data (e.g., point and time series performance traces at fine- and coarse-grained levels); 2. A set of open-source data capture, curation, and publishing tools fully integrated with a state-of-the-art workflow management system that automates data ingestion to the repository and enables users to discover, query, and process data from the repository; 3. A set of analysis algorithms and machine learning based tools to perform analysis and characterization of the gathered data, which can be used to detect anomalous performance or system faults; and 4. Best practices and recommendations for workflow evaluation, analysis, execution, and architectures.

97 MATHEMATICS AND COMPUTING↗

Panorama 360 (Final Report)

This final technical report from the lead institution, USC grant #DE-SC0012636, serves as the final technical report for collaborative institution UNC-CH grant #DE-SC0012390. The goal was to develop a repository and associated capabilities for data collection, ingestion, and analysis for a broad class of DOE applications that span experimental and simulation science workflows. In particular, this work focuses on workflows that include experimental data generation at DOE facilities. The main activities of Panorama 360 include the development of: (1) A distributed repository that stores different types of workflow execution data (e.g., point and time series performance traces at fine- and coarse-grained levels); (2) A set of open-source data capture, curation, and publishing tools fully integrated with a state-of-the-art workflow management system that automates data ingestion to the repository and enables users to discover, query, and process data from the repository; (3) A set of analysis algorithms and machine learning based tools to perform analysis and characterization of the gathered data, which can be used to detect anomalous performance or system faults; and (4) Best practices and recommendations for workflow evaluation, analysis, execution, and architectures.

97 MATHEMATICS AND COMPUTING↗

LANL HPC Data Center Monitoring (Summer 2022 Talk) [Slides]

HPC Monitoring is tasked with the understanding of everything that happens in the Data Center. They get to that understanding by: collecting logs and metrics, processing data, analyzing data, displaying information, and receiving alerts on events. Their ultimate (and probably unreachable) goal is a holistic view of the Data Center and an understanding of how everything is correlated.

97 MATHEMATICS AND COMPUTING↗

Quarterly VAP Report - April to June

The purpose of this report is to provide a concise status update for value-added products (VAP) implemented by the Atmospheric Radiation Measurement Climate Research Facility. The report is divided into the following sections: (1) new VAPs for which development has begun, (2) progress on existing VAPs, (3) future VAPs that have been recently approved, (4) other work that leads to a VAP, and (5) top requested VAPs from the Data Archive. New information is highlighted in blue text. New information about processed data by the developer is highlighted in red text. The upcoming milestones and dates are highlighted in green.

54 ENVIRONMENTAL SCIENCES↗

PHASE I MATERIALS PROPERTY DATABASE DEVELOPMENT FOR ASME CODES AND STANDARDS

To support the ASME Boiler and Pressure Vessel Codes and Standard (BPVC) in modern information era, development of a web-based materials property database is initiated under the supervision of ASME Committee on Materials. To achieve efficiency, the project heavily draws upon experience from development of the Gen IV Materials Handbook and the Nuclear System Materials Handbook. The effort is divided into two phases. Phase I is planned to deliver a materials data file warehouse that offers a depository for various files containing raw data and background information, and Phase II will provide a relational digital database that provides advanced features facilitating digital data processing and management. Population of the database will start with materials property data for nuclear applications and expand to data covering the entire ASME Code and Standards including the piping codes as the database structure is continuously optimized. The ultimate goal of the effort is to establish a sound cyber infrastructure that support ASME Codes and Standards development and maintenance.

ASME↗

MRT (Medical Reporting Tracker)

Medical Report Tracker (MRT) was created to respond to COVID19 data, process and analytic needs of our clinic staff. The report tracker captures data around symptoms and contacts, test results, and last work date and location. The MRT is built using Nintex Online, SharePoint Online and Power BI/Excel.

Devitt, Carrie↗

Towards Hybrid Online On-Demand Querying of Realtime Data with Stateful Complex Event Processing

Emerging Big Data applications in areas like e-commerce and energy industry require both online and on-demand queries to be performed over vast and fast data arriving as streams. These present novel challenges to Big Data management systems. Complex Event Processing (CEP) is recognized as a high performance online query scheme which in particular deals with the velocity aspect of the 3-V’s of Big Data. However, traditional CEP systems do not consider data variety and lack the capability to embed ad hoc queries over the volume of data streams. In this paper, we propose H2O, a stateful complex event processing framework, to support hybrid online and on-demand queries over realtime data. We propose a semantically enriched event and query model to address data variety. A formal query algebra is developed to precisely capture the stateful and containment semantics of online and on-demand queries. We describe techniques to achieve the interactive query processing over realtime data featured by efficient online querying, dynamic stream data persistence and on-demand access. The system architecture is presented and the current implementation status reported.

big data↗

The Hyper Suprime-Cam software pipeline

Here in this article, we describe the optical imaging data processing pipeline developed for the Subaru Telescope’s Hyper Suprime-Cam (HSC) instrument. The HSC Pipeline builds on the prototype pipeline being developed by the Large Synoptic Survey Telescope’s Data Management system, adding customizations for HSC, large-scale processing capabilities, and novel algorithms that have since been reincorporated into the LSST codebase. While designed primarily to reduce HSC Subaru Strategic Program (SSP) data, it is also the recommended pipeline for reducing general-observer HSC data. The HSC pipeline includes high-level processing steps that generate coadded images and science-ready catalogs as well as low-level detrending and image characterizations.

79 ASTRONOMY AND ASTROPHYSICS↗

Enriching the physics program of the CMS experiment via data scouting and data parking

Specialized data-taking and data-processing techniques were introduced by the CMS experiment in Run 1 of the CERN LHC to enhance the sensitivity of searches for new physics and the precision of standard model measurements. These techniques, termed data scouting and data parking, extend the data-taking capabilities of CMS beyond the original design specifications. The novel data-scouting strategy trades complete event information for higher event rates, while keeping the data bandwidth within limits. Data parking involves storing a large amount of raw detector data collected by algorithms with low trigger thresholds to be processed when sufficient computational power is available to handle such data. The research program of the CMS Collaboration is greatly expanded with these techniques. The implementation, performance, and physics results obtained with data scouting and data parking in CMS over the last decade are discussed in this Report, along with new developments aimed at further improving low-mass physics sensitivity over the next years of data taking.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Leveraging the checkpoint-restart technique for optimizing CPU efficiency of ATLAS production applications on opportunistic platforms

Data processing applications of the ATLAS experiment, such as event simulation and reconstruction, spend considerable amount of time in the initialization phase. This phase includes loading a large number of shared libraries, reading detector geometry and condition data from external databases, building a transient representation of the detector geometry and initializing various algorithms and services. In some cases the initialization step can take as long as 10-15 minutes. Such slow initialization has a significant negative impact on overall CPU efficiency of the production job, especially when the job is executed on opportunistic, often short-lived, resources such as commercial clouds or volunteer computing. In order to improve this situation, we can take advantage of the fact that ATLAS runs large numbers of production jobs with similar configuration parameters (e.g. jobs within the same production task). This allows us to checkpoint one job at the end of its configuration step and then use the generated checkpoint image for rapid startup of thousands of production jobs. By applying this technique we can bring the initialization time of a job from tens of minutes down to just a few seconds. In addition to that we can leverage container technology for restarting checkpointed applications on the variety of computing platforms, in particular of platforms different from the one on which the checkpoint image was created. We will describe the mechanism of creating checkpoint images of Geant4 simulation jobs with AthenaMP (the multi-process version of the ATLAS data simulation, reconstruction and analysis framework Athena) and the usage of these images for running ATLAS Simulation production jobs on volunteer computing resources (ATLAS@Home) and on Supercomputers.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

ORIGAMI Automator Primer. Automated ORIGEN Source Terms and Spent Fuel Storage Pool Analysis

Source terms and spent nuclear fuel (SNF) storage pool decay heat load analyses for operating nuclear power plants require a large number of Oak Ridge Isotope Generation and Depletion (ORIGEN) calculations. SNF source term calculations also require a significant amount of bookkeeping to track quantities such as core and assembly operating histories, spent fuel pool (SFP) residence times, heavy metal masses, and enrichments. The ORIGEN Assembly Isotopics (ORIGAMI) module in the SCALE code system provides a simple scheme for entering these data. However, given the large scope of the analysis, extensive scripting is necessary to convert formats and process data to create thousands of ORIGAMI input files (one per assembly) and to process the results into formats readily usable by follow-on analysis tools. This primer describes a project within the SCALE Fulcrum graphical user interface (GUI) called ORIGAMI Automator that was developed to automate the scripting and bookkeeping in large-scale source term analyses. The ORIGAMI Automator enables the analyst to (1) easily create, view, and edit the reactor site and assembly information, (2) automatically create and run ORIGAMI inputs, and (3) analyze the results from ORIGAMI. ORIGAMI Automator uses the standard ORIGEN binary concentrations files produced by ORIGAMI, with concentrations available at all time points in each assembly’s life. The GUI plots results such as mass, concentration, activity, and decay heat using a powerful new ORIGEN Post-Processing Utility for SCALE (OPUS) GUI component. This document includes a description and user guide for the GUI, a step-by-step tutorial for a simplified scenario, and appendices that document the file structures used.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

AmeriFlux BASE Flux/Met Data QA/QC and Processing (AMF-BASE-QAQC) v1.0.0

The AmeriFlux BASE Flux/Met Data QA/QC and Processing (AMF-BASE-QAQC) code provides tools to review and prepare continuous flux/met data submitted to the AmeriFlux Management Project for publication as the AmeriFlux BASE data product. The code provides 3 core functionalities: Format QA/QC assesses submitted data files for compliance with the required submission format; Data QA/QC assesses the data quality; BASE Publish prepares the data for publication.

Christianson, Danielle↗

Quantitative probing of the fast particle motion during the solidification of battery electrodes

Synchrotron x-ray tomography represents one of the most powerful imaging techniques that can visualize the three-dimensional microstructural complexity with a high spatial resolution. To date, however, the temporal resolution of this technique has been handicapped by the intrinsic limitations in the data acquisition speed and the lack of a suitable data processing method. The present study aims to fundamentally resolve this challenge by implementing a data analysis method that is based on differentiating discrete tomography reconstruction from a continuous scan reconstruction. With this methodological development, we capture and quantify the particle motion during the solidification of battery electrodes and reveal the statistics of the dynamically evolving motion in the drying process, which has been challenging to resolve. We discover that the particle motion exhibits a strong dependence on its geometric location within a drying electrode. Our results also imply that the final electrode quality can be controlled by balancing the solvent evaporation rate and the particle mobility in the region close to the drying surface.

25 ENERGY STORAGE↗