Automatic Data Processing and Data Display System for the HERMES III Accelerator.
Abstract not provided.
Coffey, Sean K↗
SEARCH · Engineering Papers
Results for “Data Processing”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Buoy - California - Wind Sentinel (130), Morro Bay - Processed Data
Processed data for various datasets.
17 WIND ENERGY↗
MCPC Friction Stir Welding (FSW) Process Data
Processing parameters and machine log data for the MCPC LDRD Agile investment is collected material samples processed. This dataset captures the selected processing parameters, machine logs captured during material processing, and descriptions of how characterization samples were extracted from processed plates of material. The collect characterization data is captured in other datasets.
316 Stainless Steel↗
Lidar - California - Leosphere Windcube 866 (130), Morro Bay - Processed Data
Processed daily lidar dataset.
17 WIND ENERGY↗
Performance Analysis of Data Processing in Distributed File Systems with Near Data Processing
In the era of big data, the escalating volume and velocity of data generation pose significant challenges in data processing. Traditional systems like Spark and Hadoop manage the increasing amount and velocity of data by improving data placement and processing speeds. However, they face inherent limitations due to the essential data movement required for processing. In this paper, we explore the Skyhook framework, a novel extension of the Ceph distributed system, which significantly reduces the need for data movement. We present an extensive case study using the Skyhook framework, applying it with the TPC-H and K-means clustering algorithms. More specifically, we leverage the TPC-H benchmark to distinguish between CPU-intensive and I/O-intensive tasks. We explore the integration of K-means clustering into SQL, coupled with a near-data processing system to offload the computational burden of the K-means clustering algorithm to storage nodes. We conduct a comprehensive performance evaluation of distributed data processing applications across three processing approaches: traditional layout (baseline), optimized layout, and near-data processing. Additionally, we introduce the use of the FIO tool to simulate real-world system workloads, enabling the measurement of performance metrics such as average latency and CPU utilization. Our research is a significant advance in understanding how to optimize data processing systems to meet the demands of the modern data landscape.
Hou, Shiyue↗
Sandwiched Rényi divergence satisfies data processing inequality
Sandwiched (quantum) α-Rényi divergence has been recently defined in the independent works of Wilde et al. [“Strong converse for the classical capacity of entanglement-breaking channels,” preprint http://arxiv.org/abs/arXiv:1306.1586 (2013)] and Müller-Lennert et al. [“On quantum Rényi entropies: a new definition, some properties and several conjectures,” preprint http://arxiv.org/abs/arXiv:1306.3142v1 (2013)]. This new quantum divergence has already found applications in quantum information theory. Here we further investigate properties of this new quantum divergence. In particular, we show that sandwiched α-Rényi divergence satisfies the data processing inequality for all values of α > 1. Moreover we prove that α-Holevo information, a variant of Holevo information defined in terms of sandwiched α-Rényi divergence, is super-additive. Our results are based on Hölder's inequality, the Riesz-Thorin theorem and ideas from the theory of complex interpolation. We also employ Sion's minimax theorem.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Data processing workflow for time of flight polarized neutrons inelastic measurements
We discuss the data processing workflow for polarized neutron scattering measurements performed at HYSPEC spectrometer at the Spallation Neutron Source, Oak Ridge National Laboratory. The effects of the focusing Heusler crystal polarizer and the wide-angle supermirror transmission polarization analyzer are added to the data processing flow of the non-polarized case. The implementation is done using the Mantid software package.
75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
Automated data processing architecture for the Gemini Planet Imager Exoplanet Survey
The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.
79 ASTRONOMY AND ASTROPHYSICS↗
ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing: ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing
Managing the large volumes of data produced by emerging scientific and engineering simulations running on leadership-class resources has become a critical challenge. The data have to be extracted off the computing nodes and transported to consumer nodes so that it can be processed, analyzed, visualized, archived, and so on. Several recent research efforts have addressed data-related challenges at different levels. One attractive approach is to offload expensive input/output operations to a smaller set of dedicated computing nodes known as a staging area. However, even using this approach, the data still have to be moved from the staging area to consumer nodes for processing, which continues to be a bottleneck. In this paper, we investigate an alternate approach, namely moving the data-processing code to the staging area instead of moving the data to the data-processing code. Specifically, we describe the ActiveSpaces framework, which provides (1) programming support for defining the data-processing routines to be downloaded to the staging area and (2) runtime mechanisms for transporting codes associated with these routines to the staging area, executing the routines on the nodes that are part of the staging area, and returning the results. We also present an experimental performance evaluation of ActiveSpaces using applications running on the Cray XT5 at Oak Ridge National Laboratory. Finally, we use a coupled fusion application workflow to explore the trade-offs between transporting data and transporting the code required for data processing during coupling, and we characterize sweet spots for each option.
97 MATHEMATICS AND COMPUTING↗
HEP Software Foundation Community White Paper Working Group - Data Processing Frameworks
Data processing frameworks are an essential part of HEP experiments' software stacks. Frameworks provide a means by which code developers can undertake the essential tasks of physics data processing, accessing relevant inputs and storing their outputs, in a coherent way without needing to know the details of other domains. Frameworks provide essential core services for developers and help deliver a configurable working application to the experiments' production systems. Modern HEP processing frameworks are in the process of adapting to a new computing landscape dominated by parallel processing and heterogeneity, which pose many questions regarding enhanced functionality and scaling that must be faced without compromising the maintainability of the code. In this paper we identify a program of work that can help further clarify the key concepts of frameworks for HEP and then spawn R&D activities that can focus the community's efforts in the most efficient manner to address the challenges of the upcoming experimental program.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
FPGA-based computing system for processing data in size, weight, and power constrained environments
Technologies that are well-suited for use in size, weight, and power (SWAP)-constrained environments are described herein. A host controller dispatches data processing instructions to hardware acceleration engines (HAEs) of one or more field programmable gate arrays (FPGAs) and further dispatches data transfer instructions to a memory controller, such that the HAEs perform processing operations on data stored in local memory devices of the HAEs in parallel with other data being transferred from external memory devices coupled to the FPGA(s) to the local memory devices.
Napier, Matthew↗
Data processing methods and data acquisition for samples larger than the field of view in parallel-beam tomography
Parallel-beam tomography systems at synchrotron facilities have limited field of view (FOV) determined by the available beam size and detector system coverage. Scanning the full size of samples bigger than the FOV requires various data acquisition schemes such as grid scan, 360-degree scan with offset center-of-rotation (COR), helical scan, or combinations of these schemes. Though straightforward to implement, these scanning techniques have not often been used due to the lack of software and methods to process such types of data in an easy and automated fashion. The ease of use and automation is critical at synchrotron facilities where using visual inspection in data processing steps such as image stitching, COR determination, or helical data conversion is impractical due to the large size of datasets. Here, we provide methods and their implementations in a Python package, named Algotom, for not only processing such data types but also with the highest quality possible. The efficiency and ease of use of these tools can help to extend applications of parallel-beam tomography systems.
36 MATERIALS SCIENCE↗
Data Processing Package for Cyclic Integrated Reversible Bending Fatigue Testing
A data processing software package has been introduced. The package was developed using MATLAB with the aid of the Curve Fitting Toolbox. The package is made up of four modules: pre-processing, data processing for static testing, data processing for monitoring, and data processing for measurements. CIRFT data are structured with multiple levels of architecture involving group, specimen, session, and scan/block. The degree of complexity of the data structure depends on whether a test is static or cyclic.The test results are presented in figures, scatter plots, and tables. For static testing, the output in tables provides bending mechanical properties and characteristic points of moment–curvature relation: flexural rigidities in linear segments of loading and unloading stages, intersection points between characteristic segments of the curve, and equivalent stress and strain quantities. For cyclic testing, the table output lists control and fatigue life and responsive/dependent quantities including moment, curvature, flexural rigidity, flexural hysteresis, rigidity phase angle, and equivalent stresses and strains. In addition, derivatives such as half-gage length and sensor spacing correction are included. The output also provides standard deviations of the reported quantities when they are applicable or available. The data processing package can serve as a fundamental characterization tool in mechanical study of materials. The package is intended primarily for data processing for the CIRFT process and can also be used in applications for which similar testing requirements exist.
36 MATERIALS SCIENCE↗
PhytoOracle: Scalable, modular phenomics data processing pipelines
As phenomics data volume and dimensionality increase due to advancements in sensor technology, there is an urgent need to develop and implement scalable data processing pipelines. Current phenomics data processing pipelines lack modularity, extensibility, and processing distribution across sensor modalities and phenotyping platforms. To address these challenges, we developed PhytoOracle (PO), a suite of modular, scalable pipelines for processing large volumes of field phenomics RGB, thermal, PSII chlorophyll fluorescence 2D images, and 3D point clouds. PhytoOracle aims to ( i ) improve data processing efficiency; ( ii ) provide an extensible, reproducible computing framework; and ( iii ) enable data fusion of multi-modal phenomics data. PhytoOracle integrates open-source distributed computing frameworks for parallel processing on high-performance computing, cloud, and local computing environments. Each pipeline component is available as a standalone container, providing transferability, extensibility, and reproducibility. The PO pipeline extracts and associates individual plant traits across sensor modalities and collection time points, representing a unique multi-system approach to addressing the genotype-phenotype gap. To date, PO supports lettuce and sorghum phenotypic trait extraction, with a goal of widening the range of supported species in the future. At the maximum number of cores tested in this study (1,024 cores), PO processing times were: 235 minutes for 9,270 RGB images (140.7 GB), 235 minutes for 9,270 thermal images (5.4 GB), and 13 minutes for 39,678 PSII images (86.2 GB). These processing times represent end-to-end processing, from raw data to fully processed numerical phenotypic trait data. Repeatability values of 0.39-0.95 (bounding area), 0.81-0.95 (axis-aligned bounding volume), 0.79-0.94 (oriented bounding volume), 0.83-0.95 (plant height), and 0.81-0.95 (number of points) were observed in Field Scanalyzer data. We also show the ability of PO to process drone data with a repeatability of 0.55-0.95 (bounding area).
59 BASIC BIOLOGICAL SCIENCES↗
CLARA: A Contemporary Approach to Physics Data Processing
In traditional physics data processing (PDP) systems, data location is static and is accessed by analysis applications. In comparison, CLARA (CLAS12 Reconstruction and Analysis framework) is an environment where data processing algorithms filter continuously flowing data. In CLARA's domain of loosely coupled services, data is not stored, but rather flows from one service to another, mutating constantly along the way. Agents, performing event processing, can then subscribe to particular data/events at any stage of the data transformation, and make intricate decisions (e.g. particle ID) by correlating events from multiple, parallel data streams and/or services. This paper presents a PDP application development framework based on service oriented and event driven architectures. This system allows users to design (Java, C++, and Python languages are supported) and deploy data processing services, as well as dynamically compose PDP applications using available services. The PDP service bus provides a layer on top of a distributed pub-sub middleware implementation, which allows complex service composition and integration without writing code. Examples of service creation and deployment, along with the CLAS12 track reconstruction application design will be presented.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
GLAST (FERMI) Data-Processing Pipeline
The Data Processing Pipeline ('Pipeline') has been developed for the Gamma-Ray Large Area Space Telescope (GLAST) which launched June 11, 2008. It generically processes graphs of dependent tasks, maintaining a full record of its state, history and data products. The Pipeline is used to automatically process the data down-linked from the satellite and to deliver science products to the GLAST collaboration and the Science Support Center and has been in continuous use since launch with great success. The pipeline handles up to 2000 concurrent jobs and in reconstructing science data produces approximately 750GB of data products using 1/2 CPU-year of processing time per day.
99 GENERAL AND MISCELLANEOUS↗
Effects of non-homogeneous flow on ADCP data processing in a hydroturbine forebay
Observations of the flow conditions in the forebay of a hydroelectric power station indicate significant regions of non-homogeneous velocities near the intakes and shoreline. The effect of these non-homogeneous regions on the velocity measurement of an acoustic Doppler current profiler (ADCP) is investigated. By using a numerical model of an ADCP operating in a velocity field calculated using computational fluid dynamics (CFD), the errors due to the spatial variation of the flow velocity are identified. The numerical model of the ADCP is referred to herein as a Virtual ADCP (VADCP). Two scenarios are modeled in the numerical analyses presented. Firstly the measurement error of the VADCP is calculated for a single instrument adjacent to the short converging intake of the powerhouse. Secondly, the flow discharge through the forebay is estimated from a transect of VADCP instruments at dif- ferent distances from the powerhouse. The influence of instrument location and orientation are investigated for both cases. A velocity error of over up to 94% of the reference velocity is calculated for a VADCP modeled adjacent to an operating intake. Qualitative agreement is observed between the calculated VADCP velocities and reference velocities by an offset of one intake height upstream of the powerhouse.
13 HYDRO ENERGY↗