Radar / Processed Data
This dataset contains processed raw data from the UND 94 GHz W-band Cloud Doppler Radar.
17 WIND ENERGY↗
SEARCH · Engineering Papers
Results for “processed data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Intelligent experiments through real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and future EIC detectors
This R&D project, initiated by the DOE Nuclear Physics AI-Machine Learning initiative in 2022, leverages AI to address data processing challenges in high-energy nuclear experiments (RHIC, LHC, and future EIC). Our focus is on developing a demonstrator for real-time processing of high-rate data streams from sPHENIX experiment tracking detectors. The limitations of a 15 kHz maximum trigger rate imposed by the calorimeters can be negated by intelligent use of streaming technology in the tracking system. The approach efficiently identifies low momentum rare heavy flavor events in high-rate p+p collisions (3MHz), using Graph Neural Network (GNN) and High Level Synthesis for Machine Learning (hls4ml). Success at sPHENIX promises immediate benefits, minimizing resources and accelerating the heavy-flavor measurements. The approach is transferable to other fields. For the EIC, we develop a DIS-electron tagger using Artificial Intelligence - Machine Learning (AI-ML) algorithms for real-time identification, showcasing the transformative potential of AI and FPGA technologies in high-energy nuclear and particle experiments real-time data processing pipelines.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NLR Data Processing Pipeline for MADIS [SWR-26-050]
The NLR Data Processing Pipeline for MADIS software package is for downloading, processing, and performing QA/QC on MADIS data. Designed to handle the following steps: 1) Download all MADIS data as compressed netcdf files for a given time period. 2) Unpack netcdf files into timeseries csvs for each coordinate within the given bounding box. 3) Process the csvs to filter according to quality control checks and convert variables to correct units. 4) Write processed csvs to a single nc file.
Benton, Brandon [National Laboratory of the Rockie↗
Lidar Wind Cube / Processed Data
This dataset contains processed and standardized data from DOE Buoy 140 deployed during WFIP3.
17 WIND ENERGY↗
PvaPy streaming framework for real-time data processing
User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.
EPICS pvAccess↗
Phlex: Parallel, Hierarchical, and Layered EXecution of data-processing algorithms
Phlex is a computing framework supporting the parallel, hierarchical, and layered execution of data-processing algorithms. It is based on the functional-programming paradigm, thus guaranteeing thread-safety when invoking user-defined pure functions. Phlex allows users to specify arbitrary graph-based hierarchies of data organization, enabling more flexible processing of data as required by the constraints of the program.
Knoepfel, KyleJ. [Fermi National Accelerator Labor↗
WFIP3 - CACO site - NREL Thermodynamic profiler (Assist II-12) / Processed Data
This dataset contains processed NREL thermodynamic profiler (Assist II-12) data from the WFIP3 CACO site.
17 WIND ENERGY↗
Intelligent Experiments through Real-Time AI: Fast Data Processing and Autonomous Detector Control for High-Energy Nuclear Experiments
The aim of this project is to develop software and hardware for fast real-time data processing and autonomous detector control and calibration for the sPHENIX and the future EIC experiments. Below summarizes Georgia Tech team efforts in the past year: 1. We developed a real-time clustering algorithm and FPGA-based pipeline architecture for processing fired pixel data from ALPIDE sensors in sPHENIX experiments. Our Columnar Clustering Co-Design introduces a hardware-aware, stream-friendly approach that segments pixel data by column pairs using a Column Pair Clustering (CPC) strategy, followed by Cluster Stitching to merge adjacent subclusters. Implemented in Vitis HLS, the pipeline comprises five stages—read-in, subclustering, stitching, analysis, and write-out—connected by tagged HLS streams with custom end-of-event signaling for robust synchronization. We designed a pipelined dataflow model optimized for throughput, low latency, and minimal buffering, enabling scalable clustering across events of arbitrary size. Our system maintains spatial precision via center-of-mass and shape key extraction and efficiently handles edge cases such as fragmented or nested clusters. Compared against DBSCAN in both software and hardware, our approach demonstrates competitive performance under FPGA constraints. 2. We also conducted a comprehensive algorithm-to-hardware co-design of connected component analysis tailored for sPHENIX experiments, focusing on real-time, low-latency processing using FPGAs and High-Level Synthesis (HLS). Starting from a Python-based particle tracking pipeline, the team translated the core logic—graph traversal via DFS and Union-Find—into an HLS-compatible C++ model, replacing dynamic memory and recursion with static arrays and pipelined control flow. The final design includes a fully streamed and dataflow-compatible Union-Find kernel optimized across five iterations, incorporating loop pipelining, array partitioning, AXI/FIFO interface tuning, and function flattening. Experimental results show up to 14.8× speedup over the CPU baseline, reducing per-graph latency to 1.58 μs and demonstrating strong resource efficiency with only ~7k LUTs and zero BRAM usage. The design maintains functional correctness against the Python reference using a Python-based C-simulation framework and Mean Squared Error metrics. This work validates the potential of HLS-driven FPGA designs for edge-level HEP data acquisition, laying a scalable foundation for future integration with real-time detector pipelines and multi-graph processing systems.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
Summary of the 5th IAEA technical meeting on fusion data processing, validation and analysis (FDPVA)
The purpose of the 5th International Atomic Energy Agency technical meeting on fusion data processing, validation and analysis (FDPVA) (Ghent University, Ghent, Belgium, 12–15 June 2023) was to provide a platform during which a set of topics relevant to FDPVA were discussed with the view of meeting the needs of next step fusion devices such as ITER. The validation and analysis of experimental data obtained from diagnostics used to characterize fusion plasmas are crucial for a knowledge-based understanding of the physical processes governing the dynamics of these plasmas. This paper presents the recent progress and achievements in the domain of plasma diagnostics data analysis and synthetic diagnostics reported at the meeting, including concept description of new devices; fusion databases; integrated data analysis; inverse problems; uncertainty propagation, verification and validation; probabilistic methods and machine learning. The relevant results underline trends observed in the current major fusion confinement devices.
fusion databases↗
Validation of Modern Nuclear Data Processing in SCALE
The nuclear data (ND) community is continuously developing more accurate, diversified, and comprehensive data for radiation transport modeling to support the nuclear science community. As these community efforts progress, it is crucial that ND processing tools like AMPX (used for SCALE [1] ND) also be developed in parallel to incorporate these new data into transport codes and actually deliver those data to end users. AMPX is a mature, well-tested code that was developed by many people at Oak Ridge National Laboratory (ORNL) over the course of the past few decades. A large portion of the AMPX codebase, however, was outdated, difficult to maintain, and incompatible with modern code development tools. Some of the most important parts of the AMPX code have now been replaced with modern C++ code that can be maintained more cost-effectively and can be tested more rigorously.
AMPX↗
A Data Processing Pipeline To Extract A Knowledge Graph From Heterogeneous Data For Socio-technical Analysis Of Critical Infrastructure Influence
The code is written in Python and consists of the following pipeline that is implemented in Apache Airflow. This pipeline intends to understand the companies that are directly or indirectly involved with a type of critical infrastructure system at some point in that system's lifecycle. The pipeline takes a configuration file that specifies a list of initial companies to consider, a geographic region of interest, and a set of SEC form types as well as other data sources (e.g. CrunchBase) from which to extract entities and relations. There are four main components to this pipeline as currently implemented: Entity Extraction, Network Construction, Analysis, and Visualization. First, Entity Extraction, is implemented as the `topear-extract_organizations` Apache Airflow workflow. Given an initial query that specifies a geographic region of interest and a time interval, the software will extract CI facilities of interest and organizations that have a direct influence relationship to those facilities (e.g. ownership). During the course of the LDRD, we focused on Electric Vehicle charging stations and this information is available via the Department of Energy (DOE) database on fueling stations maintained by NREL. Within the context of the DOE CESER project, we have focused on Battery Energy Storage Systems (BESS). Second, the Network Extraction component will iteratively construct a social network graph given the set of organizations and people extracted in the previous step. Organizations (and eventually People if desired) are then fed as a query to the `topgear-construct_social_network` Apache Airflow workflow which given a set of initial companies and data sets (e.g. SEC EDGAR form types, OpenCorporates, Crunchbase). This Airflow workflow will iteratively query such data sources to discover relationships with new organizations and people. For example, this module can iteratively query SEC EDGAR for metadata that documents the number of each type of form for the given set of companies and their location. This forms metadata represents a catalog of data sources from SEC EDGAR for the extracted social network knowledge graph. The pipeline then downloads these forms from the website and saves them in a build directory for further processing. These documents are then parsed for entities and relations. Again, we note that in additional to SEC data sources, this step can also pull in information on organizations via API services such as CrunchBase and OpenCorporates or bulk data sources. At the end of this step, the resultant social network, the Critical Infrastructure network, and the edges that encode relationships between organizations and CI facilities, form the Adversarial Socio-Technical Network (ASTN) that informs the analysis. Third, the Analysis component processes these generated ASTN. Previously, that has included the ability to compare prevalence of different vendors for a given infrastructure component type across different regions as well as identify common public and private investors across those vendors. This was demonstrated for EV Charging Stations across several different metropolitan areas within an IEEE PES GridEdge publication. More recently, we have looked at ways to identify infrastructure owners and operators of BESS with the most nameplate capacity across different states as well as other indictors of risk resulting from changes in ownership over time. Finally, the Visualization component consists of an HTML/CSS/JS framework by which users can interact geospatial, operational, and organizational relationships across a given portfolio of Critical Infrastructure facilities. The objective is to provide a library of UI/UX modules that can be repurposed for stakeholder-specific dashboards. All of the modules are related via a common event model that enables UI actions in one view to percolate across the other views.
Weaver, Gabriel [Idaho National Laboratory (INL), ↗
Intelligent Experiments Through Real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and Future EIC Detectors (Final Report)
The overall vision of this project was to integrate real-time artificial intelligence (AI) directly into the data acquisition and detector-control systems of nuclear physics experiments, including both fast online event selection and an autonomous detector-control feedback loop. The work carried out under the award focused on the fast online event-selection half of that vision: the efficient recording of low-momentum heavy-flavor (HF) hadron decays in proton-proton collisions at the sPHENIX experiment at the Relativistic Heavy Ion Collider (RHIC)—an observable that requires fast tracking and topological trigger selection not previously demonstrated at RHIC, and that is essential for QCD studies at future facilities such as the Electron-Ion Collider (EIC). The autonomous detector-control (GPU-based feedback) component named in the project title remained a design concept and was not implemented under this award. The Massachusetts Institute of Technology (MIT) group led the offline simulation and data processing needed to train the machine-learning (ML) models, the translation of trained models to Field-Programmable Gate Array (FPGA) firmware using the hls4ml framework, and the physics validation of heavy-flavor reconstruction. Over the award period, the team developed and hardware-tested the principal components of an AI-based heavy-flavor trigger on simulated and recorded sPHENIX tracker data: a software Bipartite Graph Attention Network (BiGAT) trigger model reaching > 95% signal efficiency at 99% background rejection; an FPGA-native hit clusterizer matching the offline clustering; smaller networks synthesized to FPGA within the required sub-10 µs latency; and an assembled decoder–clusterizer–inference firmware chain exercised on the FELIX readout board. A complete, fully integrated hardware demonstrator was not finished within the award period. This report documents the project goals, the MIT group’s contributions, the technical accomplishments, and the outlook toward applications at the future EIC ePIC detector.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
Preparation of the Multi-Site Data Processing at the Vera C. Rubin Observatory
The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) Camera is scheduled to start taking data in the summer of 2025. The Data Release Production will run the LSST Science Pipe software at data facilities in the US, France and the UK. The LSST Science Pipeline consists of complex directed acyclic graphs (DAGs) of tasks. Rubin will use the Production and Distributed Analysis (PanDA) workflow and workload management system to orchestrate this complex workflow and the distribution of workloads to the data facilities. When run end-to-end by a team of data production staff, this processing (the Science Pipelines, distributed by the workflow and workload management system) is referred to as a 'campaign'. This paper describes the central services and data facility specific services that support this multi-site data process model, including the service deployment infrastructure, the workload and workflow system, the Campaign Management tools, and connection to Rubin Data Management. This paper will also mention the experience of processing the Rubin Commissioning Camera data. All these are part of the effort to scale up the processing capabilities for the expected very large data volume from the LSST Camera.
Yang, Wei [SLAC]↗
Machine-Protection System for the PIP-II Linear Accelerator: Architecture and Data Processing Schemes
PIP-II at Fermilab features a brand-new, 800 MeV leading-edge Linear Accelerator (Linac) that will enable the Fermilab complex to deliver more than a megawatt of beam power to the Long Base-line Neutrino Facility (LBNF), requiring a robust Machine Protection System (MPS) to prevent beam-induced damage to delicate cryomodules and vacuum components. Signals in various accelerator components, in both analog and digital format, are collected and processed to assest the operating conditions of the accelerator. In the situation a potential damage is foreseen, the beam permit signal will be drop. A critical element of this system is the fast Analog Machine Protection System, a high-bandwidth platform designed for real-time beam loss monitoring. The system utilizes a modular, FPGA-based architecture to digitize signals from beam-sensing devices, such as AC Current Transformers (ACCTs), non-invasive Ring Pickups (RPUs) and beam scrapers. To eliminate both high frequency as well as 60 Hz noise that might be picked up in the industrial environment, data processing schemes beyond regular digital filtering are planned and implemented.
Wu, Jinyuan [Fermilab] (ORCID:0000000344329521)↗
Colorado State University (CSU) X-Band Precipitation Radar Plan Position Indicator Data Processed with Corrected Moments in Antenna Coordinates (CMAC) Value-Added Product Report
In 2010 the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility procured 3- and 5-cm wavelength radars for documenting the macrophysical, microphysical, and dynamical structure of precipitating systems. To maximize the scientific impact, ARM supported the development of an application chain to correct for various phenomena in order to retrieve the “point” values of moments of the radar spectrum and polarimetric measurements. We have now used the lessons learned from the processing of the 3- and 5-cm wavelength radars obtained by ARM to help process X-band radar data from the Surface Atmospheric Integrated Field Laboratory (SAIL) field campaign. This report details the motivation, science, and progress to date as well as charting a path forward.
54 ENVIRONMENTAL SCIENCES↗
pathSQE : an automated workflow for single-crystal inelastic neutron scattering data processing and analysis
Inelastic neutron scattering (INS) experiments utilizing modern time-of-flight spectrometers enable the comprehensive mapping of the energy (E)- and momentum (Q)-resolved dynamical structure factor of single crystals, probing both the lattice and magnetic excitations. Yet, the large size and complexity of four-dimensional INS data are challenging current analysis workflows, often resulting in an underutilization of the measured information. To help address this issue, this paper introduces new software interfaced with the Mantid framework, pathSQE, designed to streamline the processing, analysis and interpretation of 4D single-crystal INS data. By automating key tasks such as 1D/2D slicing, symmetrization, Brillouin zone folding, data visualization, prioritization and filtering, and comparisons with simulations, pathSQE facilitates and accelerates INS data analysis workflows. Here, this paper outlines the features and implementation and provides several illustrations of the use of pathSQE on data collected on single crystals using direct-geometry time-of-flight spectrometers at the Spallation Neutron Source, including Ge, FeSi, MnO and SnS single-crystal measurements on the ARCS, HYSPEC and CNCS neutron spectrometers. Beyond streamlining post-experiment data processing, pathSQE establishes an automated and modular processing pipeline that could support future real-time experiment steering.
36 MATERIALS SCIENCE↗
Leveraging FPGA Advantages for Quicker Data Processing for LBNF
The Long Baseline Neutrino Facility (LBNF) will deliver a 2.4 MW muon neutrino beam from Fermilab to the Deep Underground Neutrino Experiment (DUNE), requiring unprecedented precision in beamline alignment to achieve DUNE's neutrino oscillation measurement goals. Vertical misalignments of beamline components as small as 0.5 mm can contribute 6-7\% uncertainty in predicted neutrino flux, necessitating sub-0.1 mm alignment monitoring capabilities. The Horn Location Sensor (HLS) system employs frequency sweep interferometry (FSI) in a distributed hydrostatic leveling network to achieve the required precision under harsh radiation conditions up to 5000 kRad/year. Traditional FSI implementations suffer from laser sweep nonlinearities that degrade resolution and require computationally intensive post-processing corrections using gas reference cells. This work presents a real-time FPGA-based implementation of the HLS data acquisition and processing system using a sweep tracker interferometer for dynamic sweep linearization. The system utilizes a PYNQ-Z2 FPGA with programmable logic implementing parallel 16k-point FFT processing across four channels, synchronized by the sweep tracker signal to eliminate post-processing requirements. Spectral performance testing demonstrates significant improvements in peak sharpness compared to traditional fixed-frequency digitization. The FPGA implementation enables real-time displacement monitoring with processing speeds orders of magnitude faster than software-based approaches, essential for the operational requirements of LBNF's eventual distributed sensor network. This advancement in real-time FSI processing directly supports DUNE's precision neutrino physics program by providing the rapid feedback necessary for maintaining stringent beamline alignment tolerances during high-power beam operations.
Rossel, Jacob↗
Global Geo-processed Data of Aquifer Properties by 0.5° Grid, Country and Water Basins
This repository of global hydrogeologic datasets contains aquifer properties on 0.5° scale, including depth to groundwater (Fan et al., 2013), aquifer thickness (de Graaf et al., 2015), WHYMap aquifer classes (Richts et al., 2011), recharge (Döll and Fiedler, 2008; Gleeson et al., 2016), lakes (Messager et al., 2016), porosity and permeability (Gleeson et al., 2014), digitized and geo-processed from their respective sources. Globally gridded aquifer properties could be used independently to estimate global groundwater availability or used as critical inputs to the superwell model to simulate groundwater extraction and provide estimates of pumped volumes and unit costs under user-specific scenarios. Key resources related to this data are: Niazi, H., Ferencz, S. B., Graham, N. T., Yoon, J., Wild, T. B., Hejazi, M., Watson, D. J., & Vernon, C. R. (2025). Long-term hydro-economic analysis tool for evaluating global groundwater cost and supply: Superwell v1.1. Geoscientific Model Development, 18(5), 1737-1767. https://doi.org/10.5194/gmd-18-1737-2025 superwell model repository which uses this data to simulate groundwater extraction and provides estimates of the global extractable volumes and unit-costs ($/km3) of accessible groundwater production under user-specified extraction scenarios. Repository Overview Main output: aquifer_properties_rec.csv contains all processed outputs, including aquifer properties like porosity, permeability, recharge, lake areas, aquifer thickness, and depth to groundwater. shapefiles.zip: contains all digitized GIS databases and shapefile for all aquifer properties prep_inputs.R and prep_inputs_recharge_lakes.R: R scripts that process the shapefiles to produce the aquifer_properties_rec.csv file plot_inputs.R: R script for plotting the maps and conducting preliminary analysis on the available groundwater volume basin_to_country_mapping.csv, basin_country_region_mapping.csv and continent_county_mapping.csv provide the mapping between continents, 32 energy-economic macro regions, countries, and water basins for post-processing aquifer_properties_rec.csv Maps: Each map visualizes the spatial distribution of one of the aquifer properties across the globe map_in_Porosity.png map_in_Permeability.png map_in_Aquifer_thickness.png map_in_Depth_to_water.png map_in_Recharge.png map_in_Grid_area_km.png map_in_Lake_area_km.png map_in_WHYClass.png Sample inputs sample_inputs.py: this script samples inputs from the aquifer_properties_rec dataset, ensuring the sampled and original inputs maintain the same distributions sampled_data_100.csv contains 100 sampled data points and sampled_data_100.png compares their distributions Dataset Overview The main outputs are consolidated in a comprehensive aquifer_properties_rec.csv file and include the following fields: GridCellID: Unique identifier for each (roughly 0.5°) grid cell Continent: Continent name Country: Country name GCAM_basin_ID: Identifier for GCAM hydrologic basin Basin_long_name: Full name of the basin WHYClass: Hydrogeologic classification based on WHYMap aquifer classes (Richts et al., 2011) Porosity: Soil porosity (%) (Gleeson et al., 2014) Permeability: Soil permeability (in square meters; Gleeson et al., 2014) Aquifer_thickness: Thickness of the aquifer (in meters; de Graaf et al., 2015) Depth_to_water: Depth to groundwater (in meters; Fan et al., 2013) Recharge: long-term annual averaged recharge rates (in m/yr; Döll and Fiedler, 2008; Gleeson et al., 2016) Grid_area: Area of the grid cell (in square meters) Lakes_area: Area of inland lakes (in square meters; Messager et al., 2016) Key References The datasets are digitized versions of global hydrogeologic properties from the following key literature sources: Depth to Groundwater: Fan, Y., Li, H., & Miguez-Macho, G. (2013). Global Patterns of Groundwater Table Depth. Science, 339(6122), 940-943. https://doi.org/10.1126/science.1229881 Aquifer Thickness: de Graaf, I. E. M., Sutanudjaja, E. H., van Beek, L. P. H., & Bierkens, M. F. P. (2015). A high-resolution global-scale groundwater model. Hydrol. Earth Syst. Sci., 19(2), 823-837. https://doi.org/10.5194/hess-19-823-2015 Porosity and Permeability: Gleeson, T., Moosdorf, N., Hartmann, J., & van Beek, L. P. H. (2014). A glimpse beneath earth's surface: GLobal HYdrogeology MaPS (GLHYMPS) of permeability and porosity. Geophysical Research Letters, 41(11), 3891-3898. https://doi.org/10.1002/2014GL059856 Aquifer classes: Richts, A., Struckmeier, W. F., & Zaepke, M. (2011). WHYMAP and the Groundwater Resources Map of the World 1:25,000,000. In J. A. A. Jones (Ed.), Sustaining Groundwater Resources: A Critical Element in the Global Water Crisis (pp. 159-173). Springer Netherlands. https://doi.org/10.1007/978-90-481-3426-7_10 Recharge: Döll, P., & Fiedler, K. (2008). Global-scale modeling of groundwater recharge. Hydrol. Earth Syst. Sci., 12(3), 863-885. https://doi.org/10.5194/hess-12-863-2008; Gleeson, T., Befus, K. M., Jasechko, S., Luijendijk, E., & Cardenas, M. B. (2016). The global volume and distribution of modern groundwater. Nature Geoscience, 9(2), 161-167. https://doi.org/10.1038/ngeo2590 Inland Lakes: Messager, M. L., Lehner, B., Grill, G., Nedeva, I., & Schmitt, O. (2016). Estimating the volume and age of water stored in global lakes using a geo-statistical approach. Nature Communications, 7(1), 13603. https://doi.org/10.1038/ncomms13603 Cite as Niazi, H., Watson, D., Hejazi, M., Yonkofski, C., Ferencz, S., Vernon, C., Graham, N., Wild, T., & Yoon, J. (2024). Global Geo-processed Data of Aquifer Properties by 0.5° Grid, Country and Water Basins. MultiSector Dynamics-Living, Intuitive, Value-adding, Environment. https://doi.org/10.57931/2484226 Contact Reach out to Hassan Niazi or open an issue in the superwell repository for questions or suggestions.
aquifer thickness↗