Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,387 records · Page 77

Performance of Heterogeneous Algorithm Scheduling in CMSSW

The CMS experiment started to utilize Graphics Processing Units (GPU) to accelerate the online reconstruction and event selection running on its High Level Trigger (HLT) farm in the 2022 data taking period. The projections of the HLT farm to the High-Luminosity LHC foresee a significant use of compute accelerators in the LHC Run 4 and onwards in order to keep the cost, size, and power budget of the farm under control. This direction of leveraging compute accelerators has synergies with the increasing use of HPC resources in HEP computing, as HPC machines are employing more and more compute accelerators that are predominantly GPUs today. In this work we review the features developed for the CMS data processing framework, CMSSW, to support the effective utilization of both compute accelerators and many-core CPUs within a highly concurrent task-based framework. We measure the impact of various design choices for the scheduling of heterogeneous algorithms on the event processing throughput, using the Run-3 HLT application as a realistic use case.

Bocci, Andrea↗

Pagoda

Pagoda is a set of parallel tools as well as an application programmer interface for parallel data analysis of high-resolution climate data. Pagoda implements data parallel versions of some of the commonly used tools from the NetCDF Operators. These tools support data processing operations including subsetting, record and ensemble averages, binary operations, and interpolation. Pagoda also provides an application programmer interface for developing custom data analysis programs.

Source record↗

General Interaction Data Interface

The General Interaction Data Interface (GIDI) is a C++ API that reads and writes nuclear reaction data stored in the General Nuclear Data Structure (GNDS). GNDS is a new "format" for storing nuclear reaction data developed by the Working Party on Evaluation Co-operation (OECD/NEA/WPEC) Subgroup 38. GNDS can store evaluated nuclear data as well as nuclear data processed for deterministic (multi-group) and Monte Carlo transport. GIDI handles and provides accesses to all forms of the GNDS nuclear data. For multi-group data, GIDI provides functions to sum the data over all reactions. It also has functions for collapsing multi-group data to a small subset of groups and can transport correct multi-group data using the Pendlebury/Underhill formalism. The GNDS definition supports data for a PROjectile hitting a TARget for a given Evaluation. In GIDI this is called a PROTARE. For example, a neutron hitting O-16 from the ENDF/B-VIII.0 evaluation. To support a collection of protares, GIDI adds a Map class. The map class reads in a "map" file which specifies a list of projectile/target/evalaution's and their associated protare files. With a map instance, a user can request a specific projectile/target/evaluation and the map instance will return the requested protare.

Beck, Bret↗

Meld: A project for exploring how to meet DUNE's framework needs

Existing data-processing frameworks for HEP experiments are largely based on collider-physics concepts, which may be based on rigid, event-based data hierarchies. These data organizations are not always helpful for neutrino experiments, which must sometimes work around such restrictions by manually splitting apart events into constructs that are better suited for neutrino physics. The purpose of Meld is to explore more flexible data organizations by treating a frameworks job as: (1) A graph of data-product sequences connected by (2) User-defined functions that serve as operations to (3) Framework-provided higher-order functions.

Knoepfel, KyleJ. [Fermi National Accelerator Labor↗

“Translational Opportunities in CPS Transportation”

This talk will describe opportunities for translating research from open-road experiments with modified adaptive cruise controllers. The data and controllers used for the previous research are based on use cases and example drives from within the US. We explore processes, data sharing, experiments, and other techniques that we think will drive trans-Pacific partnerships with driving data.

Sprinkle, Jonathan↗

Pagoda

Pagoda is a set of parallel tools as well as an application programmer interface for parallel data analysis of high-resolution climate data. Pagoda implements data parallel versions of some of the commonly used tools from the NetCDF Operators. These tools support data processing operations including subsetting, record and ensemble averages, binary operations, and interpolation. Pagoda also provides an application programmer interface for developing custom data analysis programs.

Source record↗

Evolution of standardization and dissemination of cryo-EM structures and data jointly by the community, PDB, and EMDB

Cryogenic electron microscopy (cryo-EM) methods began to be used in the mid-1970s to study thin and periodic arrays of proteins. Following a half-century of development in cryo-specimen preparation, instrumentation, data collection, data processing and modeling software, cryo-EM has become a routine method for solving structures from large biological assemblies to small biomolecules at near to true atomic resolution. This review explores the critical roles played by the Protein Data Bank (PDB) and Electron Microscopy Data Bank (EMDB) in partnership with the community to develop the necessary infrastructure to archive cryo-EM maps and associated models. Public access to cryo-EM structure data has in turn facilitated better understanding of structure-function relationships and advancement of image processing and modeling tool development. The partnership between the global cryo-EM community and PDB and EMDB leadership has synergistically shaped the standards for metadata, one-stop deposition of maps and models, and validation metrics to assess the quality of cryo-EM structures. The advent of cryo-electron tomography (cryo-ET) for in situ molecular cell structures at a broad resolution range and their correlations with other imaging data introduces new data archival challenges in terms of data size and complexity in the years to come.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Sap flow data from San Lorenzo, Panama (PA-SLZ), 2019

This data package contains sap flow data from the NGEE Tropics site in San Lorenzo, Panama (PA-SLZ) from 22 January 2019 to 31 December 2019. Sap flow measurements quantify plant-level transpiration, which varies across spatial and temporal scales. It is a key trait in understanding plant hydraulic functioning. Here sap flow was measured using ‘Granier-type’ thermal dissipation probes. Raw data downloaded from each data logger are in the 'Raw data' folder as CSV files, along with the Campbell data logger programs which can be read when opened with a text editor. The data was processed using the AquaFlux package in R. R code for preparing the data for use in AquaFlux is in the 'R code for pre-processing data folder'; it is saved as a text file that can be copy and pasted into R. The processed data for each data logger is in the 'Processed data' folder as CSV files. Documentation for the Granier sensors and AquaFlux package is in the ‘Manuals’ folder as PDF files.

54 ENVIRONMENTAL SCIENCES↗

Evolution of the ATLAS TDAQ online software framework towards Phase-II upgrade: Use of Kubernetes as an orchestrator of the ATLAS Event Filter computing farm

The ATLAS experiment at the LHC at CERN continuously evolves its TDAQ system to meet the challenges of new physics goals and technological advancements. As ATLAS prepares for the Phase-II Run 4 of the LHC, significant enhancements in the TDAQ Controls and Configuration (TDAQ-CC) tools have been designed to ensure efficient data collection, processing, and management. This abstract presents the evolution of ATLAS TDAQ-CC system leading up to Phase-II Run 4. As part of the evolution towards Phase-II, Kubernetes has been chosen to orchestrate the Event Filter (EF) farm. By leveraging Kubernetes, ATLAS can dynamically allocate computing resources, scale processing capacity in response to changing data taking conditions and ensure high availability of data processing services. The integration of the Kubernetes with the TDAQ Run Control framework enables perfect synchronisation between the experiment’s data acquisition components and the computing infrastructure. We will discuss the architectural considerations and implementation challenges involved in Kubernetes integration with the ATLAS TDAQ-CC system. We will highlight the benefits of using Kubernetes as an EF farm orchestrator, including improved resource utilization, enhanced fault tolerance, and simplified deployment and management of data processing workflows. In addition, we will report on the extensive testing of Kubernetes that was conducted using a farm of 2500 servers within the experiment data taking environment, demonstrating its scalability and robustness in handling the demands of the ATLAS TDAQ system for Phase-II. The adoption of Kubernetes represents a significant step forward in the evolution of ATLAS TDAQ-CC system, aligning with industry best practices in container orchestration.

Corso Radu, Alina [Univ. of California, Irvine, CA↗

The Future of a Myriad of Accelerated Biodiscoveries Lies in AI‐Powered Mass Spectrometry and Multiomics Integration

The intersection of modern artificial intelligence (AI) and mass spectrometry (MS) is set to transform the MS‐based “omics” research fields, particularly proteomics, metabolomics, lipidomics, and glycomics, enabling advancements across a wide range of domains, from health to environment and industrial biotechnology. Beginning with an overview of key challenges inherent in MS software pipelines, this personal perspective explores how AI‐driven solutions can address them to enhance data processing, integration and interpretation. It proposes a paradigm shift in molecular identification and quantitation algorithms, leveraging AI to enable holistic interpretation of MS‐based multiomics data. While centered on MS‐based omics, this holistic AI‐driven paradigm is also critical for connecting dynamic biochemical changes to genomics and transcriptomics contexts, reinforcing the integrative value of MS in multiomics research. Ultimately, this AI‐driven approach could enhance efficiency, accuracy, and molecular breadth of coverage, deepening our systems‐level understanding of biological processes and accelerating a myriad of biodiscoveries.

47 OTHER INSTRUMENTATION↗

TU-C-17A-03: An Integrated Contour Evaluation Software Tool Using Supervised Pattern Recognition for Radiotherapy

Purpose: Radiotherapy (RT) contours delineated either manually or semiautomatically require verification before clinical usage. Manual evaluation is very time consuming. A new integrated software tool using supervised pattern contour recognition was thus developed to facilitate this process. Methods: The contouring tool was developed using an object-oriented programming language C# and application programming interfaces, e.g. visualization toolkit (VTK). The C# language served as the tool design basis. The Accord.Net scientific computing libraries were utilized for the required statistical data processing and pattern recognition, while the VTK was used to build and render 3-D mesh models from critical RT structures in real-time and 360° visualization. Principal component analysis (PCA) was used for system self-updating geometry variations of normal structures based on physician-approved RT contours as a training dataset. The inhouse design of supervised PCA-based contour recognition method was used for automatically evaluating contour normality/abnormality. The function for reporting the contour evaluation results was implemented by using C# and Windows Form Designer. Results: The software input was RT simulation images and RT structures from commercial clinical treatment planning systems. Several abilities were demonstrated: automatic assessment of RT contours, file loading/saving of various modality medical images and RT contours, and generation/visualization of 3-D images and anatomical models. Moreover, it supported the 360° rendering of the RT structures in a multi-slice view, which allows physicians to visually check and edit abnormally contoured structures. Conclusion: This new software integrates the supervised learning framework with image processing and graphical visualization modules for RT contour verification. This tool has great potential for facilitating treatment planning with the assistance of an automatic contour evaluation module in avoiding unnecessary manual verification for physicians/dosimetrists. In addition, its nature as a compact and stand-alone tool allows for future extensibility to include additional functions for physicians’ clinical needs.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

The SNS/HFIR Web Portal System – How Can it Help Me?

Abstract. In a busy world, continuing with the status-quo, to do things the way we are already familiar, often seems to be the most efficient way to conduct our work. We look for the value-add to decide if investing in a new method is worth the effort. How shall we evaluate if we have reached this tipping point for change? For contemporary researchers, understanding the properties of the data is a good starting point. The new generation of neutron scattering instruments being built are higher resolution and produce one or more orders of magnitude larger data than the previous generation of instruments. For instance, we have grown out of being able to perform some important tasks with our laptops – the data are too big and the computations would simply take too long. These large datasets can be problematic as facility users now begin to grapple with many of the same issues faced by more established computing communities. These issues include data access, management, and movement, data format standards, distributed computing, and collaboration among others. The Neutron Science Portal has been architected, designed, and implemented to provide users with an easy-to-use interface for managing and processing data, while also keeping an eye on meeting modern cybersecurity requirements imposed on institutions. The cost of entry for users has been lowered by utilizing a web interface providing access to backend portal resources. Users can browse or search for data which they are allowed to see, data reduction applications can be run without having to load the software, sample activation calculations can be performed for SNS and HFIR beamlines, McStas simulations can be run on TeraGrid and ORNL computers, and advanced analysis applications such as those being produced by the DANSE project can be run. Behind the scenes is a “live cataloging” system which automatically catalogs and archives experiment data via the data management system, and provides proposal team members access to their experiment data. The complexity of data movement and utilizing distributed computing resources has been taken care on behalf of users. Collaboration is facilitated by providing users a read/writeable common area, shared across all experiment team members. To date, these shared areas are the fastest growing data spaces. The portal currently has over 370 registered users, almost 5TB of experiment and user data, approximately 660K files cataloged, and had almost 10,000 unique visits last year. Future directions for enhancing portal robustness include examining how to mirror data and portal services, better facilitation of collaborations via virtual organizations, enhancing disconnected service via “thick client” applications, and better inter-facility connectivity to support cross-cutting research. The portal has established itself in the SNS/HFIR user community, and the development team strives to continue to improve the quality of features and services provided in order to better serve the community.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Integration of Titan supercomputer at OLCF with ATLAS Production System

The PanDA (Production and Distributed Analysis) workload management system was developed to meet the scale and complexity of distributed computing for the ATLAS experiment. PanDA managed resources are distributed worldwide, on hundreds of computing sites, with thousands of physicists accessing hundreds of Petabytes of data and the rate of data processing already exceeds Exabyte per year. While PanDA currently uses more than 200,000 cores at well over 100 Grid sites, future LHC data taking runs will require more resources than Grid computing can possibly provide. Additional computing and storage resources are required. Therefore ATLAS is engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. In this paper we will describe a project aimed at integration of ATLAS Production System with Titan supercomputer at Oak Ridge Leadership Computing Facility (OLCF). Current approach utilizes modified PanDA Pilot framework for job submission to Titan's batch queues and local data management, with lightweight MPI wrappers to run single node workloads in parallel on Titan's multi-core worker nodes. It provides for running of standard ATLAS production jobs on unused resources (backfill) on Titan. The system already allowed ATLAS to collect on Titan millions of core-hours per month, execute hundreds of thousands jobs, while simultaneously improving Titans utilization efficiency. We will discuss the details of the implementation, current experience with running the system, as well as future plans aimed at improvements in scalability and efficiency.

97 MATHEMATICS AND COMPUTING↗

Tools for supporting solution scattering during the COVID-19 pandemic

During the COVID-19 pandemic, synchrotron beamlines were forced to limit user access. Performing routine measurements became a challenge. At the Life Science X-ray Scattering (LiX) beamline, new instrumentation and mail-in protocols have been developed to remove the access barrier to solution scattering measurements. Our efforts took advantage of existing instrumentation and coincided with the larger effort at NSLS-II to support remote measurements. Given the limited staff–user interaction for mail-in measurements, additional software tools have been developed to ensure data quality, to automate the adjustments in data processing, as users would otherwise rely on the experience of the beamline staff, and produce a summary of the initial assessments of the data. This report describes the details of these developments.

99 GENERAL AND MISCELLANEOUS↗

TexTonic: Interactive Visualization for Exploration and discovery of Very Large Text Collections

TexTonic is a visual analytic system for interactive exploration of very large unstructured text collections. TexTonic visualizes hierarchical clusters of representative terms, snippets, and documents in a single, multi-scale spatial layout. Exploration is supported by interacting with the visualization and directly manipulating the terms in the visualization using semantic interactions. These semantic interactions steer the underlying analytic model by translating user interactions within the visualization to contextual updates to the supporting data model. The combination of semantic interactions and information visualization at multiple levels of the data hierarchy helps users manage information overload so that they can more effectively explore very large text collections. In this paper we describe TexTonic’s data processing and analytic pipeline, user interface and interaction design principles, and the results of a user study conducted mid-development with experienced data analysts. We also discuss the implications TexTonic could have on visual exploration and discovery tasks

human centered computing, Visual analytics, Inform↗

Laser ultrasonic imaging of subsurface defects with the linear sampling method

Laser ultrasonics is a remote nondestructive evaluation technique suitable for real-time monitoring of fabrication processes in semiconductor metrology, advanced manufacturing, and other applications where non-contact, high fidelity measurements are required. Here we investigate laser ultrasonic data processing approaches to reconstruct images of subsurface side drilled holes in aluminum alloy specimens. We demonstrate through simulation that the model-based linear sampling method (LSM) can perform accurate shape reconstruction of single and multiple holes and produce images with well-defined boundaries. We experimentally confirm that LSM produces images that represent the internal geometric features of an object, some of which may be missed by conventional imaging.

42 ENGINEERING↗

Artificial Intelligence in Nuclear Physics

Artificial Intelligence (AI) and Machine Learning (ML) are rapidly developing fields providing data-driven algorithms to predict, classify, and make decisions based on data. Nuclear Physics Research is data-driven and AI/ML techniques have been implemented for experiment and accelerator control, in theoretical applications, and in data processing and analysis. These algorithms open possibilities for automation, thereby augmenting human capabilities. Additionally, Open Science is enabled by simultaneous analyses of multiple data sources, leading to scientific knowledge. This talk will summarize current applications of AI/ML in nuclear physics, as well as accelerator applications, and will cover upcoming initiatives and research in AI/ML.

Jeske, Torri↗

Physics-Guided Machine Learning (PGML) for Improved Aerostructure Manufacturing

The Physics-Guided Machine Learning (PGML) for Improved Aerostructure Manufacturing project objective is to automatically tune machining parameter predictions from physics-based models using process data and Bayesian machine learning. The intent is to enable a step change in aerospace manufacturing by combining machine learning, physics-based process models, and sensors/data in a comprehensive digital environment that simultaneously considers the computer numerically controlled (CNC) machining center capabilities, the workpiece material and geometry, and the workpiece support (fixturing). The project hypothesis is that this combination will enable improved performance in machining operations.

42 ENGINEERING↗