SEARCH · Engineering Papers

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

All sources NASA NTRS DOE OSTI

DOE OSTI2013

Oak Ridge Bio-surveillance Toolkit (ORBiT): Integrating Big-Data Analytics with Visual Analysis for Public Health Dynamics

In this position paper, we describe the design and implementation of the Oak Ridge Bio-surveillance Toolkit (ORBiT): a collection of novel statistical and machine learning tools implemented for (1) integrating heterogeneous traditional (e.g. emergency room visits, prescription sales data, etc.) and non-traditional (social media such as Twitter and Instagram) data sources, (2) analyzing large-scale datasets and (3) presenting the results from the analytics as a visual interface for the end-user to interact and provide feedback. We present examples of how ORBiT can be used to summarize ex- tremely large-scale datasets effectively and how user interactions can translate into the data analytics process for bio-surveillance. We also present a strategy to estimate parameters relevant to dis- ease spread models from near real time data feeds and show how these estimates can be integrated with disease spread models for large-scale populations. We conclude with a perspective on how integrating data and visual analytics could lead to better forecasting and prediction of disease spread as well as improved awareness of disease susceptible regions.

Ramanathan, Arvind↗

DOE OSTI2014

F3D Image Processing and Analysis for Many - and Multi-core Platforms

F3D is written in OpenCL, so it achieve[sic] platform-portable parallelism on modern mutli-core CPUs and many-core GPUs. The interface and mechanims to access F3D core are written in Java as a plugin for Fiji/ImageJ to deliver several key image-processing algorithms necessary to remove artifacts from micro-tomography data. The algorithms consist of data parallel aware filters that can efficiently utilizes[sic] resources and can work on out of core datasets and scale efficiently across multiple accelerators. Optimizing for data parallel filters, streaming out of core datasets, and efficient resource and memory and data managements over complex execution sequence of filters greatly expedites any scientific workflow with image processing requirements. F3D performs several different types of 3D image processing operations, such as non-linear filtering using bilateral filtering and/or median filtering and/or morphological operators (MM). F3D gray-level MM operators are one-pass constant time methods that can perform morphological transformations with a line-structuring element oriented in discrete directions. Additionally, MM operators can be applied to gray-scale images, and consist of two parts: (a) a reference shape or structuring element, which is translated over the image, and (b) a mechanism, or operation, that defines the comparisons to be performed between the image and the structuring element. This tool provides a critical component within many complex pipelines such as those for performing automated segmentation of image stacks. F3D is also called a "descendent" of Quant-CT, another software we developed in the past. These two modules are to be integrated in a next version. Further details were reported in: D.M. Ushizima, T. Perciano, H. Krishnan, B. Loring, H. Bale, D. Parkinson, and J. Sethian. Structure recognition from high-resolution images of ceramic composites. IEEE International Conference on Big Data, October 2014.

Source record↗

DOE OSTI2014

F3D Image Processing and Analysis for Many - and Multi-core Platforms

Source record↗

DOE OSTI2025

A Data Science and Machine Learning Platform Supporting Large Particle Accelerator Control and Diagnostics Applications Final Report: SBIR Initial Phase II DE-SC0022583

The Machine Learning Data Platform (MLDP) is a product providing full-stack support for data science, Machine Learning, and Artificial Intelligence (ML/AI) applications at particle accelerator and large experimental physics facilities. It supports ML/AI applications from front-end, high-speed acquisition of heterogeneous, time-series data, through data archiving and management, to back-end analysis. The MLDP embodies a “data-science ready” platform for data analysis and ML/AI applications in diagnosis, modelling, control, and optimization of these facilities. It provides data scientists and applications a consistent, datacentric interface to archive data standardizing implementation and deployment of ML/AI algorithms to different operations configurations within the same facility, or between facilities. Being an open-source, public-domain project, the MLDP is intended for broadest possible impact by increasing accessibility and minimizing the required expertise for installation and operation. The MLDP can also be deployed at user facilities for experimental data collection, archiving, and analysis. It is capable of acquisition and archiving of heterogeneous data from experimental equipment (e.g., images, arrays, structures, etc.) along with system hardware configurations (e.g., scalars, tables), control system process variables, and any metadata required for provenance. Thus, the MLDP can manage experimental data through its entire lifecycle, from acquisition and archiving, through analysis and investigation, to release and final publication.

43 PARTICLE ACCELERATORS↗

DOE OSTI2024

Machine Learning-Enabled Image Classification for Automated Electron Microscopy

Abstract Traditionally, materials discovery has been driven more by evidence and intuition than by systematic design. However, the advent of “big data” and an exponential increase in computational power have reshaped the landscape. Today, we use simulations, artificial intelligence (AI), and machine learning (ML) to predict materials characteristics, which dramatically accelerates the discovery of novel materials. For instance, combinatorial megalibraries, where millions of distinct nanoparticles are created on a single chip, have spurred the need for automated characterization tools. This paper presents an ML model specifically developed to perform real-time binary classification of grayscale high-angle annular dark-field images of nanoparticles sourced from these megalibraries. Given the high costs associated with downstream processing errors, a primary requirement for our model was to minimize false positives while maintaining efficacy on unseen images. We elaborate on the computational challenges and our solutions, including managing memory constraints, optimizing training time, and utilizing Neural Architecture Search tools. The final model outperformed our expectations, achieving over 95% precision and a weighted F-score of more than 90% on our test data set. This paper discusses the development, challenges, and successful outcomes of this significant advancement in the application of AI and ML to materials discovery.

Materials Science↗

DOE OSTI2023

Smart Mobility in the Cloud: Enabling Real-Time Situational Awareness and Cyber-Physical Control Through a Digital Twin for Traffic

This article presents the design, implementation, and use cases of the Chattanooga Digital Twin (CTwin) towards the vision for next-generation smart city applications for urban mobility management. CTwin is an end-to-end web-based platform that incorporates various aspects of the decision-making process for optimizing urban transportation systems in Chattanooga, Tennessee, to reduce traffic congestion, incidents, and vehicle fuel consumption. The platform serves as a cyberinfrastructure to collect and integrate multi-domain urban mobility data from various online repositories and Internet of Things (IoT) sensors, covering multiple urban aspects (e.g., traffic, natural hazards, weather, and safety) that are relevant to urban mobility management. The platform enables advanced capabilities for: (a) real-time situational awareness on traffic and infrastructure conditions on highways and urban roads, (b) cyber-physical control for optimizing traffic signal timing, and (c) interactive visual analytics on big urban mobility data and various metrics for traffic prediction and transportation performance evaluation. The platform is designed using a multi-level componentization paradigm and is implemented using modular and adaptive architecture, rendering it as a generalizable and extendable prototype for other urban management applications. We present several use cases to demonstrate CTwin's core capabilities for supporting decision-making in smart urban mobility management.

33 ADVANCED PROPULSION SYSTEMS↗

DOE OSTI2015

SPR Hydrostatic Column Model Verification and Validation.

A Hydrostatic Column Model (HCM) was developed to help differentiate between normal "tight" well behavior and small-leak behavior under nitrogen for testing the pressure integrity of crude oil storage wells at the U.S. Strategic Petroleum Reserve. This effort was motivated by steady, yet distinct, pressure behavior of a series of Big Hill caverns that have been placed under nitrogen for extended period of time. This report describes the HCM model, its functional requirements, the model structure and the verification and validation process. Different modes of operation are also described, which illustrate how the software can be used to model extended nitrogen monitoring and Mechanical Integrity Tests by predicting wellhead pressures along with nitrogen interface movements. Model verification has shown that the program runs correctly and it is implemented as intended. The cavern BH101 long term nitrogen test was used to validate the model which showed very good agreement with measured data. This supports the claim that the model is, in fact, capturing the relevant physical phenomena and can be used to make accurate predictions of both wellhead pressure and interface movements.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2015

BactoGeNIE: A large-scale comparative genome visualization for big displays

The volume of complete bacterial genome sequence data available to comparative genomics researchers is rapidly increasing. However, visualizations in comparative genomics--which aim to enable analysis tasks across collections of genomes--suffer from visual scalability issues. While large, multi-tiled and high-resolution displays have the potential to address scalability issues, new approaches are needed to take advantage of such environments, in order to enable the effective visual analysis of large genomics datasets. In this paper, we present Bacterial Gene Neighborhood Investigation Environment, or BactoGeNIE, a novel and visually scalable design for comparative gene neighborhood analysis on large display environments. We evaluate BactoGeNIE through a case study on close to 700 draft Escherichia coli genomes, and present lessons learned from our design process. In conclusion, BactoGeNIE accommodates comparative tasks over substantially larger collections of neighborhoods than existing tools and explicitly addresses visual scalability. Given current trends in data generation, scalable designs of this type may inform visualization design for large-scale comparative research problems in genomics.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2020

Multi-Functional Distributed Fiber Sensors for Pipeline Monitoring and Methane Detections. Final Report

As an abundant and cheap fossil energy source, natural gas has become a significant energy supply to support the United States’ economy. However, the large-scale extraction and utilization of natural gas also impose significant challenges on methane leakage. This problem is exacerbated by aging gas utility delivery systems, including interstate high-pressure pipelines, storage, and transmission facilities. This project aims to develop a cost-effective fiber optical sensing method that can perform multi-parameter real-time measurements of natural gas pipelines across long interrogation distances up to 100 km with 1-meter spatial resolution. This sensing tool can evaluate overall pipeline efficiency and reduce methane emissions for mid-stream methane infrastructures. To accomplish this objective, research and development efforts funded by this project have resulted in the following accomplishments: This project successfully has developed new functional sensory polymer materials using Metal-Organic Frameworks (MOFs) that can be coated on optical fiber through the reel-to-reel coating process. Functional polymer-coated optical fibers can perform sensitive methane detection through evanescence wave interaction and strain-based measurements to achieve 1% detection sensitivities. The new sensors fibers support both distributed measurements and multiplexed fiber sensors array for multi-point measurements. This project developed and optimized a new multi-core optical fiber that supports simultaneous and distributed measurements of strain and temperatures with 1-meter spatial resolutions across up to 100-km interrogation distance. This new fiber, combined with sensory polymercoated fiber, could perform both distributed temperature and methane detections. This project developed a new artificial intelligence big data algorithm approach that can effectively analyze high-resolution data harnessed by distributed fiber sensors to protect natural gas pipelines against external threats and detect internal defects induced by corrosion. Working with our industry partner, this project developed new optical fibers that support fiber sensor fabrications through polymer coating after the fibers are drawn. These new fibers eliminate the need for direct sensor fabrication when the fiber is fabricated on a fiber draw tower, which drastically expands fiber sensors' applicability. This research project has significantly advanced the distributed fiber sensing technology. It will dramatically increase the applicability and adaptability of distributed fiber sensors for a wide array of applications in energy, sustainability, and environmental science, including structural health monitoring of natural gas pipelines, oil infrastructures, hydrogen facilities, and environmental monitoring of carbon storage sites, water supply systems, and others.

03 NATURAL GAS↗

DOE OSTI2017

Constraining nuclear data via cosmological observations: Neutrino energy transport and big bang nucleosynthesis

Here, we introduce a new computational capability that moves toward a self-consistent calculation of neutrino transport and nuclear reactions for big bang nucleosynthesis (BBN). Such a self-consistent approach is needed to be able to extract detailed information about nuclear reactions and physics beyond the standard model from precision cosmological observations of primordial nuclides and the cosmic microwave background radiation. We also calculate the evolution of the early universe through the epochs of weak decoupling, weak freeze-out and big bang nucleosynthesis (BBN) by simultaneously coupling a full strong, electromagnetic, and weak nuclear reaction network with a multi-energy group Boltzmann neutrino energy transport scheme. The modular structure of our approach allows the dissection of the relative contributions of each process responsible for evolving the dynamics of the early universe. Such an approach allows a detailed account of the evolution of the active neutrino energy distribution functions alongside and self-consistently with the nuclear reactions and entropy/heat generation and flow between the neutrino and photon/electron/positron/baryon plasma components. Our calculations reveal nonlinear feedback in the time evolution of neutrino distribution functions and plasma thermodynamic conditions. We discuss the time development of neutrino spectral distortions and concomitant entropy production and extraction from the plasma. These effects result in changes in the computed values of the BBN deuterium and helium-4 yields that are on the order of a half-percent relative to a baseline standard BBN calculation with no neutrino transport. This is an order of magnitude larger effect than in previous estimates. For particular implementations of quantum corrections in plasma thermodynamics, our calculations show a 0.4% increase in deuterium and a 0.6% decrease in 4 He over our baseline. The magnitude of these changes are on the order of uncertainties in the nuclear physics for the case of deuterium and are potentially significant for the error budget of helium in upcoming cosmological observations.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2020

Adaptive Anomaly Detection for Dynamic Clinical Event Sequences

Over the past decade, health information technology (IT) has enabled the amount of digital information stored in electronic health records (EHRs) to expand greatly. However, according to some studies, hazards in health IT can lead to changes in clinical decisions, care processes, and care outcomes, as well as other issues. Thus, the effects of health IT hazards on patient safety have been at the forefront of recent patient safety research. Nonetheless, hazard detection in health IT remains a challenge. In this paper, the authors assume that safety-related issues in health IT would exhibit anomalous characteristics in EHR data. Although all hazards will exhibit some anomalous characteristics, not all anomalies can be regarded as hazards. The authors hypothesize that errors in health IT could lead to interruptions in the sequence of clinical actions. To this end, the problem of detecting anomalous sequences in big EHR data is considered. This paper focuses on dynamic event sequences, which are a series of clinical actions in motion. The authors propose an adaptive anomaly detection approach that uses higher-order network representation to detect anomalous sequences. Furthermore, the authors propose a contiguous subsequence anomaly detection approach that identifies abnormal subsequences in the detected anomalous sequences. The proposed approaches are tested by using synthetic and real-world EHR data. The proposed methods outperform existing state of the art anomaly detection techniques. To reduce the computational complexity associated with the operational implementation of the proposed approaches, the Apache Spark environment was leveraged, and a much shorter run time together with improved performance were achieved, especially for data with more than 60,000 sequences.

Niu, Haoran↗

DOE OSTI2025

Barge Site - Avian Radar System / Derived Data

This is a combined data set of 67,410 bird/bat tracks from an avian radar system deployed on a research barge (MERLIN True3D, DeTect, Panama City, Florida, USA) and concurrent wind measurements from two scanning lidars (WindCube v2.1, Vaisala, Vantaa, Finland, and Halo XR+, Halo Photonics, Lannion, France). The research barge (16.5 m x 61 m) was deployed as part of the Wind Forecast Improvement Project (WFIP-3) off the northeast coast of the United States south of Massachusetts (40.9 deg N, 70.79 deg W). This data set comprises 5 weeks of data between August 27th 2024 and September 27th 2024. Radar data were provided by DeTect and Lidar data were accessed through the Wind Data Hub (wfip3/barg.WINDPROF.z01.a0) The data have been filtered and sorted into two size groups ("big" and "small") based on a clustering approach. See Snortland, A., Clerc, J., Hein, C., & Cotter, E. (2025). Wind as Driver of Bird and Bat Abundance, Flight Direction, Altitude, and Speed on the North Atlantic Shelf. arXiv preprint arXiv:2511.14983 for complete details. Data are provided in 2 files: "Birds" and "Birds_hourly" Birds: This file contains information about each of the 67,410 flying animal tracks detected by the radar during the data collection period, including parameters measured by the radar and wind information interpolated from the lidar wind measurements. We note that the raw radar dataset contained 301,618 tracks; tracks in this processed dataset were filtered based on the requirements described in Snortland et al. (2025). Birds_hourly: This file contains timeseries of the number of tracks detected per hour over the course of the data collection period, including wind conditions and sun position for each hour. These data were used for generalized additive modeling in Snortland et al. (2025).

17 WIND ENERGY↗

DOE OSTI2021

MindSynchro

This report presents the developments and results of MindSynchro project as part of DOE OE FOA 1861. DOE and Pacific Northwest National Laboratory (PNNL) have made available to FOA awardees datasets containing years of real historical data recorded from various phasor measurement units (PMUs) which are installed in three large US interconnections: Texas (IC A), Western (IC B), and Eastern (IC C). The main goal of the project, which was successfully achieved, was to develop methods for detection and identification of events which are relevant for power grid operation. Tasks performed for achieving the project goals included data exploration and pre-processing, the development and application of physics-based features, data analysis and labeling based on unsupervised learning approaches, training and testing of DSSL models for classification of events which are relevant for power grid operation, and deployment of solutions to cloud environments. The methods developed in the project can potentially provide relevant benefits to power grid asset owners/operators in general in terms of situational awareness. Two main types of outcomes can be provided by these tools: Identification of specific relevant power grid event types: Semi-supervised ML methods developed in the project can adequately employ not only the relatively scarce labeled data but also the large amount of available unlabeled data to train models for detection of specific event types. Such methods enable the application of trained models for the detection of events in a population of PMUs much larger than that associated to the labeled events. Support in data labeling / label validation: Labels are critical for training of models for identification of specific types of events. However, labeling large amounts of data is a manual and tedious process. This means that such process is error prone and is not scalable. Methods developed in the project, based on ensembles of clustering models, have been successfully employed for turning manual labeling into a scalable process. Accurate identification of specific relevant events can provide the operators with immediate situational awareness that could otherwise require hours or days of analysis from domain experts. We envision that such methods could be initially employed in support of post-mortem analysis of events and, as confidence is gained, they could be employed for online/real-time support, providing, among other benefits, insights for avoiding major events which could happen due to a combination of smaller ones. On the longer term, related methods could potentially be employed to improve protection and control.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2013

Integrating Online Compression to Accelerate Large-Scale Data Analytics Applications

Abstract—With growing dataset sizes, and as computing cycles are increasing faster than storage and wide-area bandwidths, compression appears like a promising approach for improving the performance of large-scale data analytics applications. In this context, this paper makes the following contributions. First, we develop a new compression methodology, which exploits the similarities between spatial and/or temporal neighbors in a simulation dataset, and enables high compression ratios and low decompression costs. Second, we have developed a framework that can be used to incorporate a variety of compression and decompression algorithms. This framework also supports a simple API to allow integration with an existing application or data processing middleware. Once a compression algorithm is implemented, this framework can allow multi-threaded retrieval, multi-threaded data decompression, and use of informed prefetching and caching. By integrating this framework with a data-intensive middleware, we have applied our compression methodology and framework to three applications over two datasets, including a GCRM climate model dataset. We obtained an average compression ratio of 51.68%, and up to 53.27% improvement in execution time of data analysis applications.

Big Data, I/O, HPC↗

DOE OSTI2016

Making Advanced Scientific Algorithms and Big Scientific Data Management More Accessible

Synchrotrons such as the Advanced Light Source (ALS) at Lawrence Berkeley National Laboratory are known as user facilities. They are sources of extremely bright X-ray beams, and scientists come from all over the world to perform experiments that require these beams. As the complexity of experiments has increased, and the size and rates of data sets has exploded, managing, analyzing and presenting the data collected at synchrotrons has been an increasing challenge. The ALS has partnered with high performance computing, fast networking, and applied mathematics groups to create a"super-facility", giving users simultaneous access to the experimental, computational, and algorithmic resources to overcome this challenge. This combination forms an efficient closed loop, where data despite its high rate and volume is transferred and processed, in many cases immediately and automatically, on appropriate compute resources, and results are extracted, visualized, and presented to users or to the experimental control system, both to provide immediate insight and to guide decisions about subsequent experiments during beam-time. In this paper, We will present work done on advanced tomographic reconstruction algorithms to support users of the 3D micron-scale imaging instrument (Beamline 8.3.2, hard X-ray micro-tomography).

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

An effective online data monitoring and saving strategy for large-scale climate simulations

Large-scale climate simulation models have been developed and widely used to generate historical data and study future climate scenarios. These simulation models often have to run for a couple of months to understand the changes in the global climate over the course of decades. This long-duration simulation process creates a huge amount of data with both high temporal and spatial resolution information; however, how to effectively monitor and record the climate changes based on these large-scale simulation results that are continuously produced in real time still remains to be resolved. Due to the slow process of writing data to disk, the current practice is to save a snapshot of the simulation results at a constant, slow rate although the data generation process runs at a very high speed. This study proposes an effective online data monitoring and saving strategy over the temporal and spatial domains with the consideration of practical storage and memory capacity constraints. Finally, our proposed method is able to intelligently select and record the most informative extreme values in the raw data generated from real-time simulations in the context of better monitoring climate changes.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2015

UPR/Mayaguez High Energy Physics

For the period of sixteen years covered by this report (June 1, 1997 - July 31, 2013) the High Energy Physics Group at the University of Puerto Rico’s Mayaguez Campus (UPRM) carried out an extensive research program that included major experiments at Fermi National Accelerator Laboratory (Fermilab), the Cornell Electron-positron Collider and CERN. In particular, these were E831 (FOCUS) at Fermilab, CLEOc at Cornell and the Compact Muon Solenoid (CMS) at the Large Hadron Collider (LHC) at CERN. The group’s history is one of successful execution and growth. Beginning with one faculty researcher in 1985, it eventually included four faculty researchers, one post-doctoral research associate, two undergraduates and as many as six graduate students at one time working on one of the experiments that discovered the Higgs boson. Some of this expansion was due to the group’s leveraging of funds from the Department of Energy’s core grant to attract funds from National Science Foundation programs not targeted to high energy physics. Besides the group’s research productivity, its other major contribution was the training of a large number of MS students who later went on to successful technical careers in industry as well as academia including many who obtained PhD degrees at US universities. In an attempt to document this history, this final report gives a general description of the Group’s work prior to June 1, 2010, the starting date for the last grant renewal period. Much more detail can, of course, be found in the annual reports submitted up to that date. The work during the last grant period is discussed in detail in a separate section. To summarize the group’s scientific accomplishments, one can point to the results of the experiments. Both FOCUS and CLEOc were designed to carry out precise measurements of processes involving the heavy quarks, charm and bottom. Heavy quarks are particularly interesting because, due to their mass, theoretical calculations based on the Standard Model have less uncertainty than those for the light quarks. Precise heavy quark experiments can therefore yield some of the best tests of the Standard Model and of the approximations that are made in calculating measurable observables. Both FOCUS and CLEOc were highly successful achieving significant improvement in the precision of measurements such as lifetimes and decay branching ratios. For example, FOCUS obtained a data sample that contained ten times as many heavy quark decay events as its predecessor. CMS was a big shift in the group’s research. During the first decade of the century it became clear that the LHC would be the world’s highest energy accelerator offering a unique opportunity for discovery. Given the UPRM’s group record of achievement, it was successful in obtaining admission to the CMS collaboration in March, 2006, becoming the first institution to do so that did not have a PhD program. CMS is one of two major experiments at the LHC. Although the plans are for these experiments to run for many years with increased energy and event rates, they have already achieved one of their principal goals. The test for the existence of the Higgs boson, a particle which plays a unique role in the Standard Model but had not been observed, was answered in the affirmative in 2012.The particular contributions of the UPRM group to these experiments make up the majority of this report although other contributions such as the training of students, outreach to the general community and the organization of scientific meetings are also discussed.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2017

Python and HPC for High Energy Physics Data Analyses

High level abstractions in Python that can utilize computing hardware well seem to be an attractive option for writing data reduction and analysis tasks. In this paper, we explore the features available in Python which are useful and efficient for end user analysis in High Energy Physics (HEP). A typical vertical slice of an HEP data analysis is somewhat fragmented: the state of the reduction/analysis process must be saved at certain stages to allow for selective reprocessing of only parts of a generally time-consuming workflow. Also, algorithms tend to to be modular because of the heterogeneous nature of most detectors and the need to analyze different parts of the detector separately before combining the information. This fragmentation causes difficulties for interactive data analysis, and as data sets increase in size and complexity (O10 TiB for a “small” neutrino experiment to the O10 PiB currently held by the CMS experiment at the LHC), data analysis methods traditional to the field must evolve to make optimum use of emerging HPC technologies and platforms. Mainstream big data tools, while suggesting a direction in terms of what can be done if an entire data set can be available across a system and analysed with high-level programming abstractions, are not designed with either scientific computing generally, or modern HPC platform features in particular, such as data caching levels, in mind. Our example HPC use case is a search for a new elementary particle which might explain the phenomenon known as “Dark Matter”. Here, using data from the CMS detector, we will use HDF5 as our input data format, and MPI with Python to implement our use case.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗