SEARCH · Engineering Papers

Results for “hierarchical data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

All sources NASA NTRS DOE OSTI

DOE OSTI2023

Progressive Tree-Based Compression of Large-Scale Particle Data

Scientific simulations and observations using particles have been creating large datasets that require effective and efficient data reduction to store, transfer, and analyze. However, current approaches either compress only small data well while being inefficient for large data, or handle large data but with insufficient compression. Toward effective and scalable compression/decompression of particle positions, we introduce new kinds of particle hierarchies and corresponding traversal orders that quickly reduce reconstruction error while being fast and low in memory footprint. Our solution to compression of large-scale particle data is a flexible block-based hierarchy that supports progressive, random-access, and error-driven decoding, where error estimation heuristics can be supplied by the user. For low-level node encoding, we introduce new schemes that effectively compress both uniform and densely structured particle distributions. Our proposed methods thus target all three phases of a tree-based particle compression pipeline, namely tree construction, tree traversal, and node encoding. In conclusion, the improved efficacy and flexibility of these methods over existing compressors are demonstrated through extensive experimentation, using a wide range of scientific particle datasets.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Multi-Label Classification with Constraint-Based Learning for Hierarchical Consistency

We explore the limitations of traditional crossentropy loss in a hierarchical multi-label classification setting and introduce a novel loss function. This function is designed to integrate hierarchical constraints directly into the training process. By incorporating such constraints into the loss, our approach slightly improves the logical consistency of predictions in structured domains. We demonstrate the efficacy of our approach through experiments on primary site and histology classification by using electronic pathology reports. These results show that our proposed hierarchical loss function enhances the model's ability to produce predictions that are logically consistent with the natural data hierarchies, and it slightly improves predictive accuracy. Our framework may be extended to other hierarchical domains, however the performance gains are context specific.

Shivanna, Abhishek [ORNL] (ORCID:0009000665228593)↗

DOE OSTI2025

Uncovering multiscale structure-property correlations via active learning in scanning tunneling microscopy

Atomic arrangements and local sub-structures fundamentally influence emergent material functionalities. These structures are conventionally probed using spatially resolved studies and the property correlations are deciphered by a researcher based on sequential explorations, thereby limiting the efficiency and scope. Here we demonstrate a multi-scale Bayesian deep-learning based framework that automatically correlates material structure with its electronic properties using scanning tunneling microscopy (STM) measurements in real-time. Its predictions are used to autonomously direct exploration toward regions of the sample that optimize a given material property. This method is deployed on a low-temperature ultra-high vacuum STM to understand the structure-property relationship in a europium-based semimetal, EuZn 2 As 2 , a promising candidate relevant to magnetism-driven topological phenomena. The framework employs a sparse-sampling approach to efficiently construct the scalar-property space using minimal measurements, about 1–10% of the data required in standard hyperspectral methods. Moreover, we formulate the problem hierarchically across length scales, implementing autonomous workflow to locate mesoscopic and atomic structures that correspond to a target material property. This framework offers the choice to design scalar-property from the spectroscopic data to steer sample exploration. Our findings reveal correlations of the electronic properties unique to surface terminations, local defect density, and point defects.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2022

Substructure in the stellar halo near the Sun: I. Data-driven clustering in integrals-of-motion space

Context. Merger debris is expected to populate the stellar haloes of galaxies. In the case of the Milky Way, this debris should be apparent as clumps in a space defined by the orbital integrals of motion of the stars. Aims. Our aim is to develop a data-driven and statistics-based method for finding these clumps in integrals-of-motion space for nearby halo stars and to evaluate their significance robustly. Methods. We used data from Gaia EDR3, extended with radial velocities from ground-based spectroscopic surveys, to construct a sample of halo stars within 2.5 kpc from the Sun. We applied a hierarchical clustering method that makes exhaustive use of the single linkage algorithm in three-dimensional space defined by the commonly used integrals of motion energy E, together with two components of the angular momentum, L z and L ⊥ . To evaluate the statistical significance of the clusters, we compared the density within an ellipsoidal region centred on the cluster to that of random sets with similar global dynamical properties. By selecting the signal at the location of their maximum statistical significance in the hierarchical tree, we extracted a set of significant unique clusters. By describing these clusters with ellipsoids, we estimated the proximity of a star to the cluster centre using the Mahalanobis distance. Additionally, we applied the HDBSCAN clustering algorithm in velocity space to each cluster to extract subgroups representing debris with different orbital phases. Results. Our procedure identifies 67 highly significant clusters (> 3σ), containing 12% of the sources in our halo set, and 232 subgroups or individual streams in velocity space. In total, 13.8% of the stars in our data set can be confidently associated with a significant cluster based on their Mahalanobis distance. Inspection of the hierarchical tree describing our data set reveals a complex web of relations between the significant clusters, suggesting that they can be tentatively grouped into at least six main large structures, many of which can be associated with previously identified halo substructures, and a number of independent substructures. This preliminary conclusion is further explored in a companion paper, in which we also characterise the substructures in terms of their stellar populations. Conclusions. Our method allows us to systematically detect kinematic substructures in the Galactic stellar halo with a data-driven and interpretable algorithm. The list of the clusters and the associated star catalogue are provided in two tables available at the CDS.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2026

Scientific Data Compression for Large Scale Computational Fluid Dynamics (CFD) Simulations

This Cooperative Research and Development Agreement (CRADA) between Oak Ridge National Laboratory (ORNL) and General Electric (GE) investigated methods for reducing the size of large computational fluid dynamics (CFD) simulation datasets using scientific data compression techniques. The work focused on adapting the MultiGrid Adaptive Reduction of Data (MGARD) compression framework and integrating it with high-performance I/O and visualization tools used in CFD workflows. MGARD uses hierarchical multilevel decomposition to enable error-controlled compression of floating-point scientific data while preserving quantities of interest. During the project, MGARD compression was integrated with the ADIOS I/O framework and visualization tools such as ParaView to enable efficient storage, transfer, and analysis of simulation data. The collaboration also explored approaches for improving compression performance for CFD data defined on unstructured meshes. Results demonstrate that scientific data compression can significantly reduce storage requirements and improve data management for large-scale CFD simulations.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

AI to Automate ModEx for Optimal Predictive Improvement and Scientific Discovery

Focal Areas: Data acquisition and assimilation enabled by machine learning, AI, advanced experimental optimization, unsupervised learning, and hardware-related AI efforts; Predictive modeling through AI techniques and AI-derived model components; Using AI to design a hierarchical model prediction system consisting and model selection; and, Interrogating complex data (observed and simulated) using AI, big data analytics, and other advanced methods such as explainable AI and physics- or knowledge-guided AI.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2023

Sequential Image Recovery Using Joint Hierarchical Bayesian Learning

Abstract Recovering temporal image sequences (videos) based on indirect, noisy, or incomplete data is an essential yet challenging task. We specifically consider the case where each data set is missing vital information, which prevents the accurate recovery of the individual images. Although some recent (variational) methods have demonstrated high-resolution image recovery based on jointly recovering sequential images, there remain robustness issues due to parameter tuning and restrictions on the type of sequential images. Here, we present a method based on hierarchical Bayesian learning for the joint recovery of sequential images that incorporates prior intra- and inter-image information. Our method restores the missing information in each image by “borrowing” it from the other images. More precisely, we couple sequential images by penalizing their pixel-wise difference. The corresponding penalty terms (one for each pixel and pair of subsequent images) are treated as weakly-informative random variables that favor small pixel-wise differences but allow occasional outliers. As a result, all of the individual reconstructions yield improved accuracy. Our method can be used for various data acquisitions and allows for uncertainty quantification. Some preliminary results indicate its potential use for sequential deblurring and magnetic resonance imaging.

Xiao, Yao↗

DOE OSTI2021

Methods and devices for preventing computationally explosive calculations in a computer for model parameters distributed on a hierarchy of geometric simplices

A computer-implemented method of preventing computationally explosive calculations. The method includes obtaining, by a processor of the computer, measured data of one of a physical process or a physical object; performing hierarchical numerical modeling of a physical process inclusive of an Earth model containing at least one of (a) infrastructure in the ground and (b) a formation feature in the ground, wherein predicted data is generated; comparing the measured data to the predicted data to calculate an estimated error; analyzing the estimated error via an inversion process to update the at least one of the Earth model and infrastructure model so as to reduce the estimated error and to determine a final composite Earth model of at least one of the infrastructure and the feature; and using the final composite Earth model to characterize at least one of the process and the physical object.

Source record↗

DOE OSTI2022

Robust Carbon Dioxide Plume Imaging Using Joint Tomographic Inversion of Seismic Onset Time and Distributed Pressure and Temperature Measurements (Final Report)

We develop and demonstrate rapid and cost-effective methodologies for spatiotemporal tracking of CO2 plumes during geologic sequestration using joint inversion of seismic data and distributed pressure and temperature measurements. Key elements of our methodology are: (a) a computationally efficient approach to pressure and temperature propagation, (b) analysis of time lapse seismic data using a novel ‘seismic onset time’ approach to detect fluid front propagation, and (c) data assimilation and uncertainty assessment via joint inversion of pressure, temperature and time lapse seismic data, and (d) validating the numerical tomographic inversion using a CO2 injection demonstration projects, specifically data collected from the from the Petra Nova Parish Holdings CCUS project in the West Ranch Field, Texas and the Chester-16 reef CO2 injection site in Northern Michigan which is part of the DOE Midwestern Carbon Sequestration Project. The research team is led by Texas A&M University and includes Battelle as a subcontractor with support from Shell, Anadarko, Chevron and JX Nippon. A carbon dioxide (CO2) water-alternating-gas (WAG) pilot was conducted to gain insights into tertiary oil recovery potential via CO2 flood in the West Ranch Field as part of the Petra Nova project, the world’s largest post-combustion CO2 capture and utilization initiative. With a fluvial formation geology and large contrasts in permeability, this is a challenging and novel application of CO2 enhanced oil recovery (EOR). We build a predictive dynamic model of the subsurface that incorporates the multiphase and compositional data acquired during the pilot operation. The calibrated model is used for the carbon dioxide plume imaging. The study began with an initialization of the pilot sector model extracted from a calibrated full-field model. The pilot model calibration follows a two-step hierarchical workflow. First, we performed a large-scale update of the permeability distribution by integrating available bottomhole pressure and multiphase production data. In the second step, local permeability field is fine-tuned using a streamline-based method to match CO2 breakthrough times at the producers. The predictive capability of the calibrated model was verified through two blind validation tests: (1) the model showed good agreement with saturation logs acquired at two observation wells; and (2) the model reproduced the CO2 recovery as a fraction of the injected CO2. The use of seismic onset times has shown great promise for integrating near-continuous seismic surveys for updating geologic models. In this study, we analyze the impact of seismic survey frequency on the onset time approach aiming to extend the application of onset time to infrequent seismic surveys. In addition, we quantitatively examine the nonlinearity of the onset time method and compare it to the commonly used amplitude inversion method. We carry out a sensitivity analysis of seismic survey frequency based on the complete seismic survey data (over 175 surveys) of steam injection in a heavy oil reservoir (Peace River Unit) in Canada. Our results show that an adequate onset time map can be obtained from the infrequent seismic surveys by interpolation between seismic surveys as long as there is no change in the dominant underlying physics between the successive surveys. The study also shows that nonlinearity of the onset time method can be -smaller than that of the amplitude inversion method by several orders of magnitude. Application to the Brugge benchmark case shows that the onset time method obtains comparable permeability update as the traditional seismic amplitude inversion method with faster computation and improved convergence characteristics. We extend the streamline-based data integration approach to incorporate distributed temperature sensor (DTS) data using the concept of thermal tracer travel time. Then, a hierarchical workflow composed of evolutionary and streamline methods is employed to jointly history match the DTS and pressure data. Finally, CO2 saturation and streamline maps are used to visualize the CO2 plume movement during the sequestration process. The hierarchical workflow is applied to a carbon sequestration project in a carbonate reef reservoir within the Northern Niagaran Pinnacle Reef Trend in Michigan, USA. The monitoring data set consists of distributed temperature sensing (DTS) data acquired at the injection well and a monitoring well, flowing bottom-hole pressure data at the injection well, and time-lapse pressure measurements at several locations along the monitoring well. The history matching results indicate that the CO2 movement is mostly restricted to the intended zones of injection which is consistent with an independent warm-back analysis of the temperature data. In addition to employing simulation models and inverse methods for CO2 plume imaging, we also initialized a data-driven technology for detecting inter-well connectivity based on production and pressure data. Our machine-learning framework is built on the statistical recurrent unit (SRU) model and interprets well-based injection/production data into inter-well connectivity without relying on a geologic model. We test it on synthetic and field-scale CO2 EOR projects utilizing the water-alternating-gas (WAG) process. The validation of the proposed data-driven inter-well connectivity assessment is performed using synthetic data from simulation models where inter-well connectivity can be easily measured using the streamline-based flux allocation. The SRU model is shown to offer excellent prediction performance on the synthetic case. Despite significant measurement noise and frequent well shut-ins imposed in the field-scale case, the SRU model offers good prediction accuracy, the overall relative error of the phase production rates at most producers ranges from 10% to 30%. It is shown that the dominant connections identified by the data-driven method and streamline method are in close agreement. Texas A&M University, the lead organization in the project, was primarily responsible for the development of tomographic approaches for CO2 plume mapping in conjunction with distributed pressure, temperature and seismic onset time data. Battelle, as a subcontractor, was primarily responsible for the development of analytical and empirical methods for analyzing transient injection rate and pressure data from point/line sources such as injection and monitoring wells. An additional area of emphasis for Battelle was the use of machine learning for such tasks as inferring reservoir connectivity information from injection-production data, and identifying variable importance for machine learning-based proxy models developed from full-physics simulations. The two organizations also collaborated on the application of the tomographic inversion methodology for a field data set.

02 PETROLEUM↗

DOE OSTI2024

Reduced‐Order Probabilistic Emulation of Physics‐Based Ring Current Models: Application to RAM‐SCB Particle Flux

Abstract In this work, we address the computational challenge of large‐scale physics‐based simulation models for the ring current. Reduced computational cost allows for significantly faster than real‐time forecasting, enhancing our ability to predict and respond to dynamic changes in the ring current, valuable for space weather monitoring and mitigation efforts. Additionally, it can also be used for a comprehensive investigation of the system. Thus, we aim to create an emulator for the Ring current‐Atmosphere interactions Model with Self‐Consistent magnetic field (RAM‐SCB) particle flux that not only improves efficiency but also facilitates forecasting with reliable estimates of prediction uncertainties. The probabilistic emulator is built upon the methodology developed by Licata and Mehta (2023), https://doi.org/10.1029/2022sw003345 . A novel discrete sampling is used to identify 30 simulation periods over 20 years of solar and geomagnetic activity. Focusing on a subset of particle flux, we use Principal Component Analysis for dimensionality reduction and Long Short‐Term Memory (LSTM) neural networks to perform dynamic modeling. Hyperparameter space was explored extensively resulting in about 5% median symmetric accuracy across all data sets for one‐step dynamic prediction. Using a hierarchical ensemble of LSTMs, we have developed a reduced‐order probabilistic emulator (ROPE) tailored for time‐series forecasting of particle flux in the ring current. This ROPE offers accurate predictions of omnidirectional flux at a single energy with no pitch angle information, providing robust predictions on the test set with an error score below 11% and calibration scores under 8% with bias under 2% providing a significant speed up as compared to the full RAM‐SCB run.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

Characterizing manufacturing sector disruptions with targeted mitigation strategies

It has become clear in recent decades that manufacturing supply chains are increasingly vulnerable to disruptions of varying geographical scales and intensities. These disruptions—whether intentional, accidental, or resulting from natural disasters—cause failures and capacity reductions to manufacturing infrastructure, with lasting effects that can cascade throughout the manufacturing network. An overall lack of understanding of solutions to mitigate disturbances has rendered the challenge of reducing manufacturing supply chain vulnerability even more difficult. Additionally, the variability of disruptions and their impacts complicates policy maker and stakeholder efforts to plan for specific disruptive scenarios. It is necessary to comprehend different kinds of disturbances and group them based on stakeholder-provided metrics to support planning processes and modeling efforts that promote adaptable, resilient manufacturing supply chains. This paper reviews existing methods for risk management in manufacturing supply chains and the economic and environmental impacts of disruptions. In addition, we develop a framework using agglomerative hierarchical clustering to classify disruptions using U.S. manufacturing network data between 2000 and 2021 and characteristic metrics defined in the literature. Our review identifies five groups of disruptions and discusses both general mitigation methods and strategies targeting each identified group. Further, we highlight gaps in the literature related to estimating and including environmental costs in disaster preparedness and mitigation planning. We also discuss the lack of easily available metrics to quantify environmental impacts of disruptions and how such metrics could be included into our methodology.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2021

Micro on a macroscale: relating microbial-scale soil processes to global ecosystem function

ABSTRACT Soil microorganisms play a key role in driving major biogeochemical cycles and in global responses to climate change. However, understanding and predicting the behavior and function of these microorganisms remains a grand challenge for soil ecology due in part to the microscale complexity of soils. It is becoming increasingly clear that understanding the microbial perspective is vital to accurately predicting global processes. Here, we discuss the microbial perspective including the microbial habitat as it relates to measurement and modeling of ecosystem processes. We argue that clearly defining and quantifying the size, distribution and sphere of influence of microhabitats is crucial to managing microbial activity at the ecosystem scale. This can be achieved using controlled and hierarchical sampling designs. Model microbial systems can provide key data needed to integrate microhabitats into ecosystem models, while adapting soil sampling schemes and statistical methods can allow us to collect microbially-focused data. Quantifying soil processes, like biogeochemical cycles, from a microbial perspective will allow us to more accurately predict soil functions and address long-standing unknowns in soil ecology.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

Joint Resource Modeling and Assessment for Hybrid Distributed Solar and Wind Systems

The inherent variability and uncertainty in distributed energy resources can presents myriad challenges to the planning and operations of power systems. These risks are poised to become larger as the penetration of renewable energy sources rises in the power generation mix. Hybrid solar-wind energy systems are able to mitigate some of these risks by their complementary resource availability. Surface solar and wind fields are coupled and correlated in both space and time. Appropriately estimating the hybrid solar wind energy system requires simulating the spatio-temporal structure of these fields that can be produced for each time horizon. We introduce a novel joint spatio-temporal stochastic differential equation (SPDE) approach that captures the spatio-temporal dynamics of solar and wind fields and their joint dependency over a domain for each time step. In the case study on Colorado, we consider nonstationary three-level hierarchical spatio temporal models for both hourly solar irradiance data and wind speed data in Colorado. Dependence between the solar irradiance data and wind speed data is captured by a shared spatio-temporal random effect. Our approach performs well in terms of the prediction score criterion.

joint modeling↗

DOE OSTI2025

The Aerosol Model Benchmarking Repository: A toolkit for model intercomparison

The Aerosol Model Benchmarking Repository and Standards (AMBRS) project was initiated to provide tools and to establish community standards for benchmarking aerosol models. This report describes a set of open-source tools for building, running, and analyzing aerosol box model simulations in a standardized framework. The framework consists of three core components: AMBuilder, a CMake-based build system that compiles supported models consistently; AMBRS, a Python module that defines unified numerical experiments and executes them with aligned inputs; and PyParticle, an aerosol analysis package that standardizes output, computes diagnostics, and visualizes simulation results. Together, these tools enable reproducible intercomparison of aerosol schemes and support process-level evaluation of how model simplifications affect predictions of size distributions, cloud condensation nuclei activity, and other relevant properties relevant for the Earth-Energy system. Beyond its role in benchmarking, AMBRS provides a platform for studying aerosol processes across scales and can be used to generate training data for AI/ML applications in support of a broader hierarchical aerosol modeling strategy.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Distributed Transient Safety Verification via Robust Control Invariant Sets: A Microgrid Application

Modern safety-critical energy infrastructures are increasingly operated in a hierarchical and modular control framework which allows for limited data exchange between the modules. In this context, it is important for each module to synthesize and communicate constraints on the values of exchanged information in order to assure system-wide safety. To ensure transient safety in inverter-based microgrids, we develop a set invariance-based distributed safety verification algorithm for each inverter module. Applying Nagumo's invariance condition, we construct a robust polynomial optimization problem to jointly search for safety-admissible set of control set-points and design parameters, under allowable disturbances from neighbors. We use sum-of-squares (SOS) programming to solve the verification problem and we perform numerical simulations using grid-forming inverters to illustrate the algorithm.

Bouvier, Jean-Baptiste H.↗

DOE OSTI2025

Conformal Hierarchical Simulation-Based Inference with Local Validity

Trustworthy and interpretable uncertainty quantification is a long-standing challenge in artificial intelligence. Simulation-based inference (SBI) comprises a broad swath of approaches for estimating latent parameters with uncertainties. Although flexible neural density estimators in SBI can be remark- ably expressive capturing highly structured, high-dimensional posteriors their credible regions can be badly mis-calibrated and are often only accompanied by heuristic coverage checks. We present the first SBI framework that delivers finite-sample local valid coverage guarantees that hold in the neighborhood of each observation. Our framework can couple any off-the-shelf hierarchical SBI engine with a confor- mal Bayesian post-processing step that operates on the posterior predictive density. A kernel-weighted conformity score adapts the conformal quantile to the local geometry of the data, yielding prediction sets that are simultaneously (i) marginally calibrated, (ii) locally valid, and (iii) hierarchical, handling global and observation-specific parameters in a single pass. Through experiments on synthetic data and benchmarks from neuroscience and physics, we show that our approach attains 1 − α coverage, where prior SBI methods under- or over-cover. Our approach also maintains a competitive, credible set size with minimal computational overhead. Finally, our approach can be used to make predictions on real data and give valid credible regions modulo weight-initialization-based model mis-specification.

Trivedi, Shubhendu [Fermilab]↗

DOE OSTI2025

Identifying Vehicle Signals in Continuous Seismic Data Using Unsupervised Machine-Learning Techniques

Seismic sensors deployed near roadways effectively capture ground vibrations generated by passing vehicles. Although both traditional and machine‐learning algorithms have been utilized for analyzing such signals, independent validation of detected vehicle events remains limited. We applied two unsupervised machine‐learning algorithms, uniform manifold approximation and projection for dimension reduction, and hierarchical density‐based spatial clustering of applications with noise, to continuous seismic data collected along a road on the main campus of Oak Ridge National Laboratory. The algorithms identified seven distinct cluster labels across the entire dataset. By comparing these cluster labels with precipitation records from a nearby weather station and image‐derived labels from a local camera system, we identified one cluster associated with rainfall and another with vehicle activity. Our algorithms identified a greater number of vehicle‐related labels compared to the camera‐derived labels because seismic data are unaffected by poor lighting conditions. The arrival times of the newly detected vehicle signals corresponded well with the road’s speed limit, supporting our findings. Our algorithm outperformed the short‐term average/long‐term average method and k‐means clustering. Our results suggest that seismic data, when analyzed with machine‐learning algorithms, can complement existing vehicle monitoring systems, particularly under challenging environmental conditions.

Chai, Chengping [Oak Ridge National Laboratory (OR↗

DOE OSTI2021

Scaling pair count to next galaxy surveys

ABSTRACT Counting pairs of galaxies or stars according to their distance is at the core of real-space correlation analyses performed in astrophysics and cosmology. Upcoming galaxy surveys (LSST, Euclid) will measure properties of billions of galaxies challenging our ability to perform such counting in a minute-scale time relevant for the usage of simulations. The problem is only limited by efficient access to the data, hence belongs to the big data category. We use the popular Apache Spark framework to address it and design an efficient high-throughput algorithm to deal with hundreds of millions to billions of input data. To optimize it, we revisit the question of non-hierarchical sphere pixelization based on cube symmetries and develop a new one dubbed the ‘Similar Radius Sphere Pixelization’ (SARSPix) with very close to square pixels. It provides the most adapted indexing over the sphere for all distance-related computations. Using LSST-like fast simulations, we compute autocorrelation functions on tomographic bins containing between a hundred million to one billion data points. In each case, we achieve the construction of a standard pair-distance histogram in about 2 min, using a simple algorithm that is shown to scale, over a moderate number of nodes (16–64). This illustrates the potential of this new techniques in the field of astronomy where data access is becoming the main bottleneck. They can be easily adapted to other use-cases as nearest-neighbours search, catalogue cross-match or cluster finding. The software is publicly available from https://github.com/astrolabsoftware/SparkCorr.

79 ASTRONOMY AND ASTROPHYSICS↗