Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “FILES”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,369 records · Page 76

Surface water nitrogen and sediment potential nitrate reduction rates, nutrient stocks, and stable isotopes from nine wetlands at the Tanglewood Biological Station, Alabama

This dataset supports a broader study investigating wetland hydrologic and biogeochemical responses to inundation disturbances. Bimonthly surface water and sediment sampling events were conducted at nine wetland sites situated within the Tanglewood Biological Station in Alabama from April 2023 to February 2024. The contents included in the data package include surface water nitrogen (nitrogen oxides and ammonium) and sediment potential nitrate reduction rates (measured as potential denitrification and dissimilatory nitrate reduction to ammonium processing), nutrient stocks (total carbon, total nitrogen, and organic matter), and stable isotopes (carbon and nitrogen). Water level data related to each wetland location can be found at https://data.ess-dive.lbl.gov/view/doi:10.15485/2530253 (Kirker et al., 2024) and related water geochemistry data can be found at https://data.ess-dive.lbl.gov/datasets/doi:10.15485/3001967 (Forbes et al., 2025). In addition to a readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. This dataset is comprised of (1) file-level metadata; (2) data dictionary; (3) field metadata and international generic sample numbers (IGSNs); (4) readme; (5) the field protocol; and (6) a subfolder with sample data. The sample data subfolder contains (1) sediment potential denitrification rate, (2) sediment potential dissimilatory nitrate reduction to ammonium (DNRA) rate, (3) sediment total carbon and nitrogen content, (4) sediment stable isotopes (delta nitrogen-15 and delta carbon-13), (5) sediment percent organic matter, (6) surface water nitrous oxides, (7) surface water ammonium, and (8) methods codes. All files are .csv or .pdf.

Ammonium↗

Model Data Archive for Manuscript Titled "Evaluation of a Coupled Surface–Subsurface Hydrologic Model Using Dense Water‑Level Sensors in a Mixed Urban–Rural Watershed"

This archive provides scripts, input files, and datasets used for the implementation and evaluation of a fully coupled surface–subsurface hydrologic model in the Neches River Basin, southeast Texas. The study uses the Advanced Terrestrial Simulator (ATS) to simulate coupled surface–subsurface hydrologic processes over a mixed urban–rural watershed and evaluates model performance using a dense network of 136 in situ water-level sensors, nine U.S. Geological Survey (USGS) stream gauges, and SSEBop-derived evapotranspiration estimates during the period October 2014–June 2024. The workflow is implemented primarily in Python 3 using the Watershed Workflow package. The Jupyter notebooks can be executed using open-source software such as Anaconda JupyterLab or Visual Studio Code. Other data files include TXT, CSV, XML, SHP, TIF, NetCDF, HDF5, and ExodusII files, which can be processed using the provided Python scripts. ATS input files are provided in XML format and can be edited using any commonly used text editor. This archive contains: *Scripts and input files used to generate the ATS model setup, including watershed discretization, mesh generation, parameter mapping, and model configuration. *Jupyter notebooks used for preprocessing observational data, evaluating streamflow, water levels, and evapotranspiration, computing performance metrics, and generating the figures presented in the manuscript. *ATS simulation outputs and processed observational datasets, including OneRain and DD6 water-level sensors, USGS streamflow observations, GIS data, and supporting spatial datasets used throughout the study.

Dense water-level sensor network↗

Soil nitrogen mineralization rates, nutrient stocks, stable isotopes, and water volumetric measurements across terrestrial-aquatic interfaces from three wetlands at the Tanglewood Biological Station, Alabama

This dataset supports a broader study investigating wetland hydrologic and biogeochemical responses to inundation events. Soil samples were collected across four sampling events along terrestrial-aquatic gradients at three wetland sites located within the Tanglewood Biological Station in Alabama from April 2024 to June 2025. The contents in this data package include soil in-situ nitrogen mineralization rates (measurements of net nitrification, net ammonification, and net mineralization), nutrient stocks (total carbon, total nitrogen, and organic matter), stable isotopes (carbon and nitrogen), and water volumetric measurements (water-filled pore space). Water level data related to each wetland location can be found at https://data.ess-dive.lbl.gov/view/doi:10.15485/2530253 (Kirker et al., 2024), related water geochemistry data can be found at https://data.ess-dive.lbl.gov/datasets/doi:10.15485/3001967 (Forbes et al., 2025), and related surface water sediment chemistry data can be found at https://data.ess-dive.lbl.gov/datasets/doi:10.15485/3377325 (Molina Serpas et al., 2026). In addition to this readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. This dataset is comprised of (1) file-level metadata; (2) data dictionary; (3) field metadata and international generic sample numbers (IGSNs); (4) readme; (5) the field protocol; and (6) a subfolder with sample data. The sample data subfolder contains (1) net nitrification rate, (2) net ammonification rate, (3) areal net mineralization rate, (4) percent organic matter, (5) water-filled pore space, (6) total carbon content, (7) total nitrogen content, (8) stable carbon isotope (delta carbon-13), and (9) stable nitrogen isotope (delta nitrogen-15), and (10) methods codes. All files are .csv or .pdf.

13-C↗

SPRUCE Vegetation Community Survey Data from SPRUCE Experimental Plots, 2014-2020

This dataset provides vegetation survey data collected across vegetation community plots within the 10 chambered treatment plots (6, 19, 11, 20, 4, 13, 8, 16, 10, 17) and 3 unchambered ambient plots (7, 14, 21) from the Spruce and Peatland Responses Under Changing Environments (SPRUCE) experimental site within the Marcell Experimental Forest in northern Minnesota, USA. Collections occurred from 2014-2020 (2014-04-27 to 2020-08-06). The vegetation community plots, three in each experimental plot, were established in 2014 and designated to be undisturbed and reserved for long-term sampling. Vegetation survey data were collected at the peak of each growing season by placing a 1x2 meter rectangular grid over each community plot. The 1x2 meter grid is subset into 50 20x20 cm grid cells. Observations are of the presence of each species in each grid cell for each subplot. That’s a total of 39 subplots across the 13 experimental plots. From 2018 to 2020, shrub stem density was counted and percent shrub cover was estimated in five selected grid cells in each vegetation community plot. Also included are photographs of each community plot taken at the time of data collection for each measurement year. These data provide insight into the responses of plant community composition to warming and increased carbon dioxide. These data were use in analyses in McPartland et al (2018; 2020). This dataset contains three data files in comma-separate values (*.csv) format and seven compressed (*.zip) files containing 877 JPEG (*.jpg) files. Additional metadata are provided: a data dictionary and a file-level metadata file in comma-separate values (.csv) format and a user guide in PDF (*.pdf) format.

54 ENVIRONMENTAL SCIENCES↗

Detecting anomalous packets in network transfers: investigations using PCA, autoencoder and isolation forest in TCP

Large-scale scientific workflows rely heavily on high-performance file transfers. These transfers require strict quality parameters such as guaranteed bandwidth, no packet loss or data duplication. To have successful file transfers, methods such as predetermined thresholds and statistical analysis need to be done to determine abnormal patterns. Network administrators routinely monitor and analyze network data for diagnosing and alleviating these, making decisions based on their experience. However, as networks grow and become complex, monitoring large data files and quickly processing them, makes it improbable to identify errors and rectify these. Abnormal file transfers have been classified by simply setting alert thresholds, via tools such as PerfSonar and TCP statistics (Tstat). This paper investigates the feasibility of unsupervised feature extraction methods for identifying network anomaly patterns with three unsupervised classification methods—principal component analysis, autoencoder and isolation forest. Here, we collect file transfer statistics from two experiment sets—synthetic iPerf generated traffic and 1000 Genome workflow runs, with synthetically introduced anomalies. Our results show that while PCA and a simple autoencoder finds it difficult to detect clusters, the tree-variant isolation forest is able to identify anomalous packets by breaking down TCP traces into tree classes early.

97 MATHEMATICS AND COMPUTING↗

Videos, photos, and AI-derived grain size data associated with “High-throughput AI Video Surveys Enable Reproducible Multiscale Sediment Size Mapping, with Implications for Hydrobiogeochemical Parameterization”

NOTE: The manuscript associated with this data package is currently in review. The data may be revised based on reviewer feedback. Upon manuscript acceptance, this data package will be updated with the final dataset and additional metadata. This data package is associated with the manuscript “High-throughput AI Video Surveys Enable Reproducible Multiscale Sediment Size Mapping, with Implications for Hydrobiogeochemical Parameterization” under review. This data package includes five data types: 1) raw photos and videos from drone survey and walking smartphone surveys; 2) images derived from raw videos; 3) manual labeling of reference scales; 4) metadata for all images and photo resolution derived from artificial intelligence (AI) models or manual labels, 5) grain size data obtained from AI models for all photos, 6) metadata and grain size data after quality control, 7) summaries of sample efficiency for all data, and 8) computational fluid dynamics (CFD) data used to support hydro-biogeochemical (HBGC) parameter estimation. Such data is used to 1) demonstrate significant improvements in accuracy, efficiency, and quality control for grain size data collection with the help of AI models, 2) study the spatial heterogeneity of grain size and observation reproducibility based on tens of thousands of data points generated by the AI models, and 3) evaluate the impacts of grain size heterogeneity on key HBGC parameters across sediment-to-reach and hourly-to-yearly scales. In particular, the data package contains 116 folders and 179696 files. The files include 41 videos in .mov format, 64047 photos in .jpg format, 13541 video-derived photos in .png format, 12747 segmentation mask data in .tif format, 12747 segmentation data in .json format, 24771 .csv files that with metadata and grain size for each individual photo as well as water depth and velocity data from CFD and observation, 51791 .txt files of raw AI predicted labels, and 11 flight record data in .srt format. The summary for all metadata and grain size statistics information is included in “Scales_V3_NG.csv” and “Statistics_V3_NG.csv”. The summary for data that pass data quality control (QC) level 0-2 is included in “QCStatistics_V3_NG.csv”. The QC level 0 represents photos whose photo resolution is positive, excluding photos that miss reference scale. The QC level 1 means reference scale circularity uncertainty is less than 5% for smartphone images while representing photo resolution is larger than 0.44 mm/pixel for drone images. The QC level 2 means excluding photos whose grain number is less than 100, a minimum number of grains recommended by classic literature. The summary for each video’s name, length, frame rates, survey area, grain number, survey efficiency, etc. can be found in “QCSummary_V3_NG.csv”. The summary for site name, GPS coordinates, and number of images at each site can be found in “SitesSummary_V3_*.csv” files. Overall computational efficiency summary is reported in Table 4 of accompanying manuscript. Additionally, the nitrate concentration data used in this work was downloaded from an existing dataset published on ESS-DIVE (Boat-Dragged Sensor Hanford Reach.csv; Conner A. et al., 2020). We thank the United States Forest Service, Washington Department of Fish and Wildlife, Washington Department of Natural Resources, Cowiche Canyon Conservatory, Port of Benton, and the Confederated Tribes and Bands of the Yakama Nation for access to field locations where the data were collected. We also thank the Yakama Nation Tribal Council and Yakama Nation Fisheries for working with us to facilitate data collection and optimization of data usage according to their values and worldview.

54 ENVIRONMENTAL SCIENCES↗

Updates to R-matrix Evaluations of Fissile Actinides: 233,235 U, 239 Pu [Slides]

This presentation begins by discussing the importance and significance of cross section fluctuations. It also discusses the motivation and overarching goals which are to establish the highest fidelity nuclear evaluated data. In conclusion, a procedure to accurately model the measured cross section fluctuations for inclusion into evaluated nuclear data files was developed and tested on 239 Pu total cross sections. The procedure consisted of defining population of levels as well as related amplitudes extrapolated from the systematics of the resonance parameters in the resolved resonance region. By using realistic population of extrapolated levels, the fit of measured total cross section produced a continuous S-matrix function S( E ) in Reich-Moore approximation. The S-matrix function averaged over a suitable energy interval was used to obtain averaged cross sections consistent with the measured data. Transmittable formatted files (ENDF, GNDS) with LSSF=1 will consist of File 3 which are smoothed cross sections and File 2 which are average resonance parameters fitted to the energy bins defined in File 3. The capability to store probability tables is a way to show the fluctuating cross sections.

07 ISOTOPE AND RADIATION SOURCES↗

A brief parallel I/O tutorial.

This document provides common best practices for the efficient utilization of parallel file systems for analysts and application developers. A multi-program, parallel supercomputer is able to provide effective compute power by aggregating a host of lower-power processors using a network. The idea, in general, is that one either constructs the application to distribute parts to the different nodes and processors available and then collects the result (a parallel application), or one launches a large number of small jobs, each doing similar work on different subsets (a campaign). The I/O system on these machines is usually implemented as a tightly-coupled, parallel application itself. It is providing the concept of a 'file' to the host applications. The 'file' is an addressable store of bytes and that address space is global in nature. In essence, it is providing a global address space. Beyond the simple reality that the I/O system is normally composed of a small, less capable, collection of hardware, that concept of a global address space will cause problems if not very carefully utilized. How much of a problem and the ways in which those problems manifest will be different, but that it is problem prone has been well established. Worse, the file system is a shared resource on the machine - a system service. What an application does when it uses the file system impacts all users. It is not the case that some portion of the available resource is reserved. Instead, the I/O system responds to requests by scheduling and queuing based on instantaneous demand. Using the system well contributes to the overall throughput on the machine. From a solely self-centered perspective, using it well reduces the time that the application or campaign is subject to impact by others. The developer's goal should be to accomplish I/O in a way that minimizes interaction with the I/O system, maximizes the amount of data moved per call, and provides the I/O system the most information about the I/O transfer per request.

99 GENERAL AND MISCELLANEOUS↗

SPRUCE Data-constrained Projections of Methane Fluxes in a Northern Minnesota Peatland in Response to Elevated CO2 and Warming: Modeling Archive

This modeling archive dataset contains results from a study developed a data-informed process-based model using methane chamber measurement data from the Spruce and Peatland Responses Under Changing Environments (SPRUCE) experimental site within the Marcell Experimental Forest in northern Minnesota, USA. The study used existing manual CH4 flux measurements collected from the ambient environment from 2011-2014 and developed a data-informed process-based methane module. The module was incorporated into the Terrestrial ECOsystem (TECO) model before its parameters were constrained with multiple years of methane flux data for forecasting CH4 emission under five warming and two elevated CO2 experimental treatments at SPRUCE. This effort modeled wetland methane (CH4) fluxes with process modeling to better understand uncertainties and the effects of increased temperature and carbon dioxide (CO2) on methane fluxes. Ma et al. (2017) contains results from this modeling effort. This dataset contains 6 files: (1) a user guide in PDF (*.pdf) format, (2) a readme in PDF (*.pdf) format, (3) the posterior distribution of parameters values in comma-separate values (*.csv) format, (4) a compressed (*.zip) folder containing model simulated results of methane emission for different pathways under different environmental conditions, (5) source code of the TECO model used in the study in Fortran 90 (*.f90) format, and (6) a file-level metadata file in comma-separate values (*.csv). In the August 2026 update to this dataset, the associated Fortran 90 code script, TECO_SPRUCE_ME.f90, and a user guide were added to this dataset. Two file names were changed: 2.Posterior_distribution.csv to 2_Posterior_distribution.csv and 3.Simulations_data.zip to 3_Simulations_data.zip and Files_1_3_legends is now provided as a PDF (*.pdf) instead of in Microsoft Word format.

54 ENVIRONMENTAL SCIENCES↗

Analysis of Trade-Off Between Power Saving and Response Time in Disk Storage Systems

It is anticipated that in the near future disk storage systems will surpass application servers and will become the primary consumer of power in the data centers. Shutting down of inactive disks is one of the more widespread solutions to save power consumption of disk systems. This solution involves spinning down or completely shutting off disks that exhibit long periods of inactivity and placing them in standby mode. A file request from a disk in standby mode will incur an I/O cost penalty as it takes time to spin up the disk before it can serve the file. In this paper, we address the problem of designing and implementing file allocation strategies on disk storage that save energy while meeting performance requirements of file retrievals. We present an algorithm for solving this problem with guaranteed bounds from the optimal solution. Our algorithm runs in O(nlogn) time where n is the number of files allocated. Detailed simulation results and experiments with real life workloads are also presented.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Scalable Data Management of the Uintah Simulation Framework for Next-Generation Engineering Problems with Radiation

The need to scale next-generation industrial engineering problems to the largest computational platforms presents unique challenges. This paper focuses on data management related problems faced by the Uintah simulation framework at a production scale of 260K processes. Uintah provides a highly scalable asynchronous many-task runtime system, which in this work is used for the modeling of a 1000 megawatt electric (MWe) ultra-supercritical (USC) coal boiler. At 260K processes, we faced both parallel I/O and visualization related challenges, e.g., the default file-per-process I/O approach of Uintah did not scale on Mira. In this paper we present a simple to implement, restructuring based parallel I/O technique. We impose a restructuring step that alters the distribution of data among processes. The goal is to distribute the dataset such that each process holds a larger chunk of data, which is then written to a file independently. This approach finds a middle ground between two of the most common parallel I/O schemes–file per process I/O and shared file I/O–in terms of both the total number of generated files, and the extent of communication involved during the data aggregation phase. To address scalability issues when visualizing the simulation data, we developed a lightweight renderer using OSPRay, which allows scientists to visualize the data interactively at high quality and make production movies. Finally, this work presents a highly efficient and scalable radiation model based on the sweeping method, which significantly outperforms previous approaches in Uintah, like discrete ordinates. The integrated approach allowed the USC boiler problem to run on 260K CPU cores on Mira.

97 MATHEMATICS AND COMPUTING↗

Mission possible: Unify HPC and Big Data stacks towards application-defined blobs at the storage layer

HPC and Big Data stacks are completely separated today. The storage layer offers opportunities for convergence, as the challenges associated with HPC and Big Data storage are similar: trading versatility for performance. This motivates a global move towards dropping file-based, POSIX-IO compliance systems. However, on HPC platforms this is made difficult by the centralized storage architecture using file-based storage. In this paper we advocate that the growing trend of equipping HPC compute nodes with local storage redistributes the cards by enabling object storage to be deployed alongside the application on the compute nodes. Furthermore, such integration of application and storage not only allows fine-grained configuration of the storage system, but also improves application portability across platforms. In addition, the single-user nature of such application-specific storage obviates the need for resource-consuming storage features like permissions or file hierarchies offered by traditional file systems. In this article we propose and evaluate Blobs (Binary Large Objects) as an alternative to distributed file systems. Finally, we factually demonstrate that it offers drop-in compatibility with a variety of existing applications while improving storage throughput by up to 28%.

97 MATHEMATICS AND COMPUTING↗

Time sequence gamma-spectrometry of irradiated salt

Persisted Memory Mapping File (MMF) techniques have developed for the processing and analysis of time-stamped list (TLIST) mode data collected from an irradiated salt sample. This is a new technique that is unique to this research and enables additional information to be obtained from a conventional pulse height analysis (PHA) gamma-spectrometry acquisition. A salt sample was irradiated using a DT neutron generator and immediately measured using TLIST acquisition for 1 month to generate a 22 GB data file. MMF techniques were used to process the file without size restrictions at a rate of 388.5 MB minute-1, with instant access to any position within the file, separating the file into a series of time sequenced spectra suitable for PHA analysis. This enabled 14 activation products to be identified at different time intervals (41Ar, 80Br, 80mBr, 47Ca, 34mCl, 38Cl, 42K, 43K, 56Mn, 22Na, 24Na, 85Sr, 87mSr and 69mZn), and 6 radionuclides (80Br, 47Ca, 34mCl, 56Mn, 24Na and 87mSr) to be definitively confirmed based upon their measured half-life. Using this technique, the detection of 47Ca, and in-growth of its 47Sc progeny, was demonstrated as useful for calculating the time since irradiation.

Burnett, Jonathan L.↗

OpenMSI: A High-Performance Web-Based Platform for Mass Spectrometry Imaging

Mass spectrometry imaging (MSI) enables researchers to directly probe endogenous molecules directly within the architecture of the biological matrix. Unfortunately, efficient access, management, and analysis of the data generated by MSI approaches remain major challenges to this rapidly developing field. Despite the availability of numerous dedicated file formats and software packages, it is a widely held viewpoint that the biggest challenge is simply opening, sharing, and analyzing a file without loss of information. Here we present OpenMSI, a software framework and platform that addresses these challenges via an advanced, high-performance, extensible file format and Web API for remote data access (http://openmsi.nersc.gov). The OpenMSI file format supports storage of raw MSI data, metadata, and derived analyses in a single, self-describing format based on HDF5 and is supported by a large range of analysis software (e.g., Matlab and R) and programming languages (e.g., C++, Fortran, and Python). Careful optimization of the storage layout of MSI data sets using chunking, compression, and data replication accelerates common, selective data access operations while minimizing data storage requirements and are critical enablers of rapid data I/O. The OpenMSI file format has shown to provide >2000-fold improvement for image access operations, enabling spectrum and image retrieval in less than 0.3 s across the Internet even for 50 GB MSI data sets. To make remote high-performance compute resources accessible for analysis and to facilitate data sharing and collaboration, we describe an easy-to-use yet powerful Web API, enabling fast and convenient access to MSI data, metadata, and derived analysis results stored remotely to facilitate high-performance data analysis and enable implementation of Web based data sharing, visualization, and analysis.

59 BASIC BIOLOGICAL SCIENCES↗

Automatic log analysis with NLP for the CMS workflow handling

The central Monte-Carlo production of the CMS experiment utilizes the WLCG infrastructure and manages daily thousands of tasks, each up to thousands of jobs. The distributed computing system is bound to sustain a certain rate of failures of various types, which are currently handled by computing operators a posteriori. Within the context of computing operations, and operation intelligence, we propose a Machine Learning technique to learn from the operators with a view to reduce the operational workload and delays. This work is in continuation of CMS work on operation intelligence to try and reach accurate predictions with Machine Learning. We present an approach to consider the log files of the workflows as regular text to leverage modern techniques from Natural Language Processing (NLP). In general, log files contain a substantial amount of text that is not human language. Therefore, different log parsing approaches are studied in order to map the log files’ words to high dimensional vectors. These vectors are then exploited as feature space to train a model that predicts the action that the operator has to take. This approach has the advantage that the information of the log files is extracted automatically and the format of the logs can be arbitrary. In this work the performance of the log file analysis with NLP is presented and compared to previous approaches.

Layer, Lukas↗

Extra Dimensions: 3D in PDF Documentation

Experimental science is replete with multi-dimensional information which is often poorly represented by the two dimensions of presentation slides and print media. Past efforts to disseminate such information to a wider audience have failed for a number of reasons, including a lack of standards which are easy to implement and have broad support. Adobe's Portable Document Format (PDF) has in recent years become the de facto standard for secure, dependable electronic information exchange. It has done so by creating an open format, providing support for multiple platforms and being reliable and extensible. By providing support for the ECMA standard Universal 3D (U3D) and the ISO PRC file format in its free Adobe Reader software, Adobe has made it easy to distribute and interact with 3D content. Until recently, Adobe's Acrobat software was also capable of incorporating 3D content into PDF files from a variety of 3D file formats, including proprietary CAD formats. Furthermore, this functionality is no longer available in Acrobat X, having been spun off to a separate company. Incorporating 3D content now requires the additional purchase of a separate plug-in. In this talk we present alternatives based on open source libraries which allow the programmatic creation of 3D content in PDF format. While not providing the same level of access to CAD files as the commercial software, it does provide physicists with an alternative path to incorporate 3D content into PDF files from such disparate applications as detector geometries from Geant4, 3D data sets, mathematical surfaces or tesselated volumes.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Online Data Handling and Storage at the CMS Experiment

During the LHC Long Shutdown 1, the CMS Data Acquisition (DAQ) system underwent a partial redesign to replace obsolete network equipment, use more homogeneous switching technologies, and support new detector back-end electronics. The software and hardware infrastructure to provide input, execute the High Level Trigger (HLT) algorithms and deal with output data transport and storage has also been redesigned to be completely file- based. All the metadata needed for bookkeeping are stored in files as well, in the form of small documents using the JSON encoding. The Storage and Transfer System (STS) is responsible for aggregating these files produced by the HLT, storing them temporarily and transferring them to the T0 facility at CERN for subsequent offline processing. The STS merger service aggregates the output files from the HLT from ~62 sources produced with an aggregate rate of ~2GB/s. An estimated bandwidth of 7GB/s in concurrent read/write mode is needed. Furthermore, the STS has to be able to store several days of continuous running, so an estimated of 250TB of total usable disk space is required. In this article we present the various technological and implementation choices of the three components of the STS: the distributed file system, the merger service and the transfer system.

Andre, J. M.↗

Accelerating Scientific Workflows on HPC Platforms with In Situ Processing

Scientific workflows drive most modern large-scale science breakthroughs by allowing scientists to define their computations as a set of jobs executed in a given order based on their data dependencies. Workflow management systems (WMSs) have become key to automating scientific workflows-executing computational jobs and orchestrating data transfers between those jobs running on complex high-performance computing (HPC) platforms. Traditionally, WMSs use files to communicate between jobs: a job writes out files that are read by other jobs. However, HPC machines face a growing gap between their storage and compute capabilities. To address that concern, the scientific community has adopted a new approach called in situ, which bypasses costly parallel filesystem I/O operations with faster in-memory or in-network communications. When using in situ approaches, communication and computations can be interleaved. In this work, we leverage the Decaf in situ dataflow framework to accelerate task-based scientific workflows managed by the Pegasus WMS, by replacing file communications with faster MPI messaging. We propose a new execution engine that uses Decaf to manage communications within a sub-workflow (i.e., set of jobs) to optimize inter-job communications. We consider two workflows in this study: (i) a synthetic workflow that benchmarks and compares file- and MPI-based communication; and (ii) a realistic bioinformatics workflow that computes mu-tational overlaps in the human genome. Experiments show that in situ communication can improve the bioinformatics workflow execution time by 22% to 30% compared with file communication. Our results motivate further opportunities and challenges for bridging traditional WMSs with in situ frameworks.

Decaf↗