Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “metadata”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,189 records · Page 66
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

ServiceX A Distributed, Caching, Columnar Data Delivery Service

We will describe a component of the Intelligent Data Delivery Service being developed in collaboration with IRIS-HEP and the LHC experiments. ServiceX is an experiment-agnostic service to enable on-demand data delivery specifically tailored for nearly-interactive vectorized analysis. This work is motivated by the data engineering challenges posed by HL-LHC data volumes and the increasing popularity of python and Spark-based analysis workflows.ServiceX gives analyzers the ability to query events by dataset metadata. It uses containerized transformations to extract just the data required for the analysis. This operation is colocated with the data to avoid transferring unnecessary branches over the WAN. Simple filtering operations are supported to further reduce the amount of data transferred.Transformed events are cached in a columnar datastore to accelerate delivery of subsequent similar requests. ServiceX will learn commonly related columns and automatically include them in the transformation to increase the potential for cache hits by other users.Selected events are streamed to the analysis system using an efficient wire protocol that can be readily consumed by a variety of computational frameworks. This reduces time-to-insight for physics analysis by delegating to ServiceX the complexity of event selection, slimming, reformatting, and streaming.

Galewsky, B.↗
DOE OSTI2025

dCache project status and update

The dCache project delivers an open-source, massively scalable, distributed storage system deployed internationally to satisfy today’s scientists’ ever-demanding storage requirements. Its multifaceted approach supports different use cases with the same storage, from high throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and longterm data persistence on tertiary storage. Even though dCache was initially developed for HEP experiments, today, it is used by various scientific communities, including astrophysics, biomed, and life science, each with their specific requirements. To match the needs of these new communities and keep up with the scaling demands of existing experiments, dCache is permanently evolving. With this contribution, we would like to highlight the recent developments in dCache regarding integration with CERN Tape Archive (CTA), advanced metadata handling, token-based authorization support, bulk API for QoS transitions, REST API to control interaction with the tape system, and future development directions.

Mkrtchyan, Tigran [DESY]↗
DOE OSTI2025

Label-based Virtual Directories In dCache

Traditional filesystems organize data in directories. These directories are typically a collection of files whose grouping is based on a single criterion, e.g., the starting date of an experiment, experiment name, beamline ID, measurement device, or instrument. However, each file in a directory can belong to several logical groups, such as a special event type, experiment condition, or a part of a selected dataset. dCache is a storage system developed to store large amounts of scientific data, used by many HEP and Photon Science experiments. With recent developments in dCache, we have introduced a concept of file tagging, which dynamically groups files with the same label into virtual directories. The file labels can be added, removed, renamed, and deleted through the admin interface or via REST API. The files in virtual directories are exposed through all protocols supported by dCache. This contribution will describe the details of the implementation for file tagging in dCache and present our future development plans on automatic metadata extractions, a feature that will significantly simplify data management. Additionally, we are exploring the future use of virtual directories as a way to translate scientific data catalogs into filesystem views for direct data analysis.

Sahakyan, Marina [DESY]↗
DOE OSTI2023

Templates of expected measurement uncertainties for prompt fission neutron spectra

In this paper, we provide templates of uncertainty sources expected to appear for three measurement types of prompt fission neutron spectra (PFNS): (1) shape measurements, (2) clean-ratio shape, that is the monitor PFNS are measured in nearly exactly the same surrounding as the PFNS of interest, and (3) indirect ratios, where the detector efficiency is backed out from PFNS monitor measurements. Information is also listed that is needed to faithfully include PFNS in nuclear data evaluations to guide experimenters on how to best report data and metadata for their measurements. These templates also suggest a typical range of pertinent uncertainty values and their correlations in case realistic uncertainties cannot be estimated from information on the measurement itself. The templates were based on a literature review, information found in EXFOR for 252 Cf, 235, 238 U, and 239 Pu PFNS, and enhanced by expertise from experimenters contributing to these PFNS templates.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2017

An HDF5-Based Framework for the Distribution and Analysis of Ultrasonic Concrete Data

There are many commercial ultrasonic tomography devices (UTDs) available for use in nondestructive evaluation (NDE) of reinforced concrete structures. These devices emit, measure, and store ultrasonic signals typically in the 25 kHz to 5 MHz frequency range. UTDs are characterized by a composition of multiple transducers, also known as a transducer array or phased array. Often, UTDs data are in a proprietary format. Consequently, NDE research data is limited to those who have prior non-disclosure agreements or the appropriate licenses. Thus, there is a need for a proper universal data framework to exist such that proprietary file datasets for different concrete specimens can be converted, organized, and stored with relative metadata for individual or collaborative NDE research. Building upon the Hierarchical Data Format (HDF5) model, we have developed a UTD data management framework and Graphic User Interface (GUI) to promote the algorithmic reconstruction of ultrasonic data in a controlled environment for easily reproducible and publishable results.

concrete NDE↗
DOE OSTI2018

Time filtering of event based neutron scattering data: A pathway to study the dynamic structural responses of materials

Time-resolved diffraction has become a vital tool for probing dynamic responses to an applied stimulus. Such experiments traditionally use hardware solutions to histogram measured data into their respective bin. We will show that a major advantage of event-based data acquisition, which time-stamps measured diffraction data with 100 ns accuracy, is much preferred over hardware histogramming of the data by enabling postprocessing for advanced custom binning using a software solution. This approach is made even more powerful by coupling measured diffraction data with metadata about the applied stimuli and material response. In this work, we present a time-filter approach that leverages the power of event-based diffraction collection to reduce stroboscopic data measured over many hours into equally weighted segments that represent subsets of the response to a single cycle of the applied stimulus. We demonstrate this approach by observing ferroelectric/ferroelastic domain wall motion during electric field cycling of BaTiO 3 . Furthermore, the developed approach can readily be expanded to investigate other dynamic phenomena using complex sample environments.

47 OTHER INSTRUMENTATION↗
DOE OSTI2023

The fast camera (Fastcam) imaging diagnostic systems on the DIII-D tokamak

Two camera systems are installed on the DIII-D tokamak at the toroidal positions of 90° (90° system) and 225° (225° system), respectively. The cameras have two types of relay optics, namely, a coherent optical fiber bundle and a periscope system. The periscope system provides absolute intensity calibration stability while sacrificing resolution (10 lp/mm), while the fiber system provides high resolution (16 lp/mm) while sacrificing calibration stability. The periscope is available only for the 90° system. The optics of the 225° system were designed for view stability, repeatability, and easy maintenance. The cameras are located inside optimized neutron, x ray and magnetic shielding in order to reduce electronics damage, reboots, and magnetic and neutron interference, increasing the overall system reliability. An automated filter wheel, providing remote filter change, allows for remote wavelength selection. A software suite automates camera acquisition and data storage, allowing for remote operation and reduced operator involvement. System metadata is used to streamline the data analysis workflow, particularly for intensity calibration. Here, the spatial calibration uses multiple observable wall features, resulting in a reconstruction accuracy ≤2 cm.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2023

Fermionic quantum processing with programmable neutral atom arrays

Simulating the properties of many-body fermionic systems is an outstanding computational challenge relevant to material science, quantum chemistry, and particle physics.-5.4pc]Please note that the spelling of the following author names in the manuscript differs from the spelling provided in the article metadata: D. González-Cuadra, D. Bluvstein, M. Kalinowski, R. Kaubruegger, N. Maskara, P. Naldesi, T. V. Zache, A. M. Kaufman, M. D. Lukin, H. Pichler, B. Vermersch, Jun Ye, and P. Zoller. The spelling provided in the manuscript has been retained; please confirm. Although qubit-based quantum computers can potentially tackle this problem more efficiently than classical devices, encoding nonlocal fermionic statistics introduces an overhead in the required resources, limiting their applicability on near-term architectures. In this work, we present a fermionic quantum processor, where fermionic models are locally encoded in a fermionic register and simulated in a hardware-efficient manner using fermionic gates. We consider in particular fermionic atoms in programmable tweezer arrays and develop different protocols to implement nonlocal gates, guaranteeing Fermi statistics at the hardware level. We use this gate set, together with Rydberg-mediated interaction gates, to find efficient circuit decompositions for digital and variational quantum simulation algorithms, illustrated here for molecular energy estimation. Finally, we consider a combined fermion-qubit architecture, where both the motional and internal degrees of freedom of the atoms are harnessed to efficiently implement quantum phase estimation as well as to simulate lattice gauge theory dynamics.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2016

Analysis of soil eDNA functional genes: potential to increase profitability and sustainability of pastoral agriculture

Management of soil biological resources to optimise plant production, efficiency of nutrient inputs, and system sustainability is an emerging opportunity for pastoral agriculture. To attain these goals, suitable tools that can assess the functional state of the soil ecosystem must be developed and standardised approaches to their application adopted. Towards this end, we have undertaken comprehensive, high-density functional-gene microarray analysis (GeoChip5) of environmental DNA (eDNA) extracted from 50 pastoral soils. When combined with soil, environmental and management metadata, the information can be used to provide insights into soil biological processes spanning greenhouse gas emissions, through to natural suppression of plant root diseases. To give an example of a structured workflow of analysis in a pastoral system context, we analysed the GeoChip data using a combination of approaches spanning routine univariate methods through to more complex multivariate and network-based analysis. Analyses were restricted to comparing effects of land-use (dairy or ‘other’ farming systems), and exploring relationships of the GeoChip data with the soil properties from each sample. These exemplar analyses introduce a pathway for the application of eDNA approaches (GeoChip or others) to deliver outcomes for pastoral agricultural in New Zealand.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2014

Qualities of sequential chromospheric brightenings observed in Hα and UV images

Chromospheric flare ribbons observed in Hα appear well-organized when first examined: ribbons impulsively brighten, morphologically evolve, and exponentially decay back to pre-flare levels. Upon closer inspection of the Hα flares, there is often a significant number of compact areas brightening in concert with the flare eruption but are spatially separated from the evolving flare ribbon. One class of these brightenings is known as sequential chromospheric brightenings (SCBs). SCBs are often observed in the immediate vicinity of erupting flares and are associated with coronal mass ejections. In the past decade there have been several previous investigations of SCBs. These studies have exclusively relied upon Hα images to discover and analyze these ephemeral brightenings. This work employs the automated detection algorithm of Kirk et al. to extract the physical qualities of SCBs in observations of ground-based Hα images and complementary Atmospheric Imaging Assembly images in He II, C IV, and 1700 Å. The metadata produced in this tracking process are then culled using complementary Doppler velocities to isolate three distinguishable types of SCBs. From a statistical analysis, we find that the SCBs at the chromospheric Hα layer appear earlier and last longer than their corresponding signatures measured in AIA. From this multi-layer analysis, we infer that SCBs are spatially constrained to the mid-chromosphere. We also derive an energy budget to explain SCBs which have a postulated energy of not more than 0.01% of the total flare energy.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Extending the distributed computing infrastructure of the CMS experiment with HPC resources

Particle accelerators are an important tool to study the fundamental properties of elementary particles. Currently the highest energy accelerator is the LHC at CERN, in Geneva, Switzerland. Each of its four major detectors, such as the CMS detector, produces dozens of Petabytes of data per year to be analyzed by a large international collaboration. The processing is carried out on the Worldwide LHC Computing Grid, that spans over more than 170 compute centers around the world and is used by a number of particle physics experiments. Recently the LHC experiments were encouraged to make increasing use of HPC resources. While Grid resources are homogeneous with respect to the used Grid middleware, HPC installations can be very different in their setup. In order to integrate HPC resources into the highly automatized processing setups of the CMS experiment a number of challenges need to be addressed. For processing, access to primary data and metadata as well as access to the software is required. At Grid sites all this is achieved via a number of services that are provided by each center. However at HPC sites many of these capabilities cannot be easily provided and have to be enabled in the user space or enabled by other means. At HPC centers there are often restrictions regarding network access to remote services, which is again a severe limitation. The paper discusses a number of solutions and recent experiences by the CMS experiment to include HPC resources in processing campaigns.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2014

A Data Handling System for Modern and Future Fermilab Experiments

Current and future Fermilab experiments such as Minerva, NOνA, and MicroBoone are now using an improved version of the Fermilab SAM data handling system. SAM was originally used by the CDF and D0 experiments for Run II of the Fermilab Tevatron to provide file metadata and location cataloguing, uploading of new files to tape storage, dataset management, file transfers between global processing sites, and processing history tracking. However SAM was heavily tailored to the Run II environment and required complex and hard to deploy client software, which made it hard to adapt to new experiments. The Fermilab Computing Sector has progressively updated SAM to use modern, standardized, technologies in order to more easily deploy it for current and upcoming Fermilab experiments, and to support the data preservation efforts of the Run II experiments.

Illingworth, R. A.↗
DOE OSTI2014

CMS Space Monitoring

During the first LHC run, CMS stored about one hundred petabytes of data. Storage accounting and monitoring help to meet the challenges of storage management, such as efficient space utilization, fair share between users and groups and resource planning. We present a newly developed CMS space monitoring system based on the storage metadata dumps produced at the sites. The information extracted from the storage dumps is aggregated and uploaded to a central database. A web based data service is provided to retrieve the information for a given time interval and a range of sites, so it can be further aggregated and presented in the desired format. The system has been designed based on the analysis of CMS monitoring requirements and experiences of the other LHC experiments. In this paper, we demonstrate how the existing software components of the CMS data placement system, PhEDEx, have been re-used, dramatically reducing the development effort.

Ratnikova, N.↗
DOE OSTI2015

Online Data Handling and Storage at the CMS Experiment

During the LHC Long Shutdown 1, the CMS Data Acquisition (DAQ) system underwent a partial redesign to replace obsolete network equipment, use more homogeneous switching technologies, and support new detector back-end electronics. The software and hardware infrastructure to provide input, execute the High Level Trigger (HLT) algorithms and deal with output data transport and storage has also been redesigned to be completely file- based. All the metadata needed for bookkeeping are stored in files as well, in the form of small documents using the JSON encoding. The Storage and Transfer System (STS) is responsible for aggregating these files produced by the HLT, storing them temporarily and transferring them to the T0 facility at CERN for subsequent offline processing. The STS merger service aggregates the output files from the HLT from ~62 sources produced with an aggregate rate of ~2GB/s. An estimated bandwidth of 7GB/s in concurrent read/write mode is needed. Furthermore, the STS has to be able to store several days of continuous running, so an estimated of 250TB of total usable disk space is required. In this article we present the various technological and implementation choices of the three components of the STS: the distributed file system, the merger service and the transfer system.

Andre, J. M.↗
DOE OSTI2015

File-Based Data Flow in the CMS Filter Farm

During the LHC Long Shutdown 1, the CMS Data Acquisition system underwent a partial redesign to replace obsolete network equipment, use more homogeneous switching technologies, and prepare the ground for future upgrades of the detector front-ends. The software and hardware infrastructure to provide input, execute the High Level Trigger (HLT) algorithms and deal with output data transport and storage has also been redesigned to be completely file- based. This approach provides additional decoupling between the HLT algorithms and the input and output data flow. All the metadata needed for bookkeeping of the data flow and the HLT process lifetimes are also generated in the form of small “documents” using the JSON encoding, by either services in the flow of the HLT execution (for rates etc.) or watchdog processes. These “files” can remain memory-resident or be written to disk if they are to be used in another part of the system (e.g. for aggregation of output data). We discuss how this redesign improves the robustness and flexibility of the CMS DAQ and the performance of the system currently being commissioned for the LHC Run 2.

Andre, J. M.↗
DOE OSTI2015

A Scalable Monitoring for the CMS Filter Farm Based on Elasticsearch

A flexible monitoring system has been designed for the CMS File-based Filter Farm making use of modern data mining and analytics components. All the metadata and monitoring information concerning data flow and execution of the HLT are generated locally in the form of small documents using the JSON encoding. These documents are indexed into a hierarchy of elasticsearch (es) clusters along with process and system log information. Elasticsearch is a search server based on Apache Lucene. It provides a distributed, multitenant-capable search and aggregation engine. Since es is schema-free, any new information can be added seamlessly and the unstructured information can be queried in non-predetermined ways. The leaf es clusters consist of the very same nodes that form the Filter Farm thus providing natural horizontal scaling. A separate central” es cluster is used to collect and index aggregated information. The fine-grained information, all the way to individual processes, remains available in the leaf clusters. The central es cluster provides quasi-real-time high-level monitoring information to any kind of client. Historical data can be retrieved to analyse past problems or correlate them with external information. We discuss the design and performance of this system in the context of the CMS DAQ commissioning for LHC Run 2.

Andre, J. M.↗
DOE OSTI2016

Exascale Storage Systems the SIRIUS Way

As the exascale computing age emerges, data related issues are becoming critical factors that determine how and where we do computing. Popular approaches used by traditional I/O solution and storage libraries become increasingly bottlenecked due to their assumptions about data movement, re-organization, and storage. While, new technologies, such as "burst buffers", can help address some of the short-term performance issues, it is essential that we reexamine the underlying storage and I/O infrastructure to effectively support requirements and challenges at exascale and beyond. In this paper we present a new approach to the exascale Storage System and I/O (SSIO), which is based on allowing users to inject application knowledge into the system and leverage this knowledge to better manage, store, and access large data volumes so as to minimize the time to scientific insights. Central to our approach is the distinction between the data, metadata, and the knowledge contained therein, transferred from the user to the system by describing "utility" of data as it ages.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Scidac-Data: Enabling Data Driven Modeling of Exascale Computing

Here, the SciDAC-Data project is a DOE-funded initiative to analyze and exploit two decades of information and analytics that have been collected by the Fermilab data center on the organization, movement, and consumption of high energy physics (HEP) data. The project analyzes the analysis patterns and data organization that have been used by NOvA, MicroBooNE, MINERvA, CDF, D0, and other experiments to develop realistic models of HEP analysis workflows and data processing. The SciDAC-Data project aims to provide both realistic input vectors and corresponding output data that can be used to optimize and validate simulations of HEP analysis. These simulations are designed to address questions of data handling, cache optimization, and workflow structures that are the prerequisites for modern HEP analysis chains to be mapped and optimized to run on the next generation of leadership-class exascale computing facilities. We present the use of a subset of the SciDAC-Data distributions, acquired from analysis of approximately 71,000 HEP workflows run on the Fermilab data center and corresponding to over 9 million individual analysis jobs, as the input to detailed queuing simulations that model the expected data consumption and caching behaviors of the work running in high performance computing (HPC) and high throughput computing (HTC) environments. In particular we describe how the Sequential Access via Metadata (SAM) data-handling system in combination with the dCache/Enstore-based data archive facilities has been used to develop radically different models for analyzing the HEP data. We also show how the simulations may be used to assess the impact of design choices in archive facilities.

97 MATHEMATICS AND COMPUTING↗