Results for “data management” — Page 93

DOE OSTI2025

Toward a persistent event-streaming system for high-performance computing applications

High-performance computing (HPC) applications have traditionally relied on parallel file systems and file transfer services to manage data movement and storage. Alternative approaches have been proposed that use direct communications between application components, trading persistence and fault tolerance for speed. Event-driven architectures, as popularized in enterprise contexts, present a compelling middle ground, avoiding the performance cost and API constraints of parallel file systems while retaining persistence and offering impedance matching between application components. However, adapting streaming frameworks to HPC workloads requires addressing challenges unique to HPC systems. This paper investigates the potential for a streaming framework designed for HPC infrastructures and use cases. We introduce Mofka, a persistent event-streaming framework designed specifically for HPC environments. Mofka combines the capabilities of a traditional streaming service with optimizations tailored to the HPC context, such as support for massively multicore nodes, efficient scaling for large producer-consumer workflows, RDMA-enabled high-performance network communications, specialized network fabrics with multiple links per node, and efficient handling of large scientific data payloads. Built using the Mochi suite of HPC data service components, Mofka provides a lightweight, modular, and high-performance solution for persistent streaming in HPC systems. We present the architecture of Mofka and evaluate its performance against Kafka and Redpanda using benchmarks on diverse platforms, including Argonne's Polaris and Oak Ridge's Frontier supercomputers, showing up to 8× improvement in throughput in some scenarios. We then demonstrate its utility in several real-world applications: a tomographic reconstruction pipeline, a workflow for the discovery of metal-organic frameworks for carbon capture, and the instrumentation of Dask workflows for provenance tracking and performance analysis.

HPC↗

DOE OSTI2023

The Field Epidemiology Training Program’s Contribution to Essential Public Health Functions in Seven National Public Health Institutes

Objective: This study explores how Field Epidemiology Training Programs (FETP) whose National Public Health Institutes (NPHI) are supported by U.S. Centers for Disease Control and Prevention (CDC) have contributed to strengthening essential public health functions. Methods: We conducted 96 semi-structured interviews with public health experts including NPHI staff, non-NPHI government staff, and staff from non-governmental and international organizations in Cambodia, Colombia, Liberia, Mozambique, Nigeria, Rwanda, and Zambia. We managed data using MAXQDA and employed direct content analysis to derive themes. Results: Three overarching themes emerged in relation to FETPs’ role within the NPHIs’ public health functions. These themes included contribution to improving country surveillance systems, role in providing leadership in outbreak responses, and strengthening countries’ and the NPHIs’ surveillance workforce capacity. Participants also shared challenges around FETPs’ implementation and suggestions for improvement. Conclusion: The results demonstrate the value of FETPs in strengthening public health systems through building workforce capacity and improving surveillance systems. By identifying the successes of FETPs in contributing to essential public health functions, our findings might inform current and future FETP implementation and its integration into NPHIs.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2023

A Theoretical Open Architecture Framework and Technology Stack for Digital Twins in Energy Sector Applications

Digital twin is often viewed as a technology that can assist engineers and researchers make data-driven system and network-level decisions. Across the scientific literature, digital twins have been consistently theorized as a strong solution to facilitate proactive discovery of system failures, system and network efficiency improvement, system and network operation optimization, among others. With their strong affinity to the industrial metaverse concept, digital twins have the potential to offer high-value propositions that are unique to the energy sector stakeholders to realize the true potential of physical and digital convergence and pertinent sustainability goals. Although the technology has been known for a long time in theory, its practical real-world applications have been so far limited, nevertheless with tremendous growth projections. In the energy sector, there have been theoretical and lab-level experimental analysis of digital twins but few of those experiments resulted in real-world deployments. There may be many contributing factors to any friction associated with real-world scalable deployment in the energy sector such as cost, regulatory, and compliance requirements, and measurable and comparable methods to evaluate performance and return on investment. Those factors can be potentially addressed if the digital twin applications are built on the foundations of a scalable and interoperable framework that can drive a digital twin application across the project lifecycle: from ideation to theoretical deep dive to proof of concept to large-scale experiment to real-world deployment at scale. This paper is an attempt to define a digital twin open architecture framework that comprises a digital twin technology stack (D-Arc) coupled with information flow, sequence, and object diagrams. Those artifacts can be used by energy sector engineers and researchers to use any digital twin platform to drive research and engineering. This paper also provides critical details related to cybersecurity aspects, data management processes, and relevant energy sector use cases.

Gourisetti, Sri, Nikhil Gupta (ORCID:0000000188778↗

DOE OSTI2019

Portal and Repository for Information on Marine Renewable Energy (PRIMRE)

The Marine Renewable Energy (MRE) industry is in the early stages of development corresponding to low technology readiness levels (TRLs) where the ability of the MRE community (developers, researchers, academics, stakeholders, investors, and regulators) to work together to share knowledge, experience, and lessons learned is critical to the advancement of the entire MRE industry. Through collaboration on solving common problems, the MRE community has the potential to reduce cost and accelerate technology development. Currently, the US Department of Energy (US DOE) Water Power Technologies Office (WPTO) is addressing the challenge of storing, curating, and accessing MRE information by sponsoring development of MRE databases and information portals such as Tethys (https://tethys.pnnl.gov), OpenEI (https://openei.org), the MHK Data Repository (MHKDR, https://mhkdr.openei.org/), and a searchable MRE code catalog and open source code repository (MHKiT, currently under development), to name a few. These sites host scientific papers, news articles, reports, databases, open source codes, and stakeholder engagement information, but they are only a step towards facilitating global discovery and use. In short, there is an abundance of information available online, however it is located on many disparate sites and repositories that make the discovery of those data and information difficult. A DOE national laboratory team from the National Renewable Energy Laboratory (NREL), Pacific Northwest National Laboratory (PNNL), and Sandia National Laboratories (Sandia) is addressing the issues of data discoverability, shared knowledge, and interconnection of existing MRE databases and information portals. To meet the needs of the MRE community, as identified through multiple community outreach and engagement events, and described in this paper, the multi-lab team has developed an implementation plan for PRIMRE, the Portal and Repository for Information on Marine Renewable Energy. PRIMRE will provide broad access to information on engineering and technologies, resource characterization, device performance, and environmental effects of MRE projects. PRIMRE will facilitate the commercial development of the MRE industry by increasing the accessibility and discoverability of this information, integrating the databases and information portals described in this paper, and developing standards and guidelines. Providing consistent, easy access to information can help reduce duplication of effort and enable the MRE community to learn from past failures and build upon the successes of others to innovate and advance the commercialization of MRE technologies.

16 TIDAL AND WAVE POWER↗

DOE OSTI2023

Grand challenges in the digitalisation of wind energy

The availability of large amounts of data is starting to impact how the wind energy community works. From turbine design to plant layout, construction, commissioning, and maintenance and operations, new processes and business models are springing up. This is the process of digitalisation, and it promises improved efficiency and greater insight, ultimately leading to increased energy capture and significant savings for wind plant operators, thus reducing the levelised cost of energy. Digitalisation is also impacting research, where it is both easing and speeding up collaboration, as well as making research results more accessible. This is the basis for innovations that can be taken up by end users. But digitalisation faces barriers. This paper uses a literature survey and the results from an expert elicitation to identify three common industry-wide barriers to the digitalisation of wind energy. Comparison with other networked industries and past and ongoing initiatives to foster digitalisation show that these barriers can only be overcome by wide-reaching strategic efforts, and so we see these as “grand challenges” in the digitalisation of wind energy. They are, first, creating FAIR data frameworks; secondly, connecting people and data to foster innovation; and finally, enabling collaboration and competition between organisations. The grand challenges in the digitalisation of wind energy thus include a mix of technical, cultural, and business aspects that will need collaboration between businesses, academia, and government to solve. Working to mitigate them is the beginning of a dynamic process that will position wind energy as an essential part of a global clean energy future.

17 WIND ENERGY↗

DOE OSTI2024

Source Data for Calculations, UV-vis-NIR, and NMR for A Tetrahedral Neptunium(V) Complex

Data associated with the Nature Chemistry publication (DOI: 10.1038/s41557-024-01529-6 ) was made publicly available in a FigShare repository (10.6084/m9.figshare.24309463) to ensure reproducibility and support the PI's commitment to open-science that was included in the data management plan of the project

Niklas, Julie E.↗

DOE OSTI2026

Supporting Data: Structural and Theoretical Assessment of Covalency in a Pu(III) Borohydride Complex

Data associated with the Journal of the American Chemical Society publication (DOI: 10.1021/jacs.4c09888 ) was made publicly available in a FigShare repository (DOI: 10.6084/m9.figshare.26997388) to ensure reproducibility and support the PI's commitment to open-science that was included in the data management plan of the project

Vlaisavljevich, Bess [University of Iowa]↗

DOE OSTI2025

Supporting Data: Computational Investigation of the Chemical Bond between An(III) Ions and Soft-Donor Ligands

Data associated with the Inorganic Chemistry publication (DOI: 10.1021/acs.inorgchem.4c03924) was made publicly available in a FigShare repository (DOI: 10.6084/m9.figshare.28195664) to ensure reproducibility and support the PI's commitment to open-science that was included in the data management plan of the project.

Roy Chowdhury, Sabyasachi [Department of Chemistry↗

DOE OSTI2017

DMTN-036: jointcal: Simultaneous Astrometry & Photometry for thousands of Exposures with Large CCD Mosaics

The jointcal package simultaneously optimizes the astrometric and photometric calibrations of a set of astronomical images. In principle and often in practice, this approach produces distortion and thoroughput models which are more precise than when fitted independently. This is especially true when the images are deeper than the astrometric reference catalogs. In the "Astromatic" software suite, this simultaneous astrometry functionality is fulfilled by "SCAMP". The code we describe here has similar aims, but follows a slightly different route. Jointcal is built on top of the the LSST Data Management software stack.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

Replication Data for: Elucidating the composition of PtAg surface alloys with atomic-scale imaging and spectroscopy

The data underlying this published work have been made publicly available in this repository as part of the IMASC Data Management Plan. This work was supported as part of the Integrated Mesoscale Architectures for Sustainable Catalysis (IMASC), an Energy Frontier Research Center funded by the U.S. Department of Energy, Office of Science, Basic Energy Sciences under Award # DE-SC0012573.