Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data-centric”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

SYMBIOSYS: A Methodology for Performance Analysis of Composable HPC Data Services

Microservices are a powerful new way of building, customizing, and deploying distributed services owing to their flexibility and maintainability. Several large-scale distributed platforms have emerged to serve the growing needs of data-centric workloads and services in commercial computing. Concurrently, high-performance computing (HPC) systems and software are rapidly evolving to meet the demands of diversified applications and heterogeneity. The interplay of hardware factors, software configuration parameters, and the flexibility offered with a microservice architecture makes it nontrivial to estimate the optimal service instantiation for a given application workload. Further, this problem is exacerbated when considering that these services operate in a dynamic and heterogeneous HPC environment. An optimally integrated service can be vastly more performant than a haphazardly integrated one. Existing performance tools for HPC either fail to understand the request-response model of communication inherent to microservices or they operate within a narrow scope, limiting the insight that can be gleaned from employing them in isolation. We propose a methodology for integrated performance analysis of HPC microservices frameworks and applications called SYMBIOSYS. We describe its design and implementation within the context of the Mochi framework. This integration is achieved by combining distributed callpath profiling and tracing with a performance data exchange strategy that collects fine-grained, low-level metrics from the RPC communication library and network layers. The result is a portable, low-overhead performance analysis setup that provides a holistic profile of the dependencies among microservices and how they interact with the Mochi RPC software stack. Using HEPnOS, a production-quality Mochi data service, we demonstrate the low-overhead operation of SYMBIOSYS at scale and use it to identify the root causes of poorly performing service configurations.

microservices↗

MICCO: An Enhanced Multi-GPU Scheduling Framework for Many-Body Correlation Functions

Calculation of many-body correlation functions is one of the critical kernels utilized in many scientific computing areas, especially in Lattice Quantum Chromodynamics (Lattice QCD). It is formalized as a sum of a large number of contraction terms each of which can be represented by a graph consisting of vertices describing quarks inside a hadron node and edges designating quark propagations at specific time intervals. Due to its computation- and memory-intensive nature, real-world physics systems (e.g., multi-meson or multi-baryon systems) explored by Lattice QCD prefer to leverage multi-GPUs. Different from general graph processing, many-body correlation function calculations show two specific features: a large number of computation-/data-intensive kernels and frequently repeated appearances of original and intermediate data. The former results in expensive memory operations such as tensor movements and evictions. The latter offers data reuse opportunities to mitigate the data-intensive nature of many-body correlation function calculations. However, existing graph-based multi-GPU schedulers cannot capture these data-centric features, thus resulting in a sub-optimal performance for many-body correlation function calculations. To address this issue, this paper presents a multi-GPU scheduling framework, MICCO, to accelerate contractions for correlation functions particularly by taking the data dimension (e.g., data reuse and data eviction) into account. This work first performs a comprehensive study on the interplay of data reuse and load balance, and designs two new concepts: local reuse pattern and reuse bound to study the opportunity of achieving the optimal trade-off between them. Based on this study, MICCO proposes a heuristic scheduling algorithm and a machine-learning-based regression model to generate the optimal setting of reuse bounds. Specifically, MICCO is integrated into a real-world Lattice QCD system, Redstar, for the first time running on multiple GPUs. The evaluation demonstrates MICCO outperforms other state-of-art works, achieving up to 2.25× speedup in synthesized datasets, and 1.49× speedup in real-world correlation functions.

Wang, Qihan↗

Programming Abstractions for Managing Workflows on Tiered Storage Systems

Scientific workflows in High Performance Computing (HPC) environments are processing large amounts of data. The storage hierarchy on HPC systems is getting deeper, driven by new technologies (NVRAMs, SSDs, etc.) There is a need for new programming abstractions that allow users to seamlessly manage data at the workflow level on multi-tiered storage systems, and provide optimal workflow performance and use of storage resources. In previous work, we introduced a software architecture Managing Data on Tiered Storage for Scientific Workflows (MaDaTS) that used a Virtual Data Space (VDS) abstraction to hide the complexities of the underlying storage system while allowing users to control data management strategies. In this article, we detail the data-centric programming abstractions that allow users to manage a workflow around its data on the storage layer. The programming abstractions simplify data management for scientific workflows on multi-tiered storage systems, without affecting workflow performance or storage capacity. We measure the overheads and effectiveness introduced by the programming abstractions of MaDaTS. Our results show that these abstractions can optimally use the storage capacity in lesser capacity storage tiers, and simplify data management without adding any performance overheads.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DaCeML

Machine learning powered by data-centric parallel programming.

Rausch, Oliver↗

pnnl/arena

CFA ARENA is a novel programming model with the support of a runtime targeting asynchronous data-centric execution paradigm in a distributed system. All the machine nodes in ARENA are connected by a ring network to bring the specialized computation to the data rather than the reverse to minimize data movement. The programming interfaces are implemented using C++

Tan, Cheng↗

DEEPEN Global Standardized Categorical Exploration Datasets for Magmatic Plays

DEEPEN stands for DE-risking Exploration of geothermal Plays in magmatic ENvironments. As part of the development of the DEEPEN 3D play fairway analysis (PFA) methodology for magmatic plays (conventional hydrothermal, superhot EGS, and supercritical), weights needed to be developed for use in the weighted sum of the different favorability index models produced from geoscientific exploration datasets. This was done using two different approaches: one based on expert opinions, and one based on statistical learning. This GDR submission includes the datasets used to produce the statistical learning-based weights. While expert opinions allow us to include more nuanced information in the weights, expert opinions are subject to human bias. Data-centric or statistical approaches help to overcome these potential human biases by focusing on and drawing conclusions from the data alone. The drawback is that, to apply these types of approaches, a dataset is needed. Therefore, we attempted to build comprehensive standardized datasets mapping anomalies in each exploration dataset to each component of each play. This data was gathered through a literature review focused on magmatic hydrothermal plays along with well-characterized areas where superhot or supercritical conditions are thought to exist. Datasets were assembled for all three play types, but the hydrothermal dataset is the least complete due to its relatively low priority. For each known or assumed resource, the dataset states what anomaly in each exploration dataset is associated with each component of the system. The data is only a semi-quantitative, where values are either high, medium, or low, relative to background levels. In addition, the dataset has significant gaps, as not every possible exploration dataset has been collected and analyzed at every known or suspected geothermal resource area, in the context of all possible play types. The following training sites were used to assemble this dataset: - Conventional magmatic hydrothermal: Akutan (from AK PFA), Oregon Cascades PFA, Glass Buttes OR, Mauna Kea (from HI PFA), Lanai (from HI PFA), Mt St Helens Shear Zone (from WA PFA), Wind River Valley (From WA PFA), Mount Baker (from WA PFA). - Superhot EGS: Newberry (EGS demonstration project), Coso (EGS demonstration project), Geysers (EGS demonstration project), Eastern Snake River Plain (EGS demonstration project), Utah FORGE, Larderello, Kakkonda, Taupo Volcanic Zone, Acoculco, Krafla. - Supercritical: Coso, Geysers, Salton Sea, Larderello, Los Humeros, Taupo Volcanic Zone, Krafla, Reyjanes, Hengill. **Disclaimer: Treat the supercritical fluid anomalies with skepticism. They are based on assumptions due to the general lack of confirmed supercritical fluid encounters and samples at the sites included in this dataset, at the time of assembling the dataset. The main assumption was that the supercritical fluid in a given geothermal system has shared properties with the hydrothermal fluid, which may not be the case in reality. Once the datasets were assembled, principal component analysis (PCA) was applied to each. PCA is an unsupervised statistical learning technique, meaning that labels are not required on the data, that summarized the directions of variance in the data. This approach was chosen because our labels are not certain, i.e., we do not know with 100% confidence that superhot resources exist at all the assumed positive areas. We also do not have data for any known non-geothermal areas, meaning that it would be challenging to apply a supervised learning technique. In order to generate weights from the PCA, an analysis of the PCA loading values was conducted. PCA loading values represent how much a feature is contributing to each principal component, and therefore the overall variance in the data.

15 GEOTHERMAL ENERGY↗

2020 natural gas LCA data appendices

This collection is the data-centric appendices for the report Life Cycle Analysis of Natural Gas Extraction and Power Generation: U.S. 2020 Emissions Profile. It consists of Appendix A: Additional Modeling Parameters [spreadsheet]; Appendix B: Water Burdens [spreadsheet]; Appendix D: Simulation of Liquids Unloading [python script and spreadsheet]; Appendix E: Detailed GHG Results for All Scenarios [spreadsheet]; Appendix F: Full Inventory Results [spreadsheet]; and Appendix I: Stage-Level Natural Gas Loss and Consumption Rates [spreadsheet].

Appendices↗

2020 natural gas LCA appendices Rev1

This collection is the data-centric appendices for the report Life Cycle Analysis of Natural Gas Extraction and Power Generation: U.S. 2020 Emissions Profile. It consists of Appendix A: Additional Modeling Parameters [spreadsheet]; Appendix B: Water Burdens [spreadsheet]; Appendix D: Simulation of Liquids Unloading [python script and spreadsheet]; Appendix E: Detailed GHG Results for All Scenarios [spreadsheet]; Appendix F: Full Inventory Results [spreadsheet]; and Appendix I: Stage-Level Natural Gas Loss and Consumption Rates [spreadsheet]. These results have been updated from the previous version (https://edx.netl.doe.gov/dataset/2020-natural-gas-lca-data-appendices) to correct a modeling error where the same post-processing natural gas composition was used instead of the intended regional compositions.

Appendices↗

CY15 Livermore Computing Focus Areas

The LC team undertook a survey of primary Center drivers for CY15. Identified key drivers included enhancing user experience and productivity, pre-exascale platform preparation, process improvement, data-centric computing paradigms and business expansion. The team organized critical supporting efforts into three cross-cutting focus areas; Improving Service Quality; Monitoring, Automation, Delegation and Center Efficiency; and Next Generation Compute and Data Environments In each area the team detailed high level challenges and identified discrete actions to address these issues during the calendar year. Identifying the Center’s primary drivers, issues, and plans is intended to serve as a lens focusing LC personnel, resources, and priorities throughout the year.

97 MATHEMATICS AND COMPUTING↗

A Unified Data-Driven Approach for Programming In Situ Analysis and Visualization Final Scientific/Technical Report

The goal of Kitware’s work has been the study of a unified data-driven approach for programming applications and in situ analysis and visualization. We have especially focused on understanding the interplay between data-centric programming model requirements at extreme-scale and the overall impact of those requirements on the design, capabilities, flexibility, and implementation details for both applications and the supporting in situ infrastructure. We have successfully demonstrated the use of pipeline abstractions within Legion and developed a number of algorithms that were integrated within the Legion simulation framework. This report focuses on describing milestones, mainly those focusing on exploring pipeline abstractions in the context of the Legion programming model.

97 MATHEMATICS AND COMPUTING↗

Basic Research Needs for Microelectronics

Since the invention of the integrated circuit in 1958, advances in microelectronics have followed Moore’s law and other scaling laws. These have led to over a billion-fold increase in the number of transistors on a chip. The Department of Energy’s Office of Science (DOE-SC) programs have always been at the cutting edge of microelectronics, making major contributions to the scientific understanding, materials, and advanced instrumentation that enabled innovations to promote scaling. This has driven transformative advances in microelectronics for the challenging demands of DOE’s high performance computing and science facilities. Now, strong evidence exists that scaling is approaching its physical and economic limits, and yet, the growth of data-centric computing and sensor networks is redefining computing workloads and microelectronics needs. In addition, greatly improved microelectronics are needed for the nation’s electricity grid if it is to be energy-efficient, resilient to natural phenomena and intentional attack, and agile in adapting to fluctuations in demand and power generation. Sustained and rapid progress in microelectronics science and technology from millivolt to megavolt scales is thus essential if we are to continue pushing the boundaries of science within DOE, and more significantly, to continue to lead the global information and power technology revolution. In October 2018, DOE-SC convened a workshop to address Basic Research Needs for Microelectronics by a “co-design” approach. Participants focused on scientific issues associated with advanced microelectronics technologies for applications relevant to the DOE mission, including computing, power grid management, and science facility workloads. (Topics of direct relevance to quantum information science and quantum computing were outside the scope of this workshop.) The workshop was organized around microelectronics needs for three areas: 1) future DOE-SC facilities, 2) high performance computing beyond exascale (a billion billion calculations per second), and 3) power control, conversion, and detection for a modernized electrical grid and related high power applications. Panels were formed around each of these areas, with a fourth panel formed to address Crosscutting Research. Workshop participants were asked to focus on a co-design innovation ecosystem in which materials, chemistries, devices, systems, architectures, algorithms, and software are researched and developed in a closely integrated fashion, with feedback and interdisciplinary collaboration at each interface in this microelectronics technology hierarchy. The panels identified the five Priority Research Directions on the following page.

36 MATERIALS SCIENCE↗

National-Tribal Critical Infrastructure Protection: Collaboration for Extraordinary National Security Benefit

This paper examines national and tribal collaborative opportunities to get ahead of the critical infrastructure insecurity problem. Recommendations are viewed through the lens of the Sandia Labs Tribal Cyber-Energy initiative and national security projects. Recommendations include 1) Collaboratively address national priority and shared challenges to gain faster and better solutions to national priority problems on a smaller yet comprehensive American Indian and Alaskan Native sovereign single-point of authority scale 2) Utilize newer standards-based technologies to provide scalable, capable, and manageable solutions for greatly expanded and connected national critical infrastructures 3) Employ Cyber-Physical-Resilient design preliminary analysis to define concept- to-disposition design requirements for preemptive critical infrastructure risk mitigation and baked-in security; 4) Develop data-centric protection to provide increased information asset protection as data shifts from data-owner operated on-premises infrastructure to virtual service provider data-steward owned and operated off-premises infrastructure; and 5) Balance shared solutions with the National Institute of Science and Technology (NIST) Cybersecurity and Risk Management frameworks, and the System Security Engineering Guidelines. As yet unallocated federal funding would support research, development, the timely application of National-Tribal critical infrastructure protection, and critical infrastructure Cyber disruption response and recovery with extraordinary mutual benefits for the foreseeable future. The Critical Infrastructure Insecurity Problem: Rapid modernization and expansive connectivity are due to advances in Information and Communications Technologies that have sweeping cyber impact across all critical infrastructure sectors. Supervisory Control and Data Acquisition and Industrial Control Systems are particularly impacted as systems long separated from the Internet are now being connected and computerized. Virtualization and mobility create a Data Everywhere-User Anywhere paradigm that has evaporated the enterprise network perimeter. There are multi-front technological challenges at play, where long depended on technologies simply don't scale to current needs resulting in a digital dichotomy of competing old and new standards. New standards-based technologies scale but are not as well-known or as widely deployed, which leaves decision makers, stakeholders, and the workforce in a quandary, caught mid-stream between the technological past and the virtual future. Rapid and expansive cyber threat accompanies disruptive change in connectivity and computational dependencies. A lack of action will exacerbate the problem if new technologies roll out without baked-in security design. The Risk: If National-Tribal CIP collaboration to design in security is not done, then an ongoing state of insufficient bolt-on security and elevated threat exposure will remain for years to come.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Operational Technology Behavioral Analytics (OTBA) (Final Technical Report DE-FE0031640)

This final report provides a summary of the methodology, findings, lessons learned, and insights from an investigation into the feasibility of the Operational Technology Behavioral Analytics (OTBA) cybersecurity approach. The concept was evaluated with data from the National Carbon Capture Center (NCCC) – a U.S. Department of Energy (DOE) funded facility that is managed and operated by Southern Company Services, Inc. at Alabama Power Company’s E. C. Gaston generating power plant in Wilsonville, Alabama. Appropriate data sources for the post-combustion carbon capture system were identified. Infrastructure was deployed to monitor, capture and archive data for the system. Critical parameters for each subsystem were identified and analyzed. Machine-learning algorithms were used to establish and characterize normal operations and subsequently identify anomalies. This effort yielded valuable insights and formed the basis of a data-centric strategy for detecting cyber-attacks along with a coordinated response philosophy. A significant takeaway is that the OTBA cybersecurity approach is quite portable; it can be applied to other critical infrastructure beyond fossil power generation.

20 FOSSIL-FUELED POWER PLANTS↗

Design-Space Data: Informing Common Design Decisions with Pre-Simulated Data

Design Space Exploration (DSE) analysis techniques represent a data-centric approach to integrating performance analysis in early design phases when there is the greatest potential to cheaply improve the energy efficiency of a building. We focus on a novel extension of DSE called Universal Design Space Exploration (UDSE), which leverages massive databases of pre-simulated analysis that represent all possible outcomes of common analysis workflows. These databases, called Design Spaces, become “universal” when a single pre-simulated design space can be re-applied to future unknown projects. Unlike current simulation methods, which require a design to exist before it can be analyzed and often take minutes or hours to simulate, UDSE leverages pre-simulation to deliver rapid and relevant insight as new designs are conceptualized. The data underpinning UDSE enables advanced statistical and Artificial Intelligence methods, allowing UDSE to deliver a greater understanding of the larger problem being explored, rather than simply delivering analysis of several pre-conceived design options. We believe that UDSE can provide instantaneous, relevant analysis for all building design projects at negligible cost. This paper has two main goals, to develop a relevant Universal Design Space that showcases the potential of UDSE and to release this data freely to industry and academia; thereby lowering the barrier to entry to digital literacy in statistics, ML and AI within the architecture, engineering, construction (AEC) industry.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Data Structure Alchemy

In an increasingly more data-driven world, the project set out to uncover the first principles of data-structure design, chart the immense design space they form, and build automation that can synthesize an optimal structure, or even a whole storage engine, for any given workload, hardware platform, and cost target. Data structures are at the center of every computational system and are directly responsible for its performance. Two core technical thrusts were defined: 1) Mapping design spaces for key data-centric abstractions (filters, hash functions, storage-engine layouts, neural-network topologies, blockchain protocols, image layouts, etc.). 2) Developing search & synthesis algorithms, initially analytical cost models, later neural-guided bi-level optimisers that navigate sextillions of candidate designs in seconds and materialise the best one as ready‐to-run code. This report distills the key insights, accomplishments, and impact.

97 MATHEMATICS AND COMPUTING↗

Energy Emergency and Preparedness Data: Frequently Asked Questions (FAQs) and Quick Guidance on Crisis Communications

State Energy Offices and Public Utility Commissions rely on timely, accurate, and actionable information to perform their energy emergency response duties and execute their roles as state energy security planners. In support of this need, the National Association of State Energy Officials (NASEO) and the National Association of Regulatory Utility Commissions (NARUC) hosted an Energy Security and Data Analysis Workshop in Washington, DC to identify energy security response and planning data sources; and to share successful methods of data use and integration in state, federal, and private sector tools. Following the workshop, NASEO and NARUC hosted two topical data-centric webinars based on state priorities to identify best practices in Geographic Information Systems (GIS) and Crisis Communications programs leveraged in energy assurance planning and response. The Crisis Communications webinar covered best practice tactics for how states can respond during energy emergencies and other crises. Based on the workshop and webinars, this document summarizes commonly used data sources and includes frequently asked questions which may be used by state energy officials (i.e., consisting of staff from Public Utility Commissions and Governor-designated State Energy Offices) to help guide them in developing or improving their Crisis Communications capabilities. A strong public information program is a key crisis management tool. Timely and accurate information helps prevent confusion and uncertainty and encourages public support and cooperation. As energy subject matter experts, state energy officials are vital in the distilling, clarifying, and conveying energy sector information and implications to decision-makers and the public. Other participants in an effective public information program include the Governor’s Office, other state agencies, local governments, energy providers, local businesses, state legislature, and the federal government. It is essential to provide stakeholders and the public with information about the nature, severity, and duration of an emergency because inadequate understanding and awareness can lead to undesirable actions that could further exacerbate the situation. Before a state government can provide information to the public, it must gather information, describe the emergency accurately, and develop recommendations to manage the situation.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

The Data Foundry: Secure Collaboration for the Geothermal Industry: Preprint

The Data Foundry provides secure, cloud-based storage and universal access to digital information, enabling the greater geothermal industry to collaborate seamlessly with the Department of Energy (DOE), national labs, universities, and private organizations. Originally developed to support the EGS Collab project, the Data Foundry has been expanded to support FORGE, EDGE, and other DOE-funded projects, some collaborative, some private, by providing each project with a secure space and the ability to fine-tune individual access controls. In response to user feedback, it now also features improved integration with DOE’s Geothermal Data Repository (GDR), to provide a clear and convenient pathway from collaboration to publication, and to register collaborative data projects with well-known data registries like Data.gov and the National Geothermal Data System (NGDS). This paper will explore how recent concerns raised by data-centric, proprietary projects have informed development on the Data Foundry and highlight improvements designed to streamline workflows, improve access control, and promote the timely dissemination of information to the geothermal industry.

Data Foundry↗

Modernizing the Nuclear Industry and New Ways of Working

Nuclear energy is recognized as the most feasible energy source towards achieving nation-wide net-zero goals (COP 208, MIT). Additionally, the demand for reliable, consistent energy supply is soaring with the construction of energy hungry data centers and AI tools. This means the construction of new nuclear and the continuation of current nuclear are a high national priority. However, this leaves the nuclear industry with a workforce challenge. The rising demand for nuclear power is threatened by an underpopulated workforce. There exist a few contributors to the decreased workforce such as skewed workforce demographics resulting in large-scale workforce retirement. Also, some plants have observed difficulty hiring and retaining skill sets that typically comprise the nuclear power workforce – specifically skilled craftsman. Those who are graduating with desirable skills are seeking employment in sectors that are more modern and culturally more aligned with younger generational values. Lastly, within the nuclear sector, skilled employees will be a competitive commodity as advanced plants come online offering work environments that incorporate modern technologies, skill sets, and opportunities for career advancement. These factors emphasize the need for legacy plants to modernize with a focus on the workplace environment and culture that meets younger generations’ skills, cultural expectations, and desire for advancement opportunities. Nuclear plants are actively engaged in deploying advancements that improve the management of systems, structures, and components as well as process improvements and technologies that can reduce the costs of operation and maintenance. However, the advancements must also be analyzed, reviewed and deployed in a manner that considers the impact on how people within the organization collaborate, communicate, make decisions, and solve problems. Neglecting to consider the cultural impact of modernization risks poor adoption, unrealized opportunities, or insignificant change toward helping attract new employees. The traditional way of working is not necessarily the way new generations want to engage with their employer. For instance, in a survey performed by North American Young Generation in Nuclear the top three reasons for younger nuclear employees to seek other job opportunities was seeking better work-life balance, a lack of advancement opportunities, and work culture and leadership style differences (Smyth et al. 2022). Integrated Operations for Nuclear (ION) is an approach for the nuclear industry to create long-term strategic modernization plans and analyze each advancement for the impact on people and processes and measure how those impacts flow up to support high-level, long-term plant goals and requirements. One principle of ION is to replace the current labor-centric operation style in legacy nuclear plants with data-centric collaborative operation styles. This paper will explain how transitioning operations to leverage centralized skills and responsibility, multi-skilled teams, and collaborative decision making can flatten the hierarchical structure in plants, enable greater individual efficacy while also offering more experiences and advancement opportunities to staff. Adopting the ION way of working shifts the industry mind-set towards more networked, collaborative, and modern way of working that will attract skilled, next-generation workers to operate the legacy nuclear fleet.

99 - GENERAL AND MISCELLANEOUS↗