Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “runtime performance”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 667 records · Page 37
All sourcesNASA NTRSDOE OSTI
NASA NTRS2023

Launch Complex 34, SWMU CCO542022 DNAPL Source Zone Operations, Maintenance, and Monitoring, Site-Wide Long-Term Monitoring, and Hot Spot 6 Air Sparge System Annual Performance Monitoring Report Cape Canaveral Space Force Station, Florida

This Annual Performance Monitoring Report (PMR) for the Dense Non-Aqueous Phase Liquid (DNAPL) Source Zone (DSZ), Site-Wide Long-Term Monitoring (LTM), and Hot Spot (HS) 6 Air Sparge (AS) System presents the results of Year 13 operation of the hydraulic containment (HC) Interim Measure (IM), the results of performance monitoring direct-push technology (DPT) sampling and monitoring well sampling conducted in the DSZ, results of the biennial site-wide LTM event, and the results of operations and performance sampling of the HS 6 AS IM at Launch Complex 34 (LC34), located at Cape Canaveral Space Force Station (CCSFS), Florida. This site has been designated Solid Waste Management Unit (SWMU) CC054 under the Kennedy Space Center (KSC) Resource Conservation and Recovery Act (RCRA) Corrective Action Program. For the site-wide biennial LTM event, a total of 55 monitoring wells were sampled for volatile organic compounds (VOCs) in February 2023 and one well was sampled for polychlorinated biphenyls (PCBs) in December 2022. One well planned for VOC sampling was found to be destroyed and could not be sampled (CW0002). The LTM wells are screened in two lithologic zones: Layer 1 (0 to 25 feet below land surface [bls]) and Layer 2 (25 to 30 ft bls), and are located in the outlying areas of LC34 to monitor groundwater conditions within the Low-Concentration Plume (LCP), defined as concentrations exceeding Groundwater Cleanup Target Levels (GCTLs), and the High-Concentration Plume (HCP), defined as concentrations exceeding Natural Attenuation Default Concentrations (NADCs). Results from the biennial sampling event indicated overall plume stability and delineation for the plume, which extends over 300 acres. The operational period for Year 13 of the HCS was from April 1, 2022 to March 31, 2023. Operational runtime for the system was 90 percent during Year 13, with downtime events attributed to planned maintenance, system repairs, and power outages. As of March 31, 2023, a total of 313,673,241 cumulative gallons of groundwater containing 88,337 pounds of VOCs have been removed by the HCS. Influent concentrations of trichloroethene (TCE) have decreased since startup from approximately 280,000 μg/L (January 2010) to 12,000 μg/L (March 2023). During the reporting period, all effluent concentrations from the HCS (aqueous and vapor) were below regulatory reporting limits, indicating the system continues to operate as intended. Performance monitoring was conducted in December 2022 within the DSZ to evaluate TCE contamination. Groundwater samples were collected via DPT at nine locations, consistent with previous events between 2017 and 2021. Full vertical profile sampling was completed at each DPT from 8 to 98 feet bls, at 5-foot intervals. The DPT performance monitoring results are summarized in this PMR. The results revealed TCE remains at concentrations greater than 11,000 μg/L in the DSZ (1-percent solubility, indicative of DNAPL) at eight of the nine DPT locations and at depths ranging from 8 to 98 feet bls. In addition to DPT sampling, groundwater samples were collected from deep monitoring wells in the DSZ area (Layers 7 and 8) in December 2022 to verify vertical delineation. Layer 7/8 monitoring well results were non-detect in December 2022, with exception of three wells (IW0162, IW043D2, and IW044D2), where TCE, cis-1,2-dichloroethene (cDCE), and/or vinyl chloride (VC) were detected above GCTLs. These wells are screened 105 to 115 feet bls, which is below the existing recovery well capture zone. TCE was first detected in IW0162 in December 2021 and has since been sampled at least monthly to monitor TCE concentrations. The maximum TCE concentration during this operational period was 15,000 μg/L at IW0162 in March 2023. Because of the increased TCE concentrations in this well, a new recovery well (RW21D), screened 86 to 106 feet bls, was installed in January 2023 and incorporated into existing HCS operations. The HS 6 AS IM was initiated in 2018 with 160 AS wells, and expanded in 2019 with an additional 140 AS wells. Quarterly performance monitoring was reduced to semi-annual in 2020. The HS 6 AS system remained operational during the reporting period covered under this report. The results of the HS 6 system operation and semi-annual performance monitoring are summarized in this report. Semi-annual monitoring results collected in April and October 2022 show concentrations of contaminants of concern (cDCE, trans-1,2-dichloroethene, and vinyl chloride) are generally decreasing and not impacting the surface water drainage canal, indicating the HS 6 IM is meeting objectives. A Phase Two Expansion of the HS 6 AS IM was recently completed. As of the date of this report, the expansion became operational in August 2023 and the first quarter of monitoring was conducted in November 2023. Details of the construction, start-up, and performance monitoring will be included in a future Annual PMR. Overall, the tasks associated with Year 13 operation of the HC IM, operation of the HS 6 AS IM, and biennial site-wide sampling were performed in accordance with recommendations included in the previous 2021 LC34 (Year 12) annual report. Evaluation of results from the HC IM and HS 6 IM show that these systems are operating as designed and meeting performance objectives. Results from the site-wide biennial LTM program also show that the overall network of monitoring wells is adequate to continue monitoring plume-wide conditions.

Complex 34↗
DOE OSTI2020

Extending the Frontier of Quantum Computers with Qutrits

Here, we advocate for a fundamentally different way to perform quantum computation by using three-level qutrits instead of qubits. In particular, we substantially reduce the resource requirements of quantum computations by exploiting a third state for temporary variables (ancilla) in quantum circuits. Past work with qutrits has demonstrated only constant factor improvements, owing to the log 2 (3) binary-to-ternary compression factor. We present a novel technique using qutrits to achieve a logarithmic runtime decomposition of the Generalized Toffoli gate using no ancilla - an exponential improvement over the best qubit-only equivalent. Our approach features a 70× improvement in total two-qudit gate count over the qubit-only decomposition. This results in improvements for important algorithms for arithmetic and QRAM. Simulation results under realistic noise models indicate over 90% mean reliability (fidelity) for our circuit, versus under 30% for the qubit-only baseline. These results suggest that qutrits offer a promising path toward extending the frontier of quantum computers.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2018

Applications of the Dynamic N-Dimensional K-Vector

The n-dimensional k-vector (NDKV) is an appealing alternative to binary tress for resolving complex queries in large relational databases. The method has excelled in several applications involving static databases. The present paper extends the theory supporting the NDKV to handle dynamic databases, where the data is updated frequently. This includes deleting records, adding new entries, or editing existing elements. The merit of this new version of the NDKV, the dynamic n-dimensional k-vector (DNDKV), is that it is no longer necessary to recompute the entire k-vector (the main structure that indexes the data) every time a record changes. The algorithm updates the four constituents of the standard NDKV on the fly: the database, sorted database, index, and k-vector tables. As a result, the DNDKV becomes comparable in terms of capabilities and flexibility to stateof-the-art storage engines relying on structured query languages (SQL). The performance of the DNDKV is assessed by running typical read/write operations on a database that contains millions of pre-computed missions to celestial bodies. This database requires frequent updates whenever an orbit solution is refined or new bodies are discovered. The DNDKV is faster than rebuilding the k-vector tables completely, provided that the number of elements being added or removed is not excessively large. Direct runtime comparisons with MySQL suggest that the DNDKV is several times faster for reading but might be slower for writing and updating the database. One limit of the technique is the elements being added must be within the range of the current k-vector tables. If this is not the case, the technique cannot be used and the k-vector tables must be rebuilt from scratch.

Mortari, Daniele↗
DOE OSTI2023

On the Feasibility of Simulation-Driven Portfolio Scheduling for Cyberinfrastructure Runtime Systems

Runtime systems that automate the execution of applications on distributed cyberinfrastructures need to make scheduling decisions. Researchers have proposed many scheduling algorithms, but most of them are designed based on analytical models and assumptions that may not hold in practice. The literature is thus rife with algorithms that have been evaluated only within the scope of their underlying assumptions but whose practical effectiveness is unclear. It is thus difficult for developers to decide which algorithm to implement in their runtime systems.To obviate the above difficulty, we propose an approach by which the runtime system executes, throughout application execution, simulations of this very execution. Each simulation is for a different algorithm in a scheduling algorithm portfolio, and the best algorithm is selected based on simulation results. The main objective of this work is to evaluate the feasibility and potential merit of this portfolio scheduling approach, even in the presence of simulation inaccuracy, when compared to the traditional one-algorithm approach. We perform this evaluation via a case study in the context of scientific workflows. Our main finding is that portfolio scheduling can outperform the best one-algorithm approach even in the presence of relatively large simulation inaccuracies.

Casanova, Henri↗
DOE OSTI2017

Use of DAGMan in CRAB3 to improve the splitting of CMS user jobs

CRAB3 is a workload management tool used by CMS physicists to analyze data acquired by the Compact Muon Solenoid (CMS) detector at the CERN Large Hadron Collider (LHC). Research in high energy physics often requires the analysis of large collections of files, referred to as datasets. The task is divided into jobs that are distributed among a large collection of worker nodes throughout the Worldwide LHC Computing Grid (WLCG). Splitting a large analysis task into optimally sized jobs is critical to efficient use of distributed computing resources. Jobs that are too big will have excessive runtimes and will not distribute the work across all of the available nodes. However, splitting the project into a large number of very small jobs is also inefficient, as each job creates additional overhead which increases load on infrastructure resources. Currently this splitting is done manually, using parameters provided by the user. However the resources needed for each job are difficult to predict because of frequent variations in the performance of the user code and the content of the input dataset. As a result, dividing a task into jobs by hand is difficult and often suboptimal. In this work we present a new feature called “automatic splitting” which removes the need for users to manually specify job splitting parameters. We discuss how HTCondor DAGMan can be used to build dynamic Directed Acyclic Graphs (DAGs) to optimize the performance of large CMS analysis jobs on the Grid. We use DAGMan to dynamically generate interconnected DAGs that estimate the processing time the user code will require to analyze each event. This is used to calculate an estimate of the total processing time per job, and a set of analysis jobs are run using this estimate as a specified time limit. Some jobs may not finish within the alloted time; they are terminated at the time limit, and the unfinished data is regrouped into smaller jobs and resubmitted.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2016

Usage Pattern-Driven Dynamic Data Layout Reorganization

As scientific simulations and experiments move toward extremely large scales and generate massive amounts of data, the data access performance of analytic applications becomes crucial. A mismatch often happens between write and read patterns of data accesses, typically resulting in poor read performance. Data layout reorganization has been used to improve the locality of data accesses. However, current data reorganizations are static and focus on generating a single (or set of) optimized layouts that rely on prior knowledge of exact future access patterns. We propose a framework that dynamically recognizes the data usage patterns, replicates the data of interest in multiple reorganized layouts that would benefit common read patterns, and makes runtime decisions on selecting a favorable layout for a given read pattern. This framework supports reading individual elements and chunks of a multi-dimensional array of variables. Our pattern-driven layout selection strategy achieves multi-fold speedups compared to reading from the original dataset.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2022

Launch Complex 34, SWMU Cc054 2021 DNAPL Source Zone Operations, Maintenance, and Monitoring, and Hot Spot 6 Air Sparge System Annual Performance Monitoring Report Cape Canaveral Space Force Station, Florida

This Annual Performance Monitoring Report (PMR) for the Dense Non-Aqueous Phase Liquid (DNAPL) Source Zone (DSZ) and Hot Spot 6 (HS 6) Air Sparge (AS) System presents the results of Year 12 operation of the hydraulic containment (HC) Interim Measure (IM), the results of performance monitoring direct-push technology (DPT) sampling and monitoring well sampling conducted in the DSZ, and the results of operations and performance sampling of the HS 6 AS IM at Launch Complex 34 (LC34), located at Cape Canaveral Space Force Station (CCSFS), Florida. Site-wide biennial LTM sampling was not conducted during this reporting period and is scheduled to be conducted in December 2022. The timeframe for activities documented in this PMR extends from April 1, 2021 to March 31, 2022. LC34 has been designated Solid Waste Management Unit CC054 under the Kennedy Space Center (KSC) Resource Conservation and Recovery Act Corrective Action Program. The objective of the HC IM at LC34 is to contain the DSZ and deep dissolved-phase trichloroethene (TCE) high concentration plume via operation of a hydraulic containment system (HCS). The pre-IM design 300 micrograms per liter (μg/L) TCE groundwater contour was used to establish the deep zone capture area for deep recovery wells, and the shallow zone capture area was defined by the DSZ. The system began operating in 2010, and in 2015, the system was expanded to provide HC for areas within the 300 μg/L TCE groundwater isocontours of HS 3 and 4. In 2018 and 2019, an investigation was conducted to re-characterize the DSZ, which included investigating TCE mass in Layer 7. This data was subsequently used to optimize the pumping rates of the HCS to more adequately capture residual contaminant mass. The operational period for Year 12 of the HCS was from April 1, 2021 to March 31, 2022. Operational runtime for the system was 94 percent during Year 12, with downtime events attributed to planned maintenance, system repairs, and power outages. As of March 31, 2022, a total of 285,712,801 gallons of groundwater containing 84,933 pounds of VOCs have been removed by the HCS. Influent concentrations of TCE have decreased since startup from approximately 280,000 µg/L (January 2010) to 12,000 µg/L (March 2022). During the reporting period, all effluent concentrations from the HCS (aqueous and vapor) were below regulatory reporting limits, indicating the system continues to operate as intended. Performance monitoring was conducted in December 2021 within the DSZ to evaluate TCE contamination. Groundwater samples were collected via DPT at nine locations, consistent with previous events in 2017, 2018, 2019, and 2020. Full vertical profile sampling was completed at each DPT from 8 to 98 ft bls, at 5 foot intervals. The DPT performance monitoring results are summarized in this PMR. The results revealed TCE remains at concentrations greater than 11,000 µg/L in the DSZ (1-percent solubility, indicative of DNAPL) at eight of the nine DPT locations at depths ranging from 8 to 98 ft bls. An overall increasing trend of TCE concentrations was observed in DPT samples during this reporting period, which may be due to several recovery wells that were turned off during the AS Pilot Study in the DSZ that operated from July 2021 to February 2022 (documented separately from this report). The maximum TCE concentration in 2021 was 15,400,000 µg/L in the 58 ft bls depth interval at DPT597 (previous maximum result in 2020 was 1,690,000 at 48 ft bls at DPT596). During the 2021 DPT event, the overall majority of TCE contamination was identified in the 58 ft bls interval (below Layer 4), where in the previous year the majority of mass was observed in Layer 4. This trend appears to indicate continued mass discharge from Layer 4 (fine-grained unit). In addition to DPT sampling, monitoring well samples were collected from deep wells in the DSZ area (Layers 7 and 8) to verify vertical delineation. All monitoring well results were non-detect or below cleanup levels, with exception of one well (IW0162, screened 105 to 115 ft bls, which is below the existing recovery well capture zone) where TCE was identified above cleanup target levels. The HS 6 AS system remained operational during the reporting period covered under this report. The HS 6 AS IM was initiated in 2018 with 160 AS wells, and expanded in 2019 with an additional 140 AS wells. Quarterly performance monitoring was reduced to semi-annual prior to this operational period. The results of the HS 6 system operation and semi-annual performance monitoring are summarized in this report. Semi-annual monitoring results collected in April and October 2021 show concentrations of contaminants of concern (cis-1,2-dichloroethene, trans-1,2- dichloroethene, and vinyl chloride) are generally decreasing and not impacting the surface water drainage canal, indicating the HS 6 IM is meeting objectives. Overall, the tasks associated with Year 12 operation of the HC IM and operation of the HS 6 AS IM were performed in accordance with the recommendations of the 2020 LC34 (Year 11) Operations, Maintenance, and Monitoring Report for DNAPL Source Zone, Site Wide LongTerm Monitoring, and Hot Spot 6 Air Sparging System PMR (NASA, 2021c). Evaluation of results from the HC IM and HS 6 IM show that these systems are operating as designed and meeting performance objectives.

trichloroethene↗
NASA NTRS2018

Imaging for Hypersonic Experimental Aeroheating Testing (IHEAT) Version 4.0: User Manual

The IHEAT v4.0 software is a data reduction code for global thermography data acquired in the NASA Langley Aerothermodynamics Laboratory (LAL) hypersonic wind tunnels. IHEAT uses red and green color-intensity data from two-dimensional images of wind tunnel models to compute temperatures and heat-transfer rates using a semi-infinite, one-dimensional heat transfer approximation at each image pixel. Multiple automated tools in IHEAT v4.0 decrease the time required to reduce the data from a phosphor thermography wind tunnel run. Data at one or all of the image pixel locations can be exported to computer files for further analysis. The prior version of IHEAT, v3.2, was written in PV-WAVE® (now owned by Rogue Wave® Software) in 1994 and was limited in functionality to fit within the memory constraints of the available computers at the time. IHEAT v4.0 is written in MATLAB® by MathWorks® and contains several new features that leverage the increase in available memory of the current computers. A Piecewise tool permits the user to extract data along a segmented line cut that can follow interesting features in the image better than the single, straight line cuts that were possible with the legacy Length and Profile tools. The new Load Run and Batch tools facilitate batch processing by loading in all of the input files and images for a run at the same time. Load Run permits the user to process the available run images manually, while Batch automatically saves heat transfer data from all of the images based on the analysis previously performed on a single frame. IHEAT v4.0 also can automatically calculate the temporal collapse of reference line cuts from the time history heating data for a run to indicate the appropriate frame to reduce for each run. The IHEAT v4.0 source code was compiled into a standalone executable file that can be accessed remotely from several computers with different operating systems, simultaneously. The software is run through the MATLAB® Compiler Runtime engine, and therefore, IHEAT does not require a software license to run. Any software commands executed in the IHEAT v4.0 code will not affect other similar applications running on the same machine. Similarly, changes to the parent software do not affect a compiled code. These features of IHEAT v4.0 are improvements over the legacy v3.2 code, which required regular maintenance to avoid losing functionality as the PVWAVE ® programming language was upgraded.

Mason, Michelle L.↗
DOE OSTI2017

Block-Parallel Data Analysis with DIY2

DIY2 is a programming model and runtime for block-parallel analytics on distributed-memory machines. Its main abstraction is block-structured data parallelism: data are decomposed into blocks; blocks are assigned to processing elements (processes or threads); computation is described as iterations over these blocks, and communication between blocks is defined by reusable patterns. By expressing computation in this general form, the DIY2 runtime is free to optimize the movement of blocks between slow and fast memories (disk and flash vs. DRAM) and to concurrently execute blocks residing in memory with multiple threads. This enables the same program to execute in-core, out-of-core, serial, parallel, single-threaded, multithreaded, or combinations thereof. This paper describes the implementation of the main features of the DIY2 programming model and optimizations to improve performance. DIY2 is evaluated on benchmark test cases to establish baseline performance for several common patterns and on larger complete analysis codes running on large-scale HPC machines.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Extreme Heterogeneity 2018 - Productive Computational Science in the Era of Extreme Heterogeneity: Report for DOE ASCR Workshop on Extreme Heterogeneity

The 2018 Basic Research Needs Workshop on Extreme Heterogeneity identified five Priority Research Directions for realizing the capabilities needed to address the challenges posed in this era of rapid technological change. This report captures the outcomes of that workshop. In the context of extreme heterogeneity, it defines basic research needs and opportunities in computer science research to develop smart and trainable operating and runtime systems, programming environments, and predictive tools that will make future systems easier to adapt to scientists’ computing needs and easier for facilities to deploy securely.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Verification and Performance Impact of the New Parallel MCNP6.3 Particle Track Output Capability for Subcritical Multiplication Simulations [Slides]

A separate MCNP6.3 V&V document reports on all the default calculations for all test suites. This report does not include the subcritical multiplication benchmark suite. After some additional clean-up and finalizing the post-processing and documentation steps, the subcritical multiplication benchmark suite will be released in the next version of our vnvstats repository. We tested the new HDF5 PTRAC feature in MCNP6.3 and found encouraging outcomes. Identical results coming out of the simulation with respect to the legacy PTRAC results. The overall runtime for all simulations is reduced by ~20% with the new HDF5 PTRAC capability. We consider giving the new HDF5 PTRAC features a try and using it for all subcritical multiplication and any other relevant (PTRAC) calculations.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2003

Experimental Evaluation of Verification and Validation Tools on Martian Rover Software

We report on a study to determine the maturity of different verification and validation technologies (V&V) on a representative example of NASA flight software. The study consisted of a controlled experiment where three technologies (static analysis, runtime analysis and model checking) were compared to traditional testing with respect to their ability to find seeded errors in a prototype Mars Rover. What makes this study unique is that it is the first (to the best of our knowledge) to do a controlled experiment to compare formal methods based tools to testing on a realistic industrial-size example where the emphasis was on collecting as much data on the performance of the tools and the participants as possible. The paper includes a description of the Rover code that was analyzed, the tools used as well as a detailed description of the experimental setup and the results. Due to the complexity of setting up the experiment, our results can not be generalized, but we believe it can still serve as a valuable point of reference for future studies of this kind. It did confirm the belief we had that advanced tools can outperform testing when trying to locate concurrency errors. Furthermore the results of the experiment inspired a novel framework for testing the next generation of the Rover.

Brat, Guillaume↗
NASA NTRSDate not supplied

CropManage Application for Vineyard Irrigation Decision-Support

CropManage is a free web-application developed by U.C. Cooperative Extension to support evapotranspiration based irrigation scheduling and nutrient management for major specialty crops. Prescribed phenology curves are used to develop daily estimates of canopy cover within a given field, based on days since planting (annual crops) or budbreak (trees, vines). These curves are modulated by a MaxCan parameter representing seasonal maximum canopy cover. Crop development observations can be used to adjust for such factors as weather anomalies or non-standard agronomic practice, as needed. Canopy cover is converted to crop coefficient and combined with reference evapotranspiration to derive daily water consumption. Guidance on crop water requirement is then conveyed to users in terms of system runtime issued on-demand for a given date, largely based on total evapotranspiration since last irrigation event. In this study, CropManage was adapted to vineyards by adding modules accounting for early-season soil moisture depletion and cover crop presence. A crop stress parameter was added to accommodate deficit irrigation practice, allowing the user to specify percentage departure from full water requirement along with start/stop dates. An initial verification exercise was performed on three winegrape vineyards located in California’s Central Coast (2020), North Coast (2020) and Central Valley (2019). Daily crop evapotranspiration was monitored by eddy-covariance fluxtowers. MaxCan was measured by ground and satellite observation. Stress regime was specified by grower practice where available, otherwise stress levels were inferred from applied water records. Mean absolute error and mean bias error of modeled cumulative evapotranspiration were computed with respect to the eddy covariance measurements collected throughout the growing season. Results indicate the modified CropManage water management module performs reasonably well for winegrape. Additional effort is planned to modify the nutrient module for vineyard use.

CropManage↗
DOE OSTI2015

Adding Data Management Services to Parallel File Systems

The objective of this project, called DAMASC for “Data Management in Scientific Computing”, is to coalesce data management with parallel file system management to present a declarative interface to scientists for managing, querying, and analyzing extremely large data sets efficiently and predictably. Managing extremely large data sets is a key challenge of exascale computing. The overhead, energy, and cost of moving massive volumes of data demand designs where computation is close to storage. In current architectures, compute/analysis clusters access data in a physically separate parallel file system and largely leave it scientist to reduce data movement. Over the past decades the high-end computing community has adopted middleware with multiple layers of abstractions and specialized file formats such as NetCDF-4 and HDF5. These abstractions provide a limited set of high-level data processing functions, but have inherent functionality and performance limitations: middleware that provides access to the highly structured contents of scientific data files stored in the (unstructured) file systems can only optimize to the extent that file system interfaces permit; the highly structured formats of these files often impedes native file system performance optimizations. We are developing Damasc, an enhanced high-performance file system with native rich data management services. Damasc will enable efficient queries and updates over files stored in their native byte-stream format while retaining the inherent performance of file system data storage via declarative queries and updates over views of underlying files. Damasc has four key benefits for the development of data-intensive scientific code: (1) applications can use important data-management services, such as declarative queries, views, and provenance tracking, that are currently available only within database systems; (2) the use of these services becomes easier, as they are provided within a familiar file-based ecosystem; (3) common optimizations, e.g., indexing and caching, are readily supported across several file formats, avoiding effort duplication; and (4) performance improves significantly, as data processing is integrated more tightly with data storage. Our key contributions are: SciHadoop which explores changes to MapReduce assumption by taking advantage of semantics of structured data while preserving MapReduce’s failure and resource management; DataMods which extends common abstractions of parallel file systems so they become programmable such that they can be extended to natively support a variety of data models and can be hooked into emerging distributed runtimes such as Stanford’s Legion; and Miso which combines Hadoop and relational data warehousing to minimize time to insight, taking into account the overhead of ingesting data into data warehousing.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2023

Towards Lightweight Data Integration Using Multi-Workflow Provenance and Data Observability

Modern large-scale scientific discovery requires multidisciplinary collaboration across diverse computing facilities, including High Performance Computing (HPC) machines and the Edge-to-Cloud continuum. Integrated data analysis plays a crucial role in scientific discovery, especially in the current AI era, by enabling Responsible AI development, FAIR, Reproducibility, and User Steering. However, the heterogeneous nature of science poses challenges such as dealing with multiple supporting tools, cross-facility environments, and efficient HPC execution. Building on data observability, adapter system design, and provenance, we propose MIDA: an approach for lightweight runtime Multi-workflow Integrated Data Analysis. MIDA defines data observability strategies and adaptability methods for various parallel systems and machine learning tools. With observability, it intercepts the dataflows in the background without requiring instrumentation while integrating domain, provenance, and telemetry data at runtime into a unified database ready for user steering queries. We conduct experiments showing end-to-end multi-workflow analysis integrating data from Dask and MLFlow in a real distributed deep learning use case for materials science that runs on multiple environments with up to 276 GPUs in parallel. We show near-zero overhead running up to 100,000 tasks on 1,680 CPU cores on the Summit supercomputer.

Santos Souza, Renan↗
DOE OSTI2015

Bus.py: A GridLAB-D Communication Interface for Smart Distribution Grid Simulations

As more Smart Grid technologies (e.g., distributed photovoltaic, spatially distributed electric vehicle charging) are integrated into distribution grids, static distribution simulations are no longer sufficient for performing modeling and analysis. GridLAB-D is an agent-based distribution system simulation environment that allows fine-grained end-user models, including geospatial and network topology detail. A problem exists in that, without outside intervention, once the GridLAB-D simulation begins execution, it will run to completion without allowing the real-time interaction of Smart Grid controls, such as home energy management systems and aggregator control. We address this lack of runtime interaction by designing a flexible communication interface, Bus.py (pronounced bus-dot-pie), that uses Python to pass messages between one or more GridLAB-D instances and a Smart Grid simulator. This work describes the design and implementation of Bus.py, discusses its usefulness in terms of some Smart Grid scenarios, and provides an example of an aggregator-based residential demand response system interacting with GridLAB-D through Bus.py. The small scale example demonstrates the validity of the interface and shows that an aggregator using said interface is able to control residential loads in GridLAB-D during runtime to cause a reduction in the peak load on the distribution system in (a) peak reduction and (b) time-of-use pricing cases.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

ReEDS Performance Improvement

The Regional Energy Deployment System (ReEDS) is an open-source, spatially explicit, long-term capacity expansion model for the bulk electric power system of the contiguous United States, encompassing multiple scenarios with technological and political assumptions (see https://github.com/NREL/ReEDS-2.0). With the increased needs for capabilities, higher temporal and spatial resolutions to model the evolution of the power system with modern technologies and low-carbon pathways, ReEDS' model solution times have increased significantly from 4-6 hours in 2018 to 18-48+ hours in 2023 . Also, the model size for commonly-run ReEDS scenarios reached 22 and 28 million equations and variables, respectively. These runtimes can be especially challenging under certain scenario settings (e.g., very high temporal or spatial resolution) or with limited computational power. In this presentation, we will discuss several methods we used to improve model runtime, including data preparation, model modification, and solver tuning. The implementation of these methods shrank the model size to 7.2 and 7.3 million equations and variables, respectively. Furthermore, this led to a 77% reduction in the model's run time for commonly-run ReEDS scenarios. We will discuss the process of identifying areas for solve time improvements and how the specific enhancements for the ReEDS model might be applied to other similar large-scale models.

ENERGY PLANNING, POLICY, AND ECONOMY,MATHEMATICS A↗
DOE OSTI2015

On noise and the performance benefit of nonblocking collectives

Relaxed synchronization offers the potential of maintaining application scalability by allowing many processes to make independent progress when some processes suffer delays. Yet, the benefits of this approach in important parallel workloads have not been investigated in detail. In this paper, we use a validated simulation approach to explore the noise mitigation effects of idealized nonblocking collectives in workloads where these collectives are a major contributor to total execution time. In conclusion, although nonblocking collectives are unlikely to provide significant noise mitigation to applications in the low-OS-noise environments expected in next-generation HPC systems, we show that they can potentially improve application runtime with respect to other noise types.

97 MATHEMATICS AND COMPUTING↗