Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 793 records · Page 44
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

Enabling Diverse Software Stacks on Supercomputers using High Performance Virtual Clusters.

While large-scale simulations have been the hallmark of the High Performance Computing (HPC) community for decades, Large Scale Data Analytics (LSDA) workloads are gaining attention within the scientific community not only as a processing component to large HPC simulations, but also as standalone scientific tools for knowledge discovery. With the path towards Exascale, new HPC runtime systems are also emerging in a way that differs from classical distributed com- puting models. However, system software for such capabilities on the latest extreme-scale DOE supercomputing needs to be enhanced to more appropriately support these types of emerging soft- ware ecosystems. In this paper, we propose the use of Virtual Clusters on advanced supercomputing resources to enable systems to support not only HPC workloads, but also emerging big data stacks. Specifi- cally, we have deployed the KVM hypervisor within Cray's Compute Node Linux on a XC-series supercomputer testbed. We also use libvirt and QEMU to manage and provision VMs directly on compute nodes, leveraging Ethernet-over-Aries network emulation. To our knowledge, this is the first known use of KVM on a true MPP supercomputer. We investigate the overhead our solution using HPC benchmarks, both evaluating single-node performance as well as weak scaling of a 32-node virtual cluster. Overall, we find single node performance of our solution using KVM on a Cray is very efficient with near-native performance. However overhead increases by up to 20% as virtual cluster size increases, due to limitations of the Ethernet-over-Aries bridged network. Furthermore, we deploy Apache Spark with large data analysis workloads in a Virtual Cluster, ef- fectively demonstrating how diverse software ecosystems can be supported by High Performance Virtual Clusters.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

Using Kamodo for CCMC ITM Output and Beyond

Kamodo is an official NASA open source python software package that functionalizes diverse datasets from models and observations in a consistent way, enabling advanced scientific analysis and visualization with simplistic syntax. Here we demonstrate this ability using several ITM models available through the Community Coordinated Modeling Center (CCMC). Users can now interact directly with model outputs, and satellites can be virtually flown through model output to allow many types of model/model and data/model comparisons. We will also provide information about significant updates and improvements to Kamodo and future plans.

Open Source Software↗
DOE OSTI2016

Dependability Assessment of Software for Safety Instrumentation and Control Systems

In its 2013 recommendations to the International Atomic Energy Agency (IAEA), the Technical Working Group on Nuclear Power Plant Instrumentation and Control (TWG-NPPIC) identified assessment and evaluation of software dependability as a topic of interest for IAEA Member States. To implement the recommendation of the TWG-NPPIC, the IAEA Scientific Secretary for the TWG initiated a consultancy activity in 2014 to develop a technical report describing a dependability assessment framework suitable for safety instrumentation and control (I and C) systems in nuclear power plants (NPPs). The development of the software dependability assessment report has involved three consultancy meetings at IAEA headquarters in Vienna, Austria. The conduct of these meetings allowed consultation among subject matter experts to generate consensus technical information that serves as the content of the report. The technical contributors consist of international experts from academia, industry, regulatory bodies and research organizations. In September 2014, a technical meeting was held in Daejeon, Republic of Korea, to present interim findings of the consultancy effort and to solicit input from representatives of IAEA Member States. The information and technical comments contributed during the technical meeting were addressed in subsequent development of the report. This paper gives an overview of key technical elements of the IAEA report. The IAEA report presents a framework by which a dependability assessment of safety software can be performed. The specified guiding principles provide the basis for establishing an assessment strategy and implementing the assessment process. The assessment framework provides a systematic, coherent structure within which to organize technical and process evidence to satisfy arguments in support of claims about software dependability properties. During the consultancy activity, the participating subject matter experts focused on high level principles, a structured approach, and identification of suitable methods and techniques. However, it was recognized that what remains is to concretize the framework through specific examples and usage experience. Thus, following publication of the report, it is recommended that case studies be performed to illustrate and more precisely define specific phases and elements of the assessment framework. For example, current national assessment approaches could be mapped into the assessment framework to serve as case studies and help to identify any aspects of the assessment framework that needs further definition or adjustment. Additionally, a challenge problem could be defined to engage stakeholders to apply the assessment framework with a common focus. This could help to better illustrate strategic options and approaches to the assessment phases. It also could serve to suggest other considerations and potential sources of evidence not fully covered in the report. Based on findings from case studies, challenge problems and country experience, it should be possible to develop generalized templates to support application of the assessment framework. These findings can be collected to serve as the basis for a User's Guide on dependability assessment of safety software. (authors)

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2014

IOPro: a parallel I/O profiling and visualization framework for high-performance storage systems

Efficient execution of large-scale scientific applications requires high-performance computing systems designed to meet the I/O requirements. To achieve high-performance, such data-intensive parallel applications use a multi-layer layer I/O software stack, which consists of high-level I/O libraries such as PnetCDF and HDF5, the MPI library, and parallel file systems. To design efficient parallel scientific applications, understanding the complicated flow of I/O operations and the involved interactions among the libraries is quintessential. Such comprehension helps identify I/O bottlenecks and thus exploits the potential performance in different layers of the storage hierarchy. To profile the performance of individual components in the I/O stack and to understand complex interactions among them, we have implemented a GUI-based integrated profiling and analysis framework, IOPro. IOPro automatically generates an instrumented I/O stack, runs applications on it, and visualizes detailed statistics based on the user-specified metrics of interest. We present experimental results from two different real-life applications and show how our framework can be used in practice. By generating an end-to-end trace of the whole I/O stack and pinpointing I/O interference, IOPro aids in understanding I/O behavior and improving the I/O performance significantly.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

S4PST: Sustainability for Programming Systems and Tools: May Workshop Report

The US Department of Energy (DOE) Exascale Computing Project (ECP) has fostered and strengthened the use of modern software engineering practices for developing applications and libraries, and this effort has resulted in the coordinated and interoperable E4S1 and xSDK2 ecosystems. Although this approach is cost-effective, it relies on robust programming systems and tools (PST) as the underlying foundation for our HPC software. At present, our primary PST stack consists of traditional high-performance computing (HPC) languages, namely Fortran, C, C++, and the popular Python language for data analysis and AI workflows. These languages support various programming frameworks and run-time abstractions that enable parallelism and concurrency across multiple node architectures and thousands of nodes through a variety of interconnect systems. However, to accommodate users’ diverse needs, certain aspects of the HPC ecosystem are delegated to vendor-specific or third-party implementations that extend beyond a particular scientific domain. This broader scope results in a multitude of specifications and variations, which leads to a complex orchestration of many-ecosystems. Unfortunately, this complexity in the ecosystem imposes additional overhead costs on consumers during the latter stages of the development cycle. In addition to the software ecosystem challenge, the upcoming conclusion of the ECP by December 2023 has raised significant concerns within the HPC programming systems community, from both the economic and social perspectives. The ECP has implemented a management structure for software development and funding decisions across all ECP participants by following a conventional hierarchical and centralized approach. However, this structure has prompted certain considerations within the community, particularly in anticipation of the Software Sustainability initiative by the DOE’s Advanced Scientific Computing Research Program (ASCR). For the success of this new initiative, it is of utmost importance to secure consistent funding and foster close engagement with researchers and core developers of existing programming-system products. This collaboration is vital to maintaining the critical capabilities of the current software during the transition phase while proactively adapting to future technology and workforce trends. The community recognizes the significance of adapting to emerging trends and is aware of the inherent fragility of the HPC software ecosystem, particularly in relation to programming systems that cater to all users. The ability to adapt and evolve is essential to staying relevant and effectively addressing these technical, economic, and social challenges. The S4PST team, which represents one of the six ASCR Software Sustainability seedling projects, is dedicated to tackling these challenges through community-based approaches that go beyond the scope of the DOE. This involves collaboration between national laboratories with academia, non-DOE institutions, hardware and system vendors, and international partners. By fostering these partnerships, we aim to create a robust and sustainable HPC software ecosystem that can effectively meet the needs of the community. This new community effort, driven by the eight DOE labs, will take on the responsibility of guiding funding decisions for programming-systems development and maintenance with transparency and consistency across all decisions. Additionally, the team will offer common technical services to the programming systems community, irrespective of their funding situations, and facilitate community-wide incubation to proactively nurture the software ecosystem. By actively engaging with stakeholders and employing a collaborative approach, we can collectively shape the future of programming systems and ensure a robust and thriving HPC software landscape. On May 11–12, 2023, the S4PST team conducted its inaugural kick-off workshop at the Innovative Computing Laboratory (ICL) in the University of Tennessee, Knoxville, hosted by Hartwig Anzt. The workshop encompassed various sessions dedicated to presentations and discussions, with the aim of comprehending the team members’ perspectives on the vision of software sustainability. Additionally, the workshop aimed to identify the technical, economic, and social requirements for sustaining the programming-systems community in the field of HPC. This report provides a summary of the S4PST effort by highlighting five major thrust areas discussed during the workshop: (i) community, (ii) technical support, (iii) training and diversity, (iv) verification, validation and correctness, and (v) emerging technologies. It also encompasses an overview of the presentations and discussions held throughout the event, our views and potential synergies with other seedling efforts, along with the outcomes and key takeaways from our initial discussions.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Microstructure-Oriented Scientific Analysis of Irradiated Concrete

The inventors are continuing development of this technology expanding a database of concrete irradiation samples. Said software may help the recertification process for nuclear power plants and may find use in characterization of cement infrastructure. UT-Battelle tech transfer will work with the inventors to identify software developers in the nuclear industry that may be interested in further development of this software.

Giorla, Alain B↗
NASA NTRS2007

Orbit Determination of LEO Satellites for a Single Pass through a Radar: Comparison of Methods

The problem of determining the orbit of a space object from measurements based on one pass through the field of view of a radar is not a new one. Extensive research in this area has been carried out in the USA and Russia since the late 50s when these countries started the development of ballistic missile defense (BMD) and Early Warning systems. In Russia these investigations got additional stimulation in the early 60s after the decision to create a Space Surveillance System, whose primary task would be the maintenance of the satellite catalog. These problems were the focus of research interest until the middle 70s when the appropriate techniques and software were implemented for all radars. Then for more than 20 years no new research papers appeared on this subject. This produced an impression that all the problems of track determination based on one pass had been solved and there was no need for further research. In the late 90s interest in this problem arose again in relation to the following. It was estimated that there would be greater than 100,000 objects with size greater than 1-2 cm and collision of an operational spacecraft with any of these objects could have catastrophic results. Thus, for prevention of hazardous approaches and collisions with valuable spacecraft the existing satellite catalog should be extended by at least an order of magnitude This is a very difficult scientific and engineering task. One of the issues is the development of data fusion procedures and the software capable of maintaining such a huge catalog in near real time. The number of daily processed measurements (of all types, radar and optical) for such a system may constitute millions, thus increasing the number of measurements by at least an order of magnitude. Since we will have ten times more satellites and measurements the computer effort required for the correlation of measurements will be two orders of magnitude greater. This could create significant problems for processing data close to real time even for modern computers. Preliminary "compression" of data for one pass through the field of view of a sensor can significantly reduce the requirements to computers and data communication. This compression will occur when all the single measurements of the sensor are replaced by the orbit determined on their basis. The single measurement here means the radar parameters (range, azimuth, elevation, and in some cases range rate) measured by a single pulse.

Khutorovsky, Z.↗
DOE OSTI2017

Canopus: A Paradigm Shift Towards Elastic Extreme-Scale Data Analytics on HPC

Scientific simulations on high performance computing (HPC) platforms generate large quantities of data. To bridge the widening gap between compute and I/O, and enable data to be more efficiently stored and analyzed, simulation outputs need to be refactored, reduced, and appropriately mapped to storage tiers. However, a systematic solution to support these steps has been lacking on the current HPC software ecosystem. To that end, this paper develops Canopus, a progressive JPEGlike data management scheme for storing and analyzing big scientific data. It co-designs the data decimation, compression and data storage, taking the hardware characteristics of each storage tier into considerations. With reasonably low overhead, our approach refactors simulation data into a much smaller, reduced-accuracy base dataset, and a series of deltas that is used to augment the accuracy if needed. The base dataset and deltas are compressed and written to multiple storage tiers. Data saved on different tiers can then be selectively retrieved to restore the level of accuracy that satisfies data analytics. Thus, Canopus provides a paradigm shift towards elastic data analytics and enables end users to make trade-offs between analysis speed and accuracy on-the-fly. We evaluate the impact of Canopus on unstructured triangular meshes, a pervasive data model used by scientific modeling and simulations. In particular, we demonstrate the progressive data exploration of Canopus using the “blob detection” use case on the fusion simulation data.

Lu, Tao↗
DOE OSTI2021

HYPPO: A Surrogate-Based Multi-Level Parallelism Tool for Hyperparameter Optimization

We present a new software, HYPPO, that enables the automatic tuning of hyperparameters of various deep learning (DL) models. Unlike other hyperparameter optimization (HPO) methods, HYPPO uses adaptive surrogate models and directly accounts for uncertainty in model predictions to find accurate and reliable models that make robust predictions. Using asynchronous nested parallelism, we are able to significantly alleviate the computational burden of training complex architectures and quantifying the uncertainty. HYPPO is implemented in Python and can be used with both TensorFlow and PyTorch libraries. We demonstrate various software features on time-series prediction and image classification problems as well as a scientific application in computed tomography image reconstruction. Finally, we show that (1) we can reduce by an order of magnitude the number of evaluations necessary to find the most optimal region in the hyperparameter space and (2) we can reduce by two orders of magnitude the throughput for such HPO process to complete.

adaptation models↗
DOE OSTI2022

pnnl/hydraconda

A framework for developing scientific computing codes for projects was developed that accommodates both explorations as well as, simultaneously, software engineering rigor. This is accomplished by organizing modular yet composable and related computational environments. Also known as Management of Multiple Computational Environments

alDosari, Majid↗
DOE OSTI2025

AuroraGPT Data Collection Interface

SF-25-043 This software package facilitates data collection for the purpose of assessing the performance of LLMs on scientific topics

Underwood, Robert [Argonne National Laboratory (AN↗
DOE OSTI2015

Integrated IO Services for Trilinos Data Structures

The Trilinos Project is an effort to develop algorithms and enabling technologies within an object- oriented software framework for the solution of large-scale, complex multi-physics engineering and scientific problems. A unique design feature of Trilinos is its focus on packages. While the abstractions make it easy to incorporate advanced processing and data manipulation tools, it is not always obvious how to take advantage of these features. The trios package incorporated two years ago offers general data management services, but has yet to offer integrated support for core Trilinos data structures, such as those offered in the Tpetra package. An initial attempt to incorporate native Trilinos data structure support into trios services revealed the complexity, from a non-mathematician's perspective, of using Trilinos. This project sought to understand the complexities and potential barriers not just for non-mathematicians that want to contribute to or use Trilinos, but potentially for new mathematically-inclined users as well that may want to offer services to support users. This report documents the challenges for trios to offer some simple data manipulation required as a precursor to any direct data services integration and makes recommendations for clarifying the performance implications and general approach to use.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

ComPort: Rigorous Testing Methods to Safeguard Software Porting (Final UW Report)

This report summarizes the University of Washington’s contributions to the ComPort project, Rigorous Testing Methods to Safeguard Software Porting, funded under the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research (ASCR), award number DE-SC0022081. UW’s work addressed numerics-driven correctness challenges arising in heterogeneous and rapidly evolving computing environments. Our efforts focused on developing interactive analysis tools, correctness-preserving rewriting systems, semantic-aware code-transformation infrastructure, and multi-language support for reproducible numerical experiments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

LDM-151: Data Management Science Pipelines Design

The LSST Science Requirements Document (the LSST SRD) specifies a set of data product guidelines, designed to support science goals envisioned to be enabled by the LSST observing program. Following these guidelines, the details of these data products have been described in the LSST Data Products Definition Document (DPDD), and captured in a formal flow-down from the SRD via the LSST System Requirements (LSR), Observatory System Specifications (OSS), to the Data Management System Requirements (DMSR). The LSST Data Management subsystem's responsibilities include the design, implementation, deployment and execution of software pipelines necessary to generate these data products. This document describes the design of the scientific aspects of those pipelines.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS1975

Future payload technology requirements study

Technology advances needed for an overall mission model standpoint as well as those for individual shuttle payloads are defined. The technology advances relate to the mission scientific equipment, spacecraft subsystems that functionally support this equipment, and other payload-related equipment, software, and environment necessary to meet broad program objectives. In the interest of obtaining commonality of requirements, the study was structured according to technology categories rather than in terms of individual payloads.

Source record↗
NASA NTRS1981

Mapping the infrared background radiation from the Shuttle

The Spacelab-2 Small Helium-Cooled Infrared Telescope will be used to map extended astronomical sources of low surface brightness emission, to measure the Shuttle induced environment and to develop techniques for managing large volumes of superfluid helium in space. The instrument is an f/4 15.2-cm Herschelian telescope with ten photoconductor detectors in the focal plane. This paper describes the hardware and software aspects of the instrument with emphasis on mission operations. In particular, a description is given of the observing plan formulated to meet the scientific and engineering objectives, the scan drive system, the precautions in design and operation necessary to prevent the sun, moon, and earth from adversely affecting the observations, the implications of thruster firings, and the on-board experiment computer application software to control the scanning of the telescope and support on-board displays.

Koch, D.↗
NASA NTRS1993

In search of meta-knowledge

Development of an Intelligent Information System (IIS) involves application of numerous artificial intelligence (AI) paradigms and advanced technologies. The National Aeronautics and Space Administration (NASA) is interested in an IIS that can automatically collect, classify, store and retrieve data, as well as develop, manipulate and restructure knowledge regarding the data and its application (Campbell et al., 1987, p.3). This interest stems in part from a NASA initiative in support of the interagency Global Change Research program. NASA's space data problems are so large and varied that scientific researchers will find it almost impossible to access the most suitable information from a software system if meta-information (metadata and meta-knowledge) is not embedded in that system. Even if more, faster, larger hardware is used, new innovative software systems will be required to organize, link, maintain, and properly archive the Earth Observing System (EOS) data that is to be stored and distributed by the EOS Data and Information System (EOSDIS) (Dozier, 1990). Although efforts are being made to specify the metadata that will be used in EOSDIS, meta-knowledge specification issues are not clear. With the expectation that EOSDIS might evolve into an IIS, this paper presents certain ideas on the concept of meta-knowledge and demonstrates how meta-knowledge might be represented in a pixel classification problem.

Lopez, Antonio M., Jr.↗
NASA NTRS1995

Data management and scientific integration within the Atmospheric Radiation Measurement Program

The Atmospheric Radiation Measurement (ARM) Program has been developed by the U.S. Department of Energy with the goal to improve the predictive capabilities of General Circulation Models (GCM's) in their treatment of clouds and radiative transfer effects. To achieve this goal, three experimental testbeds were designed for the deployment of instruments that will collect atmospheric data used to drive the GCM's. Each site, known as a Cloud and Radiation Testbed (CART), consists of a highly available, redundant data system for the collection of data from a variety of instrumentation. The first CART site was deployed in April 1992 in the Southern Great Plains (SGP), Lamont, Oklahoma, with the other two sites to follow in September 1995 in the Tropical Western Pacific and in 1997 on the North Slope of Alaska. Approximately 400 MB of data are transferred per day via the Internet from the SGP site to the ARM Experiment Center at Pacific Northwest Laboratory in Richland, Washington. The Experiment Center is central to the ARM data path and provides for the collection, processing, analysis, and delivery of ARM data. Data are received from the CART sites from a variety of instrumentation, observational systems, amd external data sources. The Experiment Center processes these data streams on a continuous basis to provide derived data products to the ARM Science Team in near real-time while providing a three-month running archive of data. A primary requirement of the ARM Program is to preserve and protect all data produced or acquired. This function is performed at Oak Ridge National Laboratory where leading edge technology is employed for the long-term storage of ARM data. The ARM Archive provides access to data for participation outside of the ARM Program. The ARM Program involves a collaborative effort by teams from various DOE National Laboratories, providing multi-disciplinary areas of expertise. This paper will discuss the collaborative methods in which the ARM teams translate the scientific goals of the Program into data products. By combining atmospheric scientists, systems engineers, and software engineers, the ARM Program has successfully designed and developed an environment where advances in understanding the parameterizations of GCM's can be made.

Gracio, Deborah K.↗