Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “community clustering”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

A step towards the final frontier: Lessons learned from acceptance testing of the first HPE/Cray EX 3000 system at ORNL

Summary In this article, we summarize the deployment of the Air Force Weather (AFW) HPC11 system at Oak Ridge National Laboratory (ORNL) including the process followed to successfully complete acceptance testing of the system. HPC11 is the first HPE/Cray EX 3000 system that has been successfully released to its user community in a federal facility. HPC11 consists of two identical 800‐node supercomputers, Fawbush and Miller, with access to two independent and identical lustre parallel file systems. HPC11 is equipped with Slingshot 10 interconnect technology and relies on the HPE Performance Cluster Manager software for system configuration. ORNL has a clearly defined acceptance testing process used to ensure that every new system deployed can provide the necessary capabilities to support user workloads. We worked closely with HPE and AFW to develop a set of tests that used the United Kingdom's Meteorological Office's Unified Model and 4‐dimensional variational data assimilation. We also included benchmarks and applications from the Oak Ridge Leadership Computing Facility portfolio to fully exercise the HPE/Cray programming environment and evaluate the functionality and performance of the system. Acceptance testing of HPC11 required parallel execution of each element on Fawbush and Miller. In addition, careful coordination was needed to ensure successful acceptance of the newly deployed lustre file systems alongside the compute resources. In this work, we present test results from specific system components and provide an overview of the issues identified, challenges encountered, and the lessons learned along the way.

Melesse Vergara, Verónica G.↗
NASA NTRS2022

pyCRTM: A Python Interface for the Community Radiative Transfer Model

The Community Radiative Transfer Model (CRTM) is a powerful and versatile scalar radiative transfer model for satellite data assimilation and remote sensing applications. It is implemented as an object-oriented Fortran library, enabling flexible code development and optimal runtime performance on clusters. The downsides of the Fortran interface are a steep learning curve for students and the reduced productivity of users that is typical for static compiled languages, in contrast to dynamic interpreted languages like Python. pyCRTM is a new software framework that directly interfaces the CRTM Fortran data structures and procedures in Python, leveraging both the simplicity and ease of use of Python syntax as well as the flexibility arising from the vast contemporary Python ecosystem. The goal of pyCRTM is to lower the barrier of entry for university students to learn and use the CRTM and to boost the productivity of researchers seeking to create new methods in radiative transfer and data assimilation, or seeking to apply the CRTM to study atmospheric phenomena without having to go through the pre-existing complexity of the CRTM Fortran interface.

Python↗
DOE OSTI2021

A Step Towards the Final Frontier: Lessons Learned from Acceptance Testing of the First HPE/Cray EX 3000 System at ORNL

In this paper, we summarize the deployment of the Air Force Weather (AFW) HPC11 system at Oak Ridge National Laboratory (ORNL) including the process followed to successfully complete acceptance testing of the system. HPC11 is the first HPE/Cray EX 3000 system that has been successfully released to its user community in a federal facility. HPC11 consists of two identical 800-node supercomputers, Fawbush and Miller, with access to two independent and identical Lustre parallel file systems. HPC11 is equipped with Slingshot 10 interconnect technology and relies on the HPE Performance Cluster Manager (HPCM) software for system configuration. ORNL has a clearly defined acceptance testing process used to ensure that every new system deployed can provide the necessary capabilities to support user workloads. We worked closely with HPE and AFW to develop a set of tests that used the United Kingdom’s Meteorological Office’s Unified Model (UM) and 4DVAR. We also included benchmarks and applications from the Oak Ridge Leadership Computing Facility (OLCF) portfolio to fully exercise the HPE/Cray programming environment and evaluate the functionality and performance of the system. Acceptance testing of HPC11 required parallel execution of each element on Fawbush and Miller. In addition, careful coordination was needed to ensure successful acceptance of the newly deployed Lustre file systems alongside the compute resources. In this work, we present test results from specific system components and provide an overview of the issues identified, challenges encountered, and the lessons learned along the way.

Melesse Vergara, Veronica↗
NASA NTRS1976

Study of systems and techniques for data base management

Data management areas were studied to identify pertinent problems and issues that will affect future NASA data users in terms of performance and cost. Specific topics discussed include the identifications of potential NASA data users other than those normally discussed, consideration affecting the clustering of minicomputers, low cost computer system for information retrieval and analysis, the testing of minicomputer based data base management systems, ongoing work related to the use of dedicated systems for data base management, and the problems of data interchange among a community of NASA data users.

Source record↗
DOE OSTI2021

Workflows Community Summit: Tightening the Integration between Computing Facilities and Scientific Workflows

Scientific workflows are used almost universally across science domains for solving complex and largescale computing and data analysis problems. The importance of workflows is highlighted by the fact that they have underpinned some of the most significant discoveries of the past decades. Many of these workflows have significant computational, storage, and communication demands, and thus must execute on a range of large-scale computer systems, from local clusters to public clouds and upcoming exascale HPC platforms. Managing these executions is often a significant undertaking, requiring a sophisticated and versatile software infrastructure. Historically, infrastructures for workflow execution consisted of complex, integrated systems, developed in-house by workflow practitioners with strong dependencies on a range of legacy technologies—even including sets of ad hoc scripts. Due to the increasing need to support workflows, dedicated workflow systems were developed to provide abstractions for creating, executing, and adapting workflows conveniently and efficiently while ensuring portability. While these efforts are all worthwhile individually, there are now hundreds of independent workflow systems. These workflow systems are created and used by thousands of researchers and developers, leading to a rapidly growing corpus of workflows research publications. The resulting workflow system technology landscape is fragmented, which may present significant barriers for future workflow users due to many seemingly comparable, yet usually mutually incompatible, systems that exist. In order to tackle some of the challenges described above, the DOE-funded ExaWorks and NSF-funded WorkflowsRI projects have organized in 2021 a series of events entitled the “Workflows Community Summit”. The third edition of the “Workflows Community Summit” explored workflows challenges and opportunities from the perspective of computing centers and facilities. This third summit builds on two prior summits (https://workflowsri.org/summits) that (i) established a high level vision for workflows research; and (ii) explored technical approaches for realizing that vision. The third summit brought together a small group of facilities representatives with the aim to understand how workflows are currently being used at each facility, how facilities would like to interact with workflow developers and users, how workflows fit with facility roadmaps, and what opportunities there are for tighter integration between facilities and workflows. This report documents and organizes the wealth of information provided by the participants before, during, and after the summit.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2000

NASA EPSCoR Nebraska Preparation Grant: Year 1. Research Cluster: Small Aircraft Transportation System/Nebraska Implementation Template (SATS-NIT)

NASA, the U.S. Department of Transportation/Federal Aviation Administration, industry stakeholders, and academia have joined forces to pursue the NASA National General Aviation Roadmap leading to a Small Aircraft Transportation System (SATS). This strategic undertaking has a 25-year goal to improve air access and bring next-generation technologies to small communities. The envisioned outcome is to improve travel between remote communities and transportation centers in urban areas by utilizing the nation's 5,400 public use general aviation airports. To facilitate this initiative, SATS stakeholders must plan, coordinate, and implement a comprehensive upgrade of public infrastructure within the framework of the national air transportation system. Ultimately, SATS may permit tripling aviation system throughput capacity by tapping the under-utilized airspace and general aviation facilities. The SATS investments, which begin in FY 2001, are designed to support the national goal of doorstep-to-destination travel at four times the speed of highways for the nation's suburban, rural, and remote communities.

Bartle, John R.↗
DOE OSTI2020

SPISEA: A Python-based Simple Stellar Population Synthesis Code for Star Clusters

We present Stellar Population Interface for Stellar Evolution and Atmospheres (SPISEA), an open-source Python package that simulates simple stellar populations. The strength of SPISEA is its modular interface which offers the user control of 13 input properties including (but not limited to) the initial mass function, stellar multiplicity, extinction law, and the metallicity-dependent stellar evolution and atmosphere model grids used. The user also has control over the initial–final mass relation in order to produce compact stellar remnants (black holes, neutron stars, and white dwarfs). We demonstrate several outputs produced by the code, including color–magnitude diagrams, HR-diagrams, luminosity functions, and mass functions. SPISEA is object-oriented and extensible, and we welcome contributions from the community. The code and documentation are available on GitHub (https://github.com/astropy/SPISEA) and ReadtheDocs (https://spisea.readthedocs.io/en/latest/), respectively.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2003

A Proposal to Investigate Outstanding Problems in Astronomy

During the past year the ACS science team has concentrated on analyzing ACS observations, writing papers, and disseminating our results to the astronomy community at conferences and workshops around the world. We also have put considerable effort in getting our results to the public via public lectures and through press releases. Taking a very broad view of our program, we are investigating the evolution of galaxies and clusters of galaxies from their birth, approximately one billion years after the beginning of the Universe, to the present. We have found and characterized a population of galaxies that are no more than 1.4 billion years old. These may well be the Universe s first generation of infant galaxies. Looking at the Universe 500,000 years later, we see what appears to be a cluster of galaxies just beginning to form (a proto-cluster) around a luminous radio galaxy. Moving forward in time and closer to the present, we are studying clusters of galaxies that are less than half the age of the Universe. Our observations and analysis lead us to the important conclusion that the elliptical galaxies in these clusters must have had their last significant star formation some three billion years earlier, which is about the time when the proto-cluster was forming. Coming still closer to home, we are observing nearby massive clusters of galaxies that are approximately 12 billion years old. The gravity from these large aggregates of dark and luminous matter is so strong it warps space-time itself, and makes the cluster act as a cosmic telescope that magnifies the distant galaxies behind the cluster. We used the magnified (or lensed) galaxies to map the distribution of the dominant matter within the clusters, which is the so-called dark matter (the matter is invisible, and its nature is unknown). We also are using these cosmic telescopes to study the distant lensed galaxies that would otherwise be too small and too faint to be seen even by Hubble and the ACS.

Ford, Holland↗
DOE OSTI2025

Structured interactions drive abrupt transitions in the spatial organization of microbial communities

Bacteria possess diverse mechanisms to regulate their motility in response to environmental and physiological signals, enabling them to navigate complex habitats and adapt their behavior. Some of these mechanisms are species specific and enable cells to modulate their movement based on the ecological identity of neighboring species. Here, we introduce a model in which bacteria interact via local signals that either enhance or suppress the motility of neighboring cells depending on species type. Through large-scale simulations and a coarse-grained stochastic model, we demonstrate the emergence of a sharp transition driven by nucleation processes: increasing the density of motility-suppressing interactions drives the system from a fully mixed, motile phase to a state characterized by large, stationary bacterial clusters. Remarkably, in systems with a large number of interacting species, this transition can be triggered solely by altering the structure of the motility-regulation interaction matrix while maintaining species and interaction densities constant. In particular, we find that heterogeneous and modular interactions promote the transition more readily than homogeneous random ones. These findings add a dimension to the theory of motility-induced phase separation and contribute to the ongoing effort to understand microbial interactions, suggesting that structured, nonrandom ones may be key to reproducing commonly observed spatial patterns in microbial communities.

bacterial communities↗
DOE OSTI2022

‘Flux+Mutability’: a conditional generative approach to one-class classification and anomaly detection

Abstract Anomaly Detection is becoming increasingly popular within the experimental physics community. At experiments such as the Large Hadron Collider, anomaly detection is growing in interest for finding new physics beyond the Standard Model. This paper details the implementation of a novel Machine Learning architecture, called Flux+Mutability, which combines cutting-edge conditional generative models with clustering algorithms. In the ‘flux’ stage we learn the distribution of a reference class. The ‘mutability’ stage at inference addresses if data significantly deviates from the reference class. We demonstrate the validity of our approach and its connection to multiple problems spanning from one-class classification to anomaly detection. In particular, we apply our method to the isolation of neutral showers in an electromagnetic calorimeter and show its performance in detecting anomalous dijets events from standard QCD background. This approach limits assumptions on the reference sample and remains agnostic to the complementary class of objects of a given problem. We describe the possibility of dynamically generating a reference population and defining selection criteria via quantile cuts. Remarkably this flexible architecture can be deployed for a wide range of problems, and applications like multi-class classification or data quality control are left for further exploration.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2026

HPC ODA Commons [SWR-26-003]

HPC ODA Commons is a community-driven platform for standardizing HPC operational data analytics. HPC sites generate enormous volumes of operational data - scheduler logs, accounting records, monitoring streams - but turning that data into actionable insight is needlessly hard. Each site builds bespoke parsers, schemas, and evaluation pipelines. Results can't be compared across institutions. Promising analytics ideas stay siloed because there's no shared language for describing the data, the experiments, or the outcomes. HPC ODA Commons fixes this by establishing community-governed contracts - versioned schemas, canonical artifacts, and benchmark recipes - that make ODA workflows discoverable, reproducible, and comparable. It pairs these standards with a practical, CLI-first toolkit that lets operators and researchers go from raw logs to standardized results without sending data off-cluster.

Menear, Kevin [National Laboratory of the Rockies ↗
DOE OSTI2022

AIMSim : An accessible cheminformatics platform for similarity operations on chemicals datasets

The recent advances in deep learning, generative modeling, and statistical learning have ushered in a renewed interest in traditional cheminformatics tools and methods. Quantifying molecular similarity is essential in molecular generative modeling, exploratory molecular synthesis campaigns, and drug-discovery applications to assess how new molecules differ from existing ones. Further, most tools target advanced users and lack general implementations accessible to the larger community. In this work, we introduce Artificial Intelligence Molecular Similarity (AIMSim), an accessible cheminformatics platform for performing similarity operations on collections of molecules called molecular datasets. AIMSim provides a unified platform to perform similarity-based tasks on molecular datasets, such as diversity quantification, outlier and novelty analysis, clustering, dimensionality reduction, and inter-molecular comparisons. AIMSim implements all major binary similarity metrics and molecular fingerprints and is provided as a Python package that includes support for command-line use as well as a Graphical User Interface for code-free utilization with fully interactive plots.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

A Cast of Thousands: How the IDEAS Productivity Project Has Advanced Software Productivity and Sustainability

Computational and data-enabled science and engineering are revolutionizing advances throughout science and society, at all scales of computing. For example, teams in the U.S. Department of Energy’s Exascale Computing Project have been tackling new frontiers in modeling, simulation, and analysis by exploiting unprecedented exascale computing capabilities—building an advanced software ecosystem that supports next-generation applications and addresses disruptive changes in computer architectures. However, concerns are growing about the productivity of the developers of scientific software. Members of the Interoperable Design of Extreme-scale Application Software project serve as catalysts to address these challenges through fostering software communities, incubating and curating methodologies and resources, and disseminating knowledge to advance developer productivity and software sustainability. This article discusses how these synergistic activities are advancing scientific discovery—mitigating technical risks by building a firmer foundation for reproducible, sustainable science at all scales of computing, from laptops to clusters to exascale and beyond.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2001

NATO Advanced Study Institute on Spectroscopy

This booklet presents an account of the course 'Spectroscopy of Systems with Spatially Confined Structures' held in Erice-Sicily, Italy, from June 15 to June 30, 2001. This meeting was organized by the International School of Atomic and Molecular Spectroscopy of the 'Ettore Majorana' Centre for Scientific Culture. The purpose of this course was to present and discuss nanometer-scale physics, a rapidly progressing field. The top-down approach of semiconductor technology will soon meet the scales of the bottom-up approaches of supramolecular chemistry and of spatially localized excitations in ionic crystals. This course dealt with the fabrication, measurement and understanding of the relevant structures and brought together the scientific communities responsible for these development. The advances in this area of physics have already let to applications in optoelectronics and will likely lead to many more. The subjects of the course included spatially resolved structures such as quantum wells, quantum wires and quantum dots, single atoms and molecules, clusters, fractal systems, and the development of related techniques like near-field spectroscopy and confocal microscopy to study such systems.

DiBartolo, Baldassare↗
DOE OSTI2021

TempestExtremes v2.1: a community framework for feature detection, tracking, and analysis in large datasets

TempestExtremes (TE) is a multifaceted framework for feature detection, tracking, and scientific analysis of regional or global Earth system datasets on either rectilinear or unstructured/native grids. Version 2.1 of the TE framework now provides extensive support for examining both nodal (i.e., pointwise) and areal features, including tropical and extratropical cyclones, monsoonal lows and depressions, atmospheric rivers, atmospheric blocking, precipitation clusters, and heat waves. Available operations include nodal and areal thresholding, calculations of quantities related to nodal features such as accumulated cyclone energy and azimuthal wind profiles, filtering data based on the characteristics of nodal features, and stereographic compositing. This paper describes the core algorithms (kernels) that have been added to the TE framework since version 1.0, including algorithms for editing pointwise trajectory files, composition of fields around nodal features, generation of areal masks via thresholding and nodal features, and tracking of areal features in time. Several examples are provided of how these kernels can be combined to produce composite algorithms for evaluating and understanding common atmospheric features and their underlying processes. These examples include analyzing the fraction of precipitation from tropical cyclones, compositing meteorological fields around extratropical cyclones, calculating fractional contribution to poleward vapor transport from atmospheric rivers, and building a climatology of atmospheric blocks.

58 GEOSCIENCES↗
NASA NTRS1983

An enhanced version of an implicit code for the Euler equations

A two-dimensional implicit finite-difference code is applied to the inviscid Euler equations to compute transonic flow past airfoils in order to provide well-documented standard test cases for the general user community. The code is an improved version of Steger's 1976 implicit code. Enhancements include the use of up-wind differencing in supersonic regions before shocks and a variable time step to accelerate convergence. An airfoil grid generation routine based on algebraic techniques is employed. The grids are clustered near shocks to improve resolution. Computed results are compared with other numerical results from the literature.

Pulliam, T. H.↗
DOE OSTI2026

From microbial diversity to functional potential using dimensionality reduction

The high dimensionality of microbial diversity data from ‘omics observations can be reduced using Machine Learning, with many recent studies showcasing ML utility for exploratory ecological feature finding and process prediction. Here, we compare the Self Organizing Map (SOM) dimensionality reduction method to the well-documented sample-based Principal Coordinate Analysis (PCoA) and taxa-based Weighted Gene Correlation Network Analysis (WGCNA) using near daily 16S rRNA gene amplicon sequencing data from the 2019 to 2020 MOSAiC International Arctic Drift Expedition. We then map k-means clustering outputs from each method to available metagenomes, extracting functionally distinct seasonal microbial ecotypes in the surface Arctic Ocean. Our results indicate the SOM method better represented expected seasonal transitions and identified a greater number of metabolically distinct functional groups than the more traditional PCoA ordination. Ultimately, we identified four community ecotypes with distinct taxonomic and functional cut-offs driven by seasonality, water mass, and substrate turnover, highlighting the importance of succession in functional diversity for the central Arctic Ocean. These results reinforce ML dimensionality reduction as a meaningful translator in the mining of historical amplicon datasets to address modern mechanistic questions and potentially provide ’omics informed ecotype diversity to leverage in mechanistic biogeochemical models.

Arctic Ocean↗
DOE OSTI2024

The Data Mine model for accessible partnerships in data science

Abstract The Data Mine at Purdue University is a pioneering experiential learning community for undergraduate and graduate students of any background to learn data science. The first data‐intensive experience embedded in a large learning community, The Data Mine had nearly 1300 students in academic year (AY) 2022–2023 and nearly 1700 students for AY 2023–2024. The Data Mine embodies data‐infused education, research, and collaboration. Students learn Python, R, SQL, and shell‐scripting, while working on weekly projects within a high‐performance computing (HPC) cluster. In the Corporate Partners cohort, students work on teams of 5–15 students, led by a paid student team leader. Each cohort follows an Agile approach, working on data‐intensive projects provided by industry partners and mentored by company employees. Students develop professional and data skills throughout the academic year, from August through April. Many students return in subsequent years to the program, increasing their tenure with a Corporate Partner. Student teams are inherently interdisciplinary; students from 133 different majors are involved in the program, ranging from new incoming students through PhD level students. These interdisciplinary teams of students bring new perspectives to challenging problems in which data science is a key part of the solution. The interdisciplinary teams foster an environment of synthesis with ideas and solutions. Students come together with different life experiences, different levels of technical skill, but also varying ways they navigate paths to solutions because of the variety of majors represented, resulting in a more creative and robust solution than a traditional data science program. This article is categorized under: Applications of Computational Statistics > Education in Computational Statistics

Betz, Margaret A.↗