Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Cloud Computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29
All sourcesNASA NTRSDOE OSTI
NASA NTRS2004

Combined Infrared Stereo and Laser Ranging Cloud Measurements from Shuttle Mission STS-85

Multiangle remote sensing provides a wealth of information for earth and climate monitoring, such as the ability to measure the height of cloud tops through stereoscopic imaging. As technology advances so do the options for developing spacecraft instrumentation versatile enough to meet the demands associated with multiangle measurements. One such instrument is the infrared spectral imaging radiometer, which flew as part of mission STS-85 of the space shuttle in 1997 and was the first earth- observing radiometer to incorporate an uncooled microbolometer array detector as its image sensor. Specifically, a method for computing cloud-top height with a precision of +/- 620 m from the multispectral stereo measurements acquired during this flight has been developed, and the results are compared with coincident direct laser ranging measurements from the shuttle laser altimeter. Mission STS-85 was the first space flight to combine laser ranging and thermal IR camera systems for cloud remote sensing.

Lancaster, R. S.↗
NASA NTRS2009

Controlling Infrastructure Costs: Right-Sizing the Mission Control Facility

Johnson Space Center's Mission Control Center is a space vehicle, space program agnostic facility. The current operational design is essentially identical to the original facility architecture that was developed and deployed in the mid-90's. In an effort to streamline the support costs of the mission critical facility, the Mission Operations Division (MOD) of Johnson Space Center (JSC) has sponsored an exploratory project to evaluate and inject current state-of-the-practice Information Technology (IT) tools, processes and technology into legacy operations. The general push in the IT industry has been trending towards a data-centric computer infrastructure for the past several years. Organizations facing challenges with facility operations costs are turning to creative solutions combining hardware consolidation, virtualization and remote access to meet and exceed performance, security, and availability requirements. The Operations Technology Facility (OTF) organization at the Johnson Space Center has been chartered to build and evaluate a parallel Mission Control infrastructure, replacing the existing, thick-client distributed computing model and network architecture with a data center model utilizing virtualization to provide the MCC Infrastructure as a Service. The OTF will design a replacement architecture for the Mission Control Facility, leveraging hardware consolidation through the use of blade servers, increasing utilization rates for compute platforms through virtualization while expanding connectivity options through the deployment of secure remote access. The architecture demonstrates the maturity of the technologies generally available in industry today and the ability to successfully abstract the tightly coupled relationship between thick-client software and legacy hardware into a hardware agnostic "Infrastructure as a Service" capability that can scale to meet future requirements of new space programs and spacecraft. This paper discusses the benefits and difficulties that a migration to cloud-based computing philosophies has uncovered when compared to the legacy Mission Control Center architecture. The team consists of system and software engineers with extensive experience with the MCC infrastructure and software currently used to support the International Space Station (ISS) and Space Shuttle program (SSP).

Martin, Keith↗
DOE OSTI2024

Utilizing Distributed Heterogeneous Computing with PanDA in ATLAS

In recent years, advanced and complex analysis workflows have gained increasing importance in the ATLAS experiment at CERN, one of the large scientific experiments at LHC. Support for such workflows has allowed users to exploit remote computing resources and service providers distributed worldwide, overcoming limitations on local resources and services. The spectrum of computing options keeps increasing across the Worldwide LHC Computing Grid (WLCG), volunteer computing, high-performance computing, commercial clouds, and emerging service levels like Platform-as-a-Service (PaaS), Container-as-a-Service (CaaS) and Function-as-a-Service (FaaS), each one providing new advantages and constraints. Users can significantly benefit from these providers, but at the same time, it is cumbersome to deal with multiple providers, even in a single analysis workflow with fine-grained requirements coming from their applications’ nature and characteristics. In this paper, we will first highlight issues in geographically-distributed heterogeneous computing, such as the insulation of users from the complexities of dealing with remote providers, smart workload routing, complex resource provisioning, seamless execution of advanced workflows, workflow description, pseudointeractive analysis, and integration of PaaS, CaaS, and FaaS providers. We will also outline solutions developed in ATLAS with the Production and Distributed Analysis (PanDA) system and future challenges for LHC Run4.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2020

Radiative Heating Rate Profiles over the Southeast Atlantic Ocean during the 2016 and 2017 Biomass Burning Seasons

Marine boundary layer clouds, including the transition from stratocumulus to cumulus, are poorly represented in numerical weather prediction and general circulation models. Further uncertainties in the cloud structure arise in the presence of biomass burning carbonaceous aerosol, as is the case over the southeast Atlantic Ocean where biomass burning aerosol is transported from the African continent. As the aerosol plume progresses across the southeast Atlantic Ocean, radiative heating within the aerosol layer has the potential to alter the thermodynamic environment and therefore the cloud structure; however, limited work has been done to quantify this along the trajectory of the aerosol plume in the region. The deployment of the First Atmospheric Radiation Measurement Mobile Facility in support of the Layered Atlantic Smoke Interactions with Clouds field campaign provided a unique opportunity to collect observations of cloud and aerosol properties during two consecutive biomass burning seasons during July through October of 2016 and 2017 over Ascension Island (7.96 °S, 14.35 °W). Using observed profiles of temperature, humidity, and clouds from the field campaign, alongside aerosol optical properties from MERRA-2 as input for the Rapid Radiation Transfer Model, profiles of the radiative heating rate due to aerosols and clouds were computed. Radiative heating is also assessed across the southeast Atlantic Ocean using an ensemble of back trajectories from the Hybrid Single Particle Lagrangian Integrated Trajectory Model. Idealized experiments using the Rapid Radiation Transfer Model with and without aerosols and a range of values for the single scattering albedo demonstrate that shortwave heating within the aerosol layer above Ascension Island can locally range between 2 and 8 K per day depending on the aerosol optical properties, though impacts of the aerosol can be felt elsewhere in the atmospheric column. When considered under clear conditions, the aerosol has a cooling effect at the TOA and based on the observed cloud properties at Ascension Island, the cloud albedo is not large enough to overcome this. Shortwave radiative heating due to biomass burning aerosol is not balanced by additional longwave cooling, and the net radiative impact results in a stabilization of the lower troposphere. However, these results are extremely sensitive to the single scattering albedo assumptions in models.

Allison B. Marquardt Collow↗
DOE OSTI2020

Radiative heating rate profiles over the southeast Atlantic Ocean during the 2016 and 2017 biomass burning seasons

Marine boundary layer clouds, including the transition from stratocumulus to cumulus, are poorly represented in numerical weather prediction and general circulation models. Further uncertainties in the cloud structure arise in the presence of biomass burning carbonaceous aerosol, as is the case over the southeast Atlantic Ocean, where biomass burning aerosol is transported from the African continent. As the aerosol plume progresses across the southeast Atlantic Ocean, radiative heating within the aerosol layer has the potential to alter the thermodynamic environment and therefore the cloud structure; however, limited work has been done to quantify this along the trajectory of the aerosol plume in the region. The deployment of the first Atmospheric Radiation Measurement (ARM) Mobile Facility (AMF1) in support of the Layered Atlantic Smoke Interactions with Clouds field campaign provided a unique opportunity to collect observations of cloud and aerosol properties during two consecutive biomass burning seasons during July through October of 2016 and 2017 over Ascension Island (7.96° S, 14.35° W). Using observed profiles of temperature, humidity, and clouds from the field campaign alongside aerosol optical properties from Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2), as input for the Rapid Radiation Transfer Model (RRTM), profiles of the radiative heating rate due to aerosols and clouds were computed. Radiative heating is also assessed across the southeast Atlantic Ocean using an ensemble of back trajectories from the Hybrid Single Particle Lagrangian Integrated Trajectory (HYSPLIT) model. Idealized experiments using the RRTM with and without aerosols and a range of values for the single-scattering albedo (SSA) demonstrate that shortwave (SW) heating within the aerosol layer above Ascension Island can locally range between 2 and 8 K d -1 depending on the aerosol optical properties, though impacts of the aerosol can be felt elsewhere in the atmospheric column. When considered under clear conditions, the aerosol has a cooling effect at the TOA, and based on the observed cloud properties at Ascension Island, the cloud albedo is not large enough to overcome this. Shortwave radiative heating due to biomass burning aerosol is not balanced by additional longwave (LW) cooling, and the net radiative impact results in a stabilization of the lower troposphere. However, these results are extremely sensitive to the single-scattering albedo assumptions in models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

3D cloud detection and tracking system for solar forecast using multiple sky imagers

We propose a system for forecasting short-term solar irradiance based on multiple total sky imagers (TSIs). The system utilizes a novel method of identifying and tracking clouds in three-dimensional space and an innovative pipeline for forecasting surface solar irradiance based on the image features of clouds. First, we develop a supervised classifier to detect clouds at the pixel level and output cloud mask. In the next step, we design intelligent algorithms to estimate the block-wise base height and motion of each cloud layer based on images from multiple TSIs. Thus, this information is then applied to stitch images together into larger views, which are then used for solar forecasting. We examine the system’s ability to track clouds under various cloud conditions and investigate different irradiance forecast models at various sites. We confirm that this system can 1) robustly detect clouds and track layers, and 2) extract the significant global and local features for obtaining stable irradiance forecasts with short forecast horizons from the obtained images. Finally, we vet our forecasting system at the 32-megawatt Long Island Solar Farm (LISF). Compared with the persistent model, our system achieves at least a 26% improvement for all irradiance forecasts between one and fifteen minutes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Collaborative Project: High-resolution Global Modeling of the Effects of Subgrid-Scale Clouds and Turbulence on Precipitating Cloud Systems

We proposed to implement, test, and evaluate recently developed turbulence parameterizations, using a wide variety of methods and modeling frameworks together with observations including ARM data. We have successfully tested three different turbulence parameterizations in versions of the Community Atmosphere Model: CLUBB, SHOC, and IPHOC. All three produce significant improvements in the simulated climate. CLUBB will be used in CAM6, and also in ACME. SHOC is being tested in the NCEP forecast model. In addition, we have achieved a better understanding of the strengths and limitations of the PDF-based parameterizations of turbulence and convection.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2016

Co-Operative Resource Allocation: Building an Open Cloud Market Using Shared Infrastructure

In this paper we present DRIVE, a distributed service-based system designed to facilitate an open economic market for federating Cloud providers. Additionally, to address the challenges associated with market ownership and operation we propose the use of a cooperative (co-op) infrastructure in which the services that make up DRIVE are hosted across participants' resources. To prevent malicious behavior we use cryptographic, secure and privacy preserving allocation protocols as a means of establishing trust in the allocation infrastructure. Finally, we investigate through simulation the effect of different strategies, pricing functions, and penalty models on allocation performance and revenue, and show that the overhead of running DRIVE's services on commodity infrastructure is modest.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Interpreting and Stabilizing Machine-Learning Parametrizations of Convection

Neural networks are a promising technique for parameterizing subgrid-scale physics (e.g., moist atmospheric convection) in coarse-resolution climate models, but their lack of interpretability and reliability prevents widespread adoption. For instance, it is not fully understood why neural network parameterizations often cause dramatic instability when coupled to atmospheric fluid dynamics. This paper introduces tools for interpreting their behavior that are customized to the parameterization task. First, we assess the nonlinear sensitivity of a neural network to lower-tropospheric stability and the midtropospheric moisture, two widely studied controls of moist convection. Second, we couple the linearized response functions of these neural networks to simplified gravity wave dynamics, and analytically diagnose the corresponding phase speeds, growth rates, wavelengths, and spatial structures. To demonstrate their versatility, these techniques are tested on two sets of neural networks, one trained with a super-parameterized version of the Community Atmosphere Model (SPCAM) and the second with a near-global cloud-resolving model (GCRM). Additionally, even though the SPCAM simulation has a warmer climate than the cloud-resolving model, both neural networks predict stronger heating/drying in moist and unstable environments, which is consistent with observations. Moreover, the spectral analysis can predict that instability occurs when GCMs are coupled to networks that support gravity waves that are unstable and have phase speeds larger than 5 m s -1 . In contrast, standing unstable modes do not cause catastrophic instability. Using these tools, differences between the SPCAM-trained versus GCRM-trained neural networks are analyzed, and strategies to incrementally improve both of their coupled online performance unveiled.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

INTEGRATED WORKFLOW MANAGEMENT FOR PARTICLE ACCELERATOR SIMULATION

Supercomputing systems are used for a wide range of computationally demanding tasks in many fields of science and engineering. They play a key role in numerical simulation, in which mathematical models are computed in order to simulate the behavior of physical systems. Scientists and engineers that use supercomputers for numerical simulation often have their productivity limited by the need to manually organize and manage extremely large amounts of data that are often produced and consumed by the software programs run on these systems. Recognizing these limitations, Kitware Inc. (Clifton Park, NY) and SLAC National Accelerator Laboratory (Menlo Park, CA) are developing an advanced software platform that can reduce the cognitive overhead required by knowledge workers when using supercomputers for numerical simulation. Phase I of the project is complete and includes the development of new capabilities for organizing simulation project files, improvements to the user interface and overall usability, and deployment of a “middle tier” server to sit between user desktop machines and supercomputers to offload much of the data management workload. The project also developed prototype software for executing sequences of numerical simulations, and a prototype for migrating supercomputing software to cloud-based computing systems to provide a potential alternative to supercomputers with different logistical and price-to-performance tradeoffs.

Tourtellott, John↗
DOE OSTI2022

Elastic Resource Management for Deep Learning Applications in a Container Cluster

The increasing demand for learning from massive datasets is restructuring our economy. Effective learning, however, involves nontrivial computing resources. Most businesses utilize commercial infrastructure providers (e.g., AWS) to host their computing clusters in the cloud, where various jobs compete for available resources. While cloud resource management is a fruitful research field that has made many advances in production, such as Kubernetes and YARN, few efforts have been invested to further optimize the system performance, especially for deep learning (DL) training jobs in a container cluster. This work introduces FlowCon, a system that is able to monitor the individual evaluation functions of DL jobs at runtime, and thus to make placement decisions on resource allocations elastically. Here, we present a detailed design and implementation of FlowCon and conduct intensive experiments over various DL models. The results demonstrate that FlowCon significantly improves DL job completion time and resource utilization efficiency, compared to default systems. According to the results, FlowCon is able to improve the completion time by up to 68.8% and meanwhile, reduce the makespan by 18.0%, in the presence of various DL job workloads.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗
NASA NTRS2019

Opportunities for Accelerating Research in the Cloud

As the data holdings of the Earth Observation System Data and Information System expand over the next several years, the typical data analysis process of downloading data to local compute resources will become increasingly inefficient. However, cloud computing promises to mitigate that by allowing the user to process close to the data. These improvements will be obtained via a variety of mechanisms: 1 - improving the ability of data transformation services to reduce the data prior to analysis; 2 - providing cloud-native analysis capabilities for common analysis functions; and 3 - providing the ability to work directly with data in Web Object Storage.

data system↗
DOE OSTI2023

Accelerating Machine Learning Inference with GPUs in ProtoDUNE Data Processing

Abstract We study the performance of a cloud-based GPU-accelerated inference server to speed up event reconstruction in neutrino data batch jobs. Using detector data from the ProtoDUNE experiment and employing the standard DUNE grid job submission tools, we attempt to reprocess the data by running several thousand concurrent grid jobs, a rate we expect to be typical of current and future neutrino physics experiments. We process most of the dataset with the GPU version of our processing algorithm and the remainder with the CPU version for timing comparisons. We find that a 100-GPU cloud-based server is able to easily meet the processing demand, and that using the GPU version of the event processing algorithm is two times faster than processing these data with the CPU version when comparing to the newest CPUs in our sample. The amount of data transferred to the inference server during the GPU runs can overwhelm even the highest-bandwidth network switches, however, unless care is taken to observe network facility limits or otherwise distribute the jobs to multiple sites. We discuss the lessons learned from this processing campaign and several avenues for future improvements.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2013

The W40 region in the gould belt: An embedded cluster and H II region at the junction of filaments

We present a multiwavelength study of the W40 star-forming region using infrared (IR) observations in the UKIRT JHK bands, Spitzer Infrared Array Camera bands, and Herschel PACS bands, 2.12 μm H{sub 2} narrowband imaging, and radio continuum observations from GMRT (610 and 1280 MHz), in a field of view (FoV) of ∼34' × 40'. Archival Spitzer observations in conjunction with near-IR observations are used to identify 1162 Class II/III and 40 Class I sources in the FoV. The nearest-neighbor stellar surface density analysis shows that the majority of these young stellar objects (YSOs) constitute the embedded cluster centered on the high-mass source IRS 1A South. Some YSOs, predominantly the younger population, are distributed along and trace the filamentary structures at lower stellar surface density. The cluster radius is measured to be 0.44 pc—matching well with the extent of radio emission—with a peak density of 650 pc{sup –2}. The JHK data are used to map the extinction in the region, which is subsequently used to compute the cloud mass—126 M {sub ☉} and 71 M {sub ☉} for the central cluster and the northern IRS 5 region, respectively. H{sub 2} narrowband imaging shows significant emission, which prominently resembles fluorescent emission arising at the borders of dense regions. Radio continuum analysis shows that this region has a blister morphology, with the radio peak coinciding with a protostellar source. Free-free emission spectral energy distribution analysis is used to obtain physical parameters of the overall photoionized region and the IRS 5 sub-region. This multiwavelength scenario is suggestive of star formation having resulted from the merging of multiple filaments to form a hub. Star formation seems to have taken place in two successive epochs, with the first epoch traced by the central cluster and the high-mass star(s)—followed by a second epoch that is spreading into the filaments as uncovered by the Class I sources and even younger protostellar sources along the filaments. The IRS 5 H II region displays indications of swept-up material that has possibly led to the formation of protostars.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

IRIS-MEMFLOW: Data Flow-Enabled Portable Memory Orchestration in IRIS Runtime for Diverse Heterogeneity

Task-based programming models and execution paradigms provide a means to decompose a computation by expressing it as a graph in which each node represents a specific computation operating on memory objects and the edges define the dependencies in the execution flow. In this execution model, independent nodes in the graph can be executed concurrently in different computing devices, making it suitable for heterogeneous systems in which computing devices with different architectures coexist. However, careful memory orchestration across heterogeneous devices is needed because copies of the same memory object may reside in multiple devices during execution. Manually ensuring such an orchestration is quite challenging. Not only must an application developer guard against race conditions, but they must also optimize data movement between the host and devices because unnecessary data movement significantly impacts performance. To mitigate these challenges, we enhance the IRIS heterogeneous runtime and introduce IRIS-MEMFLOW–a data flow–enabled portable memory abstraction for seamlessly orchestrating memory in diverse heterogeneous computing environments. By using data-flow analysis, IRIS-MEMFLOW guards against race conditions while multiple heterogeneous devices access memory objects. IRIS-MEMFLOW also optimizes data movement between the host and devices without manual intervention. As a result, IRIS provides improved programming productivity, performance, and portability for multidevice heterogeneous executions in high-performance computing and cloud systems that run diverse architectures from different vendors. The efficacy of IRIS-MEMFLOW is evaluated through experiments that show its capability in terms of programming productivity, multidevice heterogeneity, portability, and low overhead versus the state of the art.

Monil, M. A. H. [ORNL] (ORCID:0000000334194037)↗