Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “CLOUD DATA”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer low performance when subject to highly concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids, or discrete event generators. In this paper we tackle this issue. We present SLoG, shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG’s design on up to 100,000 cores of the Theta supercomputer: the results show high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.

big data↗

Planning for the utilization of the PCDS in studying the interaction of clouds (ISCCP-C data) and the Earth radiation budget (ERBE data)

The Pilot Climate Data System (PCDS) affords an opportunity to analyze data from different but highly complementary data sets. Two of these highly complementary data sets supported by the PCDS are the International Satellite Cloud Climatology Project (ISCCP) and the Earth Radiation Budget Experiment (ERBE). Both data set sponsors are aware of the utility of one data set to the other, and both projects utilize gridded data on a 2.5 deg by 2.5 deg grid. The ISCCP data have been collected since July 1983, and the NOAA-9 data for ERBE have been collected for more than a year. Therefore, there is a good chance to use these temporally overlapping data sets to investigate hypothesized relationships. Changes in cloudiness affect both cloud albedo feedback (shortwave) and the greenhouse effect (longwave). The relative importance of the effects of clouds on albedo versus outgoing longwave radiation (OLR) in determining the radiation balance has long been a matter of controversy. Now, however, changes in cloud amount as observed by the ISCCPO can be correlated to corresponding changes in the albedo and changes in the OLR from ERBE. Monthly means can be utilized in all instances.

Jacobowitz, H.↗

Cloud classification from satellite data using a fuzzy sets algorithm: A polar example

Where spatial boundaries between phenomena are diffuse, classification methods which construct mutually exclusive clusters seem inappropriate. The Fuzzy c-means (FCM) algorithm assigns each observation to all clusters, with membership values as a function of distance to the cluster center. The FCM algorithm is applied to AVHRR data for the purpose of classifying polar clouds and surfaces. Careful analysis of the fuzzy sets can provide information on which spectral channels are best suited to the classification of particular features, and can help determine likely areas of misclassification. General agreement in the resulting classes and cloud fraction was found between the FCM algorithm, a manual classification, and an unsupervised maximum likelihood classifier.

Key, J. R.↗

Cloud classification from satellite data using a fuzzy sets algorithm - A polar example

Where spatial boundaries between phenomena are diffuse, classification methods which construct mutually exclusive clusters seem inappropriate. The Fuzzy c-means (FCM) algorithm assigns each observation to all clusters, with membership values as a function of distance to the cluster center. The FCM algorithm is applied to AVHRR data for the purpose of classifying polar clouds and surfaces. Careful analysis of the fuzzy sets can provide information on which spectral channels are best suited to the classification of particular features, and can help determine like areas of misclassification. General agreement in the resulting classes and cloud fraction was found between the FCM algorithm, a manual classification, and an unsupervised maximum likelihood classifier.

Key, J. R.↗

The ARM Cloud Radar Simulator for Global Climate Models: Bridging Field Data and Climate Models

Clouds play an important role in Earth’s radiation budget and hydrological cycle. However, current global climate models (GCMs) have had difficulties in accurately simulating clouds and precipitation. To improve the representation of clouds in climate models, it is crucial to identify where simulated clouds differ from real world observations of them. This can be difficult, since significant differences exist between how a climate model represents clouds and what instruments observe, both in terms of spatial scale and the properties of the hydrometeors which are either modeled or observed. To address these issues and minimize impacts of instrument limitations, the concept of instrument “simulators”, which convert model variables into pseudo-instrument observations, has evolved with the goal to improve and to facilitate the comparison of modeled clouds with observations. Many simulators have (and continue to be developed) for a variety of instruments and purposes. A community satellite simulator package, the Cloud Feedback Model Intercomparison Project (CFMIP) Observation Simulator Package (COSP; Bodas-Salcedo et al. 2011), contains several independent satellite simulators and is being widely used in the global climate modeling community to exploit satellite observations for model cloud evaluation (e.g., Klein et al. 2013; Zhang et al. 2010). This article introduces a ground-based cloud radar simulator developed by the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) program for comparing climate model clouds with ARM observations from its vertically pointing 35-GHz radars. As compared to CloudSat radar observations, ARM radar measurements occur with higher temporal resolution and finer vertical resolution. This enables users to investigate more fully the detailed vertical structures within clouds, resolve thin clouds, and quantify the diurnal variability of clouds. Particularly, ARM radars are sensitive to low-level clouds, which are difficult for the CloudSat radar to detect due to surface contamination (Mace et al. 2007; Marchand et al. 2008). Therefore, the ARM ground-based cloud observations can provide important observations of clouds that complement measurements from space.

Zhang, Yuying↗

Relating Convective and Stratiform Rain to Latent Heating

The relationship among surface rainfall, its intensity, and its associated stratiform amount is established by examining observed precipitation data from the Tropical Rainfall Measuring Mission (TRMM) Precipitation Radar (PR). The results show that for moderate-high stratiform fractions, rain probabilities are strongly skewed toward light rain intensities. For convective-type rain, the peak probability of occurrence shifts to higher intensities but is still significantly skewed toward weaker rain rates. The main differences between the distributions for oceanic and continental rain are for heavily convective rain. The peak occurrence, as well as the tail of the distribution containing the extreme events, is shifted to higher intensities for continental rain. For rainy areas sampled at 0.58 horizontal resolution, the occurrence of conditional rain rates over 100 mm/day is significantly higher over land. Distributions of rain intensity versus stratiform fraction for simulated precipitation data obtained from cloud-resolving model (CRM) simulations are quite similar to those from the satellite, providing a basis for mapping simulated cloud quantities to the satellite observations. An improved convective-stratiform heating (CSH) algorithm is developed based on two sources of information: gridded rainfall quantities (i.e., the conditional intensity and the stratiform fraction) observed from the TRMM PR and synthetic cloud process data (i.e., latent heating, eddy heat flux convergence, and radiative heating/cooling) obtained from CRM simulations of convective cloud systems. The new CSH algorithm-derived heating has a noticeably different heating structure over both ocean and land regions compared to the previous CSH algorithm. Major differences between the new and old algorithms include a significant increase in the amount of low- and midlevel heating, a downward emphasis in the level of maximum cloud heating by about 1 km, and a larger variance between land and ocean in the new CSH algorithm.

Tao, Wei-Kuo↗

The Cloud Absorption Radiometer HDF Data User's Guide

The purpose of this document is to describe the Cloud Absorption Radiometer (CAR) Instrument, methods used in the CAR Hierarchical Data Format (HDF) data processing, the structure and format of the CAR HDF data files, and methods for accessing the data. Examples of CAR applications and their results are also presented. The CAR instrument is a multiwavelength scanning radiometer that measures the angular distributions of scattered radiation.

Li, Jason Y.↗

Cloud-Based Demodulation and Data Distribution of a Satellite Downlink

Ground station networks connected to the cloud allow space missions to have global communications coverage without operating their own infrastructure. In this work, we describe the communications architecture for the TechEdSat-13 mission, which performed the first in-space characterization of a neuromorphic processor. The mission utilizes a commercial provider for S-band downlinks. A suite of cloud services and open-source software such as GNU Radio are leveraged to demodulate signals received by an AWS ground station during passes with TechEdSat-13 and store recovered data. Once a pass is scheduled, the entire process takes place without human intervention. On-orbit results the past year of operations are presented, demonstrating the advantages of this approach over traditional operator-owned ground stations. Use of software-defined radio makes possible custom signal processing. The homogeneity of apertures and their interfaces to the cloud simplifies scaling across many sites. This abundance of candidate links lays the groundwork for intelligent scheduling agents to optimize pass selection across several factors, automatically recover from failed contacts, and gather metrics to learn from past performance.

cloud demodulation↗

Improvement of cloud cover assessment of Landsat Thematic Mapper data

Various methods for cloud cover assessment of Landsat TM data are compared and evaluated. A total of 3906 TM 241-mm black-and-white photo prints of different seasons and regions were assessed manually. These images were mostly in band 4 (0.76-0.90 microns) for daytime data and band 6 (10.4-12.5 microns) for nighttime data. The results of this manual assessment are compared to available automated and manual assessments, and the differences between results obtained by various methods are discussed.

Ludwig, R.↗

Estimating Glaciation Temperature of Deep Convective Clouds with Remote Sensing Data

Major uncertainties exist for observing and modeling ice content inside deep convective clouds (DCC). One of the difficulties has been the lack of characterization of vertical profiles of cloud hydrometeor phase. Here we propose a technique to estimate the DCC glaciation temperature using passive remote sensing data. It is based on a conceptual model of vertical hydrometeor size profiles inside DCCs. Estimates from the technique agree well with our general understanding of the problem. Furthermore, the link between vertical profiles of cloud particle size and hydrometeor thermodynamic phase is confirmed by a 3-13 cloud retrieval technique. The technique is applied to aircraft measurements of cloud side reflectance and the result was compared favorably with an independent retrieval of thermodynamic phase based on different refractive indices at 2.13 micron and 2.25 micron. Possible applications of the technique are discussed.

Yuan, Tianle↗

Application of Cloude's target decomposition theorem to polarimetric imaging radar data

We apply Cloude's decomposition to imaging radar polarimetry. We derive the general expressions for the eigenvalues and eigenvectors for the case of terrain with reflection symmetry, and show in detail how the decomposition results can guide the interpretation of scattering from vegetated areas. For multi-frequency polarimetric radar measurements of a clear-cut area, the decomposition leads us to conclude that the vegetation is probably thin compared to even the C-band radar wavelength of 6 cm. For a forested area, we notice an increased amount of even number of reflection scattering at P-band and L-band, probably the result of penetration through the coniferous canopy resulting in trunk-ground double reflection scattering. The scattering for the forested area is still dominated by scattering from randomly oriented cylinders, however. It is found that these cylinders are thicker than in the case of clear-cut areas, leading us to conclude that scattering from the branches probably dominate in this case.

Vanzyl, Jakob J.↗

Emulation and detection of physical faults and cyber-attacks on building energy systems through real-time hardware-in-the-loop experiments

The increasing use of remote or mobile access, integrated wearable technologies, data exchange, and cloud-based data analytics in modern smart buildings is steering the building industry towards open communication technologies. The increased connectivity and accessibility could lead to more cyber-attacks in smart buildings. On the other hand, physical faults (e.g., HVAC -heating, ventilation, and air-conditioning faults) may have similar adverse impacts as those from the cyber-attacks on building energy systems, such as occupant discomfort, energy wastage, and equipment downtime. However, current physical behavior-based anomaly detection methods fail to differentiate between cyber-attacks and physical faults in building energy systems. Moreover, the challenge in collecting real-world threat data with ground truth has led researchers to rely on numerical models with user-defined assumptions, which may not accurately reflect real-world conditions due to the lack of in-situ experimental datasets. To address these challenges and gaps, this paper presents a flexible hardware-in-the-loop (HIL) testbed for generating cyber-attack and physical fault datasets and demonstrating threat detection algorithms in a real building automation system (BAS) environment. This testbed combines hardware (i.e., real BAS with local HVAC controllers and a physical network) with software (i.e., high-fidelity models to represent behaviors of building envelope and HVAC energy systems), enabling emulations of realistic threats. Five HIL experiments, including one baseline without any threats, two with physical faults, and two with cyber-attacks, were conducted to generate datasets containing detailed network traffic and system states. A joint classification framework, incorporating a network analyzer and a physical HVAC fault detector, was proposed to automatically detect cyber-physical abnormalities on BAS at both the network and the physical HVAC levels. The network analyzer comprises a conditional random fields (CRF) based command validator and a statistics-based detection strategy. The fault detector employs a weather and schedule-based pattern matching and feature-based principal component analysis (WPM-FPCA) method. Evaluation of the classification using four metrics from the multi-class confusion matrix revealed an average accuracy of 90.2%, recall of 89.7%, precision of 88.5% and F1-score of 89.2%. Finally, these results demonstrate that the proposed joint classification framework can effectively differentiate between specific types of cyber-attacks (e.g., device reinitialization attack, network Denial-of-Service attack) and physical faults (e.g., air handling unit operational fault, cooling coil valve stuck) in real time for improved building energy management.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

HIGH-RESOLUTION OBSERVATIONS AND THE PHYSICS OF HIGH-VELOCITY CLOUD A0

The neutral hydrogen structure of high-velocity cloud A0 (at about -180 km s{sup -1}) has been mapped with a 9.'1 resolution. Gaussian decomposition of the profiles is used to separately map families of components defined by similarities in center velocities and line widths. About 70% of the H I gas is in the form of a narrow, twisted filament whose typical line widths are of the order of 24 km s{sup -1}. Many bright features with narrow line widths of the order of 6 km s{sup -1}, clouds, are located in and near the filament. A third category with properties between those of the filament and clouds appears in the data. The clouds are not always co-located with the broader line width filament emission as seen projected on the sky. Under the assumption that magnetic fields underlie the presence of the filament, a theorem is developed for its stability in terms of a toroidal magnetic field generated by the flow of gas along field lines. It is suggested that the axial magnetic field strength may be derived from the excess line width of the H I emission over and above that due to kinetic temperature by invoking the role of Alfven waves that create what is in essence a form of magnetic turbulence. At a distance of 200 pc the axial and the derived toroidal magnetic field strengths in the filament are then about 6 {mu}G while for the clouds they are about 4 {mu}G. The dependence of the derived field strength on distance is discussed.

79 ASTRONOMY AND ASTROPHYSICS↗

Compadre Toolkit

The Compadre Toolkit solves a minimization problem, that once solved allows a user to reconstruct a function from sample data collected from a cloud of data sites. The solution to this minimization can also be used to assemble a linear system that can be solved numerically using some other software. The minimization problem the toolkit solves is to find the optimal finite polynomial in a user specified finite dimensional space with regards to minimizing an objective function which measures a linear sampling functional applied to the polynomial, compared to the same sampling functional applied to the source function, weighted by a positive kernel that has compact support.

SciDAC↗

The Modern Research Data Portal: a design pattern for networked, data-intensive science

We describe best practices for providing convenient, high-speed, secure access to large data via research data portals. Here, we capture these best practices in a new design pattern, the Modern Research Data Portal, that disaggregates the traditional monolithic web-based data portal to achieve orders-of-magnitude increases in data transfer performance, support new deployment architectures that decouple control logic from data storage, and reduce development and operations costs. We introduce the design pattern; explain how it leverages high-performance data enclaves and cloud-based data management services; review representative examples at research laboratories and universities, including both experimental facilities and supercomputer sites; describe how to leverage Python APIs for authentication, authorization, data transfer, and data sharing; and use coding examples to demonstrate how these APIs can be used to implement a range of research data portal capabilities. Sample code at a companion web site, https://docs.globus.org/mrdp , provides application skeletons that readers can adapt to realize their own research data portals.

97 MATHEMATICS AND COMPUTING↗

Investigation of TM Band-to-band Registration Using the JSC Registration Processor

The JSC registration processor performs scene-to-scene (or band-to-band) correlation based on edge images. The edge images are derived from a percentage of the edge pixels calculated from the raw scene data, excluding clouds and other extraneous data in the scene. Correlations are performed on patches (blocks) of the edge images, and the correlation peak location in each patch is estimated iteratively to fractional pixel location accuracy. Peak offset locations from all patches over the scene are then considered together, and a variety of tests are made to weed out outliers and other inconsistencies before a distortion model is assumed. Thus, the correlation peak offset locations in each patch indicate quantitatively how well the two TM bands register to each other over that patch of scene data. The average of these offsets indicate the overall accuracies of the band-to-band registration. The registration processor was also used to register one acquisition to another acquisition of multitemporal TM data acquired over the same ground track. Band 4 images from both acquisitions were correlated and an rms error of a fraction of a pixel was routinely obtained.

Yao, S. S.↗