Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “datastreams”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

Online mapping and forecasting of epidemics using open-source indicators

Open-source indicators have been proposed as a way of tracking and forecasting disease outbreaks. Some, such are meteorological data, are readily available as reanalysis products. Others, such as those derived from our online behavior (web searches, media article etc.) are gathered easily and are more timely than public health reporting. In this study we investigate how these datastreams may be combined to provide useful epidemiological information. The investigation is performed by building data assimilation systems to track influenza in California and dengue in India. The first does not suffer from incomplete data and was chosen to explore disease modeling needs. The second explores the case when observational data is sparse and disease modeling complexities are beside the point. The two test cases are for opposite ends of the disease tracking spectrum. We find that data assimilation systems that produce disease activity maps can be constructed. Further, being able to combine multiple open-source datastreams is a necessity as any one individually is not very informative. The data assimilation systems have very little in common except that they contain disease models, calibration algorithms and some ability to impute missing data. Thus while the data assimilation systems share the goal for accurate forecasting, they are practically designed to compensate for the shortcomings of the datastreams. Thus we expect them to be disease and location-specific.

42 ENGINEERING↗
DOE OSTI2023

COMBLE Radar b1 Processing: Corrections, Calibrations, and Processing Report

The U.S. Department of Energy’s (DOE) Atmospheric Radiation Measurement (ARM) user facility recently concluded its Cold-Air Outbreaks in the Marine Boundary Layer Experiment (COMBLE), with its campaign emphasis on marine boundary-layer clouds and mixed-phase clouds during cold-air outbreaks. The COMBLE campaign featured the deployment of the first ARM Mobile Facility (AMF1) to northern Scandinavia (Andenes, Norway), including its standard complement of ARM cloud radars. In keeping with user demands stemming from the previous AMF Cloud, Aerosol, and Complex Terrain Interactions (CACTI) field campaign in Argentina, a post-campaign radar mentor effort was initiated for COMBLE. This activity was intended to improve the usability of the ARM cloud radar data sets in response to overall demands for calibrated, corrected data sets for downstream studies and retrieval applications. In addition to the terrain complexities previously important to CACTI data sets (e.g., clutter designation and/or removal), COMBLE presented new challenges to existing ARM radar mentor capabilities. These included the extension of existing methodologies (e.g., relative calibration adjustment [RCA] target techniques) to frozen environments and the potential issues with their applicability when considering mixed-phase precipitation conditions. As in the previous CACTI documentation (Hardin et al. 2020), the overall calibration and conditioning process in ARM nomenclature is referred to as generating a “b1” datastream. For the radars, these “b1” standards refer to a datastream that has been calibrated (and cross-calibrated), with effort to deliver the highest-quality (well-characterized) data possible. The “b1” radar mentor reporting (this current document) is intended to detail (i) the status/quality of the original “a1” (raw) data sets during the COMBLE AMF campaign, (ii) the corrections and calibrations that are applied to generate the b1 datastreams available on ARM’s Data Discovery, and (iii) the details of the applied methods, e.g., how radar offset/calibration numbers were determined.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2011

Report on the Global Data Assembly Center (GDAC) to the 12th GHRSST Science Team Meeting

In 2010/2011 the Global Data Assembly Center (GDAC) at NASA's Physical Oceanography Distributed Active Archive Center (PO.DAAC) continued its role as the primary clearinghouse and access node for operational Group for High Resolution Sea Surface Temperature (GHRSST) datastreams, as well as its collaborative role with the NOAA Long Term Stewardship and Reanalysis Facility (LTSRF) for archiving. Here we report on our data management activities and infrastructure improvements since the last science team meeting in June 2010.These include the implementation of all GHRSST datastreams in the new PO.DAAC Data Management and Archive System (DMAS) for more reliable and timely data access. GHRSST dataset metadata are now stored in a new database that has made the maintenance and quality improvement of metadata fields more straightforward. A content management system for a revised suite of PO.DAAC web pages allows dynamic access to a subset of these metadata fields for enhanced dataset description as well as discovery through a faceted search mechanism from the perspective of the user. From the discovery and metadata standpoint the GDAC has also implemented the NASA version of the OpenSearch protocol for searching for GHRSST granules and developed a web service to generate ISO 19115-2 compliant metadata records. Furthermore, the GDAC has continued to implement a new suite of tools and services for GHRSST datastreams including a Level 2 subsetter known as Dataminer, a revised POET Level 3/4 subsetter and visualization tool, a Google Earth interface to selected daily global Level 2 and Level 4 data, and experimented with a THREDDS catalog of GHRSST data collections. Finally we will summarize the expanding user and data statistics, and other metrics that we have collected over the last year demonstrating the broad user community and applications that the GHRSST project continues to serve via the GDAC distribution mechanisms. This report also serves by extension to summarize the activities of the GHRSST Data Assembly and Systems Technical Advisory Group (DAS-TAG).

sea surface temperature (SST)↗
DOE OSTI2016

An architecture for consolidating multidimensional time-series data onto a common coordinate grid

Consolidating measurement data for use by data models or in inter-comparison studies frequently requires transforming the data onto a common grid. Standard methods for interpolating multidimensional data are often not appropriate for data with non-homogenous dimensionality, and are hard to implement in a consistent manner for different datastreams. These challenges are increased when dealing with the automated procedures necessary for use with continuous, operational datastreams. In this paper we introduce a method of applying a series of one-dimensional transformations to merge data onto a common grid, examine the challenges of ensuring consistent application of data consolidation methods, present a framework for addressing those challenges, and describe the implementation of such a framework for the Atmospheric Radiation Measurement (ARM) program.

coordinate transformation↗
DOE OSTI2015

Degradation science: Mesoscopic evolution and temporal analytics of photovoltaic energy materials

Based on recent advances in nanoscience, data science and the availability of massive real-world datastreams, the mesoscopic evolution of mesoscopic energy materials can now be more fully studied. The temporal evolution is vastly complex in time and length scales and is fundamentally challenging to scientific understanding of degradation mechanisms and pathways responsible for energy materials evolution over lifetime. We propose a paradigm shift towards mesoscopic evolution modeling, based on physical and statistical models, that would integrate laboratory studies and real-world massive datastreams into a stress/mechanism/response framework with predictive capabilities. These epidemiological studies encompass the variability in properties that affect performance of material ensembles. Mesoscopic evolution modeling is shown to encompass the heterogeneity of these materials and systems, and enables the discrimination of the fast dynamics of their functional use and the slow and/or rare events of their degradation. As a result, we delineate paths forward for degradation science.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2015

HACC: Simulating sky surveys on state-of-the-art supercomputing architectures

Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the ‘Dark Universe’, dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers that enable these simulations. In this paper, we report on HACC (Hardware/Hybrid Accelerated Cosmology Code), a recently developed and evolving cosmology N-body code framework, designed to run efficiently on diverse computing architectures and to scale to millions of cores and beyond. HACC can run on all current supercomputer architectures and supports a variety of programming models and algorithms. It has been demonstrated at scale on Cell- and GPU-accelerated systems, standard multi-core node clusters, and Blue Gene systems. HACC’s design allows for ease of portability, and at the same time, high levels of sustained performance on the fastest supercomputers available. We present a description of the design philosophy of HACC, the underlying algorithms and code structure, and outline implementation details for several specific architectures. We show selected accuracy and performance results from some of the largest high resolution cosmological simulations so far performed, including benchmarks evolving more than 3.6 trillion particles.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Online Learning of Effective Turbine Wind Speed in Wind Farms

To develop better wind farm controllers that can meet more complex objectives, methods of modeling the wind turbine wakes at low computational expense are needed. Gaussian process (GP) regression offers a computationally inexpensive framework for learning complex functions from noisy measurements with very few datapoints. In this work, an online learning approach is presented to learn the rotor-averaged wind velocity at downstream wind turbines with GPs, using the available datastream of wind field measurements and wind turbine control set-points. This framework can readily be integrated into model-based controls methods because the model a) is updated online at low computational expense, b) assumes a mathematically favorable Gaussian form, and c) explicitly quantifies the stochastic nature of the wake field so that the trade-off between exploration and exploitation, and the uncertainty in the prediction, can be utilized. We show that a GP-learned model can match true values with errors within 0.5% on average, with as few as 5 training data points.

Gaussian process↗
DOE OSTI2020

Scalable Linked Dynamic Equilibrium (SLiDE)

SLiDE (Scalable Linked Dynamic Equilibrium) replicates the blueNOTE (National Open source Tools for general Equilibrium analysis) datastream from the Wisconsin National Data Consortium. From there, it produces a recursive dynamic computable general equilibrium model of the US states and eventually the global/national economies. The datastream operations have been set up to allow for scalability of both regions and sectors represented. The model has the capability to flexibly define functional forms similar to other general equilibrium model construction software.

Hughes, Caroline↗
DOE OSTI2020

Description of the LASSO Data Bundles Product

The U. S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility began a pilot project in May 2015 to design a routine, high-resolution modeling capability to complement ARM’s extensive suite of measurements. This modeling capability has evolved into the Large-Eddy Simulation (LES) ARM Symbiotic Simulation and Observation (LASSO) datastream. The datastream, broadly termed data bundles, contains high-resolution model output, input files, observations for evaluation, and skill scores for the simulations. The initial focus of LASSO is on shallow convection at the ARM Southern Great Plains (SGP) atmospheric observatory. The availability of LES simulations with concurrent observations serves many purposes. LES helps bridge the scale gap between DOE ARM observations and models, and the use of routine LES adds value to observations. It provides a self-consistent representation of the atmosphere and a dynamical context for the observations. Further, it elucidates unobservable processes and properties. LASSO generates a simulation library for researchers that enables statistical approaches beyond a single-case mentality. It also provides tools necessary for modelers to reproduce the LES and conduct their own sensitivity experiments. The LASSO library of data bundles is designed to facilitate a wide range of research. For an observationalist, LASSO can help inform instrument remote-sensing retrievals, conduct observation system simulation experiments (OSSEs), and test implications of radar scan strategies or flight paths. For a theoretician, LASSO can help calculate estimates of fluxes and co-variability of values, and test relationships without having to run the model personally. For a modeler, LASSO can help one know ahead of time which days have good forcing, have co-registered observations at high-resolution scales, and have simulation inputs and corresponding outputs to test parameterizations. Further details on the overall LASSO project are available at https://www.arm.gov/capabilities/modeling/lasso.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Network Models of Active Degradation Mechanisms and Pathways for Service Life Prediction of Indoor and Outdoor PV Modules

ct: PV service lifetime prediction (SLP) enables accurate calculation of levelized cost of energy (LCOE), which is crucial to rationalizing PV investment and installation. However, SLP is challeging since PV reliability in the field is affected by many combined factors, including various environmental stresses and module quality. In order to map out the active degradation mechanisms and pathways that best resemble real world conditions, we introduce the framework of a study protocol and use network models fitted to data, to enable analysis and SLP of complex PV systems with multiple active degradation mechanisms. The study protocol is the experimental design, including module variants and different exposure conditions, selection of evaluation methods, time-series data acquisition and training of network models to these data. We present SLP of minimodules in the lab and PV systems in the field. For lab SLP, minimodules with 8 variants based on manufacturer, architecture, and encapsulation were prepared and aged in modified damp heat with or without full spectrum light exposure. Stepwise I-V and Suns-Voc data acquisition tracks changes in electrical properties including Rs,IV, Isc,IV, Vmp,PIV providing insights into power loss of minimodules. Network structural equation modeling (netSEM) was utilized to construct degradation pathway models that identify active degradation mechanisms and predict power loss over time. For field SLP, datastreams of Pmp values and I-V curve datastreams of two types of modules installed in three distinctly different Köppen-Geiger climate zones for 9 years were acquired. With power loss modes corresponding to uniform current loss (ΔPIsc), recombination (ΔPVoc), series resistance (ΔPRs), and current mismatch (ΔPImis) determined, the performance loss rates (PLR) were determined using PVplr. We show how to establish a study protocol framework to ensure appropriate parametric variations and valid data collection from the variants of your complex systems. Then the data-driven netSEM model fitting provides a comprehensive mapping of multiple active degradation mechanisms, and accurate service life prediction.

network model, degradation, photovoltaic, solar↗
DOE OSTI2023

MOSAiC Radar b1 Processing: Corrections, Calibrations, and Processing Report

The U.S. Department of Energy’s (DOE) Atmospheric Radiation Measurement (ARM) user facility deployed many instruments on board a German ice breaker, the Research Vessel (RV) Polarstern, for one year from October 2019 to October 2020. The purpose of the Multidisciplinary Drifting Observatory for the Study of Arctic Climate (MOSAiC) field campaign was to study the decline in the sea-ice pack around the North Pole, and what factors may be at play. After the campaign ended, efforts were undertaken to provide a calibrated radar data set for future studies. MOSAiC presented new challenges to this process, as existing methodologies often were not applicable for the frozen environment with little to no ground clutter, and concurrent engineering updates or calibrations could not be accomplished once the ship set off. Like the previous Cloud, Aerosol, and Complex Terrain Interactions (CACTI) and Cold-air Outbreaks in the Marine Boundary Layer Experiment (COMBLE) documentation (Hardin et al. 2020, Matthews et al. 2023), the data correction and calibration process is known in ARM as creating a “b1” datastream. This means that the radar datastreams have been well characterized to the best possible quality. This report will detail the status of the raw “a1” level data sets during the MOSAiC campaign, the corrections that were applied to create the “b1” data files, and the details of the applied methods.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Merged Aerosol Value-Added Product Report

The Merged Aerosol Value-Added Product (VAP) simplifies scientists’ use of Atmospheric Radiation Measurement (ARM) User Facility aerosol data by performing several tedious, time-consuming tasks for the users. First, the VAP identifies the best data available when multiple datastreams exist for a single geophysical quantity so that ARM users do not have to research this for themselves. Second, the VAP consolidates multiple ARM aerosol datastreams into a single file for ARM data users so that they do not have to download, open, and read multiple files for their analysis. Next, the VAP transforms all measurements onto a common one-hour timestamp. The one-hour resolution matches the time resolution of the slowest instrument. Instruments with faster sampling rates than one measurement per hour are averaged over the time interval. Finally, the VAP reads the QA/QC variables and marks data with known issues as missing, so that users do not have to spend excessive time cleaning data. This includes incorporating Data Quality Reports (DQRs) that exist at the time when the VAP data is generated. DQRs are reports filed by instrument mentors or data users that indicate a problem with the output data of individual instruments.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Performance and Reliability Assessment of the U.S. Department of Energy Atmospheric Radiation Measurement (ARM) Data Advisor (ADA)

The Atmospheric Radiation Measurement (ARM) User Facility provides one of the world's largest openly accessible repositories of atmospheric observations through the ARM Data Discovery platform. Although the repository contains more than three decades of measurements collected from permanent observatories, mobile facilities, aircraft campaigns, and field experiments, identifying appropriate datasets can be challenging, particularly for new users unfamiliar with ARM instrumentation and datastream organization. To improve data accessibility, the ARM Data Center developed the ARM Data Advisor (ADA), an artificial intelligence-powered assistant designed to facilitate scientific data discovery, dataset interpretation, and user guidance. This report evaluates ADA's performance as a domain-specific scientific assistant using realistic atmospheric science workflows. The evaluation examines five key capabilities: data retrieval and curation efficiency, hallucination resistance, scientific reasoning, response to ambiguous queries, and content retention and session continuity. Representative prompts were developed to simulate typical interactions between researchers and the ARM Data Discovery platform, and ADA's responses were assessed for retrieval completeness, scientific accuracy, consistency, and practical usefulness. In these representative tests, ADA reduced the complexity of discovering and accessing ARM datasets by recommending appropriate datastreams, explaining instrumentation, interpreting metadata, and assisting with data processing workflows. ADA also exhibits strong domain knowledge of atmospheric science terminology and generally resists hallucination by acknowledging unavailable datasets and requesting clarification when appropriate. Overall, the results indicate that ADA represents a promising advancement in scientific data discovery within the ARM User Facility and has considerable potential to improve researcher productivity, particularly for new users and interdisciplinary scientists seeking efficient access to ARM observations.

Salvador, Christian [ORNL] (ORCID:0000000283287777↗
DOE OSTI2017

pgscoeff.b1

pgscoeff.b1 datastream necessary for callibration corrections to pgs.b1 datastream

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

kasacrgridppi.c1

This datastream is made by kasacrgridppi VAP. It has (time, Y, X) gridded Ka-band PPI radar data, using nearest neighbor algorithm, with calibrated (.b1) input. This datastream is very similar to kasacrgridrhi, with the output grid changed from (time, height, h_distance) to (time, Y, X). output c1 level data use calibrated b1 data as input.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1990

Highly parallel computation

Highly parallel computing architectures are the only means to achieve the computation rates demanded by advanced scientific problems. A decade of research has demonstrated the feasibility of such machines and current research focuses on which architectures designated as multiple instruction multiple datastream (MIMD) and single instruction multiple datastream (SIMD) have produced the best results to date; neither shows a decisive advantage for most near-homogeneous scientific problems. For scientific problems with many dissimilar parts, more speculative architectures such as neural networks or data flow may be needed.

Denning, Peter J.↗
NASA NTRS1995

Highly-Parallel, Highly-Compact Computing Structures Implemented in Nanotechnology

In this paper, we describe work in which we are evaluating how the evolving properties of nano-electronic devices could best be utilized in highly parallel computing structures. Because of their combination of high performance, low power, and extreme compactness, such structures would have obvious applications in spaceborne environments, both for general mission control and for on-board data analysis. However, the anticipated properties of nano-devices mean that the optimum architecture for such systems is by no means certain. Candidates include single instruction multiple datastream (SIMD) arrays, neural networks, and multiple instruction multiple datastream (MIMD) assemblies.

Crawley, D. G.↗
NASA NTRS2001

Advances in Global Water Cycle Science Made Possible by Global Precipitation Mission (GPM)

Within this decade the internationally sponsored Global Precipitation Mission (GPM) will take an important step in creating a global precipitation observing system from space. One perspective for understanding the nature of GPM is that it will be a hierarchical system of datastreams from very high caliber combined dual frequency radar/passive microwave (PMW) rain-radiometer retrievals, to high caliber PMW rain-radiometer only retrievals, and on to blends of the former datastreams with other less-high caliber PMW-based and IR-based rain retrievals. Within the context of NASA's role in global water cycle science and its own Global Water & Energy Cycle (GWEC) program, GPM is the centerpiece mission for improving our understanding of the global water cycle from a space-based measurement perspective. One of the salient problems within our current understanding of the global water and energy cycle is determining whether a change in the rate of the water cycle is accompanying changes in global temperature. As there are a number of ways in which to define a rate-change of the global water cycle, it is not entirely clear as to what constitutes such a determination, This paper presents an overview of the Global Precipitation Mission and how its datasets can be used in a set of quantitative tests within the framework of the oceanic and continental water budget equations to determine comprehensively whether substantive rate changes do accompany perturbations in global temperatures and how such rate changes manifest themselves in both water storage and water flux transport processes.

Smith, Eric A.↗