solar-data-retrieval
Retrieving and preprocessing solar data to be used for dictionary learning based on random choice of sites and sizes.
Apostolaki, Elpiniki↗
SEARCH · Engineering Papers
Results for “DATA RETRIEVAL”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Selective data retrieval based on access latency
A processor includes multiple processing units (e.g., processor cores), with each processing unit associated with at least one private, dedicated cache. The processor is also associated with a system memory that stores all data that can be accessed by the multiple processing units. A coherency manager (e.g., a coherence directory) of the processor enforces a specified coherency scheme to ensure data coherency between the different caches and between the caches and the system memory. In response to a memory access request to a given cache resulting in a cache miss, the coherency manager identifies the current access latency to the system memory as well as the current access latencies to other caches of the processor. The coherency manager transfers the targeted data to the given cache from the cache or system memory having the lower access latency.
97 MATHEMATICS AND COMPUTING↗
Streamlining land surface model Initialization: Automated data retrieval for VELMA using HMS REST API and GDAL
Not Available
Venable, Kar'retta (ORCID:0000000195038631)↗
Retrieving data from the Intellus New Mexico system, the publicly accessible data base for environmental data at LANL
Abstract not provided.
54 ENVIRONMENTAL SCIENCES↗
Heuristics for Partial-Match Retrieval Data Base Design
Explore the source record for details and available documents.
Bentley, J.↗
Technical Assistance for Remote Sensor Placement Related Projects for Structural Health Monitoring
The world’s infrastructure is aging and many aspects are cost prohibitive to replace or even repair unless critically necessary. The standard way of determining if these repairs are critical or not is usually dependent on trained visual individual inspection. This process is very time consuming and costly considering the large areas and various structures that need to be monitored. A cost effective solution to this problem is to be able to monitor structures, such as roads, buildings, and bridges with remote sensors. Developing this solution provides some unique challenges such as sensor placement, types of sensors, energy needs for placed sensors, and data retrieval. As for the aspect of sensor placement, one method is to fly semi-autonomous Unmanned Arial Vehicles (UAVs) loaded with a sensor deployment mechanism to the structures so as to place the sensor and perhaps retrieve data from previously placed sensors as well. Many tests need to be run to determine the best UAV structure and deployment system to fulfill this task. It is hoped that through a multidisciplinary engineering scope the most effective process of monitoring the aging infrastructure can be developed.
42 ENGINEERING↗
Social media based NPL system to find and retrieve ARM data: Concept paper
Information connectivity and retrieval has a role in our daily lives. The most pervasive source of online information is databases. The amount of data is growing at rapid rate and database technology is improving and having a profound effect. Almost all online applications are storing and retrieving information from databases. One challenge in supplying the public with wider access to informational databases is the need for knowledge of database languages like Structured Query Language (SQL). Although the SQL language has been published in many forms, not everybody is able to write SQL queries. Another challenge is that it may not be practical to make the public aware of the structure of the database. There is a need for novice users to query relational databases using their natural language. To solve this problem, many natural language interfaces to structured databases have been developed. The goal is to provide more intuitive method for generating database queries and delivering responses. Social media makes it possible to interact with a wide section of the population. Through this medium, and with the help of Natural Language Processing (NLP) we can make the data of the Atmospheric Radiation Measurement Data Center (ADC) more accessible to the public. We propose an architecture for using Apache Lucene/Solr [1], OpenML [2,3], and Kafka [4] to generate an automated query/response system with inputs from Twitter5, our Cassandra DB, and our log database. Using the Twitter API and NLP we can give the public the ability to ask questions of our database and get automated responses.
Devarakonda, Ranjeet↗
Error-controlled Progressive Retrieval of Scientific Data under Derivable Quantities of Interest
The unprecedented amount of scientific data has introduced heavy pressure on the current data storage and transmission systems. Progressive compression has been proposed to mitigate this problem, which offers data access with on-demand precision. However, existing approaches only consider precision control on primary data, leaving uncertainties on the quantities of interest (QoIs) derived from it. In this work, we present a progressive data retrieval framework with guaranteed error control on derivable QoIs. Our contributions are three-fold. (1) We carefully derive the theories to strictly control QoI errors during progressive retrieval. Our theory is generic and can be applied to any QoIs that can be composited by the basis of derivable QoIs proved in the paper. (2) We design and develop a generic progressive retrieval framework based on the proposed theories, and optimize it by exploring feasible progressive representations. (3) We evaluate our framework using five real-world datasets with a diverse set of QoIs. Experiments demonstrate that our framework can faithfully respect any user-specified QoI error bounds in the evaluated applications. This leads to over 2.02× performance gain in data transfer tasks compared to transferring the primary data while guaranteeing a QoI error that is less than 1E-5.
Wu, Xuan↗
Improving Progressive Retrieval for HPC Scientific Data using Deep Neural Network
As the disparity between compute and I/O on high-performance computing systems has continued to widen, it has become increasingly difficult to perform post-hoc data analytics on full-resolution scientific simulation data due to the high I/O cost. Error-bounded data decomposition and progressive data retrieval framework has recently been developed to address such a challenge by performing data decomposition before storage and reading only part of the decomposed data when necessary. However, the performance of the progressive retrieval framework has been suffering from the over-pessimistic error control theory, such that the achieved maximum error of recomposed data is significantly lower than the required error. Therefore, more data than required is fetched for recomposition, incurring additional I/O overhead. In order to tackle this issue, we propose a DNN-based progressive retrieval framework that can better identify the minimum amount of data to be retrieved. Our contributions are as follows: 1) We provide an in-depth investigation of the recently developed progressive retrieval framework; 2) We propose two designs of prediction models (named D-MGARD and E-MGARD) to estimate the amount of retrieved data size based on error bounds. 3) We evaluate our proposed solutions using scientific datasets generated by real-world simulations from two domains. Evaluation results demonstrate the effectiveness of our solution in accurately predicting the amount of retrieval data size, as well as the advantages of our solution over the traditional approach to reducing the I/O overhead. Based on our evaluation, our solution is shown to read significantly less data (5% - 40% with D-MGARD, 20% - 80% with E-MGARD).
Wang, Jinzhen↗
PANSOPHY, a JLAB SRF engineering data management system, supporting data collection, retrieval and analysis utilized by LCLS-II
Pansophy is an Engineering Data Management System that provides a comprehensive solution for managing information in the production and testing of cryomodules. It is especially suited to supporting the Data & Quality Management Systems for large projects like LCLS-II. With extensive amounts of data collected for an individual project, data retrieval to facilitate feedback and enhancement of production and processing activities is a high priority. The priority shares importance with the needs of managing the project, including production status, NCR, and Quality Management reports. Recent Pansophy enhancements have been to Data and Quality management reports and statistical analysis. Such enhancements include a database driven menu system, extended MSWord macro and preprocessing of travelers, and an extensive reporting system. The reporting system allows managers and group leaders to quickly respond to the needs of the project in areas of cavity and cryomodule production, data collection, NCR, Quality Management and schedule. Extensions include integration with the SRF inventory system PRIMeS, allowing traceability from receiving of manufactured parts to final cryomodule product.
Bookwalter, Valerie D.↗
Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition
Extreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time. The challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-the-art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and ~ 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively.
Liang, Xin↗
Satellite Retrievals
Data from the SENTINEL-1 satellite constellation (SENTINEL-1a and SENTINEL-1b), from PNNL.
17 WIND ENERGY↗
ACHILLES: Heat Transfer in PWR Core During LOCA Reflood Phase
1. NAME AND TITLE OF DATA LIBRARY ACHILLES -Heat Transfer in PWR Core During LOCA Reflood Phase. 2. NAME AND TITLE OF DATA RETRIEVAL PROGRAMS N/A 3. CONTRIBUTOR AEA Technology, Winfrith Technology Centre, Dorchester DT2 8DH United Kingdom through the OECD Nuclear Energy Agency Data Bank, Issy-les-Moulineaux, France. 4. DESCRIPTION OF TEST FACILITY The most important features of the Achilles rig were the shroud vessel, which contained the test section, and the downcomer. These may be thought of as representing the core barrel and the annular downcomer in the reactor pressure vessel. The test section comprises a cluster of 69 rods in a square array within a circular shroud vessel. The rod diameter and pitch (9.5 mm and 12.6 mm) were typical of PWR dimensions. The internal diameter of the shroud vessel was 128 mm. Each rod was electrically heated over a length of 3.66 m, which is typical of the nuclear heated length in a PWR fuel rod, and each contained 6 internal thermocouples. These were arranged in one of 8 groupings which concentrated the thermocouples in different axial zones. The spacer grids were at prototypic PWR locations. Each grid had two thermocouples attached to its trailing edge at radial locations. The axial power profile along the rods was an 11 step approximation to a "chopped cosine". The shroud vessel had 5 heating zones whose power could be independently controlled. 5. DESCRIPTION OF TESTS The Achilles experiments investigated the heat transfer in the core of a Pressurized Water Reactor during the re-flood phase of a postulated large break loss of coolant accident. The results provided data to validate codes and to improve modeling. Different types of experiments were carried out which included single phase cooling, re-flood under low flow conditions, level swell and re-flood under high flow conditions. Three series of experiments were performed. The first and the third used the same test section but the second used another test section, similar in all respects except that it contained a partial blockage formed by attaching sleeves (or "balloons") to some of the rods. 6. SOURCE AND SCOPE OF DATA Phenomena Tested - Heat transfer in the core of a PWR during a re-flood phase of postulated large break LOCA. Test Designation - Achilles Rig. The programme includes the following types of experiments: - on an unballooned cluster: -- single phase air flow -- low pressure level swell -- low flooding rate re-flood -- high flooding rate re-flood - on a ballooned cluster containing 80% blockage formed by 16 balloon sleeves -- single phase air flow -- low flooding rate re-flood 7. DISCUSSION OF THE DATA RETRIEVAL PROGRAM N/A 8. DATA FORMAT AND COMPUTER Many Computers (M00019MNYCP00). 9. TYPICAL RUNNING TIME N/A 11. CONTENTS OF LIBRARY The ACHILLES package contains test data and associated data processing software as well as the documentation listed above. 12. DATE OF ABSTRACT November 2013. KEYWORDS: DATABASES, BENCHMARKS, HEAT TRANSFER, LOSS-OF-COLLANT ACCIDENT, PWR REACTORS, REFLOODING
Source record↗
Toward full exploitation of coherent and incoherent information in Sentinel-1 TOPS data for retrieving surface displacement: Application to the 2016 Kumamoto (Japan) earthquake
Sentinel-1's continuous observation program over all major plate boundary regions makes it well suited for earthquake studies. However, decorrelation due to large displacement gradients and limited azimuth resolution of the Terrain Observation by Progressive Scan (TOPS) data challenge acquiring measurements in the near field of many earthquake ruptures and prevent measurements of displacements in the along-track direction. Here we propose to fully exploit the coherent and incoherent information of TOPS data by using standard interferometric synthetic aperture radar (InSAR), split-bandwidth interferometry in range and azimuth, swath/burst-overlap interferometry, and amplitude cross correlation to map displacements in both the line-of-sight and the along-track directions. Application to the 2016 Kumamoto earthquake sequence reveals the coseismic displacements from the far field to the near field. By adding near-field constraints, the derived slip model reveals more shallow slip than obtained when only using far-field data from InSAR, highlighting the importance of exploiting all coherent and incoherent information in TOPS data.
58 GEOSCIENCES↗
Computer systems and methods for visualizing data
A method for forming a visual plot using a hierarchical structure of a dataset. The dataset comprises a measure and a dimension. The dimension consists of a plurality of levels. The plurality of levels form a dimension hierarchy. The visual plot is constructed based on a specification. A first level from the plurality of levels is represented by a first component of the visual plot. A second level from the plurality of levels is represented by a second component of the visual plot. The dataset is queried to retrieve data in accordance with the specification. The data includes all or a portion of the dimension and all or a portion of the measure. The visual plot is populated with the retrieved data in accordance with the specification.
97 MATHEMATICS AND COMPUTING↗
Using ARM Data to Retrieve Entrainment Rates in Stratocumulus Cloud Systems (Final Report)
A stratocumulus-topped boundary layer (STBL) is a shallow convecting layer in which both the updraft and the downdraft branches of the convective circulation are saturated (cloudy) in the upper part of the boundary layer. The convective circulation is vigorous, yet the cloud-top entrainment velocity, $w_e$, is small and difficult to measure. Differences between currently used parameterizations of $w_e$ in GCMs can lead to STBLs that differ by as much as a factor of two in climatologically important properties such as liquid water path and boundary layer depth. We have developed a new method to retrieve entrainment rates in STBLs which uses vertical velocity retrievals from surface-based cloud radar, in conjunction with other surface-based measurements. These new estimates will allow a more comprehensive evaluation of parameterizations of entrainment rate for STBLs. The new method will also provide ground-based estimates of entrainment in STBLs over long time periods. We use retrievals of vertical velocity and liquid water content profiles to estimate the turbulent liquid water flux, which is then used in combination with observed cloud-top jumps of temperature, water vapor, and calculated radiative flux profiles to estimate the entrainment rate. We present results for three days for which observational data was not significantly contaminated by drizzle. The resulting average entrainment rates are 0.58 cm/s, 1.51 cm/s, and 0.31 cm/s.
54 ENVIRONMENTAL SCIENCES↗
QProR: An Efficient Framework for Quantity-of-Interest Based Progressive Retrieval with Guaranteed Error Control
Scientific applications generate an unprecedented volume of data, overwhelming the network and file systems’ bandwidth and posing challenges for efficient and scalable data retrieval and analysis. Progressive data compression offers a promising solution by enabling on-demand retrieval at reduced size. However, existing progressive methods either fail to bound the errors in essential quantities of interest (QoIs) derived from raw data or suffer from suboptimal retrieval efficiency. In this work, we propose QProR, an efficient QoI-based progressive framework that optimizes progressive retrieval for target QoIs. Our key contributions include: (1) a systematic framework that integrates error-controlled lossy compressors with bitplane encoding while decoupling the two processes for high flexibility and adaptability; (2) a novel weighted bitplane encoding method which incorperates QoI knowledge into data refactoring to enhance retrieval efficiency; (3) an optimized retrieval strategy that accounts for the varying impacts of different variables on multivariate QoIs; (4) comprehensive evaluations using six real-world datasets from multiple scientific applications and thorough comparisons against state of the arts. Experimental results demonstrate that QProR achieves up to 80.38% reduction in the retrieval size under the same requested QoI error tolerance, when compared with the best-performing existing methods. When transferring 384 GB of scientific data to remote sites, QProR delivers up to 1.68 × speedup in the end-to-end data transfer performance.
Li, Wenbo [University of Kentucky]↗
AMRZone: A Runtime AMR Data Sharing Framework For Scientific Applications
Frameworks that facilitate runtime data sharing across multiple applications are of great importance for scientific data analytics. Although existing frameworks work well over uniform mesh data, they can not effectively handle adaptive mesh refinement (AMR) data. Among the challenges to construct an AMR-capable framework include: (1) designing an architecture that facilitates online AMR data management; (2) achieving a load-balanced AMR data distribution for the data staging space at runtime; and (3) building an effective online index to support the unique spatial data retrieval requirements for AMR data. Towards addressing these challenges to support runtime AMR data sharing across scientific applications, we present the AMRZone framework. Experiments over real-world AMR datasets demonstrate AMRZone's effectiveness at achieving a balanced workload distribution, reading/writing large-scale datasets with thousands of parallel processes, and satisfying queries with spatial constraints. Moreover, AMRZone's performance and scalability are even comparable with existing state-of-the-art work when tested over uniform mesh data with up to 16384 cores; in the best case, our framework achieves a 46% performance improvement.
Zhang, Wenzhao↗