Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Queries”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,657 records · Page 92

CIVET: Continuous Integration, Verification, Enhancement, and Testing

A Git server (GitHub, GitLab, BitBucket) sends event notifications to the Civet server. These are either a " Pull Request" or a "Push" notification. Civet then checks the database to determine what tests need to be run and marks them as ready to run. Civet clients, running on dedicated machines, query the server for available jobs that are ready to run. When a client gets a job it executes the scripts attached to the job and report back to the server the output and exit status. When the client updates the server, the server will also update the Git server with the result of the job, as well as updating the main web page.

Alger, Brian↗

Geospatial-temporal semantic graph representations of trajectories from remote sensing and geolocation data

Various technologies for facilitating analysis of large remote sensing and geolocation datasets to identify features of interest are described herein. A search query can be submitted to a computing system that executes searches over a geospatial temporal semantic (GTS) graph to identify features of interest. The GTS graph comprises nodes corresponding to objects described in the remote sensing and geolocation datasets, and edges that indicate geospatial or temporal relationships between pairs of nodes in the nodes. Trajectory information is encoded in the GTS graph by the inclusion of movable nodes to facilitate searches for features of interest in the datasets relative to moving objects such as vehicles.

58 GEOSCIENCES↗

SchemaOnRead: A Package for Schema-on-Read in R

Schema-on-read is an agile approach to data storage and retrieval that defers investments in data organization until production queries need to be run by working with data directly in native form. Schema-on-read functions have been implemented in a wide range of analytical systems, most notably Hadoop. SchemaOnRead is a CRAN package that uses R’s flexible data representations to provide transparent and convenient support for the schema-on-read paradigm in R. The schema-on- read tools within the package include a single function call that recursively reads folders with text, comma separated value, raster image, R data, HDF5, NetCDF, spreadsheet, Weka, Epi Info, Pajek network, R network, HTML, SPSS, Systat, and Stata files. The provided tools can be used as-is or easily adapted to implement customized schema-on-read tool chains in R. This paper’s contribution is that it introduces and describes SchemaOnRead, the first R package specifically focused on providing explicit schema-on-read support in R.

Data Science↗

SchemaOnRead: A Package for Schema-on-Read in R

Schema-on-read is an agile approach to data storage and retrieval that defers investments in data organization until production queries need to be run by working with data directly in native form. Schema-on-read functions have been implemented in a wide range of analytical systems, most notably Hadoop. SchemaOnRead is a CRAN package that uses R’s flexible data representations to provide transparent and convenient support for the schema-on-read paradigm in R. The schema-on- read tools within the package include a single function call that recursively reads folders with text, comma separated value, raster image, R data, HDF5, NetCDF, spreadsheet, Weka, Epi Info, Pajek network, R network, HTML, SPSS, Systat, and Stata files. The provided tools can be used as-is or easily adapted to implement customized schema-on-read tool chains in R. This paper’s contribution is that it introduces and describes SchemaOnRead, the first R package specifically focused on providing explicit schema-on-read support in R.

Data Science↗

Synthetic Transaction Test Generator for Websites

Python script for querying a list of web sites and their details from Splunk and dynamically creating tests for monitoring uptime. The data generated from this script is then sent back to Splunk for creating reports and alerts.

Source record↗

Using Map Service API for Driving Cycle Detection for Wearable GPS Data: Preprint

Following advancements in smartphone and portable global positioning system (GPS) data collection, wearable GPS data have realized extensive use in transportation surveys and studies. The task of detecting driving cycles (driving or car-mode trajectory segments) from wearable GPS data has been the subject of much research. Specifically, distinguishing driving cycles from other motorized trips (such as taking a bus) is the main research problem in this paper. Many mode detection methods only focus on raw GPS speed data while some studies apply additional information, such as geographic information system (GIS) data, to obtain better detection performance. Procuring and maintaining dedicated road GIS data are costly and not trivial, whereas the technical maturity and broad use of map service application program interface (API) queries offers opportunities for mode detection tasks. The proposed driving cycle detection method takes advantage of map service APIs to obtain high-quality car-mode API route information and uses a trajectory segmentation algorithm to find the best-matched API route. The car-mode API route data combined with the actual route information, including the actual mode information, are used to train a logistic regression machine learning model, which estimates car modes and non-car modes with probability rates. The experimental results show promise for the proposed method's ability to detect vehicle mode accurately.

33 ADVANCED PROPULSION SYSTEMS↗

DEEP: Database of Energy Efficiency Performance

A database of energy efficiency performance (DEEP) is a presimulated database to enable quick and accurate assessment of energy retrofit of commercial buildings. DEEP was compiled from results of about 10 million EnergyPlus simulations. DEEP provides energy savings for screening and evaluation of retrofit measures targeting the small and medium-sized office and retail buildings in California. The prototype building models are developed for a comprehensive assessment of building energy performance based on DOE commercial reference buildings and the California DEER [sic] prototype buildings. The prototype buildings represent seven building types across six vintages of constructions and 16 California climate zones. DEEP uses these prototypes to evaluate energy performance of about 100 energy conservation measures covering envelope, lighting, heating, ventilation, air conditioning, plug loads, and domestic hot war. DEEP consists the energy simulation results for individual retrofit measures as well as packages of measures to consider interactive effects between multiple measures. The large scale EnergyPlus simulations are being conducted on the super computers at the National Energy Research Scientific Computing Center (NERSC) of Lawrence Berkeley National Laboratory. The pre-simulation database is a part of the CEC PIER project to develop a web-based retrofit toolkit for small and medium-sized commercial buildings in California, which provides real-time energy retrofit feedback by querying DEEP with recommended measures, estimated energy savings and financial payback period based on users' decision criteria of maximizing energy savings, energy cost savings, carbon reduction, or payback of investment. The pre-simulated database and associated comprehensive measure analysis enhances the ability to performance assessments of retrofits to reduce energy use for small and medium buildings and business owners who typically do not have resources to conduct costly building energy audit.

Hong, Tianzhen↗

Considerations on the Use of Custom Accelerators for Big Data Analytics

Accelerators, including Graphic Processing Units (GPUs) for gen- eral purpose computation, many-core designs with wide vector units (e.g., Intel Phi), have become a common component of many high performance clusters. The appearance of more stable and reliable tools tools that can automatically convert code written in high-level specifications with annotations (such as C or C++) to hardware de- scription languages (High-Level Synthesis - HLS), is also setting the stage for a broader use of reconfigurable devices (e.g., Field Pro- grammable Gate Arrays - FPGAs) in high performance system for the implementation of custom accelerators, helped by the fact that new processors include advanced cache-coherent interconnects for these components. In this chapter, we briefly survey the status of the use of accelerators in high performance systems targeted at big data analytics applications. We argue that, although the progress in the use of accelerators for this class of applications has been sig- nificant, differently from scientific simulations there still are gaps to close. This is particularly true for the ”irregular” behaviors exhibited by no-SQL, graph databases. We focus our attention on the limits of HLS tools for data analytics and graph methods, and discuss a new architectural template that better fits the requirement of this class of applications. We validate the new architectural templates by mod- ifying the Graph Engine for Multithreaded System (GEMS) frame- work to support accelerators generated with such a methodology, and testing with queries coming from the Lehigh University Benchmark (LUBM). The architectural template enables better supporting the task and memory level parallelism present in graph methods by sup- porting a new control model and a enhanced memory interface. We show that out solution allows generating parallel accelerators, pro- viding speed ups with respect to conventional HLS flows. We finally draw conclusions and present a perspective on the use of reconfig- urable devices and Design Automation tools for data analytics.

Data Analytics↗

Forming IDEAS: Interactive Data Exploration & Analysis System

Modern cyber security operations collect an enormous amount of logging and alerting data. While analysts have the ability to query and compute simple statistics and plots from their data, current analytical tools are too simple to admit deep understanding. To detect advanced and novel attacks, analysts turn to manual investigations. While commonplace, current investigations are time-consuming, intuition-based, and proving insufficient. Our hypothesis is that arming the analyst with easy-to-use data science tools will increase their work efficiency, provide them with the ability to resolve hypotheses with scientific inquiry of their data, and support their decisions with evidence over intuition. To this end, we present our work to build IDEAS (Interactive Data Exploration and Analysis System). We present three real-world use-cases that drive the system design from the algorithmic capabilities to the user interface. Finally, a modular and scalable software architecture is discussed along with plans for our pilot deployment with a security operation command.

Bridges, Robert↗

Towards Exascale Scientific Metadata Management

Advances in technology and computing hardware are enabling scientists from all areas of science to produce massive amounts of data using large-scale simulations or observational facilities. In this era of data deluge, effective coordination between the data production and the analysis phases hinges on the availability of metadata that describe the scientific datasets. Existing workflow engines have been capturing a limited form of metadata to provide provenance information about the identity and lineage of the data. However, much of the data produced by simulations, experiments, and analyses still need to be annotated manually in an ad hoc manner by domain scientists. Systematic and transparent acquisition of rich metadata becomes a crucial prerequisite to sustain and accelerate the pace of scientific innovation. Yet, ubiquitous and domain-agnostic metadata management infrastructure that can meet the demands of extreme-scale science is notable by its absence. To address this gap in scientific data management research and practice, we present our vision for an integrated approach that (1) automatically captures and manipulates information-rich metadata while the data is being produced or analyzed and (2) stores metadata within each dataset to permeate metadata-oblivious processes and to query metadata through established and standardized data access interfaces. We motivate the need for the proposed integrated approach using applications from plasma physics, climate modeling and neuroscience, and then discuss research challenges and possible solutions.

97 MATHEMATICS AND COMPUTING↗

Computing erasure metadata and data layout prior to storage using a processing platform

Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

System and method of storing and analyzing information

A system and method of storing and analyzing information is disclosed. The system includes a compiler layer to convert user queries to data parallel executable code. The system further includes a library of multithreaded algorithms, processes, and data structures. The system also includes a multithreaded runtime library for implementing compiled code at runtime. The executable code is dynamically loaded on computing elements and contains calls to the library of multithreaded algorithms, processes, and data structures and the multithreaded runtime library.

97 MATHEMATICS AND COMPUTING↗

EnergyPlus Performance Improvements via JSON Input Refactoring

EnergyPlus is the flagship whole-building energy simulation program developed by the U.S. Department of Energy. This paper describes the refactoring efforts around the EnergyPlus input processor to natively support JavaScript Object Notation (JSON), allow validation through use of a JSON schema validator, convert and validate typical EnergyPlus input data file (*.idf) files to JSON files, and improve related metrics such as: read time for schema and input files, parse time for processing input, and time to query/return data from EnergyPlus internal data structure. Performance improvements on sample files result in 14% and 60% reduction in schema and input file reads respectively, 36-62% reduction to process input, and 24-99+% query reduction for functions that take advantage of the new format. For the files tested, overall speedup realized as part of this JSON refactor is 1.6x to 5.4x.

Adams, Mark↗

Memory synchronization filter

Data synchronization between memories of a data processing system is achieved by transferring the data blocks from a first memory to a second memory, forming a hash list from addresses of data blocks that are written to the second memory or modified in the second memory. The hash list may be to identify a set of data blocks that are possibly written to or modified. Data blocks that are possibly modified may be written back from the second memory to the first memory in response to a synchronization event. The hash list may be updated by computing, in hardware or software, hash functions of an address of the transferred or modified data block to determine bit positions to be set. The hash list may be queried by computing hash functions of an address to determine bit positions, and checking bits in the hash list at those bit positions.

97 MATHEMATICS AND COMPUTING↗

An End-to-end Deep Convolutional Neural Network for a Multi-scale Image Matching and Localization Problem

Abstract Diverse imaging techniques are utilized in many scientific domains to acquire a rich description of the subject under study and to further discover a variety of its proper- ties. Especially, in sample systems, by probing with optical, electron or x-ray beams, the captured images describe the sample in an extremely large range of length scales. This makes the correlation from one image to another very difficult in addition to the intrinsic appearance complexity of those scientific images. In this paper, we aim to tackle this multi-scale image matching and localization problem by proposing an end-to-end deep convolutional neural net- work. Our proposed network is designed to first generate different filters according to the two queried images originated from different length scales. Then, to compute the correlation map, we use these filters to predict the correspondence between the two images. For the training and evaluation, we collect a number of electron microscopy experiments to form a multi-scaled image patch dataset comprised of various material structures. We observe about 90% accuracy for multi-scale image matching and localization while a triplet-based network shows about 78% accuracy.

97 MATHEMATICS AND COMPUTING↗

Fascinating Reporting with Postgres psql and sendmail

Among many things, psql, the Postgres command line tool, can create decent output to share with coworkers and customers, and that is where many Postgres users stop. The extreme limits of psql's formatting will be explored, delving into Unicode and HTML reports and the backslash/meta commands. Pretty reports are useless unless good data can be reported. The past and current developers of Postgres have packed a wonderful set functionality into the query language such as array aggregation and un-nesting. The staggering usefulness of "DISTINCT ON ()" will also be shown (without stepping into the quagmire of window functions). Environmentally conscious developers don't want to dump endless reports onto paper, so the power of psql is paired with the power of the UNIX/Linux sendmail command to generate electronic reports to send to coworkers and customers. The icing on the cake will be an example set of simple database tables to manage recipients of an automated reporting system.

97 MATHEMATICS AND COMPUTING↗

AGR-5/6/7 Irradiation Monitoring Web Site Design

The Nuclear Data Management and Analysis System (NDMAS) manages irradiation experiment data collected under the Advanced Gas Reactor (AGR) Fuel Development and Qualification program within the Advanced Reactor Technologies Technology Development Office. The AGR-5/6/7 irradiation experiment combines the fifth, sixth, and seventh planned irradiation campaigns of the U.S. Department of Energy AGR Fuel Development and Qualification Program. These combined irradiation tests will consist of fuel qualification tests (AGR-5/6) and a margin test (AGR-7) for the industrially produced uranium oxycarbide (UCO) tristructural isotropic (TRISO) coated particle fuel developed by the AGR program. The objective of the AGR-5/6/7 irradiation experiment is to establish the performance of UCO TRISO fuel fabricated using engineering-scale equipment in a production environment under and beyond normal operating conditions (PNL-5245). Experimental irradiation measurements for this experiment (i.e., thermocouple temperature, gas flow rate, fission product release rate, gross gamma count, and Advanced Test Reactor (ATR) operational parameters) are captured automatically to the NDMAS database. Data derived from numerical simulation or other mathematical analyses (i.e., fast fluence, burnup, temperature, and fission product release to birth ratio) are captured to the database when provided by the researchers. The NDMAS database is a relational database accessed via structured query language and managed with Microsoft SQL Server. While select users can directly access the database, that is not the most convenient way to interact with the data. The NDMAS team constructs numerous tables that combine related datasets, and delivers those data to researchers via graphs and spreadsheets on a Microsoft SharePoint-based website. For the AGR-5/6/7 experimental data, a new subsite has been developed, along with a set of web pages. These pages will be used by members of fuel irradiation team for monitoring irradiation conditions according to specification (SPC-1749), by the physics modelers to access necessary model inputs, and by the nuclear engineering community to view and access data such as fuel temperatures, fission product release from capsules, and ATR operational parameters. The deliverable represented in this document is a SharePoint website containing tables of the AGR-5/6/7 measured and calculated data, graphs displaying the data, and links to source documents and references.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale

The Adaptable I/O System (ADIOS) represents the culmination of substantial investment in Scientific Data Management, and it has demonstrated success for several important extreme-scale science cases. However, looking towards the exascale and beyond, we see the development of yet more stringent data management requirements that require new abstractions. Therefore, there is an opportunity to attempt to connect the traditional realms of HPC I/O optimization with the Database / Data Management community. As such, in this paper we offer some specific examples from our ongoing work in managing data structures, services, and performance at the extreme scale for scientific computing. Using the publish/subscribe model afforded by ADIOS, we demonstrate a set of services that connect data format, metadata, queries, data reduction, and high-performance delivery. The resulting publish/subscribe framework facilitates connection to on-line workflow systems to enable the dynamic capabilities that will be required for exascale science.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗