SEARCH · Engineering Papers

Results for “query generation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11

All sources NASA NTRS DOE OSTI

NASA NTRS1979

Development of Integrated Programs for Aerospace-vehicle design (IPAD): Integrated information processing requirements

The engineering-specified requirements for integrated information processing by means of the Integrated Programs for Aerospace-Vehicle Design (IPAD) system are presented. A data model is described and is based on the design process of a typical aerospace vehicle. General data management requirements are specified for data storage, retrieval, generation, communication, and maintenance. Information management requirements are specified for a two-component data model. In the general portion, data sets are managed as entities, and in the specific portion, data elements and the relationships between elements are managed by the system, allowing user access to individual elements for the purpose of query. Computer program management requirements are specified for support of a computer program library, control of computer programs, and installation of computer programs into IPAD.

Southall, J. W.↗

NASA NTRS1992

Generation and Exploitation of Aggregation Abstractions for Scheduling and Resource Allocation

Our research is investigating abstraction of computational theories for scheduling and resource allocation. These theories are represented in a variant of first order predicate calculus, parameterized multisorted logic, that facilitates specification of large problems. A particular problem is conceptually stated as a set of ground sentences that are consistent with a quantified theory. We are mainly investigating the automated generation of aggregation abstractions and approximations in which detailed resource allocation constraints are replaced by constraints between aggregate demand and capacity. We are also investigating the interaction of aggregation abstractions with the more thoroughly investigated abstractions of weakening operator preconditions. The purpose of the theories for aggregated demand/capacity is threefold: first, to answer queries about aggregate properties, such as gross feasibility; second, to reduce computational costs by using the solution of aggregate problems to guide the solution of detailed problems; and third, to facilitate reformulating theories to approximate problems for which there are efficient problem solving methods. We also describe novel methods for exploiting aggregation abstractions.

Linden, Theodore A.↗

DOE OSTI2022

Toward designing effective exascale scientific computing workflows: experiences and best practices

Many fields within scientific computing have embraced advances in big-data analysis and machine learning, which often requires the deployment of large, distributed and complicated workflows that may combine training neural networks, performing simulations, running inference, and performing database queries and data analysis in asynchronous, parallel and pipelined execution frameworks. Such a shift has brought into focus the need for scalable, efficient workflow management solutions with reproducibility, error and provenance handling, traceability, and checkpoint-restart capabilities, among other needs. Here, we discuss challenges and best-practices for deploying exascale-generation computational science workflows on resources at the Oak Ridge Leadership Computing Facility (OLCF). We present our experiences with large-scale deployment of distributed workflows on the Summit supercomputer, including for bioinformatics and computational biophysics, materials science, and deep learning model optimization. We also present problems and solutions created by working within a Python-centric software base on traditional HPC systems, and discuss steps that will be required before the convergence of HPC, AI, and data science can be fully realized. Our results point to a wealth of exciting new possibilities for harnessing this convergence to tackle new scientific challenges.

Coletti, Mark↗

DOE OSTI2022

Distributionally Robust Decision Making Leveraging Conditional Distributions

Distributionally robust optimization (DRO) is a powerful tool for decision making under uncertainty. It is particularly appealing because of its ability to leverage existing data. However, many practical problems call for decision-making with some auxiliary information, and DRO in the context of conditional distributions is not straightforward. We propose a conditional kernel distributionally robust optimization (CKDRO) method that enables robust decision making under conditional distributions through kernel DRO and the conditional mean operator in the reproducing kernel Hilbert space (RKHS). In particular, we consider problems where there is a correlation between the unknown variable y and an auxiliary observable variable x. Given past data of the two variables and a queried auxiliary variable, CKDRO represents the conditional distribution P(y|x) as the conditional mean operator in the RKHS space and quantifies the ambiguity set in the RKHS as well, which depends on the size of the dataset as well as the query point. To justify the use of RKHS, we demonstrate that the ambiguity set defined in RKHS can be viewed as a ball under a metric that is similar to the Wasserstein metric. The DRO is then dualized and solved via a finite dimensional convex program. The proposed CKDRO approach is applied to a generation scheduling problem and shows that the result of CKDRO is superior to common benchmarks in terms of quality and robustness.

Chen, Yuxiao↗

NASA NTRSDate not supplied

Web-based Visualization and Analytics of Petascale Data: Equity as a Tide that Lifts All Boats

Scientists generate petabytes of data daily to help uncover environmental trends or behaviors that are hard to predict. For example, understanding climate simulations based on the long-term average of temperature, precipitation, and other environmental variables is essential to predicting and establishing root causes of future undesirable scenarios and assessing possible mitigation strategies. While supercomputer centers provide a powerful infrastructure for generating petabytes of simulation output, accessing and analyzing these datasets interactively remains challenging on multiple fronts. This paper presents an approach to managing, visualizing, and analyzing petabytes of data within a browser on equipment ranging from the top NASA supercomputer to commodity hardware like a laptop. Our novel data fabric abstraction layer allows user-friendly querying of scientific information while hid-ing the complexities of dealing with file systems or cloud services.We also optimize network utilization while streaming from petas-cale repositories through state-of-the-art progressive compression algorithms. Based on this abstraction, we provide customizable dashboards that can be accessed from any device with any inter-net connection, enabling interactive visual analysis of vast amounts of data to a wide range of users - from top scientists with access to leadership-class computing environments to undergraduate students of disadvantaged backgrounds from minority-serving institutions. We focus on NASA’s use of petascale climate datasets as an example of particular societal impact and, therefore, a case where achieving equity in science participation is critical. We validate our approach by improving the ability of climate scientists to visually explore their data via two fully interactive dashboards. We further validate our approach by deploying the dashboards and simplified training materials in the classroom at a minority-serving institution.These dashboards, released in simplified form to the general public, contribute significantly to a broader push to democratize the access and use of climate data.

Data visualization↗

DOE OSTI2019

Fascinating Reporting with Postgres psql and sendmail

Among many things, psql, the Postgres command line tool, can create decent output to share with coworkers and customers, and that is where many Postgres users stop. The extreme limits of psql's formatting will be explored, delving into Unicode and HTML reports and the backslash/meta commands. Pretty reports are useless unless good data can be reported. The past and current developers of Postgres have packed a wonderful set functionality into the query language such as array aggregation and un-nesting. The staggering usefulness of "DISTINCT ON ()" will also be shown (without stepping into the quagmire of window functions). Environmentally conscious developers don't want to dump endless reports onto paper, so the power of psql is paired with the power of the UNIX/Linux sendmail command to generate electronic reports to send to coworkers and customers. The icing on the cake will be an example set of simple database tables to manage recipients of an automated reporting system.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2010

Fast and Flexible Multivariate Time Series Subsequence Search

Multivariate Time-Series (MTS) are ubiquitous, and are generated in areas as disparate as sensor recordings in aerospace systems, music and video streams, medical monitoring, and financial systems. Domain experts are often interested in searching for interesting multivariate patterns from these MTS databases which often contain several gigabytes of data. Surprisingly, research on MTS search is very limited. Most of the existing work only supports queries with the same length of data, or queries on a fixed set of variables. In this paper, we propose an efficient and flexible subsequence search framework for massive MTS databases, that, for the first time, enables querying on any subset of variables with arbitrary time delays between them. We propose two algorithms to solve this problem (1) a List Based Search (LBS) algorithm which uses sorted lists for indexing, and (2) a R*-tree Based Search (RBS) which uses Minimum Bounding Rectangles (MBR) to organize the subsequences. Both algorithms guarantee that all matching patterns within the specified thresholds will be returned (no false dismissals). The very few false alarms can be removed by a post-processing step. Since our framework is also capable of Univariate Time-Series (UTS) subsequence search, we first demonstrate the efficiency of our algorithms on several UTS datasets previously used in the literature. We follow this up with experiments using two large MTS databases from the aviation domain, each containing several millions of observations. Both these tests show that our algorithms have very high prune rates (>99%) thus needing actual disk access for only less than 1% of the observations. To the best of our knowledge, MTS subsequence search has never been attempted on datasets of the size we have used in this paper.

Bhaduri, Kanishka↗

NASA NTRS2010

Fast Multivariate Search on Large Aviation Datasets

Bhaduri, Kanishka↗

DOE OSTI2017

Polynomial chaos representation of databases on manifolds

Characterizing the polynomial chaos expansion (PCE) of a vector-valued random variable with probability distribution concentrated on a manifold is a relevant problem in data-driven settings. The probability distribution of such random vectors is multimodal in general, leading to potentially very slow convergence of the PCE. In this paper, we build on a recent development for estimating and sampling from probabilities concentrated on a diffusion manifold. The proposed methodology constructs a PCE of the random vector together with an associated generator that samples from the target probability distribution which is estimated from data concentrated in the neighborhood of the manifold. The method is robust and remains efficient for high dimension and large datasets. The resulting polynomial chaos construction on manifolds permits the adaptation of many uncertainty quantification and statistical tools to emerging questions motivated by data-driven queries.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2018

GeneLab: Omics Database for Spaceflight Experiments

Motivation - To curate and organize expensive spaceflight experiments conducted aboard space stations and maximize the scientific return of investment, while democratizing access to vast amounts of spaceflight related omics data generated from several model organisms. Results - The GeneLab Data System (GLDS) is an open access database containing fully coordinated and curated "omics" (genomics, transcriptomics, proteomics, metabolomics) data, detailed metadata and radiation dosimetry for a variety of model organisms. GLDS is supported by an integrated data system allowing federated search across several public bioinformatics repositories. Archived datasets can be queried using full-text search (e.g., keywords, Boolean and wildcards) and results can be sorted in multifactorial manner using assistive filters. GLDS also provides a collaborative platform built on GenomeSpace for sharing files and analyses with collaborators. It currently houses 172 datasets and supports standard guidelines for submission of datasets, MIAME (for microarray), ENCODE Consortium Guidelines (for RNA-seq) and MIAPE Guidelines (for proteomics).

omics↗

DOE OSTI2018

Computing erasure metadata and data layout prior to storage using a processing platform

Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2025

A Flang Plugin for Fortran Feature Characterization

As new compute systems are developed, there is still a need to compile and execute codes authored in Fortran on these leading edge systems. In order to achieve this, development of compilers that support the latest hardware is continuously under development. Though the specification of Fortran is extensive, it is helpful to compiler authors to be able to prioritize the development of key features in order to get certain codes deemed important, e.g., applications of interest to leadership computing facilities, executable on leading edge compute systems. Identifying key features though is largely done through querying software experts or users of the Fortran applications of interest, who then manually report what features are and are not present. This exercise can both time consuming and error prone. To automate this process, we present a compiler plugin to Flang, the Fortran frontend for LLVM. This plugin is a tool that operates on the parse tree representation generated by Flang and detects key features based on walking parse tree nodes that correspond to features of interest. We show the result of our tool on four applications, three of which were manually profiled by software experts. We show the discrepancies between our tool and the manual characterization of the three applications, as well as generate a characterization for an application not yet profiled. We intend to open-source our tool in order to invite the community to benefit from the tool and make contributions for other features.

Cabrera, Anthony [ORNL]↗

NASA NTRS1989

The development of a program analysis environment for Ada

A unit level, Ada software module testing system, called Query Utility Environment for Software Testing of Ada (QUEST/Ada), is described. The project calls for the design and development of a prototype system. QUEST/Ada design began with a definition of the overall system structure and a description of component dependencies. The project team was divided into three groups to resolve the preliminary designs of the parser/scanner: the test data generator, and the test coverage analyzer. The Phase 1 report is a working document from which the system documentation will evolve. It provides history, a guide to report sections, a literature review, the definition of the system structure and high level interfaces, descriptions of the prototype scope, the three major components, and the plan for the remainder of the project. The appendices include specifications, statistics, two papers derived from the current research, a preliminary users' manual, and the proposal and work plan for Phase 2.

Brown, David B.↗

NASA NTRS2021

HILIGT, Upper Limit Servers II - Implementing the Data Servers

The High-Energy Light curve Generator(HILIGT) is a new web-based tool which allows the user to generate long-term light curves of X-ray sources. It provides historical data and calculates upper limits from image data in real-time. HILIGT utilizes data from twelve satellites, both modern missions such as XMM-Newton and Swift, and earlier facilities such as ROSAT, EXOSAT, Einstein or Ariel V. Together, this enables the user to query 50 years of X-ray data and, for instance, study outburst behavior of transient sources. In this paper we focus on the individual back-end servers for each satellite, detailing the software layout, database design, catalog calls, and image footprints. We compile all relevant calibration information of these missions and provide an in-depth summary of the details of X-ray astronomical instrumentation and data.

Ole Konig↗

DOE OSTI2026

Expanding Access to Science Participation: A FAIR Framework for Petascale Data Visualization and Analytics

The massive data generated by scientists daily serve as both a major catalyst for new discoveries and innovations, as well as a significant roadblock that restricts access to the data. Here, our paper introduces a new approach to removing Big Data barriers and democratizing access to petascale data for the broader scientific community. Our novel data fabric abstraction layer allows user-friendly querying of scientific information while hiding the complexities of dealing with file systems or cloud services. We enable FAIR (Findable, Accessible, Interoperable, and Reusable) access to datasets such as NASA’s petascale climate datasets. Our paper presents an approach to managing, visualizing, and analyzing petabytes of data within a browser on equipment ranging from the top NASA supercomputer to commodity hardware like a laptop. Our novel data fabric abstraction utilizes state-of-the art progressive compression algorithms and machine-learning insights to power scalable visualization dashboards for petascale data. The result provides users with the ability to identify extreme events or trends dynamically, expanding access to scientific data and further enabling discoveries. We validate our approach by improving the ability of climate scientists to visually explore their data via three fully interactive dashboards. We further validate our approach by deploying the dashboards and simplified training materials in the classroom at a minority-serving institution. These dashboards, released in simplified form to the general public, contribute significantly to a broader push to democratize the access and use of climate data.

Computer science↗

DOE OSTI2022

Surfactant

Surfactant is a tool to generate an SBOM for the files in a directory/system based on information contained in various file format, including but not limited to PE, ELF, Mach-O, Java classes, .NET assemblies, and various source code files/scripts. Gathered information can be output in a variety of different SBOM formats, and functionality is provided to manipulate and analyze SBOMs (merging, manual additions, querying info, etc). Users are able to add new features via a plugin system.

Lloyd, LeviJ↗

DOE OSTI2019

Developing a Map Catalog to Display Water Right Information from Washington State Department of Ecology Database

The procedure for developing an ArcGIS Map catalog is described using Data Driven Pages ArcGIS extension for depicting wells and associated water rights by township and range in eastern Washington intersecting Franklin, Adams, Benton, Walla Walla, Yakima, and Klickitat counties. A series of queries against a geodatabase were written to distill the most relevant information by each well for mapping purposes. A labeling system was designed using the advanced Maplex Label Engine to account for the dense spatial distribution of wells in certain areas, and to ensure this information was legible at mapped extents and zoom level. A total of 110 maps were generated.

58 GEOSCIENCES↗

DOE OSTI2019

Toward a Better Understanding of Species Interactions through Network Biology

Within the last decade, there has been an explosion of multi-omics data generated for several microbial systems. At the same time, new methods of analysis have emerged that are based on inferring networks that link features both within and between species based on correlation in abundance. These developments prompt two important questions. What can be done with network approaches to better understand microbial species interactions? What challenges remain in applying network approaches to query the more complex systems of natural settings? Here, I briefly describe what has been done and what questions still need to be answered. Over the next 5 to 10 years, we will be in a strong position to infer networks that contain multiple kinds of omic data and describe systems with multiple species. These applications will open the door for a better understanding and use of microbiomes across a variety of fields.

Microbiology↗