Results for “Query Tool” — Page 5

NASA NTRS1994

Multivariate statistical analysis software technologies for astrophysical research involving large data bases

We developed a package to process and analyze the data from the digital version of the Second Palomar Sky Survey. This system, called SKICAT, incorporates the latest in machine learning and expert systems software technology, in order to classify the detected objects objectively and uniformly, and facilitate handling of the enormous data sets from digital sky surveys and other sources. The system provides a powerful, integrated environment for the manipulation and scientific investigation of catalogs from virtually any source. It serves three principal functions: image catalog construction, catalog management, and catalog analysis. Through use of the GID3* Decision Tree artificial induction software, SKICAT automates the process of classifying objects within CCD and digitized plate images. To exploit these catalogs, the system also provides tools to merge them into a large, complete database which may be easily queried and modified when new data or better methods of calibrating or classifying become available. The most innovative feature of SKICAT is the facility it provides to experiment with and apply the latest in machine learning technology to the tasks of catalog construction and analysis. SKICAT provides a unique environment for implementing these tools for any number of future scientific purposes. Initial scientific verification and performance tests have been made using galaxy counts and measurements of galaxy clustering from small subsets of the survey data, and a search for very high redshift quasars. All of the tests were successful, and produced new and interesting scientific results. Attachments to this report give detailed accounts of the technical aspects for multivariate statistical analysis of small and moderate-size data sets, called STATPROG. The package was tested extensively on a number of real scientific applications, and has produced real, published results.

Djorgovski, S. George↗

NASA NTRS1994

Multivariate Statistical Analysis Software Technologies for Astrophysical Research Involving Large Data Bases

We developed a package to process and analyze the data from the digital version of the Second Palomar Sky Survey. This system, called SKICAT, incorporates the latest in machine learning and expert systems software technology, in order to classify the detected objects objectively and uniformly, and facilitate handling of the enormous data sets from digital sky surveys and other sources. The system provides a powerful, integrated environment for the manipulation and scientific investigation of catalogs from virtually any source. It serves three principal functions: image catalog construction, catalog management, and catalog analysis. Through use of the GID3* Decision Tree artificial induction software, SKICAT automates the process of classifying objects within CCD and digitized plate images. To exploit these catalogs, the system also provides tools to merge them into a large, complex database which may be easily queried and modified when new data or better methods of calibrating or classifying become available. The most innovative feature of SKICAT is the facility it provides to experiment with and apply the latest in machine learning technology to the tasks of catalog construction and analysis. SKICAT provides a unique environment for implementing these tools for any number of future scientific purposes. Initial scientific verification and performance tests have been made using galaxy counts and measurements of galaxy clustering from small subsets of the survey data, and a search for very high redshift quasars. All of the tests were successful and produced new and interesting scientific results. Attachments to this report give detailed accounts of the technical aspects of the SKICAT system, and of some of the scientific results achieved to date. We also developed a user-friendly package for multivariate statistical analysis of small and moderate-size data sets, called STATPROG. The package was tested extensively on a number of real scientific applications and has produced real, published results.

Djorgovski, S. G.↗

DOE OSTI2017

Optimizing the query performance of block index through data analysis and I/O modeling

Indexing technique has become an efficient tool to enable scientists to directly access the most relevant data records. But, the time and space requirements of building and storing indexes are expensive in the traditional approaches, such as R-tree and bitmaps. Recently, we started to address this issue by using the idea of "block index", and our previous work has shown promising results from comparing it against other well-known solutions, including ADIOS, SciDB, and FastBit. In this work, we further improve the technique from both theoretical and implementation perspectives. Driven by an extensive effort in characterizing scientific datasets and modeling I/O systems, we presented a theoretical model to analyze its query performance with respect to a given block size configuration. We also introduced three optimization techniques to achieve a 2.3x query time reduction comparing to the original implementation.

Wu, Tzuhsien↗

NASA NTRS2007

Mission Options Scoping Tool for Mars Orbiters: Mass Cost Calculator (MC2)

Prior to developing the details of an advanced mission study, the mission architecture trade space is typically explored to assess the scope of feasible options. This paper describes the main features of an Excel-based tool, called the Mass-Cost-Calculator (MC2 ), which is used to perform rapid, high-level mass and cost options analyses of Mars orbiter missions. MC2 consists of a combination of databases, analytical solutions, and parametric relationships to enable quick evaluation of new mission concepts and comparison of multiple architecture options. The tool's outputs provide program management and planning teams with answers to "what if" queries, as well as an understanding of the driving mission elements, during the pre-project planning phase. These outputs have been validated against the outputs generated by the Advanced Projects Design Team (Team X) at NASA's Jet Propulsion Laboratory (JPL). The architecture of the tool allows for future expansion to other orbiters beyond Mars, and to non-orbiter missions, such as those involving fly-by spacecraft, probes, landers, rovers, or other mission elements.

orbiters↗

DOE OSTI2026

Expandable Log Analyzing Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, IL]↗

DOE OSTI2026

Expandable Log Analyzing Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, IL]↗

DOE OSTI2026

Database-Agnostic Log Analysis and Monitoring Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, US, IL; Fermilab]↗

DOE OSTI2025

A universal language for finding mass spectrometry data patterns

Despite being information rich, the vast majority of untargeted mass spectrometry data are underutilized; most analytes are not used for downstream interpretation or reanalysis after publication. The inability to dive into these rich raw mass spectrometry datasets is due to the limited flexibility and scalability of existing software tools. Here, in this study, we introduce a new language, the Mass Spectrometry Query Language (MassQL), and an accompanying software ecosystem that addresses these issues by enabling the community to directly query mass spectrometry data with an expressive set of user-defined mass spectrometry patterns. Illustrated by real-world examples, MassQL provides a data-driven definition of chemical diversity by enabling the reanalysis of all public untargeted metabolomics data, empowering scientists across many disciplines to make new discoveries. MassQL has been widely implemented in multiple open-source and commercial mass spectrometry analysis tools, which enhances the ability, interoperability and reproducibility of mining of mass spectrometry data for the research community.

Damiani, Tito [Czech Academy of Sciences (CAS), Pr↗

DOE OSTI2022

GeN-ROM—An OpenFOAM®-based multiphysics reduced-order modeling framework for the analysis of Molten Salt Reactors

This work presents a projection-based multiphysics Model Order Reduction (MOR) framework for the analysis of nuclear systems and its application to parametric simulations of Molten Salt Reactors (MSR). The framework, named GeN-ROM, is developed using OpenFOAM® and employs a Proper Orthogonal Decomposition aided Reduced-Basis technique (POD-RB). It can be used to reduce steady-state and transient multiphysics problems involving parametric fluid dynamics, heat exchange, and neutronics phenomena. For the treatment of structural elements in the hydraulic systems, a porous medium approach has been adopted. The reduction process is data-driven and snapshot information is extracted via POD to learn the solution manifold and to build global spatial basis functions. At the data collection phase, GeN-ROM makes use of the solvers available in GeN-Foam, a similarly OpenFOAM®-based multiphysics framework developed for the analysis of nuclear reactors. The global bases are used both to approximate the solution fields and to project the full-order equations onto lower-dimensional subspaces, thus considerably reducing the number of unknowns in a numerical system. This reduction leads to significant computational speedups, which is ideal for multi-query applications such as uncertainty quantification or design optimization. The developed tool has been tested using a 2D multiphysics model of the Molten Salt Fast Reactor (MSFR) with steady-state and transient scenarios, with speedups on the order of 10 – 10 5 .

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2018

Sirepo : an open-source cloud-based software interface for X-ray source and optics simulations

Sirepo , a browser-based GUI for X-ray source and optics simulations, is presented. Such calculations can be performed using SRW ( Synchrotron Radiation Workshop ), which is a physical optics computer code, allowing simulation of entire experimental beamlines using the concept of a `virtual beamline' with accurate treatment of synchrotron radiation generation and propagation through the X-ray optical system. SRW is interfaced with Sirepo by means of a Python application programming interface. Sirepo supports most of the optical elements currently used at beamlines, including recent developments in SRW . In particular, support is provided for the simulation of state-of-the-art X-ray beamlines, exploiting the high coherence and brightness of modern light source facilities. New scientific visualization and reporting capabilities have been recently implemented within Sirepo , as well as automatic determination of electron beam and undulator parameters. Publicly available community databases can be dynamically queried for error-free access to material characteristics. These computational tools can be used for the development and commissioning of new X-ray beamlines and for testing feasibility and optimization of experiments. The same interface can guide simulation on a local computer, a remote server or a high-performance cluster. Sirepo is available online and also within the NSLS-II firewall, with a growing number of users at other light source facilities. Our open source code is available on GitHub.

99 GENERAL AND MISCELLANEOUS↗

NASA NTRS1993

CDM analysis

The C Data Manager (CDM) is an advanced tool for creating an object-oriented database and for processing queries related to objects stored in that database. The CDM source code was purchased and will be modified over the course of the Arachnid project. In this report, the modified CDM is referred to as MCDM. Using MCDM, a detailed series of experiments was designed and conducted on a Sun Sparcstation. The primary results and analysis of the CDM experiment are provided in this report. The experiments involved creating the Long-form Faint Source Catalog (LFSC) database and then analyzing it with respect to following: (1) the relationships between the volume of data and the time required to create a database; (2) the storage requirements of the database files; and (3) the properties of query algorithms. The effort focused on defining, implementing, and analyzing seven experimental scenarios: (1) find all sources by right ascension--RA; (2) find all sources by declination--DEC; (3) find all sources in the right ascension interval--RA1, RA2; (4) find all sources in the declination interval--DEC1, DEC2; (5) find all sources in the rectangle defined by--RA1, RA2, DEC1, DEC2; (6) find all sources that meet certain compound conditions; and (7) analyze a variety of query algorithms. Throughout this document, the numerical results obtained from these scenarios are reported; conclusions are presented at the end of the document.

Larson, Robert E.↗

NASA NTRS2012

Modeling Active Region Evolution - A New LWS TR and T Strategic Capability Model Suite

In 2006 the LWS TR&T Program funded us to develop a strategic capability model of slowly evolving coronal active regions. In this poster we report on the overall design, and status of our new modeling suite. Our design features two coronal field models, a non-linear force free field model and a global 3D MHD code. The suite includes supporting tools and a user friendly GUI which will enable users to query the web for relevant magnetograms, download them, process them to synthesize a sequence of photospheric magnetograms and associated photospheric flow field which can then be applied to drive the coronal model innner boundary, run the coronal models and finally visualize the results.

MacNeice, Peter↗

NASA NTRS2017

Mapping by VESGEN of Blood Vessels in the Retinas of Astronauts Pre- and Post-Flight to the ISS

Research by NASA [1] established that significant risks for visual and ocular impairments associated with increased intracranial pressure (VIIP) are incurred by microgravity spaceflight, especially long-duration missions. It is well established in physiology and pathology that a fundamental role of the microvasculature is to mediate fluid transfers and remodel actively in response to environmental, immune and other stresses. We therefore hypothesize that remodeling of retinal blood vessels necessarily occurs during accommodation of microgravity-induced fluid shifts prior to subsequent development of visual and ocular impairments. Potential contributions of retinal vascular remodeling to VIIP etiology are therefore being investigated by NASA's innovative VESsel GENeration Analysis (VESGEN) software for two studies: (1) U.S. crew members before and after ISS missions, and (2) head-down tilt in human subjects before and after 70 days of bed rest. We anticipate that results of the two studies will be complete by the Investigators Workshop (January 22, 2017). METHODS: For the 2013 NASA NRA award, we are concluding the analysis of 30 degree infrared (IR) Heidelberg Spectralis images of retinal blood vessels by VESGEN (patents pending), a mature, automated software developed as a translational and basic vascular research discovery tool, particularly for retinal vascular disease. Subjects of our retrospective study include eight ISS crew members monitored for routine occupational surveillance pre- and post-flight, who provided their study consents to NASAs Lifetime Surveillance of Astronaut Health (LSAH) in coordination with approval of the VESGEN retrospective study protocol by NASAs Institutional Review Board (IRB). The ophthalmic retinal images (average image resolution, approximately 5.6 microns per pixel) are blinded as to pre and post ISS status until the second portion of our study, when VESGEN results will be correlated with other ophthalmic and medical findings for the crew members. Due to image resolution challenges, a novel Matlab tool was developed for aligning pre and post images, and comparing (querying) the two images for differences in the morphology of small vessels. RESULTS: During the past year, LSAH approved the release of all astronaut retinal images to our study for VESGEN analysis. Substantial progress on the initial blinded portion of the study is in place. We anticipate that VESGEN analysis of the 32 Spectralis IR retinal images will be complete for presentation at the 2017 IWS meeting. CONCLUSIONS: Modified retinal vascular patterning may offer early-stage predictions of ocular changes resulting in decreased visual acuity for the VIIP syndrome. Novel insights provided by VESGEN into progressively pathological and blinding vascular remodeling in the human retina currently help to guide other NIH- and NASA-supported therapeutic studies of retinal disease and modeling of the VIIP risk. Results of our vascular investigation of the retinas of astronauts pre- and post-flight may help advance the understanding of both healthy and pathological adaptations to fluid shifts in microgravity associated with the VIIP syndrome. Preliminary results indicate that imaging of higher resolution, such as the new OCT angiography (OCT-A) technology, will be required to determine conclusively the role of the smaller retinal and choroidal vessels in VIIP etiology.

VESGEN↗

DOE OSTI2024

BEAST DB: Grand-Canonical Database of Electrocatalyst Properties

We present BEAST DB, an open-source database comprised of ab initio electrochemical data computed using grand-canonical density functional theory in implicit solvent at consistent calculation parameters. The database contains over 20,000 surface calculations and covers a broad set of heterogeneous catalyst materials and electrochemical reactions. Calculations were performed at self-consistent fixed potential as well as constant charge to facilitate comparisons to the computational hydrogen electrode. This article presents common use cases of the database to rationalize trends in catalyst activity, screen catalyst material spaces, understand elementary mechanistic steps, analyze the electronic structure, and train machine learning models to predict higher fidelity properties. Users can interact graphically with the database by querying for individual calculations to gain a granular understanding of reaction steps or by querying for an entire reaction pathway on a given material using an interactive reaction pathway tool. BEAST DB will be periodically updated, with planned future updates to include advanced electronic structure data, surface speciation studies, and greater reaction coverage.

database↗

NASA NTRS2013

A Semantic Basis for Proof Queries and Transformations

We extend the query language PrQL, designed for inspecting machine representations of proofs, to also allow transformation of proofs. PrQL natively supports hiproofs which express proof structure using hierarchically nested labelled trees, which we claim is a natural way of taming the complexity of huge proofs. Query-driven transformations enable manipulation of this structure, in particular, to transform proofs produced by interactive theorem provers into forms that assist their understanding, or that could be consumed by other tools. In this paper we motivate and define basic transformation operations, using an abstract denotational semantics of hiproofs and queries. This extends our previous semantics for queries based on syntactic tree representations.We define update operations that add and remove sub-proofs, and manipulate the hierarchy to group and ungroup nodes. We show that

Proof Queries↗

DOE OSTI2019

pCalc User's Manual

pCalc is a software tool that computes travel-time predictions, ray path geometry and model queries. This software has a rich set of features, including the ability to use custom 3D velocity models to compute predictions using a variety of geometries. The pCalc software is especially useful for research related to seismic monitoring applications. The pCalc software is available on the web at: www.sandia.gov/salsa3d/Software.html The software is packaged with this user's manual and a set of example datasets, the use of which is described in this manual.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

PCalc User's Manual

PCalc is a software tool that computes travel-time predictions, ray path geometry and model queries. This software has a rich set of features, including the ability to use custom 3D velocity models to compute predictions using a variety of geometries. The PCalc software is especially useful for research related to seismic monitoring applications.

58 GEOSCIENCES↗

NASA NTRS2012

Revamping Spacecraft Operational Intelligence with Splunk

So what is Splunk? Instead of giving the technical details, which you can find online, I'll tell you what it did for me. Splunk slapped everything into one place, with one uniform format, and gave me the ability to forget about all these annoying details of where it is, how to parse it, and all that. Instead, I only need to interact with Splunk to find the data I need. This sounds simple and obvious, but it's surprising what you can do once you all of your data is indexed in one place. By having your data organized, querying becomes much easier. Let's say that I want to search telemetry for a sensor_name gtemp_1 h and to return all data that is at most five minutes old. And because Splunk can hook into a real ]time stream, this data will always be up-to-date. Extending the previous example, I can now aggregate all types of data into one view based in time. In this picture, I've got transaction logs, telemetry, and downlinked files all in one page, organized by time. Even though the raw data looks completely than this, I've defined interfaces that transform it into this uniform format. This gives me a more complete picture for the question what was the spacecraft doing at this particular time? And because querying data is simple, I can start with a big block of data and whiddle it down to what I need, rather than hunting around for the individual pieces of data that I need. When we have all the data we need, we can begin widdling down the data with Splunk's Unix-like search syntax. These three examples highlights my trial-and-error attempts to find large temperature changes. I begin by showing the first 5 temperatures, only to find that they're sorted chronologically, rather than from highest temperatures to lowest temperatures. The next line shows sorting temperatures by their values, but I find that that fs not really what I want either. I want to know the delta temperatures between readings. Looking through Splunk's user manual, I find the delta function, which lets me dynamically generate new information to use in my query. With that extra piece of information, I can now return only the telemetry readings where the temperature changed by at least 10. One other useful feature I'll mention is that all of these queries can be run through Splunk's API. So any scripting language you can think of can plug right in and make these queries. This gives us the ability to build a lot of new tools.

operational intelligence↗