Engineering topics
Schuchardt, Karen
Publications and source records attributed to Schuchardt, Karen.
Toward a Data Scalable Solution for Facilitating Discovery of Science Resources
Science is increasingly motivated by the need to process larger quantities of data. It is facing severe challenges in data collection, management, and processing, so much so that the computational demands of “data scaling” are competing with, and in many fields surpassing, the traditional objective of decreasing processing time. Example domains with large datasets include astronomy, biology, genomics, climate/weather, and material sciences. This paper presents a real-world use case in which we wish to answer queries pro- vided by domain scientists in order to facilitate discovery of relevant science resources. The problem is that the metadata for these science resources is very large and is growing quickly, rapidly increasing the need for a data scaling solution. We propose a system – SGEM – designed for answering graph-based queries over large datasets on cluster architectures, and we re- port performance results for queries on the current RDESC dataset of nearly 1.4 billion triples, and on the well-known BSBM SPARQL query benchmark.
Technical Highlights from the ExaHDF5 project
Explore the source record for details and available documents.
Geodesic IO Library
GIO is an application programmer interface (API) or library for high performance IO that is designed for climate models that operate on a geodesic grid. GIO uses a combination of configuration files and API calls to define the mapping of internal model data to output variables in standard climate data formats. The mapping from model data to model output includes method of data slicing, aggregation, and linearization using a morton ordering approach. The data is then persisted through a standard interface that can be implemented using several different high level parallel libraries. Both Parallel NetCDF and NetCDF4 interfaces are provided. GIO is designed to run on supercomputing systems or multi-processor clusters with parallel file systems.