Engineering Papers⌕ Search

Engineering topics

Chou, Jerry

Publications and source records attributed to Chou, Jerry.

5 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

Optimizing the query performance of block index through data analysis and I/O modeling

Indexing technique has become an efficient tool to enable scientists to directly access the most relevant data records. But, the time and space requirements of building and storing indexes are expensive in the traditional approaches, such as R-tree and bitmaps. Recently, we started to address this issue by using the idea of "block index", and our previous work has shown promising results from comparing it against other well-known solutions, including ADIOS, SciDB, and FastBit. In this work, we further improve the technique from both theoretical and implementation perspectives. Driven by an extensive effort in characterizing scientific datasets and modeling I/O systems, we presented a theoretical model to analyze its query performance with respect to a given block size configuration. We also introduced three optimization techniques to achieve a 2.3x query time reduction comparing to the original implementation.

Wu, Tzuhsien↗
DOE OSTI2012

Big Data Analaysis and Visualization: What Do LINACS and Tropical Storms Have In Common?

While there is wisdom in the old adage "the two constants in life are death and taxes,'' there are unavoidable truths facing modern experimental and computational science. First is the growing "impedance mismatch'' between our ability to collect and generate data, and our ability to store, manage, and gain understanding from it. The second is the fact that we cannot continue to rely on the same software technologies that have worked well for the past couple of decades for data management, analysis, and visualization. A third is that these complementary activities must be considered in a holistic, rather than Balkanized way. The inseparable interplay between data management, analysis, visualization, and high performance computational infrastructure, are best viewed through the lens of case studies from multiple scientific domains, where teams of computer and scientists combine forces to tackle challenging data understanding problems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2011

FastQuery: A Parallel Indexing System for Scientific Data

Modern scientific datasets present numerous data management and analysis challenges. State-of-the- art index and query technologies such as FastBit can significantly improve accesses to these datasets by augmenting the user data with indexes and other secondary information. However, a challenge is that the indexes assume the relational data model but the scientific data generally follows the array data model. To match the two data models, we design a generic mapping mechanism and implement an efficient input and output interface for reading and writing the data and their corresponding indexes. To take advantage of the emerging many-core architectures, we also develop a parallel strategy for indexing using threading technology. This approach complements our on-going MPI-based parallelization efforts. We demonstrate the flexibility of our software by applying it to two of the most commonly used scientific data formats, HDF5 and NetCDF. We present two case studies using data from a particle accelerator model and a global climate model. We also conducted a detailed performance study using these scientific datasets. The results show that FastQuery speeds up the query time by a factor of 2.5x to 50x, and it reduces the indexing time by a factor of 16 on 24 cores.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2011

Parallel Index and Query for Large Scale Data Analysis

Modern scientific datasets present numerous data management and analysis challenges. State-of-the-art index and query technologies are critical for facilitating interactive exploration of large datasets, but numerous challenges remain in terms of designing a system for process- ing general scientific datasets. The system needs to be able to run on distributed multi-core platforms, efficiently utilize underlying I/O infrastructure, and scale to massive datasets. We present FastQuery, a novel software framework that address these challenges. FastQuery utilizes a state-of-the-art index and query technology (FastBit) and is designed to process mas- sive datasets on modern supercomputing platforms. We apply FastQuery to processing of a massive 50TB dataset generated by a large scale accelerator modeling code. We demonstrate the scalability of the tool to 11,520 cores. Motivated by the scientific need to search for inter- esting particles in this dataset, we use our framework to reduce search time from hours to tens of seconds.

97 MATHEMATICS AND COMPUTING↗