Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “big data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 829 records · Page 46
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

Wide-area lustre file system using LNet routers

Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.

Rao, Nageswara↗
DOE OSTI2021

HAM: Hotspot-Aware Manager for Improving Communications with 3D-Stacked Memory

merging High-Performance Computing (HPC) workloads, such as graph analytics, machine learning, and big data science, are data-intensive. Data-intensive workloads usually present fine-grained memory accesses with limited or no data locality, and thus incur frequent cache misses and low utilization of memory bandwidth. 3D-stacked memory devices such as Hybrid Memory Cube (HMC) and High Bandwidth Memory (HBM) can provide significantly higher bandwidth than conventional memory modules. However, the traditional interfaces and optimization methods for JEDEC DDR devices do not allow to fully exploit the potential performance of 3D-stacked memory with the massive amount of irregular memory accesses of data-intensive applications. In this paper, we propose a novel Hotspot-Aware Manager (HAM) infrastructure for 3D-stacked memory devices capable of optimizing memory access streams via request aggregation, hotspot detection, and in-memory prefetching. %and an associated hotspot-aware page policy. We present the HAM design and implementation, and simulate it on a system using RISC-V embedded cores with attached HMC devices. We extensively evaluate HAM with over 12 benchmarks and applications representing diverse irregular memory access patterns. The results show that, on average, HAM reduces redundant requests by 37.51\% and increases the prefetch buffer hit rate by 4.2 times, compared to a baseline streaming prefetcher. On the selected benchmark set, HAM provides performance gains of 21.81\% in average (up to 34.28\%) and power savings of 35.07\% over a standard 3D-stacked memory.

Wang, Xi↗
DOE OSTI2020

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and planet-scale simulations. However, the lack of deep understanding on how modern GPUs can be connected and the real impact of state-of-the-art interconnect technology on multi-GPU application performance become a hurdle. In this paper, we fill the gap by conducting a thorough evaluation on five latest types of modern GPU interconnects: PCIe, NVLink-V1, NVLink-V2, NVLink-SLI and NVSwitch, from six high-end servers and HPC platforms: NVIDIA P100-DGX-1, V100-DGX-1, DGX-2, OLCF’s SummitDev and Summit supercomputers, as well as an SLI-linked system with two NVIDIA Turing RTX-2080 GPUs. Based on the empirical evaluation, we have observed four new types of GPU communication network NUMA effects: three are triggered by NVLink’s topology, connectivity and routing, while one is caused by PCIe chipset design issue. These observations indicate that, for a multi-GPU application running in a multi-GPU node, choosing the right GPU combination can impose considerable impact on GPU communication efficiency, as well as an application’s overall performance. Our evaluation can be leveraged in building practical multi-GPU performance models, which are vital for GPU task allocation, scheduling and migration in a shared environment (e.g., AI cloud and HPC centers), as well as communication-oriented performance tuning.

Performance Evaluation, GPU, Interconnect, NUMA, P↗
DOE OSTI2021

Coordinated resource allocation to plant growth–defense tradeoffs

Summary Plant resource allocation patterns often reveal tradeoffs that favor growth (G) over defense (D), or vice versa. Ecologists most often explain G–D tradeoffs through principles of economic optimality, in which negative trait correlations are attributed to the reconciliation of fitness costs. Recently, researchers in molecular biology have developed ‘big data’ resources including multi‐omic (e.g. transcriptomic, proteomic and metabolomic) studies that describe the cellular processes controlling gene expression in model species. In this synthesis, we bridge ecological theory with discoveries in multi‐omics biology to better understand how selection has shaped the mechanisms of G–D tradeoffs. Multi‐omic studies reveal strategically coordinated patterns in resource allocation that are enabled by phytohormone crosstalk and transcriptional signal cascades. Coordinated resource allocation justifies the framework of optimality theory, while providing mechanistic insight into the feedbacks and control hubs that calibrate G–D tradeoff commitments. We use the existing literature to describe the coordinated resource allocation hypothesis (CoRAH) that accounts for balanced cellular controls during the expression of G–D tradeoffs, while sustaining stored resource pools to buffer the impacts of future stresses. The integrative mechanisms of the CoRAH unify the supply‐ and demand‐side perspectives of previous G–D tradeoff theories.

Monson, Russell K.↗
DOE OSTI2016

Evolvix BEST Names for semantic reproducibility across code2brain interfaces

Names in programming are vital for understanding the meaning of code and big data. We define code2brain (C2B) interfaces as maps in compilers and brains between meaning and naming syntax, which help to understand executable code. While working toward an Evolvix syntax for general‐purpose programming that makes accurate modeling easy for biologists, we observed how names affect C2B quality. To protect learning and coding investments, C2B interfaces require long‐term backward compatibility and semantic reproducibility (accurate reproduction of computational meaning from coder‐brains to reader‐brains by code alone). Semantic reproducibility is often assumed until confusing synonyms degrade modeling in biology to deciphering exercises. We highlight empirical naming priorities from diverse individuals and roles of names in different modes of computing to show how naming easily becomes impossibly difficult. We present the Evolvix BEST (Brief, Explicit, Summarizing, Technical) Names concept for reducing naming priority conflicts, test it on a real challenge by naming subfolders for the Project Organization Stabilizing Tool system, and provide naming questionnaires designed to facilitate C2B debugging by improving names used as keywords in a stabilizing programming language. Our experiences inspired us to develop Evolvix using a flipped programming language design approach with some unexpected features and BEST Names at its core.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Condition Monitoring of Wind Turbine Planetary Gearboxes Under Different Operating Conditions

Digitally enhanced services for wind power could reduce Operations and Maintenance costs as well as the Levelised Cost Of Energy. Therefore, there is a continuous need for advanced monitoring techniques which can exploit the opportunities of Internet of Things and Big Data Analytics. The heart of wind turbines is an epicyclic gearbox and rolling element bearings are often responsible for machinery stops. The vibration signatures of bearings are rather weak compared to other components. As a result a number of signal processing techniques have been proposed, focusing towards accurate and early fault detection with limited false alarms and missed detections. In Envelope Analysis an envelope of the vibration signal is estimated usually after filtering around a frequency band. Different tools, such as Kurtogram, have been proposed to select the optimum filter parameters. Monitoring techniques have reached a mature level for steady speed and load operating conditions. On the other hand, under chaniging operating conditions, it is still unclear whether the change of the monitoring indicators is due to the change of the machinery's health or due to the change of operating parameters. Recently, the authors have proposed a new tool called IESFOgram, based on Cyclic Spectral Coherence for the automatic selection of the filtering band. In this paper, the performance of the tool is evaluated on the National Renewable Energy Laboratory wind turbine gearbox vibration condition monitoring benchmarking data set which includes various faults with different levels of diagnostic complexity as well as various speed and load operating conditions.

17 WIND ENERGY↗
DOE OSTI2019

Condition Monitoring of Wind Turbine Planetary Gearboxes Under Different Operating Conditions

Digitally enhanced services for wind power could reduce Operations and Maintenance (O&M) costs as well as the Levelised Cost Of Energy (LCOE). Therefore, there is a continuous need for advanced monitoring techniques which can exploit the opportunities of Internet of Things (IoT) and Big Data Analytics, revolutionizing the future of the energy sector. The heart of wind turbines is a rather complex epicyclic gearbox. Among others, extremely critical gearbox components which are often responsible for machinery stops are the rolling element bearings. The vibration signatures of bearings are rather weak compared to other components, such as gears, and as a result an extended number of signal processing techniques and tools have been proposed during the last decades, focusing towards accurate, early, and on time bearing fault detection with limited false alarms and missed detections. Envelope Analysis is one of the most important methodologies, where an envelope of the vibration signal is estimated usually after filtering around a frequency band excited by impacts due to the bearing faults. Different tools, such as Kurtogram, have been proposed in order to accurately select the optimum filter parameters (center frequency and bandwidth). Cyclic Spectral Correlation and Cyclic Spectral Coherence, based on Cyclostationary Analysis, have been proved as very powerful tools for condition monitoring. The monitoring techniques seem to have reached a mature level in case a machinery operates under steady speed and load. On the other hand, in case the operating conditions change, it is still unclear whether the change of the monitoring indicators is due to the change of the health of the machinery or due to the change of the operating parameters. Recently, the authors have proposed a new tool called IESFOgram, which is based on Cyclic Spectral Coherence and can automatically select the filtering band. Furthermore, the Cyclic Spectral Coherence is integrated along the selected frequency band leading to an Improved Envelope Spectrum. In this paper, the performance of the tool is evaluated and further extended on cases operating under different speeds and different loads. The effectiveness of the methodology is tested and validated on the National Renewable Energy Laboratory (NREL) wind turbine gearbox vibration condition monitoring benchmarking data set which includes various faults with different levels of diagnostic complexity as well as various speed and load operating conditions.

17 WIND ENERGY↗
DOE OSTI2015

SU-E-J-128: Two-Stage Atlas Selection in Multi-Atlas-Based Image Segmentation

Purpose: In the new era of big data, multi-atlas-based image segmentation is challenged by heterogeneous atlas quality and high computation burden from extensive atlas collection, demanding efficient identification of the most relevant atlases. This study aims to develop a two-stage atlas selection scheme to achieve computational economy with performance guarantee. Methods: We develop a low-cost fusion set selection scheme by introducing a preliminary selection to trim full atlas collection into an augmented subset, alleviating the need for extensive full-fledged registrations. More specifically, fusion set selection is performed in two successive steps: preliminary selection and refinement. An augmented subset is first roughly selected from the whole atlas collection with a simple registration scheme and the corresponding preliminary relevance metric; the augmented subset is further refined into the desired fusion set size, using full-fledged registration and the associated relevance metric. The main novelty of this work is the introduction of an inference model to relate the preliminary and refined relevance metrics, based on which the augmented subset size is rigorously derived to ensure the desired atlases survive the preliminary selection with high probability. Results: The performance and complexity of the proposed two-stage atlas selection method were assessed using a collection of 30 prostate MR images. It achieved comparable segmentation accuracy as the conventional one-stage method with full-fledged registration, but significantly reduced computation time to 1/3 (from 30.82 to 11.04 min per segmentation). Compared with alternative one-stage cost-saving approach, the proposed scheme yielded superior performance with mean and medium DSC of (0.83, 0.85) compared to (0.74, 0.78). Conclusion: This work has developed a model-guided two-stage atlas selection scheme to achieve significant cost reduction while guaranteeing high segmentation accuracy. The benefit in both complexity and performance is expected to be most pronounced with large-scale heterogeneous data.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2016

SU-F-P-34: Commission of Enhanced Dynamic Wedge of Varian Truebeam Linac System with Feature Study

Purpose: To evaluate an Enhanced Dynamic Wedge (EDW) as part of machine commission process with feature study. Methods: The EDW system in this study was from a Truebeam, which is the Linear accelerator manufactured by Varian Medical Systems. The EDW feature vectors includes selected elements. These elements were dosimetric output spots check, field size, wedge angles, dose rate, collimator orientation, and different energy settings. Point dose measurement was done by a PTW farmer chamber, and profiles were measured by Gafchromic EBT2 films positing at different depths of the Solidwater based on the study elements. The output spot measurements were done with PTW farmer chamber with Solidwater setting for all orientation and wedge angles in the EDW system. The profiles comparisons were done by IMRT measurement function in RIT software at version 6.3. And the films were scanned by Vidar scanner. Dosimetry calculation were done by using the same Solidwater scanned by GE LightSpeed CT in Eclipse Treatment Planning System (TPS). Then measurements were compared to simulation results in TPS. Results: The energy average percentage difference between chamber measurement and TPS was 0.16% with standard deviation (SD) at 0.93%. For selected features, the average percentage difference between film measurement and computation was 0.93% with SD at 1.55% in horizontal profiles, and 1.18% with SD at 0.98% at vertical profiles. The average gamma difference for film measurement and TPS computing results was at 0.924 with SD at 0.314. Conclusion: A feature vector was developed to describe the commission of EDW, and developing a complete set of features for sufficiency of commission of a LINAC function could provide optimal commission instance with acceptable confident level of clinical application of the machine. Given the institution specific vector pattern and big data process, it could provide wide range clinical outcome comparison information in application of EDW.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2016

SU-F-J-218: Predicting Radiation-Induced Xerostomia by Dosimetrically Accounting for Daily Setup Uncertainty During Head and Neck IMRT

Purpose: To determine if the accumulated parotid dosimetry using planning CT to daily CBCT deformation and dose re-calculation can predict for radiation-induced xerostomia. Methods: To track and dosimetrically account for the effects of anatomical changes on the parotid glands, we propagated physicians’ contours from planning CT to daily CBCT using a deformable registration with iterative CBCT intensity correction. A surface mesh for each OAR was created with the deformation applied to the mesh to obtain the deformed parotid volumes. Daily dose was computed on the deformed CT and accumulated to the last fraction. For both the accumulated and the planned parotid dosimetry, we tested the prediction power of different dosimetric parameters including D90, D50, D10, mean, standard deviation, min/max dose to the combined parotids and patient age to severe xerostomia (NCI-CTCAE grade≥2 at 6 mo follow-up). We also tested the dosimetry to parotid sub-volumes. Three classification algorithms, random tree, support vector machine, and logistic regression were tested to predict severe xerostomia using a leave-one-out validation approach. Results: We tested our prediction model on 35 HN IMRT cases. Parameters from the accumulated dosimetry model demonstrated an 89% accuracy for predicting severe xerostomia. Compared to the planning dosimetry, the accumulated dose consistently demonstrated higher prediction power with all three classification algorithms, including 11%, 5% and 30% higher accuracy, sensitivity and specificity, respectively. Geometric division of the combined parotid glands into superior-inferior regions demonstrated ∼5% increased accuracy than the whole volume. The most influential ranked features include age, mean accumulated dose of the submandibular glands and the accumulated D90 of the superior parotid glands. Conclusion: We demonstrated that the accumulated parotid dosimetry using CT-CBCT registration and dose re-calculation more accurately predicts for severe xerostomia and that the superior portion of the parotid glands may be particularly important in predicting for severe xerostomia. This work was supported in part by NIH/NCI under grant R42CA137886 and in part by Toshiba big data research project funds.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2016

INTEGRATION OF PANDA WORKLOAD MANAGEMENT SYSTEM WITH SUPERCOMPUTERS

Abstract The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the funda- mental nature of matter and the basic forces that shape our universe, and were recently credited for the dis- covery of a Higgs boson. ATLAS, one of the largest collaborations ever assembled in the sciences, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 140 data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data cen- ters are physically scattered all over the world. While PanDA currently uses more than 250000 cores with a peak performance of 0.3+ petaFLOPS, next LHC data taking runs will require more resources than Grid computing can possibly provide. To alleviate these challenges, LHC experiments are engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. We will describe a project aimed at integration of PanDA WMS with supercomputers in United States, Europe and Russia (in particular with Titan supercomputer at Oak Ridge Leadership Com- puting Facility (OLCF), Supercomputer at the National Research Center Kurchatov Institute , IT4 in Ostrava, and others). The current approach utilizes a modified PanDA pilot framework for job submission to the supercomputers batch queues and local data management, with light-weight MPI wrappers to run single- threaded workloads in parallel on Titan s multi-core worker nodes. This implementation was tested with a variety of Monte-Carlo workloads on several supercomputing platforms. We will present our current accom- plishments in running PanDA WMS at supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facility s infrastructure for High Energy and Nuclear Physics, as well as other data-intensive science applications, such as bioinformatics and astro-particle physics.

De, K↗
DOE OSTI2016

Verifying quantitative reliability for programs that execute on unreliable hardware

Emerging high-performance architectures are expected to contain unreliable components that may exhibit soft errors, which silently corrupt the results of computations. Full detection and masking of soft errors is challenging, expensive, and, for some applications, unnecessary. For example, approximate computing applications (such as multimedia processing, machine learning, and big data analytics) can often naturally tolerate soft errors. We introduce Rely, a programming language that enables developers to reason about the quantitative reliability of an application—namely, the probability that it produces the correct result when executed on unreliable hardware. Rely allows developers to specify the reliability requirements for each value that a function produces. We perform a static quantitative reliability analysis that verifies quantitative requirements on the reliability of an application, enabling a developer to perform sound and verified reliability engineering. The analysis takes a Rely program with a reliability specification and a hardware specification that characterizes the reliability of the underlying hardware components and verifies that the program satisfies its reliability specification when executed on the underlying unreliable hardware platform. We demonstrate the application of quantitative reliability analysis on six computations implemented in Rely.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Defense strategies for cloud computing multi-site server infrastructures

We consider cloud computing server infrastructures for big data applications, which consist of multiple server sites connected over a wide-area network. The sites house a number of servers, network elements and local-area connections, and the wide-area network plays a critical, asymmetric role of providing vital connectivity between them. We model this infrastructure as a system of systems, wherein the sites and wide-area network are represented by their cyber and physical components. These components can be disabled by cyber and physical attacks, and also can be protected against them using component reinforcements. The effects of attacks propagate within the systems, and also beyond them via the wide-area network.We characterize these effects using correlations at two levels using: (a) aggregate failure correlation function that specifies the infrastructure failure probability given the failure of an individual site or network, and (b) first-order differential conditions on system survival probabilities that characterize the component-level correlations within individual systems. We formulate a game between an attacker and a provider using utility functions composed of survival probability and cost terms. At Nash Equilibrium, we derive expressions for the expected capacity of the infrastructure given by the number of operational servers connected to the network for sum-form, product-form and composite utility functions.

Rao, Nageswara S.↗
DOE OSTI2018

High-Performance Sparse Matrix-Matrix Products on Intel KNL and Multicore Architectures

Sparse matrix-matrix multiplication (SpGEMM) is a computational primitive that is vastly used in areas ranging from traditional numerical applications to recent big data analysis and machine learning. While many SpGEMM algorithms have been proposed, hardware specific optimizations for multi- and many-core processors are lacking and a detailed analysis of their performance under various use cases and matrices is not available. We firstly identify and mitigate multiple bottlenecks with memory management and thread scheduling on Intel Xeon Phi (Knights Landing or KNL). Specifically targeting multi- and many-core processors, we develop a hash-table-based algorithm and optimize a heap-based shared-memory SpGEMM algorithm. We investigate their performance together with other publicly available codes. Different from the literature, our evaluation also includes use cases that are representative of real graph algorithms, such as multi-source breadth-first search or triangle counting. Our hash-table and heap-based algorithms are showing significant speedups from libraries in the majority of the cases while different algorithms dominate the other scenarios with different matrix size, sparsity, compression factor and operation type. We wrap up in-depth evaluation results and make a recipe to give the best SpGEMM algorithm for target scenario. In conclusion, a critical finding is that hash-table-based SpGEMM gets a significant performance boost if the nonzeros are not required to be sorted within each row of the output matrix.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Measurements and analytics of wide-area file transfers over dedicated connections

Distributed scientific and big-data computations are becoming increasingly dependent on access to remote files. Wide-area file transfers are supported by two basic schemes: (i) application-level tools, such as GridFTP, that provide transport services between file systems housed at geographically separated sites, and (ii) file systems mounted over wide-area networks, using mechanisms such as LNet routers that make them transparently available. In both cases, the file transfer performance critically depends on the configuration consisting of host, file, IO, and disk subsystems, which are complex by themselves, as well as on their complex compositions implemented using buffers and IO-network data transitions. We present extensive file transfer rate measurements collected over dedicated 10 Gbps connections with 0-366 ms round-trip times, using GridFTP and XDD file transfer tools, and Lustre file system extended over wide-area networks using LNet routers. Our test configurations are composed of: three types of host systems; XFS, Lustre, and ext3 file systems; and Ethernet and SONET wide-area connections. We present analytics based on the convexity-concavity of throughput profiles which provide insights into throughput and its superior or inferior trend compared to linear interpolations. We propose the utilization-concavity coefficient, a scalar metric that characterizes the overall performance of any file transfer method consisting of specific configuration and scheme. Our results enable performance optimizations by highlighting the significant roles of (i) buffer sizes and parallelism in GridFTP and XDD, and (ii) buffer utilization and credit mechanism in LNet routers.

Rao, Nageswara↗
DOE OSTI2019

Parsl: Pervasive Parallel Programming in Python

High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.

parallel programming↗
DOE OSTI2024

High-Throughput Computing: Case Study of Medical Image Processing Applications

HPC is designed for large-scale simulations using monolithic codes of tightly coupled processes highly optimized to deliver decreased time to solution. Medical image processing is not a traditional field of HPC. Similar to AI applications, medical image processing parses large datasets, typically multiple times, to support a variety of studies for classification, diagnosis or monitoring purposes. The convergence of AI, HPC and Big Data encouraged more fields using image processing to transition to HPC. However, not all applications benefit from the same optimizations. In this paper we focus on high throughput medical image processing applications that analyze a huge dataset of small MRI images and that require HPC systems to decrease the time of parsing the entire dataset and not individual MRIs. We show in this research the performance of running SLANT, an image processing application for a whole brain segmentation, on large-scale systems and highlight performance limitations. We present optimizations prioritizing throughput that exhibit a 3.5x speed-up on the Summit Supercomputer that can be used as a baseline for building a high-throughput execution framework for other HPC systems.

Predescu, Maria↗
DOE OSTI2020

Next-generation geospatial-temporal information technologies for disaster management

Traditional geographic information systems (GIS) have been disrupted by the emergence of Big Data in the form of geo-coded raster, vector, and time-series Internet-of-Things data. This article discusses the application of new scalable technologies that go far beyond relational databases and file-based storage on spinning disk or tape to incorporate both storage and processing data in the same platform. The roles of the Apache Hadoop Distributed File Systems and NoSQL key-value stores such as the Apache Hbase are discussed, along with indexing schemes that optimally support geospatial-temporal use. Here, we highlight how this new approach can rapidly search multiple GIS data layers to obtain insights in the context of early warning, impact evaluation, response, and recovery to earthquake and wildfire disasters.

97 MATHEMATICS AND COMPUTING↗