The availability of satellite data for scientific analysis
Satellite data acquisition and dissemination functions of National Space Science Data Center /NSSDC/
Vette, J. I.↗
SEARCH · Engineering Papers
Results for “Scientific Data Analysis”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
ARCHIE: Data Analysis Acceleration with Array Caching in Hierarchical Storage
© 2018 IEEE. Scientific data analysis typically involves reading massive amounts of data that was generated by simulations, experiments, and observations. Performance of reading such large volumes of data from disk-based file systems is often poor because of the slow and mechanical components in the disks. Recent supercomputing systems are adding non-volatile storage layers in a hierarchy to handle the performance gap between fast main memory and slow disk-based storage. Software libraries for managing this hierarchy not only need efficient reading of data but also reduce user-involvement for cross-layer data movement. Furthermore, these libraries need to support array data access patterns into hierarchical storage management as scientific data is often organized in array-based data structures. Existing software typically manage individual storage layers requiring significant manual process in moving data among them. In this paper, we introduce a new array caching in hierarchical storage (ARCHIE) to accelerate array data analysis in a seamless fashion. ARCHIE evaluates array access patterns and prefetches data with array semantics between storage layers. Our evaluation shows that ARCHIE outperforms state-of-the-art file systems, i.e., Lustre and DataWarp, on a production supercomputing system by up to 5.8× in accessing data by scientific analysis applications.
Dong, Bin↗
Mini-apps for high performance data analysis
Scaling-up scientific data analysis and machine learning algorithms for data-driven discovery is a grand challenge that we face today. Despite the growing need for analysis from science domains that are generating `Big Data' from instruments and simulations, building high-performance analytical workflows of data-intensive algorithms have been daunting because: (i) the `Big Data' hardware and software architecture landscape is constantly evolving, (ii) newer architectures impose new programming models, and (iii) data-parallel kernels of analysis algorithms and their performance facets on different architectures are poorly understood. To address these problems, we have: (i) identified scalable data-parallel kernels of popular data analysis algorithms, (ii) implemented `Mini-Apps' of those kernels using different programming models (e.g. Map Reduce, MPI, etc.), (iii) benchmarked and validated the performance of the kernels in diverse architectures. In this paper, we discuss two of those Mini-Apps and show the execution of principal component analysis built as a workflow of the Mini-Apps. We show that Mini-Apps enable scientists to (i) write domain-specific data analysis code that scales on most HPC hardware and (ii) and offers the ability (most times with over a 10x speed-up) to analyze data sizes 100 times the size of what off-the-shelf desktop/workstations of today can handle.
Sukumar, Sreenivas R.↗
Anticipated Changes in Conducting Scientific Data-Analysis Research in the Big-Data Era
A Big-Data environment is one that is capable of orchestrating quick-turnaround analyses involving large volumes of data for numerous simultaneous users. Based on our experiences with a prototype Big-Data analysis environment, we anticipate some important changes in research behaviors and processes while conducting scientific data-analysis research in the near future as such Big-Data environments become the mainstream. The first anticipated change will be the reduced effort and difficulty in most parts of the data management process. A Big-Data analysis environment is likely to house most of the data required for a particular research discipline along with appropriate analysis capabilities. This will reduce the need for researchers to download local copies of data. In turn, this also reduces the need for compute and storage procurement by individual researchers or groups, as well as associated maintenance and management afterwards. It is almost certain that Big-Data environments will require a different "programming language" to fully exploit the latent potential. In addition, the process of extending the environment to provide new analysis capabilities will likely be more involved than, say, compiling a piece of new or revised code.We thus anticipate that researchers will require support from dedicated organizations associated with the environment that are composed of professional software engineers and data scientists. A major benefit will likely be that such extensions are of higherquality and broader applicability than ad hoc changes by physical scientists. Another anticipated significant change is improved collaboration among the researchers using the same environment. Since the environment is homogeneous within itself, many barriers to collaboration are minimized or eliminated. For example, data and analysis algorithms can be seamlessly shared, reused and re-purposed. In conclusion, we will be able to achieve a new level of scientific productivity in the Big-Data analysis environments.
Kuo, Kwo-Sen↗
Dynamic color mapping with a multi-scale histogram: A design study with physical scientists
Many research and development activities for scientific data analysis have focused on scalability challenges and data-driven features. Conversely, data visualization that focus on models requiring human interaction rarely involve practical and largescale scientific data analysis. Therefore, a gap exists between interactive data visualization and scientific data analysis applications. In this paper, we present a design study of interactive data visualization to support scientists who visually analyze data from neutron scattering experiments. This study was conducted in multiple phases: 1) problem characterization; 2) initial design and formative evaluation; and 3) iterative design. We characterize the problems and the design requirements for the analysis of the specific physical science data. We discuss the design, development, evaluation of our visual analytics tool and as well as our iterative developments with physical scientists. We show how to bridge the gap between the two disciplines uncovering new potential to solve their challenges in this design study. We focus on a specific challenge, finding an optimal color mapping, which plays a critical role in neutron scattering science and is broadly applicable to other scientific domains. To address the challenge, we propose two interactive visualization techniques: a dynamic color scale bar (DCSB) and a multi-scale histogram (MSH).
Chae, Junghoon↗
WebWinds: A Java-based Environment for the Interactive Display and Analysis of Scientific Data
Explore the source record for details and available documents.
WebWinds Java-based Environment Interactive Displa↗
Lunar seismic data analysis
The scientific data transmitted continuously from all ALSEP (Apollo Lunar Surface Experiment Package) stations on the Moon and recorded on instrumentation tapes at receiving stations distributed around the Earth were processed. The processing produced sets of computer-compatible digital tapes, from which various other data sets convenient for analysis were generated. The seismograms were read, various types of seismic events were classified; the detected events were cataloged.
Nakamura, Y.↗
Reduction and scientific analysis of data from the charge-energy-mass (CHEM) spectrometer on the AMPTE/CCE spacecraft
The Charge-Energy-Mass (CHEM) spectrometer instrument on the AMPTE/Charge Composition Explorer (CCE) spacecraft is designed to measure the mass and charge-state abundance of magnetospheric and magnetosheath ions between 0.3 and 315 keV/e, an energy range that includes the bulk of the ring current and the dynamically important portion of the plasma sheet population. Continuing research is being conducted using the AMPTE mission data set, and in particular, that of the CHEM spectrometer which has operated flawlessly since launch and still provides excellent quality data. The requirted routine data processing and reduction, and software develpment continues to be performed. Scientific analysis of composition data in a number of magnetospheric regions including the ring current region, near-earth plasma sheet and subsolar magnetosheath continues to be undertaken. Correlative studies using data from the sister instrument SULEICA, which determines the mass and charge states of ions in the energy range of approximately 10 to 250 keV/e on the IRM, as well as other data from the CCE and IRM spacecraft, particularly in the upstream region and plasma sheet have also been undertaken.
Gloeckler, G.↗
Scalable Volume Visualization for Big Scientific Data Modeled by Functional Approximation
Considering the challenges posed by the space and time complexities in handling extensive scientific volumetric data, various data representations have been developed for the analysis of large-scale scientific data. Multivariate functional approximation (MFA) is an innovative data model designed to tackle substantial challenges in scientific data analysis. It computes values and derivatives with high-order accuracy throughout the spatial domain, mitigating artifacts associated with zero- or first-order interpolation. However, the slow query time through MFA makes it less suitable for interactively visualizing a large MFA model. In this work, we develop the first scalable interactive volume visualization pipeline, MFA-DVV, for the MFA model encoded from large-scale datasets. Our method achieves low input latency through distributed architecture, and its performance can be further enhanced by utilizing a compressed MFA model while still maintaining a high-quality rendering result for scientific datasets. We conduct comprehensive experiments to show that MFA-DVV can decrease the input latency and achieve superior visualization results for big scientific data compared with existing approaches.
big scientific dataset↗
Asynchronous and Load-Balanced Union-Find for Distributed and Parallel Scientific Data Visualization and Analysis
We present a novel distributed union-find algorithm that features asynchronous parallelism and k-d tree based load balancing for scalable visualization and analysis of scientific data. Applications of union-find include level set extraction and critical point tracking, but distributed union-find can suffer from high synchronization costs and imbalanced workloads across parallel processes. In this study, we prove that global synchronizations in existing distributed union-find can be eliminated without changing final results, allowing overlapped communications and computations for scalable processing. We also use a k-d tree decomposition to redistribute inputs, in order to improve workload balancing. We benchmark the scalability of our algorithm with up to 1,024 processes using both synthetic and application data. Here, we demonstrate the use of our algorithm in critical point tracking and super-level set extraction with high-speed imaging experiments and fusion plasma simulations, respectively.
97 MATHEMATICS AND COMPUTING↗
Mission operations technology
Developments in the area of spacecraft operations are examined that can be utilized in mission and spacecraft designs to enhance the efficiency of planetary and other mission operations. The technology enhancements described address four major systems: the spacecraft, the tracking and data system, the spaceflight operations center, and the flight projects. Specific technological advances are detailed in the areas of telemetry monitoring and fault analysis, scheduling systems, and scientific data analysis. The Spacecraft Health Automated Reasoning Prototype and the Engineering Analysis Subsystem Environment can provide real-time monitoring and analysis with limited workforce resources. The Multimission Automation for Real-time Verification of Spacecraft Engineering Link, the Operation Mission Planner, and the Sky Image Classification and Archiving Tool are other examples of systems developed by NASA that enhance operations technology by reducing the need for human input.
Varsi, Giulio↗
Scalable Topological Data Analysis and Visualization for Evaluating Data-Driven Models in Scientific Applications
With the rapid adoption of machine learning techniques for large-scale applications in science and engineering comes the convergence of two grand challenges in visualization. First, the utilization of black box models (e.g., deep neural networks) calls for advanced techniques in exploring and interpreting model behaviors. Second, the rapid growth in computing has produced enormous datasets that require techniques that can handle millions or more samples. Although some solutions to these interpretability challenges have been proposed, they typically do not scale beyond thousands of samples, nor do they provide the high-level intuition scientists are looking for. Here, we present the first scalable solution to explore and analyze high-dimensional functions often encountered in the scientific data analysis pipeline. By combining a new streaming neighborhood graph construction, the corresponding topology computation, and a novel data aggregation scheme, namely topology aware datacubes, we enable interactive exploration of both the topological and the geometric aspect of high-dimensional data. Following two use cases from high-energy-density (HED) physics and computational biology, we demonstrate how these capabilities have led to crucial new insights in both applications.
97 MATHEMATICS AND COMPUTING↗
Leveraging In-Network Computing and Programmable Switches for Streaming Analysis of Scientific Data
With the emergence of programmable network devices that match the performance of fixed function devices, several recent projects have explored in-network computing, where the processing that is traditionally done outside the network is offloaded to the network devices. In-network computing has typically been applied to network functions (e.g., load balancing, NAT, and DNS), caching, data reduction/aggregation, and coordination/consensus functions. In some cases it has been used to accelerate stream-processing tasks that involve small payloads and simple operations. In this work we focus on leveraging in-network computing for stream processing of scientific datasets with large payloads that require complex operations such as floating-point computations and logarithmic functions. We demonstrate in-network computing for a real-world scientific application performing streaming normalization of a 2-D image from a light source experiment. We discuss the challenges we encountered and potential approaches to address them.
Sankaran, Ganesh↗
Scientific Analysis of Data for the ISTP/SOLARMAX Programs
This Grant supplemented our work on data analysis from the Wind spacecraft which was one of the ISTRIA fleet of spacecraft. It was targeted at observations related to the time of solar maximum in 2000. The work we proposed to do under this grant included comparison of solar wind parameters obtained from different spacecraft in order to establish correlation lengths appropriate to the solar wind and also to compare parameters to explore solar cycle effects.
Lazarus, Alan J.↗
Data evaluation, analysis, and scientific study
Extensive work was performed in data analysis and modeling of solar active phenomena. The work consisted in the study of UV data from the Ultraviolet Spectrometer and Polarimeter (UVSP) instrument on board the Solar Maximum Mission satellite. These data were studied in conjunction with X-rays from the Hard X-ray Imaging Spectrometer (HXIS) instrument, and with H-alpha and magnetographic data from ground-based observatories. The processes we studied are the active phenomena which result from the interaction of the solar magnetic fields with the plasma in the outer regions of the solar atmosphere. These processes include some very dynamic processes such as the prominence eruptions and the 'microflares'. Our research aimed at characterizing the following: the observed phenomena, the possible physical models, and the relevance to the chromospheric and coronal heating.
Wu, S. T.↗
High-performance data format for scientific data storage and analysis
Here, in this article, we present the High-Performance Output (HiPO) data format developed at Jefferson Laboratory for storing and analyzing data from Nuclear Physics experiments. The format was designed to efficiently store large amounts of experimental data, utilizing modern fast compression algorithms. The purpose of this development was to provide organized data in the output, facilitating access to relevant information within the large data files. The HiPO data format has features that are suited for storing raw detector data, reconstruction data, and the final physics analysis data efficiently, eliminating the need to do data conversions through the lifecycle of experimental data. The HiPO data format is implemented in C++ and JAVA, and provides bindings to FORTRAN, Python, and Julia, providing users with the choice of data analysis frameworks to use. In this paper, we will present the general design and functionalities of the HiPO library and compare the performance of the library with more established data formats used in data analysis in High Energy and Nuclear Physics (such as ROOT and Parquete). In columnar data analysis, HiPO surpasses established data formats in performance and can be effectively applied to data analysis in other scientific fields.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
FasTensor (FT) v0.0.1
The FasTensor implements the native array data programming mode namely SLOPE for modern data analysis tasks. It is designed to work in parallel on supercomputer and can handle terabyte data analysis tasks. It is thousand times faster than other data analysis systems such as Apache Spark and 38% faster than TensorFlow in dealing with large scale scientific data analysis. It has wide applications in image data analysis and mesh data analysis tasks in physical simulations and other fields.
Wu, Kesheng↗
Hierarchical Analysis of Halo Center in Cosmology
Ever-increasing data size raises many challenges for scientific data analysis. Particularly in cosmological N-body simulation, finding the center of a dark matter halo suffers heavily from the large computational cost associated with the large number of particles (up to 20 million). In this work, we exploit the latent structure embed in a halo, and we propose a hierarchical approach to approximate the exact gravitational potential calculation for each particle in order to more efficiently find the halo center. Tests of our method on data from N-body simulations show that in many cases the hierarchical algorithm performs significantly faster than existing methods with a desirable accuracy.
97 MATHEMATICS AND COMPUTING↗