Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Intensive Science”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS2012

NASA Center for Climate Simulation (NCCS) Presentation

The NASA Center for Climate Simulation (NCCS) offers integrated supercomputing, visualization, and data interaction technologies to enhance NASA's weather and climate prediction capabilities. It serves hundreds of users at NASA Goddard Space Flight Center, as well as other NASA centers, laboratories, and universities across the US. Over the past year, NCCS has continued expanding its data-centric computing environment to meet the increasingly data-intensive challenges of climate science. We doubled our Discover supercomputer's peak performance to more than 800 teraflops by adding 7,680 Intel Xeon Sandy Bridge processor-cores and most recently 240 Intel Xeon Phi Many Integrated Core (MIG) co-processors. A supercomputing-class analysis system named Dali gives users rapid access to their data on Discover and high-performance software including the Ultra-scale Visualization Climate Data Analysis Tools (UV-CDAT), with interfaces from user desktops and a 17- by 6-foot visualization wall. NCCS also is exploring highly efficient climate data services and management with a new MapReduce/Hadoop cluster while augmenting its data distribution to the science community. Using NCCS resources, NASA completed its modeling contributions to the Intergovernmental Panel on Climate Change (IPCG) Fifth Assessment Report this summer as part of the ongoing Coupled Modellntercomparison Project Phase 5 (CMIP5). Ensembles of simulations run on Discover reached back to the year 1000 to test model accuracy and projected climate change through the year 2300 based on four different scenarios of greenhouse gases, aerosols, and land use. The data resulting from several thousand IPCC/CMIP5 simulations, as well as a variety of other simulation, reanalysis, and observationdatasets, are available to scientists and decision makers through an enhanced NCCS Earth System Grid Federation Gateway. Worldwide downloads have totaled over 110 terabytes of data.

Webster, William P.↗
DOE OSTI2015

Integration of Russian Tier-1 Grid Center with High Performance Computers at NRC-KI for LHC experiments and beyond HENP

The LHC experiments are preparing for the precision measurements and further discoveries that will be made possible by higher LHC energies from April 2015 (LHC Run2). The need for simulation, data processing and analysis would overwhelm the expected capacity of grid infrastructure computing facilities deployed by the Worldwide LHC Computing Grid (WLCG). To meet this challenge the integration of the opportunistic resources into LHC computing model is highly important. The Tier-1 facility at Kurchatov Institute (NRC-KI) in Moscow is a part of WLCG and it will process, simulate and store up to 10% of total data obtained from ALICE, ATLAS and LHCb experiments. In addition Kurchatov Institute has supercomputers with peak performance 0.12 PFLOPS. The delegation of even a fraction of supercomputing resources to the LHC Computing will notably increase total capacity. In 2014 the development a portal combining a Tier-1 and a supercomputer in Kurchatov Institute was started to provide common interfaces and storage. The portal will be used not only for HENP experiments, but also by other data- and compute-intensive sciences like biology with genome sequencing analysis; astrophysics with cosmic rays analysis, antimatter and dark matter search, etc.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2013

Efficient Memory Access with NumPy Global Arrays using Local Memory Access

This paper discusses the work completed working with Global Arrays of data on distributed multi-computer systems and improving their performance. The tasks completed were done at Pacific Northwest National Laboratory in the Science Undergrad Laboratory Internship program in the summer of 2013 for the Data Intensive Computing Group in the Fundamental and Computational Sciences DIrectorate. This work was done on the Global Arrays Toolkit developed by this group. This toolkit is an interface for programmers to more easily create arrays of data on networks of computers. This is useful because scientific computation is often done on large amounts of data sometimes so large that individual computers cannot hold all of it. This data is held in array form and can best be processed on supercomputers which often consist of a network of individual computers doing their computation in parallel. One major challenge for this sort of programming is that operations on arrays on multiple computers is very complex and an interface is needed so that these arrays seem like they are on a single computer. This is what global arrays does. The work done here is to use more efficient operations on that data that requires less copying of data to be completed. This saves a lot of time because copying data on many different computers is time intensive. The way this challenge was solved is when data to be operated on with binary operations are on the same computer, they are not copied when they are accessed. When they are on separate computers, only one set is copied when accessed. This saves time because of less copying done although more data access operations were done.

Daily, Jeffrey A.↗
DOE OSTI2016

The Medical Science DMZ

Objective We describe use cases and an institutional reference architecture for maintaining high-capacity, data-intensive network flows (e.g., 10, 40, 100 Gbpsþ) in a scientific, medical context while still adhering to security and privacy laws and regulations. Materials and Methods High-end networking, packet filter firewalls, network intrusion detection systems. Results We describe a “Medical Science DMZ” concept as an option for secure, high-volume transport of large, sensitive data sets between research institutions over national research networks. Discussion The exponentially increasing amounts of “omics” data, the rapid increase of high-quality imaging, and other rapidly growing clinical data sets have resulted in the rise of biomedical research “big data.” The storage, analysis, and network resources required to process these data and integrate them into patient diagnoses and treatments have grown to scales that strain the capabilities of academic health centers. Some data are not generated locally and cannot be sustained locally, and shared data repositories such as those provided by the National Library of Medicine, the National Cancer Institute, and international partners such as the European Bioinformatics Institute are rapidly growing. The ability to store and compute using these data must therefore be addressed by a combination of local, national, and industry resources that exchange large data sets. Maintaining data-intensive flows that comply with HIPAA and other regulations presents a new challenge for biomedical research. Recognizing this, we describe a strategy that marries performance and security by borrowing from and redefining the concept of a “Science DMZ”—a framework that is used in physical sciences and engineering research to manage high-capacity data flows. Conclusion By implementing a Medical Science DMZ architecture, biomedical researchers can leverage the scale provided by high-performance computer and cloud storage facilities and national high-speed research networks while preserving privacy and meeting regulatory requirements.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2013

Report on HEP/ASCR Data Summit

Representatives from the HEP and ASCR communities met at Germantown on April 2-3, 2013, to discuss issues in carrying out science with large datasets and associated data-intensive computing tasks. The authors of this report acknowledge the important contributions made by all of the ASCR and HEP participants at the data summit and thank them for their efforts. During the meeting, HEP scientists made a series of presentations highlighting how they as a community interact with data, what tools they have developed in doing so, and the challenges to be faced moving forward. From the presentations and discussions, it became clear that the emerging data-intensive challenges span all HEP frontiers. Furthermore, since HEP is building fewer machines and experiments, there is increased emphasis in further optimizing the extraction of science results from the data by staying informed about the latest technologies and taking advantage of those that are particularly relevant. The joint summit proved to be very useful, as it not only familiarized the ASCR team with HEP science, but it also made apparent to the HEP group that there are significant common issues across the three HEP research frontiers that, in the past, have traditionally been addressed separately within each domain. A number of potential areas of common interest across the ASCR and HEP communities were identified. They cover the full range from pure R&D to focused assistance in the development and optimization of specific applications. It was generally agreed that HEP data management tools and techniques are already state of the art, and meet current needs. The community has extensive experience with management and analysis of large data sets. Properly planned joint HEP-ASCR research could incorporate transformational advances in developing tools for future applications. Both communities expressed the need to have viable models for maintenance of such tools and software, once developed.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2011

Provenance Challenges for Earth Science Dataset Publication

Modern science is increasingly dependent on computational analysis of very large data sets. Organizing, referencing, publishing those data has become a complex problem. Published research that depends on such data often fails to cite the data in sufficient detail to allow an independent scientist to reproduce the original experiments and analyses. This paper explores some of the challenges related to data identification, equivalence and reproducibility in the domain of data intensive scientific processing. It will use the example of Earth Science satellite data, but the challenges also apply to other domains.

Tilmes, Curt↗
NASA NTRS1983

Telecommunications and data acquisition systems support for the Viking 1975 mission to Mars

The background for the Viking Lander Monitor Mission (VLMM) is given, and the technical and operational aspects of the tracking and data acquisition support that the Network was called upon to provide are described. An overview of the science results obtained from the imaging, meteorological, and radio science data is also given. The intensive efforts that were made to recover the mission are described.

Mudgway, D. J.↗
DOE OSTI2020

End-to-end online performance data capture and analysis for scientific workflows

With the increased prevalence of employing workflows for scientific computing and a push towards exascale computing, it has become paramount that we are able to analyze characteristics of scientific applications to better understand their impact on the underlying infrastructure and vice-versa. Such analysis can help drive the design, development, and optimization of these next generation systems and solutions. Here, we present the architecture, integrated with existing well-established and newly developed tools, to collect online performance statistics of workflow executions from various, heterogeneous sources and publish them in a distributed database (Elasticsearch). Using this architecture, we are able to correlate online workflow performance data, with data from the underlying infrastructure, and present them in a useful and intuitive way via an online dashboard. We have validated our approach by executing two classes of real-world workflows, both under normal and anomalous conditions. The first is an I/O-intensive genome analysis workflow; the second, a CPU- and memory-intensive material science workflow. Based on the data collected in Elasticsearch, we are able to demonstrate that we can correctly identify anomalies that we injected. The resulting end-to-end data collection of workflow performance data is an important resource of training data for automated machine learning analysis.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Data Intensive Scientific Workflows on a Federated Cloud: CRADA Final Report

The Fermilab Scientific Computing Division and the KISTI Global Science Experimental Data Hub Center have built a prototypical large-scale infrastructure to handle scientific workflows of stakeholders to run on multiple cloud resources. The demonstrations have been in the areas of (a) Data-Intensive Scientific Workflows on Federated Clouds, (b) Interoperability and Federation of Cloud Resources, and (c) Virtual Infrastructure Automation to enable On-Demand Services.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2022

Management and Storage of Scientific Data

Scientific discoveries rely heavily on efficient access, search, and management of massive data sets. Data management technologies have, for decades, provided foundational capabilities for scientific computing. Just as storage, input/output (I/O), and data management have been fundamental to simulation-based science for many years, so too are capable data-management technologies key to the success of today’s scientific workflows utilizing data intensive and machine learning (ML) techniques. The Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR) program has invested broadly in data-management research focused on high-performance computing (HPC) systems, from parallel file systems that store data to application software that makes these systems more productive. Still, advances in technology combined with growing diversity of supported science strongly motivate continued investment in this area. In January 2022, ASCR convened a workshop to identify priority research directions in the area of data management for high-performance and scientific computing. Attendees were challenged to identify promising approaches that would support the breadth of the DOE mission, including the explosion of artificial intelligence (AI) uses and the growing needs of experimental and observational science. Technological and science drivers were identified and considered as they relate to key aspects of data management such as interfaces, architectural design, and FAIR principles (Findable, Accessible, Interoperable, and Reusable). The thoughts of the workshop participants were distilled into a set of four priority research directions with the potential for high impact on DOE science. These research directions are summarized in the following pages.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2019

Provenance–aware workflow for data quality management and improvement for large continuous scientific data streams

Data quality assessment, management and improvement is integral part of any big data intensive scientific research to ensure accurate, reliable and reproducible science discoveries. The task of maintaining the quality of data, however, is non-trivial and pose a challenge for a program like Department of Energy’s Atmospheric Radiation Measurement (ARM) that collects data from hundreds of instruments across the world,and distributes thousands of streaming data products that are continuously growing in near-real-time in an archive 1.7 Petabyte in size and growing. We present a computational data processing workflow to collect the data quality issue via an easy and intuitive web-based portal that allows reporting of any quality issues for any site, facility or instruments at a granularity down to individual variable in the data files. Portal allows instrument specialists and scientists to provide corrective actions in form of symbolic equation. A parallel processing framework applies the data improvement to large volume of data in efficient parallel environment, while optimizing data transfer and file I/O operations. Corrected files are systematically versioned and archived. A provenance tracking module tracks and records any change made to the data during its entire life cycle which are communicated transparently to the scientific users. Developed in Python using open source technologies, software architecture enables efficient and fast management and improvement of data in an operational data center environment.

Kumar, Jitendra↗
DOE OSTI2019

Integrating Data Science Methods and Life Cycle Assessment (LCA): Application to the Power Sector - ISSST2019

This work was presented at the International Symposium on Sustainable Systems and Transportation (ISSST) 2019 Conference in Portland, OR on June 26, 2019. Data science is radically transforming the scientific landscape. Data-intensive frameworks such as LCA can uniquely benefit from the establishment of large-scale and national datasets, and the use of data-science methods. Additionally, the synthesis and application of these data repositories in LCA can aid in improving emissions inventories, technological characterization of product systems, and uncertainty quantification. In this work, a series of case studies are used to explore the intersection of data science and LCA of the power sector. The objective of this work is to provide a broad cross-section of data sources, methods, and approaches that have been applied to characterize the life cycle of power systems, but are generalizable across multiple LCA research domains.

Littlefield, James↗
DOE OSTI2023

New technologies as decision aids for the advancement of ecological risk assessment

Moore's law states that the number of transistors that can be placed on an integrated circuit doubles every two years (Moore, 1975). This has led to a steady increase in the processing power of computers over time, and technology is now enhancing and advancing software and scientific applications, which has enabled computationally intensive methods such as machine learning, data science, modeling, and simulation. The advancement of computers and data-driven algorithms is profoundly impacting people's lives. It is changing the way we work, the way we learn, and the way we interact with the world around us. Here, this editorial will discuss how scientists can benefit from the latest technology advancements and related tools by incorporating them into the ecological risk assessment (ERA) to study ecosystems as a way to create refined assessments and accelerate the turnaround times.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2004

Software architecture for large scale, distributed, data-intensive systems

This paper presents our experience with OODT, a novel software architectual style, and middlware-based implementation for data-intensive systems. To date, OODT has been successfully evaluated in several different science domains including Cancer Research with the National Cancer Institute (NCI), and Planetary Science with NASA's Planetary Data System (PDS).

software architecture↗
NASA NTRS2012

Leveraging Data Intensive Computing to Support Automated Event Services

A large portion of Earth Science investigations is phenomenon- or event-based, such as the studies of Rossby waves, mesoscale convective systems, and tropical cyclones. However, except for a few high-impact phenomena, e.g. tropical cyclones, comprehensive records are absent for the occurrences or events of these phenomena. Phenomenon-based studies therefore often focus on a few prominent cases while the lesser ones are overlooked. Without an automated means to gather the events, comprehensive investigation of a phenomenon is at least time-consuming if not impossible. An Earth Science event (ES event) is defined here as an episode of an Earth Science phenomenon. A cumulus cloud, a thunderstorm shower, a rogue wave, a tornado, an earthquake, a tsunami, a hurricane, or an EI Nino, is each an episode of a named ES phenomenon," and, from the small and insignificant to the large and potent, all are examples of ES events. An ES event has a finite duration and an associated geolocation as a function of time; its therefore an entity in four-dimensional . (4D) spatiotemporal space. The interests of Earth scientists typically rivet on Earth Science phenomena with potential to cause massive economic disruption or loss of life, but broader scientific curiosity also drives the study of phenomena that pose no immediate danger. We generally gain understanding of a given phenomenon by observing and studying individual events - usually beginning by identifying the occurrences of these events. Once representative events are identified or found, we must locate associated observed or simulated data prior to commencing analysis and concerted studies of the phenomenon. Knowledge concerning the phenomenon can accumulate only after analysis has started. However, except for a few high-impact phenomena. such as tropical cyclones and tornadoes, finding events and locating associated data currently may take a prohibitive amount of time and effort on the part of an individual investigator. And even for these high-impact phenomena, the availability of comprehensive records is still only a recent development. A major reason for the lack of comprehensive ,records for the majority of the ES phenomena is the perception that they do not pose immediate and/or severe threat to life and property and are thus not consistently tracked. monitored, and catalogued. Many phenomena even lack commonly accepted criteria for definitions. However. the lack of comprehensive records is also due to the increasingly prohibitive volume of observations and model data that must be examined. NASA Earth Observing System Data Information System (EOSDIS) alone archives several petabytes (PB) of satellite remote sensing data and steadily increases. All of these factors contribute to the difficulty of methodically identifying events corresponding to a given phenomenon and significantly impede systematic investigations. In the following we present a couple motivating scenarios, demonstrating the issues faced by Earth scientists studying ES phenomena.

Clune, Thomas L.↗
DOE OSTI2015

Using NVMe Gen3 PCIe SSD Cards in High-density Servers for High-performance Big Data Transfer Over Multiple Network Channels

This Technical Note describes how the Zettar team came up with a data transfer cluster design that convincingly proved the feasibility of using high-density servers for high-performance Big Data transfers. It then outlines the tests, operations, and observations that address a potential over-heating concern regarding the use of Non-Volatile Memory Host Controller Interface Specification (NVMHCI aka NVM Express or NVMe) Gen 3 PCIe SSD cards in high-density servers. Finally, it points out the possibility of developing a new generation of high-performance Science DMZ data transfer system for the data-intensive research community and commercial enterprises.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1990

Earth Radiation Budget Experiment - Preliminary seasonal results

Over the previous four years the Earth Radiation Budget Experiment (ERBE) instruments have been gathering data on two satellites, the Earth Radiation Budget Satellite and the the operational NOAA-9 satellite. The ERBE science team recently completed the validation of an initial sampling of these data involving intensive examination of data in four months during 1985 and 1986. The data being placed in the National Space Science Data Center to acquaint the scientific community with their availability are discussed. The ERBE archival data products are also presented.

Barkstrom, Bruce R.↗