Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 811 records · Page 45

Enforcing End-to-end I/O Policies for Scientific Workflows using Software-Defined Storage Resource Enclaves

Data-intensive knowledge discovery requires scientific applications to run concurrently with analytics and visualization codes executing in situ for timely output inspection and knowledge extraction. Consequently, I/O pipelines of scientific workflows can be long and complex because they comprise many stages of analytics across different layers of the I/O stack of high-performance computing systems. Performance limitations at any I/O layer or stage can cause an I/O bottleneck resulting in greater than expected end-to-end I/O latency. In this paper, we present the design and implementation of a novel data management infrastructure called Software-Defined Storage Resource Enclaves (SIREN) at system level to enforce end-to-end policies that dictate an I/O pipeline's performance. SIREN provides an I/O performance interface for users to specify the desired storage resources in the context of in-situ analytics. If suboptimal performance of analytics is caused by an I/O bottleneck when data are transferred between simulations and analytics, schedulers in different layers of the I/O stack automatically provide the guaranteed lower bounds on I/O throughput. Lastly, our experimental results demonstrate that SIREN provides performance isolation among scientific workflows sharing multiple storage servers across two I/O layers (burst buffer and parallel file systems) while maintaining high system scalability and resource utilization.

97 MATHEMATICS AND COMPUTING↗

Pioneer Odyssey: Encounter with a Giant

Ancient peoples, perhaps thousands of years ago, undoubtedly conceived the idea of "reaching out" to Jupiter, the largest and most brilliant of the "wandering stars." But for mankind to stretch across the half billion miles to the giant planet of the Solar System many advances in technical and organizational fields of human endeavor had to be made. Outreach to Jupiter did not become a serious possibility until the Pioneer F and G Project was formed by NASA early in 1968. And then man began to design an extension of his senses that would probe the environs of the giant of the Solar System, a truly pioneer odyssey into the virtually unknown regions beyond the orbit of Mars. In the ensuing year. a dedicated and cooperative effort of several thousand people in Govern­ment, university, and private industrial organiza­tions converted the idea into a reality. Less than twelve generations after Galileo first saw the banded disc of Jupiter and the flickering dots of its large satellites in the newly invented telescope, mankind sent a machine to make observations within that Jovian system. The two Pioneer spacecraft for the mission to Jupiter each weighed only about 570 pounds, yet carried eleven highly sophisticated instruments capable of operating unattended for many year in space. The spacecraft consumes less electrical power than a standard 100 watt lamp yet is able to accept instructions from Earth to control numerous operating modes of its scientific payload, process observations from these scientific instruments and format the observations into information usable on Earth. Even more remarkable. the space­craft transmits a radio signal of only 8 watts power - equal to a nightlight - yet the information carried by the radio signal is received back on Earth from a distance of several billion miles. The Pioneer mission could not have been a success without the special engineering, scientific and management organization created for its accomplishment. This organization was rather unique in that it first had to meet a launch date target relatively quickly and then had to function for an extremely long mission operational time, far longer than any previous mission to planets. The first task was thus to organize so that the mission could be planned and the spacecraft designed and fabricated to be ready for launch within a few weeks of a 30-month target for completion. The program also produced an organization that planned mission operations to such detail that more than 16,000 commands were transmitted flawlessly to the distant spacecraft during Jupiter encounter. And each command reached the spacecraft within one second of the planned time despite the more than 90 minutes required for the radio message to travel from Earth to the space­craft and for the spacecraft to return a confirma­tion to controllers back on Earth. The organization for Pioneer also determined the required flight path from Earth to Jupiter with such precision, and controlled the launch vehicle with such accuracy, that 21 months after launch the spacecraft was able to fly behind Jupiter's satellite lo, thereby providing the first measurement that indicated the possibility of a tenuous atmosphere about this large satellite. Finally, the Pioneer organization processed and analyzed each year sufficient information from the spacecraft to fill a book having about 3 million pages and reduced this avalanche of data from space into summaries of manageable size. And all this organization depended on people, consisted of people: the people who really made this whole mission possible. Pioneer has always depended on the dedication of many individuals from many organizations throughout the world to achieve its scientific objectives, and, as evidenced by the success of the Pioneer series, this dependence is completely justified. Relatively few individuals have an opportunity during their lives to participate in such a challenging, historic, pioneering effort; and still fewer are able to enjoy the rewards of such an activity. We who have worked on Pioneer 10 and its sister spacecraft, Pioneer 11, consider ourselves fortunate to be in both classes. For the opportunity we thank the people of the United States of America, who have supported our country's space effort and its spreading of human awareness of a vast and intriguing universe in which our own unique planet Earth is only one of myriads of worlds. This volume describing the mission to Jupiter and its results is one of the many rewards for our effort which we share with you, the reader.

Fimmel, Richard O.↗

Efficient Parallelization of Irregular Applications on GPU Architectures

With the enlarging computation capacity of general Graphics Processing Units (GPUs), leveraging GPUs to accelerate parallel applications has become a critical topic in academia and industry. However, a wide range of irregular applications with the computation-/memory-intensive nature cannot easily achieve high GPU utilization. The challenges mainly involve the following aspects: first, data dependence leads to coarse-grained kernel and inefficient parallelism; second, heavy GPU memory usage may cause frequent memory evictions and extra overhead of I/O; third, specific computation patterns produce memory redundancies; last, workload balance and data reusability conjunctly benefit the overall performance, but there may exist a dynamic trade-off between them. Targeting these challenges, this dissertation proposes multiple optimizations to accelerate two real-world applications: many-body correlation functions to simulate nuclear physics in a large-scale scientific system; the other is the eALS-based matrix factorization recommendation system. To accelerate the calculations of many-body correlation functions, this dissertation presents three frameworks in GPU memory management and multi-GPU scheduling. Firstly, an optimized systematic GPU memory management framework, MemHC, utilizes a series of new memory reduction designs in GPU memory allocation, CPU/GPU communications, and GPU memory oversubscription. Secondly, an enhanced multi-GPU scheduling framework, MICCO, particularly by taking both data dimension (e.g., data reuse and data eviction) and computation dimension into account. MICCO designs a heuristic scheduling algorithm and a machine learning-based regression model to generate the optimal settings of a proposed new concept to manage the trade-off. Thirdly, a locality-aware multi-GPU scheduling framework. This scheduler leverages pipeline batch generation with a looking-ahead strategy by building local dependency graphs for memory transfer reduction and better data reuse, achieving up to 79.92% memory cost reduction and 1.67x speedup. To parallelize the eALS-based recommendation system, this dissertation proposes an efficient CPU/GPU heterogeneous recommendation system, HEALS. HEALS employs newly designed architecture-adaptive data formats to achieve load balance and good data locality on CPU and GPU. To mitigate the data dependence, HEALS presents a CPU/GPU collaboration model for both task parallelism and data parallelism with multiple kernel computation optimizations. In summary, this dissertation efficiently accelerates two typical irregular applications on GPUs by building four frameworks, including CPU/GPU collaboration, GPU memory management, and multi-GPU scheduling.

Wang, Qihan↗

A global map of urban extent from nightlights

Urbanization, one of the major human induced land-cover and land-use changes, has a profound impact on the Earth system including biodiversity, the cycling of water and carbon and exchange of energy and water between Earth’s surface and atmosphere, all affecting weather and climate. Accurate information on urban areas and their spatial distribution at the regional and global scales is important for scientific understanding of their contribution to the changing Earth system, and for practical management and policy decisions. We developed a method to map the urban extent from the Defense Meteorological Satellite Program/Operational Linescan System (DMSP/OLS) nighttime stable-light data at the global level and derived a new global map of 1-km urban extent for year 2000. Based on this map, we found that globally, urban land area is about 0.5% of total land area but ranges widely at regional level from 0.1% in Oceania to 2.3% in Europe. At the country level, urban land area varies from lower than 0.01% to higher than 10%, but is lower than 1% for most (70%) countries. Urbanization follows land mass distribution, as anticipated, with the highest concentration found between 30°N to 45°N latitude and the largest longitudinal peak around 80°W. Based on a sensitivity analysis and comparison with other global urban area products, we found that our global product of urban area provides a reliable estimate of global urban areas and offer the potential of capturing more accurately their spatial and temporal dynamics.

54 ENVIRONMENTAL SCIENCES↗

Network science can improve the sustainable development of solar energy

Abstract The recent emergence of agrivoltaic and ecovoltaic approaches to ground-mounted photovoltaic (PV) energy provides a much-needed alternative to the environmentally disruptive practices employed in utility-scale solar development. Research on such land-sharing approaches has grown rapidly, with an emphasis on characterizing how PV arrays impact ecosystem processes and agricultural productivity. Although these studies have done well to quantify a variety of dual-use solar practices by employing site-specific sampling designs, this approach has limited our ability to synthesize results across sites, regions, and globally. We call for a network science approach for improved cross-site synthesis of dual-use solar research. We contend that a common approach for data collection and synthesis will facilitate a more rigorous investigation of the agricultural and ecological impacts of PV development across space and over time. The products of this scientifically informed approach can be directly applied to improve sustainable land management.

Bacon, Taylor (ORCID:0009000518578569)↗

ROBIN (RuciO/BIgData Express/SENSE) A Next-Generation High-Performance Data Service Platform

Increasingly, scientific research discoverydepends on the analysis of extremely large amounts of data. Thisdata is gathered through very large scientific instruments, verylarge numbers of small devices, and mid-scale instruments.These devices generate exponentially increasing volumes of data.Such data is often obtained, analyzed, stored, visualized, andtransferred at and among sites world-wide by largecollaborations of science communities. Managing and movingextremely large volumes of data across today’s networks is aspecial multidimensional challenge. Also, existing datamanagement and movement services and tools often areinadequate to address all specific requirements. This paperdescribes the various components of this challenge and howthose issues can be addressed by ROBIN, a uniquecomprehensive set of integrated services designed specificallyfor managing and moving extremely large amounts of data overlong distances, e.g., thousands of miles around the globe. Thispaper also describes the results of initial experiments using thatset of integrated services.

Wu, Wenji↗

An Advanced Network and distributed Storage Laboratory (ANDSL) for Data Intensive Science (Final Report)

The original intent of this project was to build and operate an Advanced Network and Distributed Storage Laboratory (ANDSL) for Data Intensive Science that will prepare the Open Science Grid (OSG) community for a new generation of wide area communication capabilities operating at a 100Gb rate. The plan was to engage a team of four dedicated scientists that will design, develop and operate a state of the art laboratory that will be fully integrated into the organizational and operational infrastructure of the OSG. The ANDSL was planned to include host-based hardware and software required to profile, emulate and study real-life distributed storage workloads at rates of 100 Gbps. The laboratory was designed to utilize the Department Of Energy’s Advanced Network Initiative (ANI) 100GbTestbed as connections to the fabric become available so that OSG stakeholders can use it to profile real-life data intensive science applications as well as experiment with advanced storage and data management technologies. The goal of the original program of work was to close the adoption gap, decreasing the time between availability of production 100Gbps links and the end-to-end use of such a network to enable scientific discovery.

100Gb networks↗

The EOS data and information system

The Earth Observing System (EOS) is proposed as a 1991 new initiative by NASA as part of the Mission to Planet Earth. One of the key components of the EOS program is the EOS Data and Information System (EOSDIS). Two independent Phase B studies of EOSDIS were conducted from January 1989 through April 1990. Some of the key challenges faced by EOSDIS are: satisfying the data and information needs of a diverse multidisciplinary scientific community integrating product generation algorithms for over two dozen instruments, keeping up with an orbital average data rate of over 50 Mb/sec and assuring prompt generation of standard products, reprocessing data as product generation algorithms change, and storing, and managing information about tens of Petabytes of data over the 15-year life of the mission.

Ramapriyan, H. K.↗

Evaluation of risk from space radiation with high-energy heavy ion beams

The most challenging radiation in space consists of fully ionized atomic elements with high energy for which only the few lowest energy ions can be stopped in shielding materials. The health risk from exposure to these ions and their secondary radiations generated in shield materials is poorly understood since there are few human data and a systematic study in relevant animal model systems has not been made. The accuracy of risk prediction is described as the major limiting factor in the management of space radiation risk. The expected impact of systematic studies is examined using the limited available biological data and models. Given the limitations of current predictions, models must be developed that are able to incorporate the required fundamental scientific data into accurate risk estimates. The important radiation components that can be provided for laboratory testing are identified. The use of ground-based accelerator beams to simulate space radiation is explained and quantitative scientific constraints on such facilities are derived. Three facilities, one each in the United States, in Germany and in Japan, currently have the partial capability to satisfy these constraints. A facility has been proposed using the Brookhaven National Laboratory Booster Synchrotron in the United States; in conjuction with other on-site accelerators, it will be able to provide the full range of heavy ion beams and energies required.

NASA Discipline Radiation Health↗

Visualization at Supercomputing Centers: The Tale of Little Big Iron and the Three Skinny Guys

Supercomputing Centers (SC's) are unique resources that aim to enable scientific knowledge discovery through the use of large computational resources, the Big Iron. Design, acquisition, installation, and management of the Big Iron are activities that are carefully planned and monitored. Since these Big Iron systems produce a tsunami of data, it is natural to co-locate visualization and analysis infrastructure as part of the same facility. This infrastructure consists of hardware (Little Iron) and staff (Skinny Guys). Our collective experience suggests that design, acquisition, installation, and management of the Little Iron and Skinny Guys does not receive the same level of treatment as that of the Big Iron. The main focus of this article is to explore different aspects of planning, designing, fielding, and maintaining the visualization and analysis infrastructure at supercomputing centers. Some of the questions we explore in this article include:"How should the Little Iron be sized to adequately support visualization and analysis of data coming off the Big Iron?" What sort of capabilities does it need to have?" Related questions concern the size of visualization support staff:"How big should a visualization program be (number of persons) and what should the staff do?" and"How much of the visualization should be provided as a support service, and how much should applications scientists be expected to do on their own?"

97 MATHEMATICS AND COMPUTING↗

Requirements for a global data infrastructure in support of CMIP6

The World Climate Research Programme (WCRP)'s Working Group on Climate Modelling (WGCM) Infrastructure Panel (WIP) was formed in 2014 in response to the explosive growth in size and complexity of Coupled Model Intercomparison Projects (CMIPs) between CMIP3 (2005–2006) and CMIP5 (2011–2012). This article presents the WIP recommendations for the global data infrastructure needed to support CMIP design, future growth, and evolution. Developed in close coordination with those who build and run the existing infrastructure (the Earth System Grid Federation; ESGF), the recommendations are based on several principles beginning with the need to separate requirements, implementation, and operations. Other important principles include the consideration of the diversity of community needs around data – a data ecosystem – the importance of provenance, the need for automation, and the obligation to measure costs and benefits. This paper concentrates on requirements, recognizing the diversity of communities involved (modelers, analysts, software developers, and downstream users). Such requirements include the need for scientific reproducibility and accountability alongside the need to record and track data usage. One key element is to generate a dataset-centric rather than system-centric focus, with an aim to making the infrastructure less prone to systemic failure. With these overarching principles and requirements, the WIP has produced a set of position papers, which are summarized in the latter pages of this document. They provide specifications for managing and delivering model output, including strategies for replication and versioning, licensing, data quality assurance, citation, long-term archiving, and dataset tracking. They also describe a new and more formal approach for specifying what data, and associated metadata, should be saved, which enables future data volumes to be estimated, particularly for well-defined projects such as CMIP6. The paper concludes with a future facing consideration of the global data infrastructure evolution that follows from the blurring of boundaries between climate and weather, and the changing nature of published scientific results in the digital age.

54 ENVIRONMENTAL SCIENCES↗

High Performance Databases For Scientific Applications

The goal for this task is to develop an Extensible File System (ELFS). ELFS attacks the problem of the following: 1. Providing high bandwidth performance architectures; 2. Reducing the cognitive burden faced by applications programmers when they attempt to optimize; and 3. Seamlessly managing the proliferation of data formats and architectural differences. The approach for ELFS solution consists of language and run-time system support that permits the specification on a hierarchy of file classes.

French, James C.↗

LC-MEMENTO: A Memory Model for Accelerated Architectures

With the advent of heterogeneous architectures, in particular, with the ubiquity of multi-GPU systems, it is becoming increasingly important to manage device memory efficiently in order to reap the benefits of the additional core count. To date, such responsibility mainly falls on the programmer where device-to-host data communication (and vice versa), if not done properly, may incur costly memory transfer operations and synchronization. The problem may be compounded by additional requirement to maintain system-wide memory consistency that may involve expensive synchronization overhead. In this paper, we present Location Consistency Memory Model for Enhanced Transfer Operations (LC-MEMENTO). This framework considers incorporating runtime techniques for multi-GPU memory management to support relaxed synchronization semantics and memory transfer operations automatically. Specifically, we implement a relaxed form of a memory consistency model based on the Location Consistency (LC) in an Asynchronous Many-Task Runtime (ARTS) and demonstrate that, this memory model enables additional optimization opportunities for the three representative applications encompassing different computational patterns (scientific computation, graphs, data streaming, etc.).

Memory Models, Accelerators, Adaptive Optimization↗

2019 Budget Request for the DOE Computational Science Graduate Fellowship (CSGF) Grant

The Department of Energy Computational Science Graduate Fellowship (DOE CSGF) is necessary to meet the continual challenging national workforce needs that arise as computational science and engineering problems continue to grow in scope and complexity. Computational science and engineering (CSE) is a multidisciplinary approach that uses scientific computing to solve practical problems methods and to supply technical tools across the scientific discovery spectrum. In particular, the DOE CSGF emphasizes high-performance computing (HPC) that enables CSE that advances science and engineering in directions important to the DOE and the economy in general. Over the past half-century, HPC has been an essential tool for DOE’s success. During this period, important missions, such as nuclear stockpile stewardship, have turned to HPC as an essential technology. Entire science disciplines, such as biology and cosmology, have been transformed through the augmentation of scientific observation via HPC. At government laboratories and in industry, DOE CSGF alumni are helping push traditional HPC boundaries while contributing to discoveries in high-energy physics, renewable energy, fusion-reactor design, additive manufacturing, nanomaterials for next-generation batteries and transistors, and turbine and advanced nuclear reactor modeling. In addition, HPC is used to address national health needs that will eventually point to cures both by helping cancer researchers manage and analyze huge troves of data, by simulating biological mechanisms, and by accelerating drug development — including continuing to rise to the challenge of pandemic-related research. A 2023 report from the ASCAC Subcommittee on American Competitiveness and Innovation to the ASCR office, “Can the United States Maintain Its Leadership in High-Performance Computing?” says of the Program, “The CSGF program provides a barometer for disciplines that will be of interest to future DOE computing.” An explosion in scientific and technological data has driven the need for increasingly sophisticated HPC to transform those data into scientific understanding. With access to more and more data and the proliferation of HPC, Machine Learning and Artificial Intelligence are experiencing a renaissance, complementing the now well-established use of computational simulation. Indeed, in its September 2020 subcommittee report on “AI/ML, Data Intensive Science and High-Performance Computing”, the DOE Advanced Scientific Computing Advisory Committee (ASCAC) explicitly called for a fellowship program to train computational and data scientists to tackle exascale and data-intensive computing challenges. This collaboration of empirical and theory-based modeling will increasingly inform federal policymakers whose decisions affect American society and future generations, and it requires highly skilled and intellectually agile computational scientists who can support the fast-moving DOE National Laboratory research environment. In fact, the DOE CSGF program has explicitly and consistently addressed this need.

97 MATHEMATICS AND COMPUTING↗

Marine Corps Base Camp Lejeune New River Estuary (NRE) Regions

Critical military training and testing on lands along the nation’s coastal and estuarine shorelines are increasingly placed at risk because of encroachment pressures in surrounding areas, impairments due to other anthropogenic disturbances, and changes in climate and sea level. The U.S. Department of Defense (DoD) intends to enhance and sustain its training and testing assets and also optimize its stewardship of natural resources through the development and application of an ecosystem-based management approach on DoD installations. To accomplish this goal, particularly for installations in estuarine/coastal environments, the Strategic Environmental Research and Development Program (SERDP) launched the Defense Coastal/Estuarine Research Program (DCERP) as a 10-year effort at Marine Corps Base Camp Lejeune (MCBCL) in North Carolina. The results of the second 5 years of the program (DCERP2) are presented in the DCERP2 Final Report.There were four overarching objectives of DCERP2. The first objective was to understand the effects of climate change impacts, including warming temperatures, variability in the hydrological cycle, storm events, and sea level rise on the coastal ecosystems at MCBCL from observations and measurements made over the 10-year program. The second objective was to understand the carbon cycle of the coastal and terrestrial ecosystems at MCBCL through a highly integrated sampling program. The third objective was to develop models, tools, and indicators to evaluate current and projected future ecosystem state changes and translate scientific findings into actionable information for installation managers. The last objective was to recommend adaptive management strategies to sustain ecosystem natural resources within the context of an active military installation.The New River Estuary regions were digitized using desktop GIS and elevation data to determine the watershed boundaries and bathymetry data is determined by the water depth. Geographic representation of box and flow model produced by DCERP aquatic & esturaine module team, includes micro-watershed data. A spatial approximation of the areas defined in the Estuarine Simulation Model developed by Mark Brush.

54 ENVIRONMENTAL SCIENCES↗

Remote Sensing of Earth Resources: A literature survey with indexes (1970 - 1973 supplement). Section 1: Abstracts

Abstracts of reports, articles, and other documents introduced into the NASA scientific and technical information system between March 1970 and December 1973 are presented in the following areas: agriculture and forestry, environmental changes and cultural resources, geodesy and cartography, geology and mineral resources, oceanography and marine resources, hydrology and water management, data processing and distribution systems, instrumentation and sensors, and economic analysis.

Source record↗

Concept Verification Test - Evaluation of Spacelab/Payload operation concepts

The Concept Verification Test (CVT) procedure is used to study Spacelab operational concepts by conducting mission simulations in a General Purpose Laboratory (GPL) which represents a possible design of Spacelab. In conjunction with the laboratory a Mission Development Simulator, a Data Management System Simulator, a Spacelab Simulator, and Shuttle Interface Simulator have been designed. (The Spacelab Simulator is more functionally and physically representative of the Spacelab than the GPL.) Four simulations of Spacelab mission experimentation were performed, two involving several scientific disciplines, one involving life sciences, and the last involving material sciences. The purpose of the CVT project is to support the pre-design and development of payload carriers and payloads, and to coordinate hardware, software, and operational concepts of different developers and users.

Mcbrayer, R. O.↗

Publications of the Jet Propulsion Laboratory, 1992

JPL Bibliography 39-33 describes and indexes by primary author the externally distributed technical reporting, released during calendar year 1992, that resulted from scientific and engineering work performed or managed by the Jet Propulsion Laboratory. Three classes of publications are included: (1) JPL Publication (92-series) in which the information is complete for a specific accomplishment; (2) articles from the quarterly Telecommunications and Data Acquisition (TDA) Progress Report (42-series) (each collection of articles in this class of publication presents a periodic survey of current accomplishments by the Deep Space Network as well as other developments in Earth-based radio technology); and (3) articles published in the open literature.

Source record↗