Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data storage data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

Software Defined Cyberinfrastructure

Within and across thousands of science labs, researchers and students struggle to manage data produced in experiments, simulations, and analyses. Largely manual research data lifecycle management processes mean that much time is wasted, research results are often irreproducible, and data sharing and reuse remain rare. In response, we propose a new approach to data lifecycle management in which researchers are empowered to define the actions to be performed at individual storage systems when data are created or modified: actions such as analysis, transformation, copying, and publication. We term this approach software-defined cyberinfrastructure because users can implement powerful data management policies by deploying rules to local storage systems, much as software-defined networking allows users to configure networks by deploying rules to switches.We argue that this approach can enable a new class of responsive distributed storage infrastructure that will accelerate research innovation by allowing any researcher to associate data workflows with data sources, whether local or remote, for such purposes as data ingest, characterization, indexing, and sharing. We report on early experiments with this approach in the context of experimental science, in which a simple if-trigger-then-action (IFTA) notation is used to define rules.

Foster, Ian↗

Enabling the Write-Once Read-Many (WORM) Dashboard and Web Interface for HPC Data Storage

Idaho National Laboratory (INL), supported by the Department of Energy Office of Nuclear Energy (DOE-NE) through the Nuclear Science User Facilities, provides access to supercomputer systems and data storage along with support staff for system management, software installation, cybersecurity, and user support to the broader DOE-NE user community. A key component of this is providing long-term or perpetual data storage as part of efforts to improve reproducibility in modeling and simulation in addition to supporting simulations with regulatory compliance requirements. This milestone details the deployment of a new type of storage option for users requiring tamper resistant perpetual storage of data and the associated support for data control and release via the data storage web interface. The perpetual data storage system is integrated within the Open OnDemand framework deployed at INL and is called the ?WORM? in recognition of its write-once read-many characteristics.

99 GENERAL AND MISCELLANEOUS↗

Experiences From NASA/Langley's DMSS Project

There is a trend in institutions with high performance computing and data management requirements to explore mass storage systems with peripherals directly attached to a high speed network. The Distributed Mass Storage System (DMSS) Project at the NASA Langley Research Center (LaRC) has placed such a system into production use. This paper will present the experiences, both good and bad, we have had with this system since putting it into production usage. The system is comprised of: 1) National Storage Laboratory (NSL)/UniTree 2.1, 2) IBM 9570 HIPPI attached disk arrays (both RAID 3 and RAID 5), 3) IBM RS6000 server, 4) HIPPI/IPI3 third party transfers between the disk array systems and the supercomputer clients, a CRAY Y-MP and a CRAY 2, 5) a "warm spare" file server, 6) transition software to convert from CRAY's Data Migration Facility (DMF) based system to DMSS, 7) an NSC PS32 HIPPI switch, and 8) a STK 4490 robotic library accessed from the IBM RS6000 block mux interface. This paper will cover: the performance of the DMSS in the following areas: file transfer rates, migration and recall, and file manipulation (listing, deleting, etc.); the appropriateness of a workstation class of file server for NSL/UniTree with LaRC's present storage requirements in mind the role of the third party transfers between the supercomputers and the DMSS disk array systems in DMSS; a detailed comparison (both in performance and functionality) between the DMF and DMSS systems LaRC's enhancements to the NSL/UniTree system administration environment the mechanism for DMSS to provide file server redundancy the statistics on the availability of DMSS the design and experiences with the locally developed transparent transition software which allowed us to make over 1.5 million DMF files available to NSL/UniTree with minimal system outage

Source record↗

Utilization of GIS technology to support development of flow and contaminant fate and transport models at US DOE sites - 15457

U.S. Department of Energy (DOE) sites have accumulated years of data and millions of spatial and temporal records related to the hydrological cycle, contaminant transport, and parameters of remediation technologies. In addition, numerical models which use field monitoring data to help understand flow and transport produce gigabytes of computed spatial and temporal data for each computation node. There is a need therefore for an advanced spatial data structure that can be used to address the management, processing, and analysis of spatial and temporal numerical modeling data derived from multiple sources. This can be accomplished using a geo-database. The development of an ArcSDE geo-database facilitates the centralized storage, backup, accessibility, organization, and management of model configuration and computed simulation data, putting it into a structured, coherent, and logical computer-supported system. GIS data processing and automation would enable faster and hence more complex analyses of field test data. The Applied Research Center (ARC) at Florida International University (FIU) has provided analysis of flow and transport for several watersheds at DOE's Oak Ridge Reservation (ORR), including East Fork Poplar Creek (EFPC), Upper EFPC in the Y-12 National Security Complex area (Y-12 NSC) and White Oak Creek (WOC). Integrated surface and subsurface flow, fate and transport models were developed to provide analysis of contaminant patterns within each watershed. In addition, digital monitoring data available from the Oak Ridge Environmental Information System (OREIS) related to mercury (Hg) contamination and remediation within these watersheds was used for calibration and verification of the model. Experimental studies were also carried out which provided kinetic and equilibrium data about important parameters related to Hg transport, speciation and methylation/demethylation kinetics within the watershed. Geographic Information Systems (GIS) technology was employed to support the modeling work through storage and geo-processing of spatial and temporal data required by the models and to produce hydrogeological maps for visualization. An ArcGIS geo-database was developed for centralized storage and management of experimental and computed model data and its capabilities were extended over the years using tools such as ModelBuilder combined with Python scripting to automate repetitive tasks, perform statistical analyses and generate maps and reports. The hydrologic geo-database model developed possesses a structure that enables linkage with scalable hydrologic modeling tools/applications to model hydrologic systems, and in this case, enables the testing of the potential impacts of various remediation scenarios on the ORR watersheds. The ArcSDE geo-database can also be used to automate and simplify the process of calling stored GIS and time series data. This geo-database can serve as a tool for contaminant flow and transport analyses which require large amounts of high-quality spatial and temporal data in order to ensure reliability and validity of modeling results. The existing geo-database structure developed for the hydrological modeling at ORR was designed to be replicable for application at other DOE sites. An investigation of downloadable free/open source GIS software along with required security protocols to facilitate online querying of the database was also conducted to determine methods by which project-derived data can be more easily shared with other project stakeholders such as DOE personnel and site contractors. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Integration experiments and performance studies of a COTS parallel archive system

Current and future Archive Storage Systems have been asked to (a) scale to very high bandwidths, (b) scale in metadata performance, (c) support policy-based hierarchical storage management capability, (d) scale in supporting changing needs of very large data sets, (e) support standard interface, and (f) utilize commercial-off-the-shelf (COTS) hardware. Parallel file systems have been asked to do the same thing but at one or more orders of magnitude faster in performance. Archive systems continue to move closer to file systems in their design due to the need for speed and bandwidth, especially metadata searching speeds such as more caching and less robust semantics. Currently the number of extreme highly scalable parallel archive solutions is very small especially those that will move a single large striped parallel disk file onto many tapes in parallel. We believe that a hybrid storage approach of using COTS components and innovative software technology can bring new capabilities into a production environment for the HPC community much faster than the approach of creating and maintaining a complete end-to-end unique parallel archive software solution. In this paper, we relay our experience of integrating a global parallel file system and a standard backup/archive product with a very small amount of additional code to provide a scalable, parallel archive. Our solution has a high degree of overlap with current parallel archive products including (a) doing parallel movement to/from tape for a single large parallel file, (b) hierarchical storage management, (c) ILM features, (d) high volume (non-single parallel file) archives for backup/archive/content management, and (e) leveraging all free file movement tools in Linux such as copy, move, Is, tar, etc. We have successfully applied our working COTS Parallel Archive System to the current world's first petafiop/s computing system, LANL's Roadrunner machine, and demonstrated its capability to address requirements of future archival storage systems.

99 GENERAL AND MISCELLANEOUS↗

Storage management in Ada. Three reports. Volume 1: Storage management in Ada as a risk to the development of reliable software. Volume 2: Relevant aspects of language. Volume 3: Requirements of the language versus manifestations of current implementations

The risk to the development of program reliability is derived from the use of a new language and from the potential use of new storage management techniques. With Ada and associated support software, there is a lack of established guidelines and procedures, drawn from experience and common usage, which assume reliable behavior. The risk is identified and clarified. In order to provide a framework for future consideration of dynamic storage management on Ada, a description of the relevant aspects of the language is presented in two sections: Program data sources, and declaration and allocation in Ada. Storage-management characteristics of the Ada language and storage-management characteristics of Ada implementations are differentiated. Terms that are used are defined in a narrow and precise sense. The storage-management implications of the Ada language are described. The storage-management options available to the Ada implementor and the implications of the implementor's choice for the Ada programmer are also described.

Auty, David↗

An open-source data storage and visualization platform for collaborative qubit control

Developing collaborative research platforms for quantum bit control is crucial for driving innovation in the field, as they enable the exchange of ideas, data, and implementation to achieve more impactful outcomes. Furthermore, considering the high costs associated with quantum experimental setups, collaborative environments are vital for maximizing resource utilization efficiently. However, the lack of dedicated data management platforms presents a significant obstacle to progress, highlighting the necessity for essential assistive tools tailored for this purpose. Current qubit control systems are unable to handle complicated management of extensive calibration data and do not support effectively visualizing intricate quantum experiment outcomes. In this paper, we introduce Qubit Control Storage and Visualization ( QubiCSV ), a platform specifically designed to meet the demands of quantum computing research, focusing on the storage and analysis of calibration and characterization data in qubit control systems. As an open-source tool, QubiCSV facilitates efficient data management of quantum computing, providing data versioning capabilities for data storage and allowing researchers and programmers to interact with qubits in real time. The insightful visualization are developed to interpret complex quantum experiments and optimize qubit performance. QubiCSV not only streamlines the handling of qubit control system data but also improves the user experience with intuitive visualization features, making it a valuable asset for researchers in the quantum computing domain.

97 MATHEMATICS AND COMPUTING↗

Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON

For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.

Lassnig, Mario↗

Data management applications

Kennedy Space Center's primary institutional computer is a 4 megabyte IBM 4341 with 3.175 billion characters of IBM 3350 disc storage. This system utilizes the Software AG product known as ADABAS with the on line user oriented features of NATURAL and COMPLETE as a Data Base Management System (DBMS). It is operational under the OS/VSI and is currently supporting batch/on line applications such as Personnel, Training, Physical Space Management, Procurement, Office Equipment Maintenance, and Equipment Visibility. A third and by far the largest DBMS application is known as the Shuttle Inventory Management System (SIMS) which is operational on a Honeywell 6660 (dedicated) computer system utilizing Honeywell Integrated Data Storage I (IDSI) as the DBMS. The SIMS application is designed to provide central supply system acquisition, inventory control, receipt, storage, and issue of spares, supplies, and materials.

Source record↗

Information management system: A summary discussion

An information management system is proposed for use in the space shuttle sortie, the modular space station, the tracking data relay satellite and associated ground support systems. Several different information management functions, including data acquisition, transfer, storage, processing, control and display are integrated in the system.

Sayers, R. S.↗

Virtual Reality Training: "Cybersickness" and Effects on Sensorimotor Functions

The overall goal of this study is to examine the extent to which exposure to virtual reality (VR) systems produces motion sickness and disrupts sensorimotor functions. Two of the major problems in using VRs are: 1) potential "cybersickness", a form of motion sickness, and 2) maladaptive sensorimotor coordination following virtual environment (VE) training. It is likely that users will eventually adapt to any unpleasant perceptual experiences in a virtual environment. However the most critical problem for training applications is that sensorimotor coordination strategies learned in the VE may not be similar to the responses required in the real environment. This study will evaluate and compare responses to the two types of VR delivery systems (head-mounted display [HMD] and a dome-projection system [DOME]), two exposure duration periods (30 minutes or 60 minutes), and repeated exposures (3 sessions). Specific responses that we will examine include cybersickness severity and symptom patterns, and several sensorimotor functions (eye-hea.d and eye-head-hand coordination, and postural equilibrium). To date, all hardware and software acquisition, development, integration and testing has been completed. A database has been developed and tested for the input, management and storage of all questionnaire data. All data analysis scripts have been developed and tested. Data was collected from 20 subjects in a pilot study that was conducted to determine the amount of training necessary to achieve a stable performance level. Seven subjects are currently enrolled in the study designed to examine the effects of exposure to VE systems on postural control. Data has been collected from two subjects, and it is expected that the results from ten subjects will be presented.

Harm, Deborah L.↗

NCDC mass storage systems and technologies

National Climatic Data Center (NCDC) data management issues are discussed, such as nature of holdings, history of the site, popularity of data sets, media/technology used for storage, volume distributed per month, mode of distribution, most frequently encountered problems, type of media requested/used, and evolution of media. Current holdings at NCDC are 107.8 terabytes of digital data and about 0.3 terabytes of manuscript data. The nexrad radar system is expected to generate approximately 88 terabytes per year by 1996.

Davis, Dick↗

Information Power Grid: Distributed High-Performance Computing and Large-Scale Data Management for Science and Engineering

We use the term "Grid" to refer to distributed, high performance computing and data handling infrastructure that incorporates geographically and organizationally dispersed, heterogeneous resources that are persistent and supported. This infrastructure includes: (1) Tools for constructing collaborative, application oriented Problem Solving Environments / Frameworks (the primary user interfaces for Grids); (2) Programming environments, tools, and services providing various approaches for building applications that use aggregated computing and storage resources, and federated data sources; (3) Comprehensive and consistent set of location independent tools and services for accessing and managing dynamic collections of widely distributed resources: heterogeneous computing systems, storage systems, real-time data sources and instruments, human collaborators, and communications systems; (4) Operational infrastructure including management tools for distributed systems and distributed resources, user services, accounting and auditing, strong and location independent user authentication and authorization, and overall system security services The vision for NASA's Information Power Grid - a computing and data Grid - is that it will provide significant new capabilities to scientists and engineers by facilitating routine construction of information based problem solving environments / frameworks. Such Grids will knit together widely distributed computing, data, instrument, and human resources into just-in-time systems that can address complex and large-scale computing and data analysis problems. Examples of these problems include: (1) Coupled, multidisciplinary simulations too large for single systems (e.g., multi-component NPSS turbomachine simulation); (2) Use of widely distributed, federated data archives (e.g., simultaneous access to metrological, topological, aircraft performance, and flight path scheduling databases supporting a National Air Space Simulation systems}; (3) Coupling large-scale computing and data systems to scientific and engineering instruments (e.g., realtime interaction with experiments through real-time data analysis and interpretation presented to the experimentalist in ways that allow direct interaction with the experiment (instead of just with instrument control); (5) Highly interactive, augmented reality and virtual reality remote collaborations (e.g., Ames / Boeing Remote Help Desk providing field maintenance use of coupled video and NDI to a remote, on-line airframe structures expert who uses this data to index into detailed design databases, and returns 3D internal aircraft geometry to the field); (5) Single computational problems too large for any single system (e.g. the rotocraft reference calculation). Grids also have the potential to provide pools of resources that could be called on in extraordinary / rapid response situations (such as disaster response) because they can provide common interfaces and access mechanisms, standardized management, and uniform user authentication and authorization, for large collections of distributed resources (whether or not they normally function in concert). IPG development and deployment is addressing requirements obtained by analyzing a number of different application areas, in particular from the NASA Aero-Space Technology Enterprise. This analysis has focussed primarily on two types of users: the scientist / design engineer whose primary interest is problem solving (e.g. determining wing aerodynamic characteristics in many different operating environments), and whose primary interface to IPG will be through various sorts of problem solving frameworks. The second type of user is the tool designer: the computational scientists who convert physics and mathematics into code that can simulate the physical world. These are the two primary users of IPG, and they have rather different requirements. The results of the analysis of the needs of these two types of users provides a broad set of requirements that gives rise to a general set of required capabilities. The IPG project is intended to address all of these requirements. In some cases the required computing technology exists, and in some cases it must be researched and developed. The project is using available technology to provide a prototype set of capabilities in a persistent distributed computing testbed. Beyond this, there are required capabilities that are not immediately available, and whose development spans the range from near-term engineering development (one to two years) to much longer term R&D (three to six years). Additional information is contained in the original.

Johnston, William E.↗

Enforcing End-to-end I/O Policies for Scientific Workflows using Software-Defined Storage Resource Enclaves

Data-intensive knowledge discovery requires scientific applications to run concurrently with analytics and visualization codes executing in situ for timely output inspection and knowledge extraction. Consequently, I/O pipelines of scientific workflows can be long and complex because they comprise many stages of analytics across different layers of the I/O stack of high-performance computing systems. Performance limitations at any I/O layer or stage can cause an I/O bottleneck resulting in greater than expected end-to-end I/O latency. In this paper, we present the design and implementation of a novel data management infrastructure called Software-Defined Storage Resource Enclaves (SIREN) at system level to enforce end-to-end policies that dictate an I/O pipeline's performance. SIREN provides an I/O performance interface for users to specify the desired storage resources in the context of in-situ analytics. If suboptimal performance of analytics is caused by an I/O bottleneck when data are transferred between simulations and analytics, schedulers in different layers of the I/O stack automatically provide the guaranteed lower bounds on I/O throughput. Lastly, our experimental results demonstrate that SIREN provides performance isolation among scientific workflows sharing multiple storage servers across two I/O layers (burst buffer and parallel file systems) while maintaining high system scalability and resource utilization.

97 MATHEMATICS AND COMPUTING↗

Preservation Environments

The long-term preservation of digital entities requires mechanisms to manage the authenticity of massive data collections that are written to archival storage systems. Preservation environments impose authenticity constraints and manage the evolution of the storage system technology by building infrastructure independent solutions. This seeming paradox, the need for large archives, while avoiding dependence upon vendor specific solutions, is resolved through use of data grid technology. Data grids provide the storage repository abstractions that make it possible to migrate collections between vendor specific products, while ensuring the authenticity of the archived data. Data grids provide the software infrastructure that interfaces vendor-specific storage archives to preservation environments.

Moore, Reagan W.↗

dCache: The Storage System of Choice for Data-Intensive Applications

The ever-increasing volumes of data produced by modern scientific facilities like EuXFEL and LHC put significant stress on data management infrastructure operated by laboratories and research centers. The challenges to be addressed span the entire data life cycle, from ingest and efficient data analysis to long-term preservation, typically involving large tape libraries. dCache, a storage system developed in collaboration between the Deutsches Elektronen-Synchrotron (DESY), Fermi National Accelerator Laboratory, and Nordic e-Infrastructure Collaboration (NeIC), is designed to manage a large number of disk servers and to facilitate transparent data migration to and from archival storage. Its multifaceted approach offers a unified method to support a variety of scientific use cases with the same storage infrastructure, including high-throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and long-term data preservation on tertiary storage. Initially developed for high energy physics (HEP) experiments, dCache is now used by various scientific communities, including astrophysics, biomedical research, and life sciences, each having specific requirements. This paper presents architecture, deployment strategies, performance and scalability enhancements, and recent advancements in dCache addressing the needs of scientific communities. Finally, we touch on the development and release process, ensuring the software’s high quality.

DCache↗

Using GIS for Processing, Analysis and Visualization of Hydrological Model Data - 16202

The Applied Research Center at Florida International University (ARC-FIU) is supporting the soil and groundwater remediation efforts of the U.S. Department of Energy's (DOE's) Savannah River Site (SRS) by developing a surface water model to simulate the hydrology and the fate and transport of contaminants and sediment in the Tims Branch watershed. Advances in ArcGIS software through the development of spatial GIS database (geo-database) technology, coupled with the development of data models such as ArcHydro which possesses a spatial relational database management system (RDMS) schema and relationship structure specific to hydrologic systems, provides modelers with tools and applications to assist in the processing, analysis and visualization of flow and contaminant transport data. In addition, the coupling of this type of geo-database structure with a numerical modeling package such as MIKE SHE/MIKE 11 can serve as an efficient tool that significantly reduces the time needed for data preparation. The MIKE modeling system has a geographic information systems (GIS) user interface built into its system that can directly use GIS data for model inputs. GIS-based hydrologic models can provide a spatial element that other hydrologic models lack. GIS is important for processing and analysis of large spatio-temporal datasets derived from multiple sources that are being used for setting up of models, model calibration and simulation of flow and contaminant fate and transport. GIS-based approaches in hydrological modeling can also provide the benefit of combining different layers of geographic data to create new integrated information which can be quite useful for creating dependent or independent hydrological variables. ARC-FIU has developed an ArcGIS geo-database to support the hydrological modeling work being conducted at SRS. The SRS geo-database was developed based on the ArcHydro and ArcGIS Base Map data models with modifications made for project specific input parameters. The significance of this approach was to ensure its replicability for potential application at other DOE sites. The SRS geo-database assists in the storage and management of GIS and time series data being used for hydrological model development at Savannah River Site. Other GIS tools such as ArcGIS Model Builder coupled with Python scripting have enabled the automation of many of the repetitive geo-processing tasks required for pre- and post-processing of the hydrological modeling data. Automation of many of the data processing tasks has facilitated faster and hence more complex analyses of field test data. The toolbox created is also a scalable and reusable application that can be implemented in other watersheds. This paper describes the use of ArcGIS tools for the pre-processing, spatial delineation and analysis of hydrological model data; for automation of repetitive geo-processing tasks; and for visualization of different hydrological components of the Tims Branch watershed. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗