SEARCH · Engineering Papers

Results for “data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 613 records · Page 34

All sources NASA NTRS DOE OSTI

DOE OSTI2024

Deeplynx Dag Repository

The DeepLynx DAG repository will contain several Airflow DAGs (Directed Acyclic Graphs) which will be used in the context of DeepLynx's deployed Apache Airflow instance. These DAGs will be used for multiple data management tasks for DeepLynx data, including but not limited to: - bringing data from various sources and tools into DeepLynx - managing sequential data workflows, such as running Python scripts on data to perform analysis and returning the results to DeepLynx - performing any necessary transformation or pre-processing on data coming into DeepLynx from external sources or out of DeepLynx to go to external applications

Brownlee, JarenM.↗

DOE OSTI2021

CSISAR--Complete System Integrated SAR

The CSISAR tool is GUI based and very simple to use. The algorithms are robust, and the unique processing flows, that the user is stepped through, virtually eliminate the possibility of error in GEOINT production. An integrated data manager is a key part of the CSISAR system. This data manager keeps track of the data available to a user and informs the user of what data is available and what can be done with that data. This keeps the user from having to be trained in the nuances of the algorithms. CSISAR also has an integrated product manager, which helps the user identify, view and manage previously made products. CSISAR was originally developed in 2010-2011 as a Windows based system. It was updated in 2015 to be a Linux based system. This SAND report is intended to make the Product Description and User’s Guide for CSISAR (originally included within the software) more widely available. New is a brief addition of Linux-specific installation details.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1993

NELS 2.0 - A general system for enterprise wide information management

NELS, the NASA Electronic Library System, is an information management tool for creating distributed repositories of documents, drawings, and code for use and reuse by the aerospace community. The NELS retrieval engine can load metadata and source files of full text objects, perform natural language queries to retrieve ranked objects, and create links to connect user interfaces. For flexibility, the NELS architecture has layered interfaces between the application program and the stored library information. The session manager provides the interface functions for development of NELS applications. The data manager is an interface between session manager and the structured data system. The center of the structured data system is the Wide Area Information Server. This system architecture provides access to information across heterogeneous platforms in a distributed environment. There are presently three user interfaces that connect to the NELS engine; an X-Windows interface, and ASCII interface and the Spatial Data Management System. This paper describes the design and operation of NELS as an information management tool and repository.

Smith, Stephanie L.↗

DOE OSTI2020

Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON

For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.

Lassnig, Mario↗

NASA NTRS1983

Comparative analysis of data base management systems

A study to determine if the Remote File Inquiry (RFI) system would handle the future requirements of the user community is discussed. RFI is a locally written and locally maintained on-line query/update package. The current and future on-line requirements of the user community were studied. Additional consideration was given to the types of data structuring the users required. The survey indicated the features of greatest benefit were: sort, subtotals, totals, record selection, storage of queries, global updating and the ability to page break. The major deficiencies were: one level of hierarchy, excessive response time, software unreliability, difficult to add, delete and modify records, complicated error messages and the lack of ability to perform interfield comparisons. Missing features users required were: formatted screens, interfield comparions, interfield arithmetic, multiple file access, security and data integrity. The survey team recommended Kennedy Space Center move forward to state-of-the-art software, a Data Base Management System which is thoroughly tested and easy to implement and use.

Smith, R.↗

DOE OSTI2025

HPC Campaign Management: Remote data access with user-defined error bound using ADIOS and ZFP

Remote access to large-scale scientific datasets, like those generated by combustion simulations or other high-performance computing (HPC) applications, presents a significant challenge. Downloading entire datasets is often impractical due to their size and the bandwidth limitations of typical networks. To address this challenge, we propose a novel approach that enables efficient remote access to large datasets distributed across multiple facilities. Our method enables technologies to download only the data values of a select variable, in a select region of interest, to a user-defined accuracy. For this purpose, we extended the ADIOS IO library to provide read functions with user-defined accuracy, a remote data server that understands multidimensional selections of specific variables, steps and accuracy from an ADIOS dataset, and which uses lossy compression on the remote site to reduce the data to be transferred back to the client. In addition, our extension of the ADIOS library collects metadata from multiple datasets in small files called Campaign Archives, which can be shared among project participants on any HPC, cloud or laptop, and which can easily facilitate the discovery of content and pointers to the data location as well as remote access to the data by local tools as if data was local. This feature called Campaign Management, enables a group of scientists to manage related datasets stored in multiple files, across multiple facilities as if it was in a single file/database. We demonstrate the effectiveness of our approach using a 1.5 TB dataset from the S3D combustion simulation on Frontier at the Oak Ridge Leadership Facility. Even a single variable from this dataset, at 64 GB, is too large to be processed on a standard laptop. We show two different reading patterns for 2D plots and 3D visualization, with careful settings that a scientist studying combustion data would do and show that running the same Python scripts on Frontier directly takes comparable time than running them on the local laptop with remote access to the data on Frontier.

Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗

NASA NTRS1997

Providing the Persistent Data Storage in a Software Engineering Environment Using Java/COBRA and a DBMS

An investigation was undertaken to build the software foundation for the WHERE (Web-based Hyper-text Environment for Requirements Engineering) project. The TCM (Toolkit for Conceptual Modeling) was chosen as the foundation software for the WHERE project which aims to provide an environment for facilitating collaboration among geographically distributed people involved in the Requirements Engineering process. The TCM is a collection of diagram and table editors and has been implemented in the C++ programming language. The C++ implementation of the TCM was translated into Java in order to allow the editors to be used for building various functionality of the WHERE project; the WHERE project intends to use the Web as its communication back- bone. One of the limitations of the translated software (TcmJava), which militated against its use in the WHERE project, was persistent data management mechanisms which it inherited from the original TCM; it was designed to be used in standalone applications. Before TcmJava editors could be used as a part of the multi-user, geographically distributed applications of the WHERE project, a persistent storage mechanism must be built which would allow data communication over the Internet, using the capabilities of the Web. An approach involving features of Java, CORBA (Common Object Request Broker), the Web, a middle-ware (Java Relational Binding (JRB)), and a database server was used to build the persistent data management infrastructure for the WHERE project. The developed infrastructure allows a TcmJava editor to be downloaded and run from a network host by using a JDK 1.1 (Java Developer's Kit) compatible Web-browser. The aforementioned editor establishes connection with a server by using the ORB (Object Request Broker) software and stores/retrieves data in/from the server. The server consists of a CORBA object or objects depending upon whether the data is to be made persistent on a single server or multiple servers. The CORBA object providing the persistent data server is implemented using the Java progranu-ning language. It uses the JRB to store/retrieve data in/from a relational database server. The persistent data management system provides transaction and user management facilities which allow multi-user, distributed access to the stored data in a secure manner.

Dhaliwal, Swarn S.↗

NASA NTRS2004

Data Grid Management Systems

The "Grid" is an emerging infrastructure for coordinating access across autonomous organizations to distributed, heterogeneous computation and data resources. Data grids are being built around the world as the next generation data handling systems for sharing, publishing, and preserving data residing on storage systems located in multiple administrative domains. A data grid provides logical namespaces for users, digital entities and storage resources to create persistent identifiers for controlling access, enabling discovery, and managing wide area latencies. This paper introduces data grids and describes data grid use cases. The relevance of data grids to digital libraries and persistent archives is demonstrated, and research issues in data grids and grid dataflow management systems are discussed.

Moore, Reagan W.↗

NASA NTRSDate not supplied

The Human Research Program Grant Lifecycle & Data Integration Schedule: Infographic

The Human Research Program (HRP) Grant Lifecycle process can be described using information from several government authoritative sources with overlapping generalizations. It is the responsibility of the HRP Program Planning & Control (PP&C) Office and the Data Management Integration Office (DMIO)to interpret this information into a cohesive process that can be communicated to the human research organization. PP&C and DMIO have been exploring training opportunities to facilitate understanding through the form of infographics. This communication tool combines eye-catching visuals and text making complex information and data more digestible and shareable. The infographic poster tells a short story about a federally awarded grant and is intended to help both the new and experienced HRP workforce understand the timeline for a research procurement and where their specific tasks fall within the 4 phases of the grant lifecycle. The Pre-Award, Award, Post-Award, and Closeout phases are overlayed with business swimlanes so HRP stakeholders can see where they fit into the timeline rather than working in an isolated part. This includes the Chief Scientist Office (CSO), PP&C, the Data Management Integration Office (DMIO), the Principal Investigator (PI), the Element stakeholders, the LSDA Archivists, and the Research & Operations Integration (ROI) team along with the stakeholders in the Human Health and Performance Directorate. Additionally, the poster introduces (1) the HRP Data Integration Schedule identified in the HRP Data Management Plan HRP-48047, Table 8-2 and (2) a Smartsheet Solution to automate the grants tracking business process. Emphasis is given to the Data Integration Schedule which is the basis for the milestone tasks that are to be completed during the Post-Award phase in the grant lifecycle. The poster is also an opportunity to present the Smartsheet Solution, a software collaboration and work management tool used to assign tasks and track project progress. This PP&C FY24 effort will facilitate an end-to-end solution to track grants and provide insight to the health of HRP grant research procurements using metrics, reports, and dashboards.

J Peace↗

DOE OSTI2020

Open Source Scalable Data Services and Data Fusion for Biological and Environmental Sciences (SBIR Phase I Final Scientific/ Technical Report)

The overarching goal of the project is to develop an integrated open-source scientific data management system (Apache V2 license), ResonantEco, that meets the need of biological and environmental researchers and developers for data management, curation, and data processing for analyses with a wide range of scale and complexity. ResonantEco will provide web enabled data services with features such as unified data interfaces and federated views of data and metadata for heterogeneous data sources with an interactive web client for data exploration. Our use of the term fusion is taken from geospatial (GIS) domain where data fusion is often synonymous with data integration. In particular, data integration in ResonantEco involves combining data residing in different sources and providing users with a unified view of them.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2019

Proactive Data Containers for Scientific Storage (Final Report)

Emerging HPC systems are expected to be deployed with an unprecedented level of complexity, due to a deep system memory/storage hierarchy and heterogeneity of the storage hardware. This hierarchy is expected to range from CPU cache through several levels of volatile memory to nonvolatile memory, traditional hard disks, and tape. Simple and efficient methods of data management and movement through this hierarchy is critical for scientific applications using exascale systems. Existing storage system and I/O (SSIO) technologies face severe challenges in dealing with these requirements. POSIX and MPI I/O standards that are the basis for existing I/O libraries and parallel file systems present fundamental challenges in the areas of scalable metadata operations, semantics-based data movement performance tuning, asynchronous operation, and support for scalable consistency of distributed operations. Moving toward new paradigms for SSIO in the extreme-scale era, we have proposed to investigate novel object-based data abstractions and storage mechanisms that take advantage of the deep storage hierarchy and enable proactive automated performance tuning. In order to achieve these overarching goals, we initiated an effort to develop a fundamental new data abstraction, called Proactive Data Containers (PDC). A PDC is a container within a locus of storage (memory, NVRAM, disk, etc.) that stores science data in an object-centric manner. Managing data as objects enables powerful optimization opportunities for data movement and transformations. The R&D focus of this project are: 1) formulation of object-oriented PDCs and their mapping in different levels of the exascale storage hierarchy; 2) efficient strategies for moving data in deep storage hierarchies using PDCs; 3) techniques for transforming and reorganizing data based on application requirements; and 4) novel analysis paradigms for enabling data transformations and user-defined analysis on data in PDCs. Toward achieving these overarching goals, we designed an object-centric application programing interface (API) for HPC, scalable metadata management for object-centric storage systems, and data movement optimizations such as Data Elevator for moving data between two levels of storage devices and TAPIOCA for efficient aggregation of data on compute nodes. We then implemented several components of the PDC system. They include metadata management, data placement services, remote procedure calls, data aggregation, etc. We have put them together into the overall PDC framework.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

EspressoDB: A scientific database for managing high-performance computing workflows

EspressoDB is a programmatic object-relational mapping (ORM) data management framework implemented in Python and based on the Django web framework. EspressoDB was developed to streamline data management, centralize and promote data integrity, while providing domain flexibility and ease of use. It is designed to directly integrate in utilized software to allow dynamical access to vast amount of relational data at runtime.

Chang, Chia↗

NASA NTRS1992

Information Management for a Large Multidisciplinary Project

In 1989, NASA's Langley Research Center (LaRC) initiated the High-Speed Airframe Integration Research (HiSAIR) Program to develop and demonstrate an integrated environment for high-speed aircraft design using advanced multidisciplinary analysis and optimization procedures. The major goals of this program were to evolve the interactions among disciplines and promote sharing of information, to provide a timely exchange of information among aeronautical disciplines, and to increase the awareness of the effects each discipline has upon other disciplines. LaRC historically has emphasized the advancement of analysis techniques. HiSAIR was founded to synthesize these advanced methods into a multidisciplinary design process emphasizing information feedback among disciplines and optimization. Crucial to the development of such an environment are the definition of the required data exchanges and the methodology for both recording the information and providing the exchanges in a timely manner. These requirements demand extensive use of data management techniques, graphic visualization, and interactive computing. HiSAIR represents the first attempt at LaRC to promote interdisciplinary information exchange on a large scale using advanced data management methodologies combined with state-of-the-art, scientific visualization techniques on graphics workstations in a distributed computing environment. The subject of this paper is the development of the data management system for HiSAIR.

Kennie H Jones↗

NASA NTRS1977

Substructure coupling: A different approach

The substructure analysis of the Lockheed L-1011 Dash 500 long range derivative without use of the NASTRAN level 16 multistage substructuring capability is described. It is presented as an example of how a large structural analysis can be organized into manageable independent tasks and how preprocessors, model definition conventions, and data management programs can be used to simplify the data management and model documentation. A procedure is described to analyze a single substructure by modifying the static solution to solve the interface compatibility equations simultaneously with the substructure solution.

Lahey, R. S.↗

NASA NTRS1993

Development of a naming convention and data ma nagement system

This contract covered work in data management, acquisition, and analysis for the Earth Science and Applications Division (ESAD) of the Space Science Laboratory of MSFC. Under this contract, a naming convention and data management system were developed for ESAD data sets, data visualization, and standard data format issues were investigated, and specific data analysis and management needs were addressed for several ESAD projects and data sets. In the following sections, details of the research performed over the duration of this contract are summarized.

Mcnider, Richard T.↗

NASA NTRS1981

Using CAD/CAM to improve productivity - The IPAD approach

Progress in designing and implementing CAD/CAM systems as a result of the NASA Integrated Programs for Aerospace-Vehicle Design is discussed. Essential software packages have been identified as executive, data management, general user, and geometry and graphics software. Data communication, as a means to integrate data over a network of computers of different vendors, provides data management with the capability of meeting design and manufacturing requirements of the vendors. Geometry software is dependent on developmental success with solid geometry software, which is necessary for continual measurements of, for example, a block of metal while it is being machined. Applications in the aerospace industry, such as for design, analysis, tooling, testing, quality control, etc., are outlined.

Fulton, R. E.↗

NASA NTRS1991

Data principles for the U.S. Global Change Research Program

The U.S. Interagency Working Group on Data Management for Global Change has developed a set of data management and access principles. The overall purpose of these statements of principle is to stimulate responsible stewardship for data and related information and to facilitate full and open access to them. These statements have been accepted by the U.S. Agencies responsible for the Global Change Research Program. The statements of principle are presented and discussed.

Ludwig, George H.↗

NASA NTRS1993

Session on techniques and resources for storm-scale numerical weather prediction

The session on techniques and resources for storm-scale numerical weather prediction are reviewed. The recommendations of this group are broken down into three area: modeling and prediction, data requirements in support of modeling and prediction, and data management. The current status, modeling and technological recommendations, data requirements in support of modeling and prediction, and data management are addressed.

Droegemeier, Kelvin↗