Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data lifecycle”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2001

Linking Science Analysis with Observation Planning: A Full Circle Data Lifecycle

A clear goal of the Virtual Observatory (VO) is to enable new science through analysis of integrated astronomical archives. An additional and powerful possibility of the VO is to link and integrate these new analyses with planning of new observations. By providing tools that can be used for observation planning in the VO, the VO will allow the data lifecycle to come full circle: from theory to observations to data and back around to new theories and new observations. The Scientist's Expert Assistant (SEA) Simulation Facility (SSF) is working to combine the ability to access existing archives with the ability to model and visualize new observations. Integrating the two will allow astronomers to better use the integrated archives of the VO to plan and predict the success of potential new observations. The full circle lifecycle enabled by SEA can allow astronomers to make substantial leaps in the quality of data and science returns on new observations. Our paper will examine the exciting potential of integrating archival analysis with new observation planning, such as performing data calibration analysis on archival images and using that analysis to predict the success of new observations, or performing dynamic signal-to-noise analysis combining historical results with modeling of new instruments or targets. We will also describe how the development of the SSF is progressing and what has been its successes and challenges.

Jones, Jeremy↗
NASA NTRS2001

Linking Science Analysis with Observation Planning: A Full Circle Data Lifecycle

A clear goal of the Virtual Observatory (VO) is to enable new science through analysis of integrated astronomical archives. An additional and powerful possibility of the VO is to link and integrate these new analyses with planning of new observations. By providing tools that can be used for observation planning in the VO, the VO will allow the data lifecycle to come full circle: from theory to observations to data and back around to new theories and new observations. The Scientist's Expert Assistant (SEA) Simulation Facility (SSF) is working to combine the ability to access existing archives with the ability to model and visualize new observations. Integrating the two will allow astronomers to better use the integrated archives of the VO to plan and predict the success of potential new observations more efficiently, The full circle lifecycle enabled by SEA can allow astronomers to make substantial leaps in the quality of data and science returns on new observations. Our paper examines the exciting potential of integrating archival analysis with new observation planning, such as performing data calibration analysis on archival images and using that analysis to predict the success of new observations, or performing dynamic signal-to-noise analysis combining historical results with modeling of new instruments or targets. We will also describe how the development of the SSF is progressing and what have been its successes and challenges.

Grosvenor, Sandy↗
NASA NTRSDate not supplied

The Human Research Program Grant Lifecycle & Data Integration Schedule: Infographic

The Human Research Program (HRP) Grant Lifecycle process can be described using information from several government authoritative sources with overlapping generalizations. It is the responsibility of the HRP Program Planning & Control (PP&C) Office and the Data Management Integration Office (DMIO)to interpret this information into a cohesive process that can be communicated to the human research organization. PP&C and DMIO have been exploring training opportunities to facilitate understanding through the form of infographics. This communication tool combines eye-catching visuals and text making complex information and data more digestible and shareable. The infographic poster tells a short story about a federally awarded grant and is intended to help both the new and experienced HRP workforce understand the timeline for a research procurement and where their specific tasks fall within the 4 phases of the grant lifecycle. The Pre-Award, Award, Post-Award, and Closeout phases are overlayed with business swimlanes so HRP stakeholders can see where they fit into the timeline rather than working in an isolated part. This includes the Chief Scientist Office (CSO), PP&C, the Data Management Integration Office (DMIO), the Principal Investigator (PI), the Element stakeholders, the LSDA Archivists, and the Research & Operations Integration (ROI) team along with the stakeholders in the Human Health and Performance Directorate. Additionally, the poster introduces (1) the HRP Data Integration Schedule identified in the HRP Data Management Plan HRP-48047, Table 8-2 and (2) a Smartsheet Solution to automate the grants tracking business process. Emphasis is given to the Data Integration Schedule which is the basis for the milestone tasks that are to be completed during the Post-Award phase in the grant lifecycle. The poster is also an opportunity to present the Smartsheet Solution, a software collaboration and work management tool used to assign tasks and track project progress. This PP&C FY24 effort will facilitate an end-to-end solution to track grants and provide insight to the health of HRP grant research procurements using metrics, reports, and dashboards.

J Peace↗
DOE OSTI2019

Experimental Data Curation at Large Instrument Facilities with Open Source Software

The National Synchrotron Light Source II operating at Brookhaven National Laboratory since 2014 for the US Department of Energy is one of the newest and brightest storage-ring synchrotron facility in the world. NSLS-II, like other facilities, provides pre-processing of the raw data and some analysis capabilities to its users. We describe the research collaborations and open source infrastructure developed at large instrument facilities such as NSLS-II for the purpose of curating high value scientific data along the early stages of the data lifecycle. Data acquisition and curation tasks include storing experiment configuration, detector metadata, raw data acquisition with infrastructure that converts proprietary instrument formats to industry standards. In addition, we describe a specific effort for discovering sample information at NSLS-II and tracing the provenance of analysis performed on acquired images. We show that curation tasks must be embedded into software along the data life cycle for effectiveness and ease of use, and that loosely defined collaborations evolve around shared open source tools. Finally we discuss best practices for experimental metadata capture in such facilities, data access and the new challenges of scale and complexity posed by AI-based discovery for the synthesis of new materials.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2017

Software Defined Cyberinfrastructure

Within and across thousands of science labs, researchers and students struggle to manage data produced in experiments, simulations, and analyses. Largely manual research data lifecycle management processes mean that much time is wasted, research results are often irreproducible, and data sharing and reuse remain rare. In response, we propose a new approach to data lifecycle management in which researchers are empowered to define the actions to be performed at individual storage systems when data are created or modified: actions such as analysis, transformation, copying, and publication. We term this approach software-defined cyberinfrastructure because users can implement powerful data management policies by deploying rules to local storage systems, much as software-defined networking allows users to configure networks by deploying rules to switches.We argue that this approach can enable a new class of responsive distributed storage infrastructure that will accelerate research innovation by allowing any researcher to associate data workflows with data sources, whether local or remote, for such purposes as data ingest, characterization, indexing, and sharing. We report on early experiments with this approach in the context of experimental science, in which a simple if-trigger-then-action (IFTA) notation is used to define rules.

Foster, Ian↗
DOE OSTI2021

WA-Omic_LA.1.0 - Quantitative Lipidomics, Metabolomics, and Sequencing (16S/ITS) Publication Data DOI Package

Corresponding Data Publication: "Rapid remodeling of the soil lipidome in response to a drying-rewetting event." This study reveals specific changes in lipids and metabolites that are indicative of stress adaptation, substrate use, and cellular recovery during soil drying and subsequent rewetting. Drought induced nutrient limitation was reflected in the lipidome and polar metabalome, both of which rapidly shifted (within hours) upon rewet. Reduced nutrient access in dry soil caused the replacement of glycerophospholipids with phosphorus-free lipids and impeded resource-expensive osmolyte accumulation. Elevated levels of ceramides and lipids with long chain polyunsaturated fatty acids, in dry soil suggests that lipids play an important role in fungal drought tolerance. Increasing abundance of bacterial glycerophospholipids and triacylglycerols with fatty acids typical of bacteria and polar metabolites suggest metabolic recovery in representative bacteria once the environmental conditions are conducive for growth. These results underscore the importance of the soil lipidome as a robust indicator of microbial community responses, especially at the short time scales of cell-environment reactions. Data package contents reported here are the first version and contain pre- and post-processed data acquisition and subsequent downstream analysis files using various data source instrument method techniques and Mass Spectroscopy (MS) EMSL capabilities. This publication data package DOI is a comprehensive high-throughput multi-omics data lifecycle collection containing processed data method metadata. Support files include additional data download “Read Me” file containing data descriptor information and data source application ontologies (see data dictionary). Reported data download contents are structured for compliance with project data sharing guidelines, community standards initiatives, and sponsor stakeholder policies supporting FAIR data principles. For increased data availability and interoperability, GC-MS/LC-MS mass spectrometry datasets (Thermo .raw ) were deposited at the MassIVE database repository under the related data accession MSV000086931 and can be accessed by using the API. Statistical data processing software, analysis tools, and data workflows are listed below corresponding to the host repository long-term location.

Amplicon sequencing 16S ITS LC-MS/MS lipidomics mu↗
DOE OSTI2021

WA-Omic_LA.1.0 - Quantitative Lipidomics, Metabolomics, and Sequencing (16S/ITS) Publication Data DOI Package

Corresponding Data Publication: "Rapid remodeling of the soil lipidome in response to a drying-rewetting event." This study reveals specific changes in lipids and metabolites that are indicative of stress adaptation, substrate use, and cellular recovery during soil drying and subsequent rewetting. Drought induced nutrient limitation was reflected in the lipidome and polar metabalome, both of which rapidly shifted (within hours) upon rewet. Reduced nutrient access in dry soil caused the replacement of glycerophospholipids with phosphorus-free lipids and impeded resource-expensive osmolyte accumulation. Elevated levels of ceramides and lipids with long chain polyunsaturated fatty acids, in dry soil suggests that lipids play an important role in fungal drought tolerance. Increasing abundance of bacterial glycerophospholipids and triacylglycerols with fatty acids typical of bacteria and polar metabolites suggest metabolic recovery in representative bacteria once the environmental conditions are conducive for growth. These results underscore the importance of the soil lipidome as a robust indicator of microbial community responses, especially at the short time scales of cell-environment reactions. Data package contents reported here are the first version and contain pre- and post-processed data acquisition and subsequent downstream analysis files using various data source instrument method techniques and Mass Spectroscopy (MS) EMSL capabilities. This publication data package DOI is a comprehensive high-throughput multi-omics data lifecycle collection containing processed data method metadata. Support files include additional data download “Read Me” file containing data descriptor information and data source application ontologies (see data dictionary). Reported data download contents are structured for compliance with project data sharing guidelines, community standards initiatives, and sponsor stakeholder policies supporting FAIR data principles. For increased data availability and interoperability, GC-MS/LC-MS mass spectrometry datasets (Thermo .raw ) were deposited at the MassIVE database repository under the related data accession MSV000086931 and can be accessed by using the API. Statistical data processing software, analysis tools, and data workflows are listed below corresponding to the host repository long-term location.

Amplicon sequencing 16S ITS LC-MS/MS lipidomics mu↗
DOE OSTI2021

WA-Omic_LA.1.0 - Quantitative Lipidomics, Metabolomics, and Sequencing (16S/ITS) Publication Data DOI Package

Corresponding Data Publication: "Rapid remodeling of the soil lipidome in response to a drying-rewetting event." This study reveals specific changes in lipids and metabolites that are indicative of stress adaptation, substrate use, and cellular recovery during soil drying and subsequent rewetting. Drought induced nutrient limitation was reflected in the lipidome and polar metabalome, both of which rapidly shifted (within hours) upon rewet. Reduced nutrient access in dry soil caused the replacement of glycerophospholipids with phosphorus-free lipids and impeded resource-expensive osmolyte accumulation. Elevated levels of ceramides and lipids with long chain polyunsaturated fatty acids, in dry soil suggests that lipids play an important role in fungal drought tolerance. Increasing abundance of bacterial glycerophospholipids and triacylglycerols with fatty acids typical of bacteria and polar metabolites suggest metabolic recovery in representative bacteria once the environmental conditions are conducive for growth. These results underscore the importance of the soil lipidome as a robust indicator of microbial community responses, especially at the short time scales of cell-environment reactions. Data package contents reported here are the first version and contain pre- and post-processed data acquisition and subsequent downstream analysis files using various data source instrument method techniques and Mass Spectroscopy (MS) EMSL capabilities. This publication data package DOI is a comprehensive high-throughput multi-omics data lifecycle collection containing processed data method metadata. Support files include additional data download “Read Me” file containing data descriptor information and data source application ontologies (see data dictionary). Reported data download contents are structured for compliance with project data sharing guidelines, community standards initiatives, and sponsor stakeholder policies supporting FAIR data principles. For increased data availability and interoperability, GC-MS/LC-MS mass spectrometry datasets (Thermo .raw ) were deposited at the MassIVE database repository under the related data accession MSV000086931 and can be accessed by using the API. Statistical data processing software, analysis tools, and data workflows are listed below corresponding to the host repository long-term location.

Amplicon sequencing 16S ITS LC-MS/MS lipidomics mu↗
DOE OSTI2017

FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework

Scientific applications are increasingly using cloud resources for their data analysis workflows. However, managing data effectively and efficiently over these cloud resources is challenging due to the myriad storage choices with different performance, cost trade-offs, complex application choices and complexity associated with elasticity, failure rates in these environments. The different data access patterns for data-intensive scientific applications require a more flexible and robust data management solution than the ones currently in existence. FRIEDA is a Flexible Robust Intelligent Elastic Data Management framework that employs a range of data management strategies in cloud environments. FRIEDA can manage storage and data lifecycle of applications in cloud environments. There are four different stages in the data management lifecycle of FRIEDA – (i) storage planning, (ii) provisioning and preparation, (iii) data placement, and (iv) execution. FRIEDA defines a data control plane and an execution plane. The data control plane defines the data partition and distribution strategy, whereas the execution plane manages the execution of the application using a master-worker paradigm. FRIEDA also provides different data management strategies, either to partition the data in real-time, or predetermine the data partitions prior to application execution.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2008

Modifying the Heliophysics Data Policy to Better Enable Heliophysics Research

The Heliophysics (HP) Science Data Management Policy, adopted by HP in June 2007, has helped to provide a structure for the HP data lifecycle. It provides guidelines for Project Data Management Plans and related documents, initiates Resident Archives to maintain data services after a mission ends, and outlines a route to the unification of data finding, access, and distribution through Virtual observatories. Recently we have filled in missing pieces that assure more coherence and a home for the VxOs (through the 'Heliophsyics Data and Model Consortium'), and provide greater clarity with respect to long term archiving. In particular, the new policy which has been vetted with many community members, details the 'Final Archives' that are to provide long-term data access. These are distinguished from RAs in that they provide little additional service beyond servicing data, but critical to their success is that the final archival materials include calibrated data in useful formats such as one finds in CDAWeb and various ASCII or FITS archives. Having a clear goal for legacy products, to be detailed as part of the Mission Archives Plans presented at Senior Reviews, will help to avoid the situation so common in the past of having archival products that preserve bits well but not readily usable information. We hope to avoid the need for the large numbers of 'data upgrade' projects that have been necessary in recent years.

Hayes, Jeffrey↗
DOE OSTI2024

Materials data science using CRADLE: A distributed, data-centric approach

Abstract There is a paradigm shift towards data-centric AI, where model efficacy relies on quality, unified data. The common research analytics and data lifecycle environment (CRADLE™) is an infrastructure and framework that supports a data-centric paradigm and materials data science at scale through heterogeneous data management, elastic scaling, and accessible interfaces. We demonstrate CRADLE’s capabilities through five materials science studies: phase identification in X-ray diffraction, defect segmentation in X-ray computed tomography, polymer crystallization analysis in atomic force microscopy, feature extraction from additive manufacturing, and geospatial data fusion. CRADLE catalyzes scalable, reproducible insights to transform how data is captured, stored, and analyzed. Graphical abstract

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

An Overview of NASA’s Airborne and Field Data Resource Center

A key recommendation from NASA’s 2022 Airborne and Field Data Workshop called for the development of a virtual Resource Center for all stakeholders across the data lifecycle of airborne and field Earth observations. The agency’s Earth Science Data and Information System (ESDIS) Project and Airborne Data Management Group (ADMG) have worked in concert to establish the newly launched NASA Airborne and Field Data Resource Center (AFDRC) to provide a single entry point for a wide assortment of information on and the effective, responsible stewardship of non-satellite observational data. The AFDRC compiles access to many existing resources, but does so in a newly organized way that integrates availability to increase efficiency and holistic understanding while simplifying users’ experience. Initially launched in fall of 2023, the AFDRC is a NASA Earthdata domain website that clarifies several previously disparate resources and provides newly updated information, including: Learning Resources: Educational resources to broaden understanding of the role airborne and field observations play in advancing understanding of our planet and NASA’s role in collecting and archiving these data. Support for data users to Find and Access Data: Advanced contextual browse/search capabilities that efficiently link researchers to data products suitable for their science objectives - this includes linking to NASA’s Catalog of Archived Suborbital Earth Science Investigations (CASEI). Working with Data: Tools specific to individual types of suborbital Earth Science data and their (inter-)disciplinary communities to provide access as well as guidance for their application. Stewardship Responsibilities: Resources for data producers with to lessen requirement burdens at the time of data transfer, and information for data stewards with consistent, authoritative guidance on best practices and agency- and/or community- specific archival procedures. This presentation will give an overview of the motivation for NASA’s AFDRC, approach for the design and content, iterative community-driven improvements, promote the use of the AFDRC, and solicit additional feedback from airborne and field data user communities.

Sara Lubkin↗
NASA NTRS2014

Data Preservation, Information Preservation, and Lifecyle of Information Management at NASA GES DISC

Data lifecycle management awareness is common today; planners are more likely to consider lifecycle issues at mission start. NASA remote sensing missions are typically subject to life cycle management plans of the Distributed Active Archive Center (DAAC), and NASA invests in these national centers for the long-term safeguarding and benefit of future generations. As stewards of older missions, it is incumbent upon us to ensure that a comprehensive enough set of information is being preserved to prevent the risk for information loss. This risk is greater when the original data experts have moved on or are no longer available. Preservation of items like documentation related to processing algorithms, pre-flight calibration data, or input-output configuration parameters used in product generation, are examples of digital artifacts that are sometimes not fully preserved. This is the grey area of information preservation; the importance of these items is not always clear and requires careful consideration. Missing important metadata about intermediate steps used to derive a product could lead to serious challenges in the reproducibility of results or conclusions. Organizations are rapidly recognizing that the focus of life-cycle preservation needs to be enlarged from the strict raw data to the more encompassing arena of information lifecycle management. By understanding what constitutes information, and the complexities involved, we are better equipped to deliver longer lasting value about the original data and derived knowledge (information) from them. The NASA Earth Science Data Preservation Content Specification is an attempt to define the content necessary for long-term preservation. It requires new lifecycle infrastructure approach along with content repositories to accommodate artifacts other than just raw data. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) setup an open-source Preservation System capable of long-term archive of digital content to augment its raw data holding. This repository is being used for such missions as HIRDLS, UARS, TOMS, OMI, among others. We will provide a status of this implementation; report on challenges, lessons learned, and detail our plans for future evolution to include other missions and services.

data management↗
NASA NTRS2018

Climatespark: an In-Memory Distributed Computing Framework for Big Climate Data Analytics

The unprecedented growth of climate data creates new opportunities for climate studies, and yet big climate data pose a grand challenge to climatologists to efficiently manage and analyze big data. The complexity of climate data content and analytical algorithms increases the difficulty of implementing algorithms on high performance computing systems. This paper proposes an in-memory, distributed computing framework, ClimateSpark, to facilitate complex big data analytics and time-consuming computational tasks. Chunking data structure improves parallel I/O efficiency, while a spatiotemporal index is built for the chunks to avoid unnecessary data reading and preprocessing. An integrated, multi-dimensional, array-based data model (ClimateRDD) and ETL operations are developed to address big climate data variety by integrating the processing components of the climate data lifecycle. ClimateSpark utilizes Spark SQL and Apache Zeppelin to develop a web portal to facilitate the interaction among climatologists, climate data, analytic operations and computing resources (e.g., using SQL query and Scala/Python notebook). Experimental results show that ClimateSpark conducts different spatiotemporal data queries/analytics with high efficiency and data locality. ClimateSpark is easily adaptable to other big multiple- dimensional, array-based datasets in various geoscience domains.

Hu, Fei↗
DOE OSTI2021

A Proposed Geospatial Data Preservation Strategy for DOE's Office of Legacy Management

Identifying and planning preservation and curation activities associated with geospatial data will improve the ability of the U.S. Department of Energy Office of Legacy Management (LM) to support their core mission of protecting human health and the environment. This report documents the development LM's strategy for preserving and curating geospatial data within the context of LM's data-lifecycle-management framework. The strategy consists of preservation and curation elements, specific activities, and key enabling factors that ensure LM's geospatial data is maintained. Preservation elements enable the effective preservation of LM's geospatial data and recognizes that strategies need to be flexible to adapt to ongoing changes in scale, technology, and standards. Key enabling factors are intended to highlight critical data management responsibilities that must be addressed by LM to meet its preservation and curation objectives. A summary of best practices for geospatial data preservation is provided as part of the strategy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Facility Cybersecurity Framework Best Practices

Federal facilities are increasingly adopting automation and connecting to the Internet creating an energy-internet-of-things environment that converges operational technology (OT) and information technology (IT). Today's buildings increasingly weave together networked sensors and cyber and physical systems that enable data to be collected, aggregated, exchanged, stored and monetized in new ways. Building technological advances have created new energy technology, services, markets and value creation opportunities (e.g. transactive energy, two-way grid communications, machine learning, and increased use of renewable and distributed energy resources). But as larger data sets are being exchanged at faster speeds between an increasing number of OT systems, it becomes more difficult to protect the security of the data lifecycle and the physical equipment it interacts with. These challenges are especially difficult to overcome because the economic and environmental gain (interoperability, big data, social networks and ubiquitous information sharing) are driving these prominent trends in the digital age. Often cybersecurity is an afterthought. The U.S. Department of Energy’s (DOE) Federal Energy Management Program (FEMP) funded the Pacific Northwest National Laboratory (PNNL) to develop various cybersecurity tools, trainings, and reports to aid federal facility managers – and other building owners and operators – in better applying frameworks and lessons learned from the National Institute of Standards and Technology (NIST) Cybersecurity Framework (CSF), risk management framework (RMF), DOE’s cybersecurity capability maturity model (C2M2), and a wide variety of industry best practices and guidance documents (i.e., NIST 800 series, Department of Defense United Facilities Criteria). This set of tools, collectively known as the FEMP Facility-Related Control System Cyber Toolkit (FRCS Cyber Toolkit)2, is focused on cybersecurity concerns from facility-related control systems and other operational technology (OT), such as industrial control systems (ICS). The FRCS Cyber Toolkit can be applied across six of the sixteen critical infrastructure sectors designated by the Department of Homeland Security, including government facilities, healthcare and public health, commercial facilities (e.g., public assembly, offices, lodging), financial services (e.g., banking and insurance), emergency services (e.g., fire and police stations), and information technology. With increasingly converged IT and OT systems, it is crucial to address OT cybersecurity considerations and assess how the seam of these two systems could impact the overall cybersecurity posture of a facility. The objective of this report is to provide an overview of the best possible method to use FRCS Cyber Toolkit (section 2.0) and distilled cybersecurity best practices for the federal facilities to address growing non-linear cyber threats (section 3.0). Recommendations in this document are aggregated from several NIST and other documents (see Appendix A for additional details).

97 MATHEMATICS AND COMPUTING↗