Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 541 records · Page 30

Transforming Energy Through Sustainable Mobility: Expanding Low-Carbon Transportation R&D Solutions

As the nation's premier laboratory for cutting-edge transportation decarbonization research and development solutions, the National Renewable Energy Laboratory (NREL) pioneers the creation and deployment of sustainable mobility technologies and strategies, with a focus on slashing transportation sector greenhouse gas emissions and combatting climate change. Trucks, planes, cargo ships, and other difficult-to-decarbonize vehicles are part of this essential transition for the transportation sector, which is currently the nation's largest source of the greenhouse gas emissions. NREL provides the scientific building blocks needed to spur innovation through multifaceted analysis, research, and engineering. This work acts as a catalyst to help industry bring affordable, high-performance, energy-efficient, and low-emission modes of transport and related infrastructure to market sooner. Our researchers collaborate closely with academic, government, and industry partners to design better batteries, drivetrains, and engines. They develop technologies for high-power charging, thermal management, energy storage, and power electronics. They are also reimagining fuels and combustion while creating sustainable lightweight materials. Unbiased expert guidance - backed by credible data and analysis, tools, and scientific rigor - empowers partners to make informed decisions about sustainable transportation. NREL recognizes that communities with limited mobility options face reduced access to employment opportunities, health care, and education, lowering overall quality of life. Alongside partners, NREL experts are creating transportation solutions that meet community-identified needs and increase mobility equity in historically underserved and overburdened communities. Rather than providing a one-size-fits-all solution, we take an interdisciplinary approach to mobility equity that considers the needs and challenges of diverse groups, maximizing benefits at the individual, community, and societal levels.

ADVANCED PROPULSION SYSTEMS↗

GeneLab: Multi-Omics Investigation of Rodent Research-1 Bio-Banked Tissues

NASAs Rodent Research (RR) project is playing a critical role in advancing biomedical research on the physiological effects of space environments. Due to the limited resources for conducting biological experiments aboard the International Space Station (ISS), it is imperative to use crew time efficiently while maximizing high-quality science return. NASAs GeneLab project has as its primary objectives to 1) further increase the value of these experiments using a multi-omics, systems biology-based approach, and 2) disseminate these data without restrictions to the scientific community. The current investigation assessed viability of RNA, DNA, and protein extracted from archived RR-1 tissue samples for epigenomic, transcriptomic, and proteomic assays. During the first RR spaceflight experiment, a variety of tissue types were harvested from subjects, snap-frozen or RNAlater-preserved, and then stored at least a year at -80OC after return to Earth. They were then prioritized for this investigation based on likelihood of significant scientific value for spaceflight research. All tissues were made available to GeneLab through the bio-specimen sharing program managed by the Ames Life Science Data Archive and included mouse adrenal glands, quadriceps, gastrocnemius, tibialis anterior, extensor digitorum longus, soleus, eye, and kidney. We report here protocols for and results of these tissue extractions, and thus, the feasibility and value of these kinds of omics analyses. In addition to providing additional opportunities for investigation of spaceflight effects on the mouse transcriptome and proteome in new kinds of tissues, our results may also be of value to program managers for the prioritization of ISS crew time for rodent research activities. Support from the NASA Space Life and Physical Sciences Division and the International Space Station Program is gratefully acknowledged.

GeneLab↗

What Comes First, the OWL or the Bean? Creating Reusable Scientific Software with OWL/RDF Vocabularies.

In the spring of 2013 the U.S. White House by executive order mandated: “Government information shall be managed as an asset throughout its life cycle to promote interoperability and openness, and, wherever possible and legally permissible, to ensure that data are released to the public in ways that make the data easy to find, accessible, and usable.” Key for the reusability of any scientific data is hereby the availability of metadata describing the published data in a vocabulary that is familiar to its potential users. The objective of this paper is to help scientific application developers who want to adopt the continuous stream of new community vocabularies to help make their data sharable, self-describable, and easily understood. To achieve this we suggest semantic vocabulary and application integration best practices and discuss the tradeoffs of encoding vocabularies through code versus deriving code from vocabularies.

Stephan, Eric G.↗

Reimagining the potential of Earth observations for ecosystem service assessments

The benefits nature provides to people, called ecosystem services, are increasingly recognized and accounted for in assessments of infrastructure development, agricultural management, conservation prioritization, and sustainable sourcing. These assessments are often limited by data, however, a gap with tremendous potential to be filled through Earth observations (EO), which produce a variety of data across spatial and temporal extents and resolutions. Despite widespread recognition of this potential, in practice few ecosystem service studies use EO. Here, we identify challenges and opportunities to using EO in ecosystem service modeling and assessment. Some challenges are technical, related to data awareness, processing, and access. These challenges require systematic investment in model platforms and data management. Other challenges are more conceptual but still systemic; they are byproducts of the structure of existing ecosystem service models and addressing them requires scientific investment in solutions and tools applicable to a wide range of models and approaches. We also highlight new ways in which EO can be leveraged for ecosystem service assessments, identifying promising new areas of research. More widespread use of EO for ecosystem service assessment will only be achieved if all of these types of challenges are addressed. Furthermore, this will require non-traditional funding and partnering opportunities from private and public agencies to promote data exploration, sharing, and archiving. Investing in this integration will be reflected in better and more accurate ecosystem service assessments worldwide.

54 ENVIRONMENTAL SCIENCES↗

Open-Source Data Engineering at NASA: CCMC's Approach to Managing Petabyte-Scale Heliophysics Data

The Community Coordinated Modeling Center (CCMC) at NASA Goddard Space Flight Center (GSFC) leads heliophysics research by providing open access to numerous models and their outputs. Our resources are available on-demand and continuously updated with real-time data, covering sun-earth interactions across multiple domains. These domains include coronal, heliosphere, inner and global magnetosphere, ionosphere, thermosphere, and lower atmosphere interactions. Operating in a hybrid environment, CCMC utilizes both self-owned hardware and Amazon Web Services (AWS) cloud infrastructure. Managing petabytes of data across multiple locations necessitates robust data engineering solutions. To address this challenge, CCMC has adopted industry-standard and open-source tools. We use Apache Airflow as our primary data engineering platform, Python for scripting and data processing, and GitLab for version control and CI/CD. Additionally, we employ Kubernetes for containerized services, Grafana and Prometheus for metrics and monitoring, and Terraform and Puppet for reproducible infrastructure as code. This presentation will discuss lessons learned from our data engineering experiences, platforms evaluated but found unsuitable for our scientific data requirements, and specific techniques developed to enhance data transfer speed and reliability. By using these technologies effectively, CCMC continues to advance heliophysics research through efficient data management and open-access modeling.

space weather↗

Evaluation Report on "The Department's Unclassified Cyber Security Program"

Industry experts report that security challenges and threats are continually evolving as malicious activity has become more web-based and attackers are able to rapidly adapt their attack methods. In addition, the number of data breaches continues to rise. In an effort to mitigate and address threats and protect valuable information, the Department of Energy anticipated spending about $275 million in Fiscal Year (FY) 2009 to implement cyber security measures necessary to protect its information technology resources. These systems and data are designed to support the Department's mission and business lines of energy security, nuclear security, scientific discovery and innovation, and environmental responsibility. The Federal Information Security Management Act of 2002 (FISMA) provides direction to agencies on the management and oversight of information security risks, including design and implementation of controls to protect Federal information and systems. As required by FISMA, the Office of Inspector General conducts an annual independent evaluation to determine whether the Department's unclassified cyber security program adequately protects its information systems and data. This memorandum and the attached report present the results of our evaluation for FY 2009. The Department continued to make incremental improvements in its unclassified cyber security program. Our evaluation disclosed that most sites had taken action to address weaknesses previously identified in our FY 2008 evaluation report. They improved certification and accreditation of systems; strengthened configuration management of networks and systems; performed independent assessments; and, developed and/or refined certain policies and procedures. In addition, the Department instituted a centralized incident response organization designed to eliminate duplicative efforts throughout the Department. As we have noted in previous reports, the Department continued to maintain strong network perimeter defenses against malicious intruders and other externals threats. These are positive accomplishments. However, in our judgment, additional action is required to further enhance the Department's unclassified cyber security program and help reduce risks to its systems and data. For example, our current review identified opportunities for improvements in areas such as security planning and testing, systems inventory, access controls, and configuration management. In particular, we issued a number of findings at sites managed by the National Nuclear Security Administration (NNSA). We also identified weaknesses across various Department program elements. Issues that warrant further attention include: (1) Weaknesses such as outdated security plans and not completing annual security control self-assessments were identified at several sites; (2) The Department had not yet resolved systems inventory issues and had yet to deploy a complex-wide automated asset management tool to help track information technology resources and identify interfaces between systems or networks; (3) Although certain improvements had been made to enhance access controls, we noted deficiencies such as a lack of periodic account reviews and inadequate password management at a number of sites; and (4) Previously identified weaknesses in configuration management had been corrected, however, we found problems related to weak administrator account settings and failure to install software patches, as well as incomplete implementation of the Federal Desktop Core Configuration. These internal control weaknesses existed, at least in part, because certain cyber security roles and responsibilities were not clearly delineated. Program officials also had not effectively performed monitoring and review activities essential for evaluating the adequacy of cyber security performance. In some cases, officials had not ensured that weaknesses discovered during audits and other evaluations were recorded and tracked to resolution in the organizations' Plans of Action and Milestones. Our testing disclosed that about 39 percent of existing corrective action milestones had missed estimated remediation dates, with many exceeding planned completion dates by at least one year. As a consequence, the risk of compromise to the Department's information and systems remained higher than necessary. To assist the continuing efforts to improve, we made several recommendations designed to help the Department's managers to strengthen the unclassified cyber security program and, thereby, protect its computer resources from unauthorized modification, loss, or disclosure of information. Due to security considerations, information on specific vulnerabilities and locations has been omitted from this report. Management officials at the sites evaluated were provided with detailed information regarding identified vulnerabilities, and, in many instances, initiated corrective actions.

99 GENERAL AND MISCELLANEOUS↗

MOSIQS: Persistent Memory Object Storage With Metadata Indexing and Querying for Scientific Computing

Scientific applications often require high-bandwidth shared storage to perform joint simulations and collaborative data analytics. Shared memory pools provide a chance to satisfy such needs. Recently, a high-speed network such as Gen-Z utilizing persistent memory (PM) offers an opportunity to create a shared memory pool connected to compute nodes. However, there are several challenges to use scientific applications on the shared memory pool directly such as scalability, failure-atomicity, and lack of scientific metadata-based search and query. In this paper, we propose MOSIQS, a persistent memory object storage framework with metadata indexing and querying for scientific computing. We design MOSIQS based on the key idea that memory objects on PM pool can live beyond the application lifetime and can become the sharing currency for applications and scientists. MOSIQS provides an aggregate memory pool atop an array of persistent memory devices to store and access memory objects to accelerate scientific computing. MOSIQS uses a lightweight persistent memory key-value store to manage the metadata of memory objects, which enables memory object sharing. To facilitate metadata search and query over millions of memory objects resident on memory pool, we introduce Group Split and Merge (GSM), a novel persistent index data structure designed primarily for scientific datasets. GSM splits and merges dynamically to minimize the query search space and maintains low query processing time while overcoming the index storage overhead. MOSIQS is implemented on top of PMDK. We evaluate the proposed approach on many-core server with an array of real PM devices. Experimental results show that MOSIQS gains a 100% write performance improvement and executes multi-attribute queries efficiently with 2.7× less index storage overhead offering significant potential to speed up scientific computing applications.

97 MATHEMATICS AND COMPUTING↗

Semantic Catalog of Things, Services, and Data to Support a Wind Data Management Facility

The purpose of this paper is to discuss how community vocabularies and linked open data best practices are being used to seamlessly link things, data, and off the shelf services to support scientific offshore wind energy research for the U.S. Department of Energy’s Office of Energy Efficiency and Renewable Energy (EERE) Wind and Water Power Program. This is largely made possible by leveraging collaborative advances in the Internet of Things (IoT), Semantic Web, Linked Services, Linked Open Data (LOD), and RDF vocabulary communities, which provide the foundation for our design. By adapting these linked community best practices, we designed a wind characterization data management facility capable of continually collecting, processing, and preservation of in situ and remote sensing instrume

atmosphere↗

High Energy Physics Network Requirements Review: Two-Year Update

The Energy Sciences Network (ESnet) is the high-performance network user facility for the US Department of Energy (DOE) Office of Science (SC) and delivers highly reliable data transport capabilities optimized for the requirements of data-intensive science. In essence, ESnet is the circulatory system that enables the DOE science mission by connecting all its laboratories and facilities in the US and abroad. ESnet is funded and stewarded by the Advanced Scientific Computing Research (ASCR) program and managed and operated by the Scientific Networking Division at Lawrence Berkeley National Laboratory (LBNL). ESnet is widely regarded as a global leader in the research and education networking community. ESnet interconnects DOE national laboratories, user facilities, and major experiments so that scientists can use remote instruments and computing resources as well as share data with collaborators, transfer large datasets, and access distributed data repositories. ESnet is specifically built to provide a range of network services tailored to meet the unique requirements of the DOE’s data-intensive science. In July 2023, the Energy Sciences Network (ESnet) and the High Energy Physics program (HEP) of the DOE SC organized an interim ESnet requirements review of HEP-supported activities, to follow up on the work started during the 2020 HEP Network Requirements Review. Preparation for these events included checking back with the key stakeholders: program and facility management, research groups, and technology providers. Each stakeholder group was asked to prepare updates to their previously submitted case study documents, so that ESnet could update the understanding of any changes to the current, near-term, and long-term status, expectations, and processes that will support the science activities of the program.

97 MATHEMATICS AND COMPUTING↗

AMRZone: A Runtime AMR Data Sharing Framework For Scientific Applications

Frameworks that facilitate runtime data sharing across multiple applications are of great importance for scientific data analytics. Although existing frameworks work well over uniform mesh data, they can not effectively handle adaptive mesh refinement (AMR) data. Among the challenges to construct an AMR-capable framework include: (1) designing an architecture that facilitates online AMR data management; (2) achieving a load-balanced AMR data distribution for the data staging space at runtime; and (3) building an effective online index to support the unique spatial data retrieval requirements for AMR data. Towards addressing these challenges to support runtime AMR data sharing across scientific applications, we present the AMRZone framework. Experiments over real-world AMR datasets demonstrate AMRZone's effectiveness at achieving a balanced workload distribution, reading/writing large-scale datasets with thousands of parallel processes, and satisfying queries with spatial constraints. Moreover, AMRZone's performance and scalability are even comparable with existing state-of-the-art work when tested over uniform mesh data with up to 16384 cores; in the best case, our framework achieves a 46% performance improvement.

Zhang, Wenzhao↗

A scalable framework for quantifying field-level agricultural carbon outcomes

Agriculture contributes nearly a quarter of global greenhouse gas (GHG) emissions, which is motivating interest in adopting certain farming practices that have the potential to reduce GHG emissions or sequester carbon in soil. The related GHG emission (including N 2 O and CH 4 ) and changes in soil carbon stock are defined here as “agricultural carbon outcomes”. Accurate quantification of agricultural carbon outcomes is the basis for achieving emission reductions for agriculture, but existing approaches for measuring carbon outcomes (including direct measurements, emission factors, and process-based modeling) fall short of achieving the required accuracy and scalability necessary to support credible, verifiable, and cost-effective measurement and improvement of these carbon outcomes. Here we propose a foundational and scalable framework to quantify field-level carbon outcomes for farmland, which is based on the holistic carbon balance of the agroecosystem: Agroecosystem Carbon Outcomes = Environment (E) × Management (M) × Crop (C). Following a comprehensive review of the scientific challenges associated with existing approaches, as well as their tradeoffs between cost and accuracy, we propose that the most viable path for the quantification of field-level carbon outcomes in agricultural land is through an effective integration of various approaches (e.g. diverse observations, sensor/in-situ data, and modeling), defined as the “System-of-Systems” solution. Such a “System-of-Systems” solution should simultaneously comprise the following components: (1) scalable collection of ground truth data and cross-scale sensing of environment variables (E), management practices (M), and crop conditions (C) at the local field level; (2) advanced modeling with necessary processes to support the quantification of carbon outcomes; (3) systematic Model-Data Fusion (MDF), i.e. robust and efficient methods to integrate sensing data and models at each local farmland level; (4) high computation efficiency and artificial intelligence (AI) to scale to millions of individual fields with low cost; and (5) robust and multi-tier validation systems and infrastructures to ensure solution fidelity and true scalability, i.e. the ability of a solution to perform robustly with accepted accuracy on all targeted fields. In this regard, we provide here the detailed scientific rationale, current progress, and future research and development (R&D) priorities to achieve different components of the “System-of-Systems” solution, thus accomplishing the Environment×Management×Crop framework to quantify field-level agricultural carbon outcomes.

54 ENVIRONMENTAL SCIENCES↗

Biospecimen Culling: Temporal RNA Integrity Analysis Across Spaceflight Missions Dating from 1985 to 2011

The Ames Life Science Data Archive (ALSDA) at NASA Ames Research Center is managed by the Space Biosciences Division and has been operational since 1993. The ALSDA is responsible for archiving information and biospecimens collected from life science spaceflight experiments and matching ground control experiments. They are stored in the Ames biobank, which is located in the Biospecimen Storage Facility (BSF). The ALSDA also manages a Biospecimen Sharing Program, performs curation and long-term storage operations, and makes biospecimens available to the scientific community for research purposes via the Life Science Data Archive public website (https:lsda.jsc.nasa.gov). The BSF maintains both fixed and frozen spaceflight and ground tissues, collected from recent and past spaceflight missions. Due to the ever increasing demand for space to preserve current and future flight biospecimens, the ALSDA has initiated the development of a culling plan for biospecimens currently stored in the BSF. Culling enables the ALSDA to assess the quality of archived samples, and supports the development of standardized culling procedures that improve the operational efficiency of the BSF. The culling plan focuses on generating disposition recommendations for samples in the BSF, and currently is based on measuring ribonucleic acid (RNA) integrity number (RIN). The culling process includes (1) sorting and identification of candidate samples for RIN analysis, (2) completion of RIN analysis on select samples, and (3) development of disposition recommendations for specimens based on the RIN values. Furthermore, our approach allows for unique scientific opportunities, including development of a RIN-based methodology for culling, and temporal assessment of the quality of the tissues that have been stored in BSF since the 1980s. Results of this work will also support NASA open science initiatives.

biospecimen↗

AiiDA: automated interactive infrastructure and database for computational science

Computational science has seen in the last decades a spectacular rise in the scope, breadth, and depth of its efforts. Notwithstanding this prevalence and impact, it is often still performed using the renaissance model of individual artisans gathered in a workshop, under the guidance of an established practitioner. Great benefits could follow instead from adopting concepts and tools coming from computer science to manage, preserve, and share these computational efforts. We illustrate here our paradigm sustaining such vision, based around the four pillars of Automation, Data, Environment, and Sharing. We then discuss its implementation in the open-source AiiDA platform (http://www.aiida.net), that has been tuned first to the demands of computational materials science. AiiDA’s design is based on directed acyclic graphs to track the provenance of data and calculations, and ensure preservation and searchability. Remote computational resources are managed transparently, and automation is coupled with data storage to ensure reproducibility. Last, complex sequences of calculations can be encoded into scientific workflows. We believe that AiiDA’s design and its sharing capabilities will encourage the creation of social ecosystems to disseminate codes, data, and scientific workflows.

Materials Science↗

High Throughput WAN Data Transfer with Hadoop-Based Storage

Hadoop distributed file system (HDFS) is becoming more popular in recent years as a key building block of integrated grid storage solution in the field of scientific computing. Wide Area Network (WAN) data transfer is one of the important data operations for large high energy physics experiments to manage, share and process datasets of PetaBytes scale in a highly distributed grid computing environment. In this paper, we present the experience of high throughput WAN data transfer with HDFS-based Storage Element. Two protocols, GridFTP and fast data transfer (FDT), are used to characterize the network performance of WAN data transfer.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Automatic Beam Path Analysis of Laser Wakefield Particle Acceleration Data

Numerical simulations of laser wakefield particle accelerators play a key role in the understanding of the complex acceleration process and in the design of expensive experimental facilities. As the size and complexity of simulation output grows, an increasingly acute challenge is the practical need for computational techniques that aid in scientific knowledge discovery. To that end, we present a set of data-understanding algorithms that work in concert in a pipeline fashion to automatically locate and analyze high energy particle bunches undergoing acceleration in very large simulation datasets. These techniques work cooperatively by first identifying features of interest in individual timesteps, then integrating features across timesteps, and based on the information derived perform analysis of temporally dynamic features. This combination of techniques supports accurate detection of particle beams enabling a deeper level of scientific understanding of physical phenomena than hasbeen possible before. By combining efficient data analysis algorithms and state-of-the-art data management we enable high-performance analysis of extremely large particle datasets in 3D. We demonstrate the usefulness of our methods for a variety of 2D and 3D datasets and discuss the performance of our analysis pipeline.

43 PARTICLE ACCELERATORS↗

Enabling Low-Overhead HT-HPC Workflows at Extreme Scale using GNU Parallel

GNU Parallel is a versatile and powerful tool for process parallelization widely used in scientific computing. This paper demonstrates its effective application in high-performance computing (HPC) environments, particularly focusing on its scalability and efficiency in executing large-scale high-throughput high-performance computing (HT-HPC) workflows. Through real-world examples, we highlight GNU Parallel’s performance across various HPC workloads, including GPU computing, container-based workloads, and node-local NVMe storage. Our results on two leading supercomputers, OLCF’s Frontier and NERSC’s Perlmutter, showcase GNU Parallel’s rapid process dispatching ability and its capacity to maintain low overhead even at extreme scales. We explore GNU Parallel’s application in massive parallel file transfers using a scheduled Data Transfer Node (DTN) cluster, emphasizing its broad utility in diverse scientific workflows. Beyond its direct application as a viable workflow manager, GNU Parallel can be employed in conjunction with other workflow systems as a "last-mile" parallelizing driver and as a quick prototyping tool to design and extract parallel profiles from application executions. We then argue that the potential for GNU Parallel to transform workflow management at extreme scales is substantial, paving the way for more efficient and effective scientific discoveries.

Maheshwari, Ketan↗

The non-Federal oceanographic community: An overview

A portion of the broad domestic non-Federal oceanographic community that represents a potential market for satellite remote sensor derived oceanographic data and/or marine environmental information is presented. The overview consists of listings of individuals and/or organizations who have used, or are likely to use such data or information for scientific research, offshore engineering purposes, marine resources exploration and utilization, marine related operational applications, or coastal zone management.

Swetnick, M. A.↗

Software for Managing Parametric Studies

The Information Power Grid Virtual Laboratory (ILab) is a Practical Extraction and Reporting Language (PERL) graphical-user-interface computer program that generates shell scripts to facilitate parametric studies performed on the Grid. (The Grid denotes a worldwide network of supercomputers used for scientific and engineering computations involving data sets too large to fit on desktop computers.) Heretofore, parametric studies on the Grid have been impeded by the need to create control language scripts and edit input data files painstaking tasks that are necessary for managing multiple jobs on multiple computers. ILab reflects an object-oriented approach to automation of these tasks: All data and operations are organized into packages in order to accelerate development and debugging. A container or document object in ILab, called an experiment, contains all the information (data and file paths) necessary to define a complex series of repeated, sequenced, and/or branching processes. For convenience and to enable reuse, this object is serialized to and from disk storage. At run time, the current ILab experiment is used to generate required input files and shell scripts, create directories, copy data files, and then both initiate and monitor the execution of all computational processes.

Yarrow, Maurice↗