SEARCH · Engineering Papers

Results for “data infrastructure”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 901 records · Page 50

All sources NASA NTRS DOE OSTI

NASA NTRS2016

UUI: Reusable Spatial Data Services in Unified User Interface at NASA GES DISC

Unified User Interface (UUI) is a next-generation operational data access tool that has been developed at Goddard Earth Sciences Data and Information Services Center(GES DISC) to provide a simple, unified, and intuitive one-stop shop experience for the key data services available at GES DISC, including subsetting (Simple Subset Wizard -SSW), granule file search (Mirador), plotting (Giovanni), and other legacy spatial data services. UUI has been built based on a flexible infrastructure of reusable web services self-contained building blocks that can easily be plugged into spatial applications, including third-party clients or services, to easily enable new functionality as new datasets and services become available. In this presentation, we will discuss our experience in designing UUI services based on open industry standards. We will also explain how the resulting framework can be used for a rapid development, deployment, and integration of spatial data services, facilitating efficient access and dissemination of spatial data sets.

EOSDIS↗

DOE OSTI2020

End-to-end online performance data capture and analysis for scientific workflows

With the increased prevalence of employing workflows for scientific computing and a push towards exascale computing, it has become paramount that we are able to analyze characteristics of scientific applications to better understand their impact on the underlying infrastructure and vice-versa. Such analysis can help drive the design, development, and optimization of these next generation systems and solutions. Here, we present the architecture, integrated with existing well-established and newly developed tools, to collect online performance statistics of workflow executions from various, heterogeneous sources and publish them in a distributed database (Elasticsearch). Using this architecture, we are able to correlate online workflow performance data, with data from the underlying infrastructure, and present them in a useful and intuitive way via an online dashboard. We have validated our approach by executing two classes of real-world workflows, both under normal and anomalous conditions. The first is an I/O-intensive genome analysis workflow; the second, a CPU- and memory-intensive material science workflow. Based on the data collected in Elasticsearch, we are able to demonstrate that we can correctly identify anomalies that we injected. The resulting end-to-end data collection of workflow performance data is an important resource of training data for automated machine learning analysis.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2019

Advanced Astrophysics Discovery Technology in the Era of Data Driven Astronomy

Astrophysics is at the threshold of a new epoch in which increasinglycomplex, heterogeneous datasets will challenge our existing information infrastructure and traditional approaches to analysis. The rapid advancement of graphics processing units, compact field programmable gate arrays and dedicated artificial intelligence accelerator chips is now permitting the use of scientific methods, processes and algorithms to extract knowledge and insights from structured and unstructured data in ways never before seen. Miniaturization of spacecraft architectures and supporting infrastructure is opening new observing strategies and new discovery spaces for science. The community is just beginning to awaken to these imminent challenges as evidenced by their relative lack of emphasis in the New Worlds, New Horizons ASTRO2010 decadal survey, in the ExoPAG Science Analysis Group 11 report andin the formulation of the WFIRST Data Challenge. We suggest that the Astrophysics Science Division (ASD), which has clearly recognized this new epoch of rapidly evolving information technology, could be more affirmative in its approach. We offer a modest structural solution.

Barry, Richard K.↗

DOE OSTI2022

Feasibility of critical infrastructure protection using network functions for programmable and decoupled ICS policy enforcement over WAN

Industrial control systems (ICS) represent a major component of our critical infrastructure. With the increasing need for more control and monitoring of such systems, ICS have seen an increase in connectivity to wide area networks (WAN) exposing aging equipment to rapidly evolving cybersecurity threats. Furthermore, the ICS data requires a reliability measure from the networks for critical functions for infrastructure monitoring and control. Especially when remote plant sites are involved such as pipelines, energy distribution networks, and transportation, WAN transport impairments most often provide a best effort delivery with no strict reliability guarantees. Network functions can provide a vendor agnostic, programmable critical infrastructure protection with a single maintenance, policy determination, and reliability assurance surface. A network function (NF) can be utilized for policy enforcement over the communication between remote entities and the main control office. This paper presents the research on transparent integration with existing ICS without disrupting communications, resulting in minimal downtime while decoupling the fast paced evolution of defensive security measures from the upgrade cycle of expensive long term hardware. We report our measurements on the resource requirements and overhead in the network for successful NF insertion under a wide variety of network impairments (network packet delay, reordering, and loss). Our paired NF implementation provides a policy enforcement platform extensible to cover myriad cybersecurity-related communication goals, including packet signing for verification, encryption for data privacy, packet filtering and data diode operation (i.e. protecting against eavesdropping, packet injection, and denial-of-service). Furthermore, bundling communication specifications into packet flows allows for tunability in applying policies as coarse- or fine-grained as the needs of the operator. We report on network function resource requirements in the form of required queue depth and network utilization overhead to inform the decision making against hardware cost constraints.

42 ENGINEERING↗

NASA NTRS2001

Center for Aerosol Research (AEROCENTER)

The newly established Center for Aerosol Research (AEROCENTER) located at the NASA/Goddard Space Flight Center in Greenbelt MD is dedicated to fostering interdisciplinary research in all aspects of aerosol science. AEROCENTER will be an incubator for innovative new analysis of existing data and ideas for new space missions. The plan is to tap and harvest ideas from a broad international and interdisciplinary science community and to incorporate these ideas into NASA's aerosol research effort for understanding and predicting the aerosol effect on climate and the environment. In order to achieve this goal the center aims to host several established and developing scientists for a period of 3-6 months each year. AEROCENTER will also develop a new technical infrastructure that will integrate the present aerosol research activities and data resources of GSFC/Greenbelt and GSFC/GISS, increase efficiency in the use of NASA remote sensing data, and increase the involvement of a larger national and international scientific community. The center aims to institutionalize and extend the present knowledge base within NASA into a national resource for the education and research communities.

Kleidman, Richard↗

DOE OSTI2022

Analytics-at-scale of Sensor Data for Digital Monitoring in Nuclear Plants (3 rd Annual Report)

Nuclear power plants collect and store large volumes of heterogeneous data from various components and systems. With recent advances in machine learning (ML) techniques, these data can be leveraged to develop diagnostic and short-term forecasting models to better predict future equipment condition. Maintenance operations can then be planned in advance whenever degraded performance is predicted, thus resulting in fewer unplanned outages and the optimization of maintenance activities. This enables lower maintenance costs and improves the overall economics of nuclear power. This report primarily focuses on developing a short-term forecasting process that leverages a feature selection process to distill large volumes of heterogeneous data and predict specific equipment parameters. A variety of feature selection methods, including Shapley Additive Explanations (SHAP) and variance inflation factor (VIF), were used to select the optimal features as inputs for three ML methods: long short-term memory (LSTM) networks, support vector regression (SVR), and random forest (RF). Each combination of model and input features was used to predict a pump bearing temperature both 1 and 24 hours in advance, based on actual plant system data. The optimal inputs for the LSTM and SVR were selected using the SHAP values, while the optimal input for the RF consisted solely of the response variable itself. Each model produced similar 1-hour-ahead predictions, with root mean square errors (RMSEs) of roughly 0.006. For the 24-hour-ahead predictions, differences could be seen between LSTM, SVR, and RF, as reflected by model performances of 0.036 ± 0.014, 0.0026 ± 0, and 0.063 ± 0.004 RMSE, respectively. As big data and continuous online monitoring become more widely available, the proposed feature selection process can be used for many applications beyond the prediction of process parameters within nuclear infrastructure. This report summarizes the Fiscal Year 2021 research progress encompassing the (1) data cleaning and feature selection necessary for ML applications; (2) development of short-term forecasting models to predict future plant process parameters for both single and multiple time steps ahead; and (3) validation of the feature selection methods and short-term forecasting models given new data from different systems.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2023

A Data Processing Pipeline for Socio-Technical Network Analysis [Slides]

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

A Data Processing Pipeline for Adversarial Socio-Technical Network Analysis

With the rapid adoption of emerging technologies, there is a need to catalog and model sociotechnical interdependencies that have been historically used to influence the operation of Critical Infrastructure networks including the impacts of mergers and acquisitions, hostile takeovers, and foreign investment. Our research intends to address this need with two primary contributions. First, we have developed a data curation and processing pipeline to generate sociotechnical networks extracted from a variety of data sources including SEC filings and infrastructure asset databases. The pipeline, implemented in Apache Airflow, extracts and normalizes the representation of entities and relations, specified within ontologies. Our intent is to provide an extensible, machine-actionable approach to quickly communicate such models, reproduce previous results, and adapt them to new, unanticipated situations. Second, networks produced by our pipeline enable the development of graph-theoretic metrics that consider the properties of network components in addition to its topology. Metadata associated with network components---whether semantic, temporal, or geospatial---affects the alignment of generated networks with assumptions underlying complexity metrics. Validation of generated networks relative to component types defined by an ontology, may allow the research community to adapt metrics to the semantics of the domains being studied. Generated networks may be processed as knowledge, dynamic, or spatial graphs and enables a variety of analyses including automated reasoning and measures of network complexity. Automated reasoning views extracted entities and relations as a knowledge graph; this enables application of inference rules that represent historically-attested adversarial business methods and applies that behavior to a specific geographic context. Measures of network complexity, including degree distribution, reachability analyses, temporal analysis, and community detection can be adapted to indicate adversarial organizational influence.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Hardware-Based Randomized Encoding for Sensor Authentication in Power Grid SCADA Systems

Supervisory Control and Data Acquisition (SCADA) systems are utilized extensively in critical power grid infrastructures. Modern SCADA systems have been proven to be susceptible to cyber-security attacks and require improved security primitives in order to prevent unwanted influence from an adversarial party. One section of weakness in the SCADA system is the integrity of field level sensors providing essential data for control decisions at a master station. In this paper we propose a lightweight hardware scheme providing inferred authentication for SCADA sensors by combining an analog to digital converter and a permutation generator as a single integrated circuit. Through this method we encode critical sensor data at the time of sensing, so that unencoded data is never stored in memory, increasing the difficulty of software attacks. We show through experimentation how our design stops both software and hardware false data injection attacks occurring at the field level of SCADA systems.

42 ENGINEERING↗

DOE OSTI2021

Caching data from remote memories

An approach is disclosed that caches distant memories within the storage a local node. The approach provides a memory caching infrastructure that supports virtual addressing by utilizing memory in the local node as a cache of distant memories for data granules. The data granules are accessed along with metadata and an ECC associated with the data granule. The metadata is updated to indicate storage of the selected data granule in the cache.

Johns, Charles Ray↗

DOE OSTI2021

Characterizing Impacts of Storage Faults on HPC Applications: A methodology and insights

In recent years, the increasing complexity in scientific simulations and emerging demands for training heavy artificial intelligence models require massive and fast data accesses, which urges high-performance computing (HPC) platforms to equip with more advanced storage infrastructures such as solid-state disks (SSDs). While SSDs offer high-performance I/O, it remains unclear about the reliability challenges faced by the HPC applications under the SSD-related failures, in particular, failures resulting in data corruptions. The goal of this paper is to understand the impact of SSD-related data corruptions on the behaviors of complex HPC applications. To this end, we propose FFIS, a FUSE-based fault injection framework that systematically introduces storage faults into the application layer to model the errors originated from SSDs. FFIS is able to plant different I/O related faults into the data returned from underlying file systems, which also enables the investigation on the error resilience characteristics of the scientific file format for the first time. We demonstrate the use of FFIS with three representative real HPC applications, show how each application reacts to the data corruptions, and provide insights on the error resilience of the widely-adopted HDF5 file format for the HPC applications.

Fang, Bo↗

DOE OSTI2021

A Machine Learning-Based Method to Estimate Transformer Primary-Side Voltages with Limited Customer-Side AMI Measurements

Distribution control applications such as volt/var optimization, network reconfiguration, and distribution automation require accurate knowledge of the distribution system state. The lack of sufficient sensors on the primary side of distribution networks often limits the accuracy of the control decisions by these applications. The deployment of advanced metering infrastructure (AMI) provides utilities an opportunity to translate the AMI data on the secondary onto the primary so that it can be used as pseudo-measurements to augment the limited existing measurements on the primary. This paper develops a machine learning based approach for estimating service transformer primary-side voltages by using limited secondary-side AMI measurement. The machine learning model is developed by using random forest algorithm. The estimated primary-side voltages can be used by utilities as pseudo-measurements for distribution control applications. The detailed secondary model topology, which is an essential input data for many existing algorithms, is not required for the proposed method. The performance of the proposed method is validated by using AMI measurements from the field and an actual distribution feeder model of San Diego Gas & Electric Company.

advanced metering infrastructure↗

DOE OSTI2025

EV Charging Infrastructure Energization An Overview of Approaches for Simplifying and Accelerating Timelines to Processing EV Charging Load Service Requests

The United States has seen significant growth in electric vehicle (EV) adoption, leading to increased demand for EV charging infrastructure. Over the past decade, EV charging infrastructure site developers, site hosts, and electric distribution utilities have navigated the process to integrate chargers onto the electric grid. Site developers and site hosts have raised the alarm that the integration process for high-powered EV charging projects does not meet the needs of the EV market for timeliness or cost. High-powered charging stations typically require a load service request or an agreement with the local utility to connect to the grid. The process of energizing a new high-powered charging site can be complex and time-consuming, often taking up to 2 years. This timeline is the result of current utility energization processes having been designed for construction projects that take longer to build (i.e., buildings). The specific challenges stem from various factors, including compartmentalization in application processes, the integration of EV charging process approvals with other distributed energy resources (DERs), and the need to ensure grid reliability. The energization process needs to evolve to meet the growing demand for high-powered EV charging. This white paper compiles information gathered through various conversations with key stakeholders, including utilities, utility regulators, EV charging operators, site developers, and authorities having jurisdiction (AHJ) as well as through an extensive literature review. This document identifies the challenges and provides potential solutions to streamline the process of connecting EV charging infrastructure to the power grid in the United States, serving as a starting point for future conversations around these solutions. The solutions noted in this white paper require collaborative efforts among utilities, regulators, and EV charging infrastructure developers to streamline the grid connection process for EV charging infrastructure. They are broadly organized into four areas: 1. Increase data access and transparency: Develop automated load service request tools, integrate hosting capacity and load service request analyses, incorporate EV adoption forecasts, and provide transparency on the processing queue. 2. Improve energization processes and timing: Create fast-track options based on prescreening criteria, provide flexibility or phased approvals in the load service request/interconnection process, build internal knowledge within utilities about EV charging technologies, and provide standardized workforce training. 3. Promote economic efficiency: Right size distribution components to accurately reflect the load requirements of EV charging infrastructure, make proactive investments in grid infrastructure based on EV adoption forecasts and growth projections, and consider energy equity and environmental justice factors such as equitable access to EV charging when planning infrastructure. 4. Improve grid reliability and resilience: Use load management/power control systems (PCS) at EV charging stations, adopt and implement harmonized standards for communication protocols and information models between the EV charging and grid control infrastructure, and address cybersecurity considerations by implementing robust security measures and standards for EV charging infrastructure—with particular emphasis on clarifying the security requirements for the interface to the grid. The objective of the solutions proposed in this white paper is to accelerate the timeline and decrease costs associated with connecting EV charging infrastructure to the grid. Electric utilities, utility regulators, EV charging infrastructure developers, and site hosts will first need to understand which solutions are available in their service territory, and if warranted, which combination of solutions would support their specific needs. Through the successful implementations of solutions at scale detailed here, industry will demonstrate a new and innovative ecosystem where timely deployment and energization of EV charging infrastructure with greater grid resiliency and reliability is a reality.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2026

Toward a standardization of cryostructure and cryogenic soil structure terminology for the field description of permafrost‐affected soils

This paper establishes standardized terminology and field documentation protocols for cryostructures and cryogenic soil structures in permafrost‐affected soils and provides brief guidance on descriptions of ground ice morphology and ice volume estimates. We consolidate permafrost terminology from Russian and North American literature, clarify long‐standing ambiguities, and provide explicit guidelines that align with US Department of Agriculture‐Natural Resources Conservation Service soil description standards. Our scheme makes critical distinctions between cryostructure, the distribution of ice within soil, and cryogenic soil structure, the morphological structure of soil resulting from ice formation. The scheme organizes cryostructures into three main categories: non‐segregated ice, visible segregated ice, and ice matrices. We introduce standardized codes and parameters for field descriptions of ice and soil that enable machine‐readable data collection compatible with existing soil information systems. This standardization will significantly enhance the integration of field observations into landscape‐scale assessments of permafrost stability, infrastructure vulnerability, and ecosystem response to permafrost thaw, addressing an urgent need for quantitative data to inform modeling and decision‐making in rapidly changing Arctic and subarctic environments.

Andersen, Megan L. [University of Minnesota, Saint↗

NASA NTRS2006

A-Train Data Depot: Integrating and Visualizing Atmospheric Measurements Along the A-Train Tracks

The succession of US and international satellites that follow each other, seconds to minutes apart, across the local afternoon equator crossing is called the ATrain. The A-Train consists of the following satellites, in order of equator crossing: OCO, EOS Aqua, CloudSat, CALIPSO, PARASOL, and EOS Aura. Flying in such formation increases the number of observations, validates observations, and enables coordination between science observations, resulting in a more complete virtual science platform (Kelly, 2000) The goal of this project is to create the first ever A-Train virtual data portal/center, the A-Train Data Depot, to process, archive, access, visualize, analyze and correlate distributed atmosphere measurements from various A-Train instruments along A-Train tracks. The A-Train Data Depot (ATDD) will enable the free movement of remotely located A-Train data so that they are combined to create a consolidated vertical view of the Earth s Atmosphere along the A-Train tracks. Once the infrastructure of the ATDD is in place, it will be easily evolved to serve data from all A-Train data measurements: one stop shopping. The innovative approach of analyzing and visualizing atmospheric profiles along the platforms track (i.e., time) will be accommodated by reusing the GSFC Atmospheric Composition Data and Information Services Center (ACDISC) visualization and analysis tool, GIOVANNI, existing data reduction tools, on-line archwing for fast data access, and Cooperative Institute for Research in the Atmosphere (CRA) data co-registration tools. Initial measurements utilized include CALIPSO lidar backscatter, CloudSat radar reflectivity, clear air relative humidity, water vapor and temperature from AIRS, and cloud properties and aerosols from both MODIS. This will be followed by associated measurements from MLS, OMI, HIRDLS, and TES. Given the independent nature of instrument/platform development, the ATDD project has been met with many interesting challenges that, once resolved, will provide a much greater understanding of the relative flight dynamics and data co-registration of the suite of A-Train instruments, thus greatly increasing the accuracy of A-Train data analysis. Some of these challenges will be discussed. The project s resulting visualizations and analysis illustrate the importance of managing data so that measurements from various missions can be combined to enhance the understanding of the atmosphere. A-Train data management coordination, as performed here, is extremely significant in facilitating the A-Train science of clouds, precipitation, aerosol and chemistry.

Kempler, Steven↗

NASA NTRS2006

A-Train Data Depot: Integrating, Visualizing, and Extracting Cloudsat, CALIPSO, MODIS, and AIRS Atmospheric Measurements Along the A-Train Tracks

The succession of US and international Earth observing satellites that follow each other, seconds to minutes apart, across the local afternoon equator crossing is called the ATrain. The A-Train consists of the following satellites, in order of equator crossing: OCO, EOS Aqua, CloudSat, CALIPSO, PARASOL, and EOS Aura. Flying in such formation increases the number of observations, validates observations, and enables coordination between science observations, resulting in a more complete virtual science platform (Kelly, 2000). The goal of this project is to create the first ever A-Train virtual data portal/center, the A-Train Data Depot (ATDD), to process, archive, access, visualize, analyze and correlate distributed atmosphere measurements from various A-Train instruments along A-Train tracks. The ATDD will enable the free movement of remotely located A-Train data so that they are combined to create a consolidated vertical view of the Earth's Atmosphere along the A-Train tracks. Once the infrastructure of the ATDD is in place, it will be easily evolved to serve data from all A-Train data measurements: one stop shopping. The innovative approach of analyzing and visualizing atmospheric profiles along the platforms track (i.e., time) will be accommodated by reusing the GSFC Atmospheric Composition Data and Information Services Center (ACDISC) visualization and analysis tool, GIOVANNI, existing data reduction tools, on-line archiving for fast data access, access to remote data without unnecessary data transfers, and data retrieval by users finding data desirable for further study. Initial measurements utilized include CALIPSO lidar backscatter, CloudSat radar reflectivity, clear air relative humidity, water vapor and temperature from AIRS, and cloud properties and aerosols from both MODIS. This will be foilowed by associated measurements from TVILS, =MI, HIRDLS, sad TES. Given the independent nature of instrumentlplatform development, the ATDD project has been met with many interesting challenges that, once resolved, will provide a much greater understanding of the relative flight dynamics and data co-registration of the suite of A-Train instruments, thus greatly increasing the accuracy of A-Train data analysis. Some of these challenges will be illustrated and discussed. The project's early visualizations and analysis efforts illustrate the importance of managing data so that measurements from various missions can be combined to enhance the understanding of the atmosphere. A-Train data management coordination, as performed here, is extremely significant in facilitating the A-Train science of clouds, precipitation, aerosol and chemistry.

Kempler, Steven↗

DOE OSTI2020

Open Source Platform for Live Processing of High Data Rate Electron Microscopy (SBIR Phase I Award Topic 15, Subtopic B, Final Scientific / Technical Report)

4D scanning transmission electron microscopy (STEM) is experiencing a revolution both in terms of data size, and data veracity, as are many other areas of experimental data acquisition. This presents enormous opportunities to make new scientific discoveries, but is also hamstrung by aging software that is no longer fit for purpose. Experimental data acquisition platforms must make the same investments in software infrastructure that the simulation community made decades ago to take advantage of high-performance computing (HPC) if they hope to realize the full potential of the latest advances made in detector technology. This requires rethinking the design and deployment of hardware, moving away from small, fixed commodity computing capabilities with closed interfaces to open, modular, scalable systems designed for parallelism and throughput. The technical approach will leverage best-in-class open source technologies, and develop new approaches strategically to build an open source platform for experimental data acquisition targeting high data rate electron microscopy. The project will develop a software platform that uses the power and performance of modern HPC resources to ingest the large volume of data produced by electron microscopes using direct electron detectors. A set of configurable pipelines will be used to enable the processing to be tailored to the problem being investigated with builtin visualization of data at interactive rates. This project will leverage investments made by the DOE and other agencies to provide a powerful software platform for the experimental data community working with most powerful electron microscopes, such as 4D STEM detectors. It will create a platform that serves as a reference for the community. Permissive open source licensing will make the platform available to all, with a simple deployment strategy offering the ability to keep data close to where it is generated. This advanced software platform will require customization for each deployment, creating opportunities for software services that are typically offered to extend open source software platforms.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

A Data Exchange Interface for a Standards Based Data Integration Platform

The modern electrical grid is a complex and a data rich system, requiring several sub-systems to provide the capability and support needed to perform functions such as distributed energy resource management, outage management, and fault location and isolation, to name a few. There is a need to develop software solutions and platforms that provide an integrated infrastructure to allow interoperability between these subsystems, while leveraging the abundant data available from today’s grid. In this paper, we present the development and deployment of an application service to integrate GridAPPS-D which is an open source, standards-based software platform with SurvalentONE, a commonly used DNP3 based ADMS platform that allows operation, monitoring, analysis, restoration, and optimization of network operations. We present details of the integration architecture, including its implementation, and provide results from functional testing of the architecture on a 13 bus IEEE system. The results validate that the integration is successful and is able to provide a two-way exchange of communication and translation between different communication protocols.

Singh, Alka↗