Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Management Platform”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

dCache on Steroids - Delegated Storage Solutions

For over a decade, dCache.org has delivered a robust software used at more than 80 Universities and research institutes around the world, allowing these sites to provide reliable storage services for the WLCG experiments as well as many other scientific communities. The flexible architecture of dCache allows running it in a wide variety of configurations and platforms - from a SoC based all-in-one Raspberry-Pi up to hundreds of nodes in a multipetabyte installation. Due to lack of managed storage at the time, dCache implemented data placement, replication and data integrity directly. Today, many alternatives are available: S3, GlusterFS, CEPH and others. While such solutions position themselves as scalable storage systems, they cannot be used by many scientific communities out of the box. The absence of community-accepted authentication and authorization mechanisms, the use of product specific protocols and the lack of namespace are some of the reasons that prevent wide-scale adoption of these alternatives. Most of these limitations are already solved by dCache. By delegating low-level storage management functionality to the above-mentioned new systems and providing the missing layer through dCache, we provide a solution which combines the benefits of both worlds - industry standard storage building blocks with the access protocols and authentication required by scientific communities. In this paper, we focus on CEPH, a popular software for clustered storage that supports file, block and object interfaces. CEPH is often used in modern computing centers, for example as a backend to OpenStack services. We will show prototypes of dCache running with a CEPH backend and discuss the benefits and limitations of such an approach. As a result, we will also outline the roadmap for supporting ‘delegated storage’ within the dCache releases.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
NASA NTRS1993

Telerobot local-remote control architecture for space flight program applications

The JPL Supervisory Telerobotics (STELER) Laboratory has developed and demonstrated a unique local-remote robot control architecture which enables management of intermittent communication bus latencies and delays such as those expected for ground-remote operation of Space Station robotic systems via the Tracking and Data Relay Satellite System (TDRSS) communication platform. The current work at JPL in this area has focused on enhancing the technologies and transferring the control architecture to hardware and software environments which are more compatible with projected ground and space operational environments. At the local site, the operator updates the remote worksite model using stereo video and a model overlay/fitting algorithm which outputs the location and orientation of the object in free space. That information is relayed to the robot User Macro Interface (UMI) to enable programming of the robot control macros. This capability runs on a single Silicon Graphics Inc. machine. The operator can employ either manual teleoperation, shared control, or supervised autonomous control to manipulate the intended object. The remote site controller, called the Modular Telerobot Task Execution System (MOTES), runs in a multi-processor VME environment and performs the task sequencing, task execution, trajectory generation, closed loop force/torque control, task parameter monitoring, and reflex action. This paper describes the new STELER architecture implementation, and also documents the results of the recent autonomous docking task execution using the local site and MOTES.

Zimmerman, Wayne↗
NASA NTRS2015

Towards an Open, Distributed Software Architecture for UxS Operations

To address the growing need to evaluate, test, and certify an ever expanding ecosystem of UxS platforms in preparation of cultural integration, NASA Langley Research Center's Autonomy Incubator (AI) has taken on the challenge of developing a software framework in which UxS platforms developed by third parties can be integrated into a single system which provides evaluation and testing, mission planning and operation, and out-of-the-box autonomy and data fusion capabilities. This software framework, named AEON (Autonomous Entity Operations Network), has two main goals. The first goal is the development of a cross-platform, extensible, onboard software system that provides autonomy at the mission execution and course-planning level, a highly configurable data fusion framework sensitive to the platform's available sensor hardware, and plug-and-play compatibility with a wide array of computer systems, sensors, software, and controls hardware. The second goal is the development of a ground control system that acts as a test-bed for integration of the proposed heterogeneous fleet, and allows for complex mission planning, tracking, and debugging capabilities. The ground control system should also be highly extensible and allow plug-and-play interoperability with third party software systems. In order to achieve these goals, this paper proposes an open, distributed software architecture which utilizes at its core the Data Distribution Service (DDS) standards, established by the Object Management Group (OMG), for inter-process communication and data flow. The design decisions proposed herein leverage the advantages of existing robotics software architectures and the DDS standards to develop software that is scalable, high-performance, fault tolerant, modular, and readily interoperable with external platforms and software.

Cross, Charles D.↗
NASA NTRSDate not supplied

Applications of NASA’s LANCE Earth Observations for Ecosystem Assessments

As one of NASA's open and free data systems, NASA's Land, Atmosphere Near real-time Capability for EOS (LANCE) provides a wide range of near real-time, low latency and expedited data products from NASA and other earth science satellite missions to support users in their understanding of the value and distribution of ecosystem services. LANCE data can support the System of Environmental Economic Accounting-Ecosystem Accounting (SEEA-EA) framework by both facilitating the creation of ecosystem spatial extent and condition accounts, as well as the quantification and mapping of ecosystem services, much more quickly than routine data processing allows. During the past 13 years, LANCE near real-time satellite data products (e.g., surface reflectance, albedo, vegetation height, thermal anomalies, soil moisture, snow cover etc.) have been used to produce ecosystem-related indicators such as vegetation indexes, biomass, land cover maps, fire, and flood products. These ecosystem-related indicators can be integrated into ecosystem services models, tools, and platforms to monitor land cover and land use changes in ecosystem composition in support of land and natural resource management. When using ecosystem services accounts in decision-making, low latency is essential because having up-to-date information can be critical to support decisions being made. With adequate spatial resolution, long time series length, and low latency data products, LANCE would promote the sustainable use of land and natural resources.

Tian Yao↗
DOE OSTI2015

A Computing Environment to Support Repeatable Scientific Big Data Experimentation of World-Wide Scientific Literature

A principal tenant of the scientific method is that experiments must be repeatable and relies on ceteris paribus (i.e., all other things being equal). As a scientific community, involved in data sciences, we must investigate ways to establish an environment where experiments can be repeated. We can no longer allude to where the data comes from, we must add rigor to the data collection and management process from which our analysis is conducted. This paper describes a computing environment to support repeatable scientific big data experimentation of world-wide scientific literature, and recommends a system that is housed at the Oak Ridge National Laboratory in order to provide value to investigators from government agencies, academic institutions, and industry entities. The described computing environment also adheres to the recently instituted digital data management plan mandated by multiple US government agencies, which involves all stages of the digital data life cycle including capture, analysis, sharing, and preservation. It particularly focuses on the sharing and preservation of digital research data. The details of this computing environment are explained within the context of cloud services by the three layer classification of Software as a Service , Platform as a Service , and Infrastructure as a Service .

"Software as a Service↗
DOE OSTI2018

Brick : Metadata schema for portable smart building applications

Buildings account for 32% of worldwide energy usage. A new regime of exciting new “applications” that span a distributed fabric of sensors, actuators and humans has emerged to improve building energy efficiency and operations management. These applications leverage the technological advances in embedded sensing, processing, networking and methods by which they can be coupled with supervisory control and data acquisition systems deployed in modern buildings and with users on mobile wireless platforms. There are, however, several technical challenges to confront before such a vision of smart building applications and cyber-physical systems can be realized. The sensory data produced by these systems need significant curation before it can be used meaningfully. This is largely a manual, cost-prohibitive task and hence such solutions rarely experience widespread adoption due to the lack of a common descriptive schema. Recent attempts have sought to address this through data standards and metadata schemata but fall short in capturing the richness of relationships required by applications. Here, this paper describes Brick, a uniform metadata schema for representing buildings that builds upon recent advances in the area. Our schema defines a concrete ontology for sensors, subsystems and the relationships between them, which enables portable applications. We demonstrate the completeness and effectiveness of Brick by using it to represent the entire vendor-specific sensor metadata of six diverse buildings across different campuses, comprising 17,700 data points, and running eight unmodified energy efficiency applications on these buildings.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2022

NREL Fleet Analysis Support Through Technology Integration Collaboration

This study leveraged the partnership between the United States Department of Energy's (DOE) Clean Cities Coalition Network and the Association for the Work Truck Industry (NTEA) to launch a vehicle and fleet analysis project that assisted fleets in identifying opportunities to save energy, improve efficiency, reduce costs, and meet environmental goals via short term data logging and analysis. The National Renewable Energy Laboratory (NREL) sought to establish a process that included initial data acquisition, provided data storage, and developed analytic methods to inform fleets of areas of opportunity based on approximately 30 days of in use vehicle performance data. However, long-term the project will require ongoing funding to fully develop and maintain the data sharing platform and to produce more complex analysis.

33 ADVANCED PROPULSION SYSTEMS↗
NASA NTRS2016

A Satellite Data-Driven, Client-Server Decision Support Application for Agricultural Water Resources Management

Water cycle extremes such as droughts and floods present a challenge for water managers and for policy makers responsible for the administration of water supplies in agricultural regions. In addition to the inherent uncertainties associated with forecasting extreme weather events, water planners need to anticipate water demands and water user behavior in a typical circumstances. This requires the use decision support systems capable of simulating agricultural water demand with the latest available data. Unfortunately, managers from local and regional agencies often use different datasets of variable quality, which complicates coordinated action. In previous work we have demonstrated novel methodologies to use satellite-based observational technologies, in conjunction with hydro-economic models and state of the art data assimilation methods, to enable robust regional assessment and prediction of drought impacts on agricultural production, water resources, and land allocation. These methods create an opportunity for new, cost-effective analysis tools to support policy and decision-making over large spatial extents. The methods can be driven with information from existing satellite-derived operational products, such as the Satellite Irrigation Management Support system (SIMS) operational over California, the Cropland Data Layer (CDL), and using a modified light-use efficiency algorithm to retrieve crop yield from the synergistic use of MODIS and Landsat imagery. Here we present an integration of this modeling framework in a client-server architecture based on the Hydra platform. Assimilation and processing of resource intensive remote sensing data, as well as hydrologic and other ancillary information occur on the server side. This information is processed and summarized as attributes in water demand nodes that are part of a vector description of the water distribution network. With this architecture, our decision support system becomes a light weight 'app' that connects to the server to retrieve the latest information regarding water demands, land use, yields and hydrologic information required to run different management scenarios. Furthermore, this architecture ensures all agencies and teams involved in water management use the same, up-to-date information in their simulations.

Agricultural↗
DOE OSTI2024

A baseline structure inventory with critical attribution for the US and its territories

Leveraging high performance computing, remote sensing, geographic data science, machine learning, and computer vision, Oak Ridge National Laboratory has partnered with Federal Emergency Management Agency (FEMA) to build a baseline structure inventory covering the US and its territories to support disaster preparedness, response, and recovery. The dataset contains more than 125 million structures with critical attribution, and is ready to be used by federal agencies, local government and first responders to accelerate on-the-ground response to disasters, further identify vulnerable areas, and develop strategies to enhance the resilience of critical structures and communities. Data can be freely and openly accessed through Figshare data repository, ESRI’s Living Atlas or FEMA’s Geodata platform.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
NASA NTRS1991

Framework Programmable Platform for the Advanced Software Development Workstation: Preliminary system design document

The Framework Programmable Software Development Platform (FPP) is a project aimed at combining effective tool and data integration mechanisms with a model of the software development process in an intelligent integrated software environment. Guided by the model, this system development framework will take advantage of an integrated operating environment to automate effectively the management of the software development process so that costly mistakes during the development phase can be eliminated. The focus here is on the design of components that make up the FPP. These components serve as supporting systems for the Integration Mechanism and the Framework Processor and provide the 'glue' that ties the FPP together. Also discussed are the components that allow the platform to operate in a distributed, heterogeneous environment and to manage the development and evolution of software system artifacts.

Mayer, Richard J.↗
DOE OSTI2025

Intersection of Hydrologic Change and Hydropower in the United States: Needs for Future Research and Practice

Hydropower is crucial for electric‐grid stability in the context of variable renewables but faces threats from changing hydrology. Here, we summarize the state of the science at the intersection of hydropower operations and planning, hydrologic science, and climate. We focus on the United States, outlining research, development, and training needs. Key knowledge gaps include the risk that intensification of compound extreme events poses to future generation, as well as uncertainties surrounding greenhouse gas emissions from hydropower reservoirs with relevance to hydropower's role in energy decarbonization. Quantifying such impacts and reducing uncertainty are critical where possible, but remaining irreducible or deep uncertainty will require new approaches. Future monitoring and modeling methods must provide a better understanding of the complexity inherent in large watersheds that is critical to managing both hydropower and watersheds in the context of hydrologic change. Yet, research and development will have little impact if they do not inform practice. Standardization and consolidation of platforms are essential for data, modeling, and tool translation to local scales and small operators. An enhanced industry‐academia dialog is pivotal for fostering a robust pipeline of hydropower professionals. Collaboration among researchers, policymakers, authorities, and industry stakeholders emerges as a recurring theme, highlighting the imperative for collective efforts.

13 HYDRO ENERGY↗
DOE OSTI2019

ServiceNow/CrayPort Integration v2.1.5

The National Energy Research Scientific Computing Center (NERSC) is the primary scientific computing facility for the Office of Science in the U.S. Department of Energy. NERSC houses top-ranked Cray supercomputers, and our staff works very closely with on-site Cray engineers, submitting 75 Cray cases per month on average. Until recently, submitting a Cray case was only possible by phone or email, and all subsequent updates were delivered manually. This process led to delays, errors during manual data entry, and greatly increased incident processing and resolution time. In 2018, NERSC deployed an API-based bidirectional integration that allowed to submit and update Cray cases directly from our Incident Management platform, thus streamlining our 24x7 operations and enhancing communication between our engineering teams. This solution is applicable to any Incident Management platform.

Gens, Daniel↗
DOE OSTI2021

Development of Distributed Microgrid Controllers Using HELICS [SWR-21-94]

Development of Distributed Microgrid Controllers Using HELICS uses HELICS framework develop distributed microgrid controllers (DMCs) for coordinated grid service. Development of Distributed Microgrid Controllers Using HELICS includes thirteen agents, including OpenDSS agent, data manager agent, and eleven DMC agents. Development of Distributed Microgrid Controllers Using HELICS can be used as a standard framework to develop fully distributed control solutions that dispatches simulated DERs in simulation platform.

Wang, Jing↗
DOE OSTI2015

NESM TAG Meeting Notes

March 2015 NESM TAG Notes related to 1) Structure surface representation, 2) Structure-fluid coupling algorithm, 3) Coupled time integration scheme, 4) Coupling data exchange API, 5) Portability across platforms and builds, and 6) Testing and verification.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2024

Grid Optimization Competition Challenge 3 Problem Formulation

This report contains the problem formulation for the Grid Optimization (GO) Competition Challenge 3. The Grid Optimization Competition is run by a team of researchers from a number of organizations, including the sponsor Advanced Research Projects Agency - Energy (ARPA-E), lead organization Pacific Northwest National Laboratory (PNNL), and technical contributors from Los Alamos National Laboratory (LANL), National Renewable Energy Laboratory (NREL), Texas A&M University (TAMU), Georgia Institute of Technology (GT), University of Wisconsin (UW), and others. The GO Competition poses challenge problems in the field of power grid management, invites entrants to develop solvers for these problems, invokes the solvers on a set of problem instances using common hardware, ranks the solvers according to their performance, and awards prizes according to the rankings. The overall goal of the GO Competition is to spur innovative research on high impact and computationally challenging problems in power grid management from initial development through commercial deployment. Complete information about the GO Competition can be found online at [2]. The webpage covers previous Challenges, rules, timeline, registration information, data formats, scoring methods, computational platform information, information on supported solvers and languages, sponsor information, frequently asked questions, administrator contact information, publicly available problem instances, computer code for reading and evaluating problem and solution data, a sandbox for testing solvers, and a solver submission interface, results, and publications.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

A Community Energy Operations and Planning System: Concept, Use cases, Metrics, and Benefits

Community and city leaders are interested in achieving sustainability goals, providing resilient energy infrastructure, and improving economic competitiveness. Community-level data acquisition and analysis can provide energy and associated benefits that are not possible at the single building level. However, there is a lack of organizational structure, common semantic data models, interoperable systems, and methods to support data-driven decision-making for community-scale energy supply and demand systems. We explored the need and opportunity for a Community Energy Operations and Planning System (Community EOPS), a potential data exchange platform. We conducted “customer discovery” interviews, and reviewed literature, public tools, and technology platforms to identify key energy data “users” and use cases in communities. The key users of the Community EOPS could be developers of mixed-use districts, corporate, defense and university campus energy managers, and city managers of cities that own their energy utility. The value could be for community planning and reporting (for energy data-integrated land use planning and community infrastructure investments in microgrids, storage, district heating and cooling), energy efficiency (leveraging optimizations for community scale energy supply and demand), flexible load management (grid-edge load management to offset, shift, and flatten loads for multiple buildings and EV fleets), cost savings and revenue generation (participating in grid services), and social benefits such as energy resilience, equity, and awareness. We developed a conceptual Community EOPS architecture with recommendations for streamlined and prioritized data acquisition, sharing, and integration driven by prioritized use cases, common metrics, and actionable visualizations that can provide value to a community’s users.

Singh, Reshma↗
NASA NTRS2011

Leveraging the Cloud for Robust and Efficient Lunar Image Processing

The Lunar Mapping and Modeling Project (LMMP) is tasked to aggregate lunar data, from the Apollo era to the latest instruments on the LRO spacecraft, into a central repository accessible by scientists and the general public. A critical function of this task is to provide users with the best solution for browsing the vast amounts of imagery available. The image files LMMP manages range from a few gigabytes to hundreds of gigabytes in size with new data arriving every day. Despite this ever-increasing amount of data, LMMP must make the data readily available in a timely manner for users to view and analyze. This is accomplished by tiling large images into smaller images using Hadoop, a distributed computing software platform implementation of the MapReduce framework, running on a small cluster of machines locally. Additionally, the software is implemented to use Amazon's Elastic Compute Cloud (EC2) facility. We also developed a hybrid solution to serve images to users by leveraging cloud storage using Amazon's Simple Storage Service (S3) for public data while keeping private information on our own data servers. By using Cloud Computing, we improve upon our local solution by reducing the need to manage our own hardware and computing infrastructure, thereby reducing costs. Further, by using a hybrid of local and cloud storage, we are able to provide data to our users more efficiently and securely. 12 This paper examines the use of a distributed approach with Hadoop to tile images, an approach that provides significant improvements in image processing time, from hours to minutes. This paper describes the constraints imposed on the solution and the resulting techniques developed for the hybrid solution of a customized Hadoop infrastructure over local and cloud resources in managing this ever-growing data set. It examines the performance trade-offs of using the more plentiful resources of the cloud, such as those provided by S3, against the bandwidth limitations such use encounters with remote resources. As part of this discussion this paper will outline some of the technologies employed, the reasons for their selection, the resulting performance metrics and the direction the project is headed based upon the demonstrated capabilities thus far.

Cloud Computing↗
DOE OSTI2019

Multi-Formalism Modeling for Disaster Resilience, Forecasting, and Response

Disaster response preparedness, risk mitigation, and effective post-event response are critical for national security, public health, and economic integrity. For areas impacted by extreme events, timely and accurate impact assessments of lifeline infrastructures are critical. Traditional disaster management and situational awareness platforms are not capable of handling the massive influx of available information from traditional and non-traditional sources. Next-generation analytic assessment approaches, where use of dynamically sourced, heterogeneous data with near real-time availability are integrated into a continually-updated analytics framework. The objective of this research was to develop a foundational multi-formalism modeling platform to integrate novel data sources for use in improving situational awareness after a disaster event. Specifically, to 1) provide an architecture for machine-learning based data fusion various multi-sensor datasets; 2) classify, geolocate, and temporally bin novel data sources for use in spatially-enabled machine-learning based data fusion; and 3) derive confidence-based observation data with machine-learning based prediction to fill spatial and temporal observation gaps. This project developed a reusable cloud-based pipeline architecture for ingesting, storing, and analyzing relevant data, executing a range of deep-learning models, and visualize the information geographically. The pipeline development was informed by previous PNNL investments (StreamKit) and built upon leading-edge technologies consisting of: 1) a raw datastore; 2) a data enrichment pipeline with custom and third-party analytics; 3) enriched data storage; and 4) a custom UI to display the enriched data. Machine learning/deep learning models were selected and tested to classify social media images for relevancy and presence of flood waters. Our custom trained relevance models were run on a validation dataset and the overall accuracy of the relevance classifiers ranged from 65% to 90%. We found that convolutional neural network (CNN) models were both faster and more accurate than more traditional machine learning models. We applied the relevance classifier to remove non-relevant images and then undertook a limited training of CNN models to detect flood on ~1,800 labeled images. The testing accuracy of the flood detection models were ~88%. To improve the geographic location of both social media and oblique aerial images we tested a multi-step workflow built on a pre-trained deep-learning model. To identify similarity features from test and reference scenes, we applied the Xception and Faiss models to a set of test images with inaccurate spatial locations, as well and precisely-located reference scenes from the general area. Matching reference scenes to the test scenes with the closest similarity index had limited success in improving the precision of test scene location and additional methods have been identified for testing. Data fusion between authoritative geographic data (satellite imaging) and non-authoritative crowdsourced data is an underlying theme throughout this research and methods are tested bring these data to a common geographic form through feature matching and transformations. The automated extraction of supporting information for disaster response from oblique aerial images or ground-level social media images is accomplished through semantic image segmentation and transformation. The non-authoritative data are used to validate the authoritative data collections and processing and optionally, t+0 flood models. In addition, the common form geographic data are combined with other static landscape features to develop spatially-explicit training vectors used in deep generative statistical inference models to develop complex non-linear relationships in static and dynamic data and ultimately derive a spatially continuous and probabilistic assessment of flood damage with the event domain.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗