Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data platform”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

High-data throughput reconfigurable computing platform

A reconfigurable computing platform includes a reconfigurable computing device, electro-optical transceiver, and first voltage converter disposed on a multilayer board. The electro-optical transceiver converts an optical signal at least one of to and from an electrical signal, and the electrical signal is operatively coupled to the reconfigurable computing device. The electro-optical transceiver is disposed in proximity to the reconfigurable computing device, and the first voltage converter is operatively coupled to a common voltage distributed around a periphery of the multilayer board. The first voltage converter converts the common voltage to a first operating voltage, and the first voltage converter is disposed in proximity to the reconfigurable computing device. The first operating voltage is provided to the reconfigurable computing device as a first power source. A reconfigurable computing system includes a plurality of reconfigurable computing platforms operatively coupled together using an optical signal. A corresponding method of providing a reconfigurable computing platform is also disclosed.

Tang, Shaochun↗
DOE OSTI2013

NREL Develops OpenEI.org, a Public Website Where Energy Data can be Generated, Shared, and Compared (Fact Sheet)

The National Renewable Energy Laboratory (NREL) has developed OpenEI.org, a public, open, data-sharing platform where consumers, analysts, industry experts, and energy decision makers can go to boost their energy IQs, search for energy data, share data, and get access to energy applications. The free site blends elements of social media, linked open-data practices, and MediaWiki-based technology to build a collaborative environment for creating and sharing energy data with the world. The result is a powerful platform that is helping government and industry leaders around the world define policy options, make informed investment decisions, and create new businesses.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2026

Alabama Carbon Storage: Data Sharing and Engagement (Final Report)

This report is the final technical report on Alabama Carbon Storage: Data Sharing Engagement (ACS:DSE) project activities. The goals of the ACS:DSE project are to compile geologic, geophysical, infrastructure, and other relevant CCUS datasets for the study area and develop a geologic model of the study area; develop an online platform to serve data to stakeholders; engage with the public, students, and industry to educate them about CCUS and the data platform; and ensure energy and environmental justice is central to all aspects of the project. Datasets compiled and expanded include formation depths and elevations, digital geophysical well logs, reservoir properties, geologic structures, and geologic models. The geologic data were used to create a three-dimensional geologic model, structure grids, structure contour maps, and fault trace maps. In addition to downloadable datasets, links to CCUS relevant regulatory agencies (e.g., OGB, U.S. Environmental Protection Agency) and sources for infrastructure and educational information were included on the website Educational materials on CCUS for use by K-12 teachers were produced as part of the ACS:DSE project.

01 COAL, LIGNITE, AND PEAT↗
DOE OSTI2020

Striped Data Analysis Framework

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Gutsche, Oliver↗
DOE OSTI2018

Striped Data Server for Scalable Parallel Data Analysis

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. As a result, we have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2016

Kira: Processing Astronomy Imagery Using Big Data Technology

Scientific analyses commonly compose multiple single-process programs into a dataflow. An end-to-end dataflow of single-process programs is known as a many-task application. Typically, HPC tools are used to parallelize these analyses. In this work, we investigate an alternate approach that uses Apache Spark-a modern platform for data intensive computing-to parallelize many-task applications. We implement Kira, a flexible and distributed astronomy image processing toolkit, and its Source Extractor (Kira SE) application. Using Kira SE as a case study, we examine the programming flexibility, dataflow richness, scheduling capacity and performance of Apache Spark running on the Amazon EC2 cloud. By exploiting data locality, Kira SE achieves a 4.1× speedup over an equivalent C program when analyzing a 1TB dataset using 512 cores on the Amazon EC2 cloud. Furthermore, Kira SE on the Amazon EC2 cloud achieves a 1.8× speedup over the C program on the NERSC Edison supercomputer. A 128-core Amazon EC2 cloud deployment of Kira SE using Spark Streaming can achieve a second-scale latency with a sustained throughput of 800 MB/s. Our experience with Kira demonstrates that data intensive computing platforms like Apache Spark are a performant alternative for many-task scientific applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Computing erasure metadata and data layout prior to storage using a processing platform

Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗