Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data formats”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 721 records · Page 40
All sourcesNASA NTRSDOE OSTI
DOE OSTI2012

Development of autonomous gamma dose logger for environmental monitoring

Continuous monitoring and archiving of background radiation levels in and around the nuclear installation is essential and the data would be of immense use during analysis of any untoward incidents. A portable Geiger Muller detector based autonomous gamma dose logger (AGDL) for environmental monitoring is indigenously designed and developed. The system operations are controlled by microcontroller (AT89S52) and the main features of the system are software data acquisition, real time LCD display of radiation level, data archiving at removable compact flash card. The complete system operates on 12 V battery backed up by solar panel and hence the system is totally portable and ideal for field use. The system has been calibrated with Co-60 source (8.1 MBq) at various source-detector distances. The system is field tested and performance evaluation is carried out. This paper covers the design considerations of the hardware, software architecture of the system along with details of the front-end operation of the autonomous gamma dose logger and the data file formats. The data gathered during field testing and inter comparison with GammaTRACER are also presented in the paper. AGDL has shown excellent correlation with energy fluence monitor tuned to identify {sup 41}Ar, proving its utility for real-time plume tracking and source term estimation.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2020

P SI4 1.4: Open-source software for high-throughput quantum chemistry

PSI4 is a free and open-source ab initio electronic structure program providing implementations of Hartree–Fock, density functional theory, many-body perturbation theory, configuration interaction, density cumulant theory, symmetry-adapted perturbation theory, and coupled-cluster theory. Most of the methods are quite efficient, thanks to density fitting and multi-core parallelism. The program is a hybrid of C++ and Python, and calculations may be run with very simple text files or using the Python API, facilitating post-processing and complex workflows; method developers also have access to most of PSI4 's core functionalities via Python. Job specification may be passed using The Molecular Sciences Software Institute (MolSSI) QCSCHEMA data format, facilitating interoperability. We announce a rewrite of our top-level computation driver, and concomitant adoption of the MolSSI QCARCHIVE INFRASTRUCTURE project, makes the latest version of PSI4 well suited to distributed computation of large numbers of independent tasks. The project has fostered the development of independent software components that may be reused in other quantum chemistry programs.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2015

Practices and Standards for Data and Processing at the APS

The handling of datasets at scientific user facilities is becoming progressively more challenging as advances in sources and detectors drive increasingly aggressive data acquisition rates. The ability to share and process these data depends on the ability of the originator of the data and the persons making use of that data to unequivocally agree on what these data mean. This understanding is driven by use of standardized data formats.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2015

Background history and cosmic perturbations for a general system of self-conserved dynamical dark energy and matter

We determine the Hubble expansion and the general cosmic perturbation equations for a general system consisting of self-conserved matter, ρ{sub m}, and self-conserved dark energy (DE), ρ{sub D}. While at the background level the two components are non-interacting, they do interact at the perturbations level. We show that the coupled system of matter and DE perturbations can be transformed into a single, third order, matter perturbation equation, which reduces to the (derivative of the) standard one in the case that the DE is just a cosmological constant. As a nontrivial application we analyze a class of dynamical models whose DE density ρ{sub D}(H) consists of a constant term, C{sub 0}, and a series of powers of the Hubble rate. These models were previously analyzed from the point of view of dynamical vacuum models, but here we treat them as self-conserved DE models with a dynamical equation of state. We fit them to the wealth of expansion history and linear structure formation data and compare their fit quality with that of the concordance ΛCDM model. Those with C{sub 0}=0 include the so-called ''entropic-force'' and ''QCD-ghost'' DE models, as well as the pure linear model ρ{sub D}∼H, all of which appear strongly disfavored. The models with C{sub 0}≠0 , in contrast, emerge as promising dynamical DE candidates whose phenomenological performance is highly competitive with the rigid Λ-term inherent to the ΛCDM.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Scaling neural simulations in STACS

Abstract As modern neuroscience tools acquire more details about the brain, the need to move towards biological-scale neural simulations continues to grow. However, effective simulations at scale remain a challenge. Beyond just the tooling required to enable parallel execution, there is also the unique structure of the synaptic interconnectivity, which is globally sparse but has relatively high connection density and non-local interactions per neuron. There are also various practicalities to consider in high performance computing applications, such as the need for serializing neural networks to support potentially long-running simulations that require checkpoint-restart. Although acceleration on neuromorphic hardware is also a possibility, development in this space can be difficult as hardware support tends to vary between platforms and software support for larger scale models also tends to be limited. In this paper, we focus our attention on Simulation Tool for Asynchronous Cortical Streams (STACS), a spiking neural network simulator that leverages the Charm++ parallel programming framework, with the goal of supporting biological-scale simulations as well as interoperability between platforms. Central to these goals is the implementation of scalable data structures suitable for efficiently distributing a network across parallel partitions. Here, we discuss a straightforward extension of a parallel data format with a history of use in graph partitioners, which also serves as a portable intermediate representation for different neuromorphic backends. We perform scaling studies on the Summit supercomputer, examining the capabilities of STACS in terms of network build and storage, partitioning, and execution. We highlight how a suitably partitioned, spatially dependent synaptic structure introduces a communication workload well-suited to the multicast communication supported by Charm++. We evaluate the strong and weak scaling behavior for networks on the order of millions of neurons and billions of synapses, and show that STACS achieves competitive levels of parallel efficiency.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2016

PanViz: interactive visualization of the structure of functionally annotated pangenomes

Summary: PanViz is a novel, interactive, visualization tool for pangenome analysis. PanViz allows visualization of changes in gene group (groups of similar genes across genomes) classification as different subsets of pangenomes are selected, as well as comparisons of individual genomes to pangenomes with gene ontology based navigation of gene groups. Furthermore it allows for rich and complex visual querying of gene groups in the pangenome. PanViz visualizations require no external programs and are easily sharable, allowing for rapid pangenome analyses. Availability and Implementation: PanViz is written entirely in JavaScript and is available on https:// github.com/thomasp85/PanViz. A companion R package that facilitates the creation of PanViz visualizations from a range of data formats is released through Bioconductor and is available at https:// bioconductor.org/packages/PanViz Generator.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2013

Network portal: a database for storage, analysis and visualization of biological networks

The ease of generating high-throughput data has enabled investigations into organismal complexity at the systems level through the inference of networks of interactions among the various cellular components (genes, RNAs, proteins and metabolites). The wider scientific community, however, currently has limited access to tools for network inference, visualization and analysis because these tasks often require advanced computational knowledge and expensive computing resources. We have designed the network portal (http://networks.systemsbiology.net) to serve as a modular database for the integration of user uploaded and public data, with inference algorithms and tools for the storage, visualization and analysis of biological networks. The portal is fully integrated into the Gaggle framework to seamlessly exchange data with desktop and web applications and to allow the user to create, save and modify workspaces, and it includes social networking capabilities for collaborative projects. While the current release of the database contains networks for 13 prokaryotic organisms from diverse phylogenetic clades (4678 co-regulated gene modules, 3466 regulators and 9291 cis-regulatory motifs), it will be rapidly populated with prokaryotic and eukaryotic organisms as relevant data become available in public repositories and through user input. The modular architecture, simple data formats and open API support community development of the portal.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

AXI4MLIR: User-Driven Automatic Host Code Generation for Custom AXI-Based Accelerators

Tensor algebra operations represent an important class of algorithms used across many applications, including machine learning, scientific computing, and data analytics. As a result, the efficient generation of custom accelerators for tensor operations has received increased attention. Previous efforts have produced automated tools enabling users to prototype and explore optimized accelerators. However, little effort has been focused on the host-accelerator interaction in these tools. Efficient use of hardware accelerators requires knowledge about the accelerator's capabilities (operations, data formats, and opcode support), the host CPU microarchitecture (e.g., memory hierarchy), the host-accelerator interface, and the application's features (which code regions should be mapped onto an accelerator). Manually rewriting the original applications to facilitate improved custom accelerator mapping is an error-prone and time-consuming endeavor. To cope with this, we propose AXI4MLIR, a new framework to automatically generate and optimize the communication between the host CPU and arbitrary accelerators that implement linear algebra algorithms. AXI4MLIR extends the MLIR compiler framework to automatically generate efficient host-accelerator driver code for accelerators with AXI-based interfaces. Our compiler extensions enable automatic driver code generation while carefully considering the host's memory hierarchy and target accelerator features. To demonstrate the flexibility and utility of AXI4MLIR, we test it with diverse use cases that include different types of accelerators, tiling scenarios, and dataflow schemes. We compare our experimental results to manual implementations of host-accelerator driver code and find that our approach can reduce CPU cache references by 56% and deliver up to a 1.65x speedup.

Bohm Agostini, Nicolas↗
DOE OSTI2024

Understanding Mixed Precision GEMM with MPGemmFI: Insights into Fault Resilience

Emerging deep learning workloads urgently need fast general matrix multiplication (GEMM). Thus, one of the critical features of machine-learning-specific accelerators such as NVIDIA Tensor Cores, AMD Matrix Cores, and Google TPUs is the support of mixed-precision enabled GEMM. For DNN models, lower-precision FP data formats and computation offer acceptable correctness but significant performance, area, and memory footprint improvement. While promising, the mixed-precision computation on error resilience remains unexplored. To this end, we develop a fault injection framework that systematically injects fault into the mixed-precision computation results. We investigate how the faults affect the accuracy of machine learning applications. Based on the characteristics of error resilience, we offer lightweight error detection and correction solutions that significantly improve the overall model accuracy by 75% if the models experience hardware faults. The solutions can be efficiently integrated into the accelerator's pipelines.

Fang, Bo↗
DOE OSTI2019

Strategies to Deploy and Scale Deep Learning on the Summit Supercomputer

The rapid growth and wide applicability of Deep Learning (DL) frameworks poses challenges to computing centers which need to deploy and support the software, and also to domain scientists who have to keep up with the system environment and scale up scientific exploration through DL. We offer recommendations for deploying and scaling DL frameworks on the Summit supercomputer, currently atop the Top500 list, at the Oak Ridge National Laboratory Leadership Computing Facility (OLCF). We discuss DL software deployment in the form of containers, and compare performance of native-built frameworks and containerized deployment. Software containers show no noticeable negative performance impact and exhibit faster Python loading times and promise easier maintenance. To explore strategies for scaling up DL model training campaigns, we assess DL compute kernel performance, discuss and recommend I/O data formats and staging, and identify communication needs for scalable message exchange for DL runs at scale. We recommend that users take a step-wise tuning approach beginning with algorithmic kernel choice, node I/O configuration, and communications tuning as best-practice. We present baseline examples of scaling efficiency 87% for a DL run of ResNet50 running on 1024 nodes (6144 V100 GPUs).

Yin, Junqi↗
DOE OSTI2017

Spark and HPC for High Energy Physics Data Analyses

A full High Energy Physics (HEP) data analysis is divided into multiple data reduction phases. Processing within these phases is extremely time consuming, therefore intermediate results are stored in files held in mass storage systems and referenced as part of large datasets. This processing model limits what can be done with interactive data analytics. Growth in size and complexity of experimental datasets, along with emerging big data tools are beginning to cause changes to the traditional ways of doing data analyses. Use of big data tools for HEP analysis looks promising, mainly because extremely large HEP datasets can be represented and held in memory across a system, and accessed interactively by encoding an analysis using highlevel programming abstractions. The mainstream tools, however, are not designed for scientific computing or for exploiting the available HPC platform features. We use an example from the Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider (LHC) in Geneva, Switzerland. The LHC is the highest energy particle collider in the world. Our use case focuses on searching for new types of elementary particles explaining Dark Matter in the universe. We use HDF5 as our input data format, and Spark to implement the use case. We show the benefits and limitations of using Spark with HDF5 on Edison at NERSC.

Sehrish, Saba↗
DOE OSTI2025

Enhancing Smart Home Privacy: A Tutorial on Local Differential Privacy Techniques for Frequency and Mean Estimation

The ubiquity of Internet of Things (IoT) systems has seamlessly integrated into our daily lives, particularly in smart homes where devices continuously monitor and optimize our living environments. These systems significantly contribute to home automation, energy efficiency, and overall comfort. However, this widespread connectivity poses inherent risks linked to the streaming of sensitive household data, necessitating robust privacy preservation mechanisms. This tutorial systematically examines privacy preservation through local differential privacy (LDP), with a particular focus on frequency and mean estimation techniques for smart home applications. Here, we present a comprehensive taxonomy of smart home data formats and provide detailed implementation guidance for event-based and w-event LDP mechanisms. Through practical examples using smart thermostats and HVAC systems, we demonstrate how these techniques can be effectively deployed in real-world scenarios. The tutorial concludes by examining emerging research directions, including adaptive privacy budgets and federated learning approaches, establishing a foundation for privacy-preserving smart home deployments.

Kotevska, Olivera [Oak Ridge National Laboratory (↗
DOE OSTI2017

A Scientific Data Provenance Harvester for Distributed Applications

Data provenance provides a way for scientists to observe how experimental data originates, conveys process history, and explains influential factors such as experimental rationale and associated environmental factors from system metrics measured at runtime. The US Department of Energy Office of Science Integrated end-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) project has developed a provenance harvester that is capable of collecting observations from file based evidence typically produced by distributed applications. To achieve this, file based evidence is extracted and transformed into an intermediate data format inspired in part by W3C CSV on the Web recommendations, called the Harvester Provenance Application Interface (HAPI) syntax. This syntax provides a general means to pre-stage provenance into messages that are both human readable and capable of being written to a provenance store, Provenance Environment (ProvEn). HAPI is being applied to harvest provenance from climate ensemble runs for Accelerated Climate Modeling for Energy (ACME) project funded under the U.S. Department of Energy’s Office of Biological and Environmental Research (BER) Earth System Modeling (ESM) program. ACME informally provides provenance in a native form through configuration files, directory structures, and log files that contain success/failure indicators, code traces, and performance measurements. Because of its generic format, HAPI is also being applied to harvest tabular job management provenance from Belle II DIRAC scheduler relational database tables as well as other scientific applications that log provenance related information.

Stephan, Eric G.↗
DOE OSTI2024

Bilevel Nodal Behind-the-meter Solar Disaggregation Under Unexpected Extreme Weather Conditions

As the power grid undergoes significant paradigm shift due to the increasing penetration of renewable generation, the ever-growing installation of behind-the-meter (BTM) solar generation in the power grid also has a significant impact on nodal loads, posing challenges on transmission operators. Furthermore, increasing frequent and severe extreme weather events intertwine with ubiquitous BTM solar generations and have amplified the challenges of accurately model nodal load profiles, especially under the lack of ground-truth information for verification. To tackle these challenges, this paper introduces a bilevel model that utilizes year-long data (e.g., proxy solar, zonal load, and individual node load profiles) to disaggregate metered profiles into actual demand and BTM solar generation at each transmission node. The proxy solar not only scales the BTM solar generation of individual nodes but also create a compensation term for enhancing performance on days with unexpected extreme weather events. The proposed algorithm is validated with real-world PJM Interconnection data during unexpected events like the recent Winter Storm Elliott. For quantitative evaluations, a novel Score error is introduced, which is based on mean percentages and load scales and offers a universal assessment method suitable for all nodes and different data formats (e.g., normalized or raw values).

behind-the-meter solar, load disaggregations, load↗
DOE OSTI2021

Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUs

Despite foreseeing tremendous speedups over conventional deep neural networks, the performance advantage of binarized neural networks (BNNs) has merely been showcased on general-purpose processors such as CPUs and GPUs. In fact, due to being unable to leverage bit-level-parallelism with a word-based architecture, GPUs have been criticized for extremely low utilization (1%) when executing BNNs. Consequently, the latest tensorcores in NVIDIA Turing GPUs start to experimentally support bit computation. In this work, we look into this brand new bit computation capability and characterize its unique features. We show that the stride of memory access can significantly affect performance delivery and a data-format co-design is highly desired to support the tensorcores for achieving superior performance than existing software solutions without tensorcores. We realize the tensorcore-accelerated BNN design, particularly the major functions for fully-connect and convolution layers — bit matrix multiplication and bit convolution. Evaluations on two NVIDIA Turing GPUs show that, with ResNet-18, our BTC-BNN design can process ImageNet at a rate of 5.6K images per second, 77% faster than state-of-the-art. Our BNN approach is released on https://github.com/pnnl/TCBNN.

Li, Ang↗
DOE OSTI2020

taxadb: A high‐performance local taxonomic database interface

Abstract A familiar and growing challenge in ecological and evolutionary research is that of establishing consistent taxonomy when combining data from separate sources. While this problem is already well understood and numerous naming authorities have been created to address the issue, most researchers lack a fast, consistent, and intuitive way to retrieve taxonomic names. We present taxadb R package which creates a local database, managed automatically from within R, to provide fast operations on millions of taxonomic names. taxadb provides access to established naming authorities to resolve synonyms, taxonomic identifiers, and hierarchical classification in a consistent and intuitive data format. taxadb makes operation on millions of taxonomic names fast and manageable.

Norman, Kari E. A.↗
DOE OSTI2013

Simultaneous reduction of radiation dose and scatter for CBCT by using collimators

Purpose: On-board cone-beam CT (CBCT) imaging has been widely available in radiotherapy clinic for target localization. However, the extra radiation dose from CBCT is always a concern for its frequent use. Additionally, the relatively large scatter in CBCT often degrades the image quality. By using collimators, some of the X-rays can be stopped from reaching the patient and the detectors, hence both the scatter and the patient doses are simultaneously reduced. The authors show in this work that the collimated CBCT data can be reconstructed without any noticeable artifacts for certain collimator blocking ratios and blocking patterns, and the focus of this work is to study the relationship between the image quality and these two collimator factors.Methods: A CBCT system with collimators was simulated following the typical geometry used in clinic. Different collimator designs were tested by varying the size and the number of the collimator slits, and at the same time, the ratio of transmitted beams to total beams was varied from 100% to 10%, resulting in hundreds of different simulation scenarios. Lung and pelvis phantoms created from patients CT images were used in the simulations, and an iterative reconstruction algorithm using the compressed sensing technique was adopted. The image quality was examined by root mean square errors (RMSEs) and compared with the conventional CBCT images.Results: The CBCT image quality increases as the amount of beams passing through the collimators increases, and decreases as the size of the collimator slits increases. With ultra-high resolution collimators, the RMSEs were comparable to the conventional CBCT image quality until the beam transmission rate is reduced below 25%.Conclusions: Collimators can reduce the scatters and radiation dose, however, the collimated CBCT image quality is strongly dependent on both the collimator blocking ratio and the blocking pattern. To achieve image quality comparable to the conventional CBCT, the amount of information and data format must be adequate.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2025

Open database for GPD analyses

This article summarizes the main ideas behind creating an open database proposed for use in the exploration of generalized parton distributions (GPDs). This lightweight database is well suited for GPD phenomenology and is designed to store both experimental and lattice-QCD data. It can also aid in benchmarking GPD-related developments, such as GPD models. The database utilizes a new data format based on the YAML serialization language, enabling the storage of essential information for modern analyses, such as replica values. It includes interfaces for both Python and C++, allowing straightforward integration with analysis codes.

Burkert, V. D. [Thomas Jefferson National Accelera↗