SEARCH · Engineering Papers

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24

All sources NASA NTRS DOE OSTI

DOE OSTI2018

The Geothermal Data Repository: Five Years of Open Geothermal Data, Benefits to the Community: Preprint

In the five years since its inception, the Department of Energy's (DOE) Geothermal Data Repository (GDR) has grown from the simple idea of storing public data in a centralized location to a valuable tool at the center of the DOE open data movement where it is providing a tangible benefit to the geothermal scientific community. Throughout this time, the GDR project team has been working closely with the community to refine the data submission process, improve the quality of submitted data, and embrace modern proper data management strategies to maximize the value and utility of submitted data. This paper explores some of the motivations behind various improvements to the GDR over the last 5 years, changes in data submission trends, and the ways in which these improvements have helped to drive research, fuel innovation, and accelerate the adoption of geothermal technologies.

15 GEOTHERMAL ENERGY↗

DOE OSTI2017

The Geothermal Data Repository: Five Years of Open Geothermal Data, Benefits to the Community

15 GEOTHERMAL ENERGY↗

NASA NTRS2000

Rovers as Geological Helpers for Planetary Surface Exploration

Rovers can be used to perform field science on other planetary surfaces and in hostile and dangerous environments on Earth. Rovers are mobility systems for carrying instrumentation to investigate targets of interest and can perform geologic exploration on a distant planet (e.g. Mars) autonomously with periodic command from Earth. For nearby sites (such as the Moon or sites on Earth) rovers can be teleoperated with excellent capabilities. In future human exploration, robotic rovers will assist human explorers as scouts, tool and instrument carriers, and a traverse "buddy". Rovers can be wheeled vehicles, like the Mars Pathfinder Sojourner, or can walk on legs, like the Dante vehicle that was deployed into a volcanic caldera on Mt. Spurr, Alaska. Wheeled rovers can generally traverse slopes as high as 35 degrees, can avoid hazards too big to roll over, and can carry a wide range of instrumentation. More challenging terrain and steeper slopes can be negotiated by walkers. Limitations on rover performance result primarily from the bandwidth and frequency with which data are transmitted, and the accuracy with which the rover can navigate to a new position. Based on communication strategies, power availability, and navigation approach planned or demonstrated for Mars missions to date, rovers on Mars will probably traverse only a few meters per day. Collecting samples, especially if it involves accurate instrument placement, will be a slow process. Using live teleoperation (such as operating a rover on the Moon from Earth) rovers have traversed more than 1 km in an 8 hour period while also performing science operations, and can be moved much faster when the goal is simply to make the distance. I will review the results of field experiments with planetary surface rovers, concentrating on their successful and problematic performance aspects. This paper will be accompanied by a working demonstration of a prototype planetary surface rover.

Stoker, Carol↗

DOE OSTI2013

Oak Ridge Bio-surveillance Toolkit (ORBiT): Integrating Big-Data Analytics with Visual Analysis for Public Health Dynamics

In this position paper, we describe the design and implementation of the Oak Ridge Bio-surveillance Toolkit (ORBiT): a collection of novel statistical and machine learning tools implemented for (1) integrating heterogeneous traditional (e.g. emergency room visits, prescription sales data, etc.) and non-traditional (social media such as Twitter and Instagram) data sources, (2) analyzing large-scale datasets and (3) presenting the results from the analytics as a visual interface for the end-user to interact and provide feedback. We present examples of how ORBiT can be used to summarize ex- tremely large-scale datasets effectively and how user interactions can translate into the data analytics process for bio-surveillance. We also present a strategy to estimate parameters relevant to dis- ease spread models from near real time data feeds and show how these estimates can be integrated with disease spread models for large-scale populations. We conclude with a perspective on how integrating data and visual analytics could lead to better forecasting and prediction of disease spread as well as improved awareness of disease susceptible regions.

Ramanathan, Arvind↗

DOE OSTI2014

F3D Image Processing and Analysis for Many - and Multi-core Platforms

F3D is written in OpenCL, so it achieve[sic] platform-portable parallelism on modern mutli-core CPUs and many-core GPUs. The interface and mechanims to access F3D core are written in Java as a plugin for Fiji/ImageJ to deliver several key image-processing algorithms necessary to remove artifacts from micro-tomography data. The algorithms consist of data parallel aware filters that can efficiently utilizes[sic] resources and can work on out of core datasets and scale efficiently across multiple accelerators. Optimizing for data parallel filters, streaming out of core datasets, and efficient resource and memory and data managements over complex execution sequence of filters greatly expedites any scientific workflow with image processing requirements. F3D performs several different types of 3D image processing operations, such as non-linear filtering using bilateral filtering and/or median filtering and/or morphological operators (MM). F3D gray-level MM operators are one-pass constant time methods that can perform morphological transformations with a line-structuring element oriented in discrete directions. Additionally, MM operators can be applied to gray-scale images, and consist of two parts: (a) a reference shape or structuring element, which is translated over the image, and (b) a mechanism, or operation, that defines the comparisons to be performed between the image and the structuring element. This tool provides a critical component within many complex pipelines such as those for performing automated segmentation of image stacks. F3D is also called a "descendent" of Quant-CT, another software we developed in the past. These two modules are to be integrated in a next version. Further details were reported in: D.M. Ushizima, T. Perciano, H. Krishnan, B. Loring, H. Bale, D. Parkinson, and J. Sethian. Structure recognition from high-resolution images of ceramic composites. IEEE International Conference on Big Data, October 2014.

Source record↗

DOE OSTI2014

F3D Image Processing and Analysis for Many - and Multi-core Platforms

Source record↗

DOE OSTI2025

A Data Science and Machine Learning Platform Supporting Large Particle Accelerator Control and Diagnostics Applications Final Report: SBIR Initial Phase II DE-SC0022583

The Machine Learning Data Platform (MLDP) is a product providing full-stack support for data science, Machine Learning, and Artificial Intelligence (ML/AI) applications at particle accelerator and large experimental physics facilities. It supports ML/AI applications from front-end, high-speed acquisition of heterogeneous, time-series data, through data archiving and management, to back-end analysis. The MLDP embodies a “data-science ready” platform for data analysis and ML/AI applications in diagnosis, modelling, control, and optimization of these facilities. It provides data scientists and applications a consistent, datacentric interface to archive data standardizing implementation and deployment of ML/AI algorithms to different operations configurations within the same facility, or between facilities. Being an open-source, public-domain project, the MLDP is intended for broadest possible impact by increasing accessibility and minimizing the required expertise for installation and operation. The MLDP can also be deployed at user facilities for experimental data collection, archiving, and analysis. It is capable of acquisition and archiving of heterogeneous data from experimental equipment (e.g., images, arrays, structures, etc.) along with system hardware configurations (e.g., scalars, tables), control system process variables, and any metadata required for provenance. Thus, the MLDP can manage experimental data through its entire lifecycle, from acquisition and archiving, through analysis and investigation, to release and final publication.

43 PARTICLE ACCELERATORS↗

DOE OSTI2024

Machine Learning-Enabled Image Classification for Automated Electron Microscopy

Abstract Traditionally, materials discovery has been driven more by evidence and intuition than by systematic design. However, the advent of “big data” and an exponential increase in computational power have reshaped the landscape. Today, we use simulations, artificial intelligence (AI), and machine learning (ML) to predict materials characteristics, which dramatically accelerates the discovery of novel materials. For instance, combinatorial megalibraries, where millions of distinct nanoparticles are created on a single chip, have spurred the need for automated characterization tools. This paper presents an ML model specifically developed to perform real-time binary classification of grayscale high-angle annular dark-field images of nanoparticles sourced from these megalibraries. Given the high costs associated with downstream processing errors, a primary requirement for our model was to minimize false positives while maintaining efficacy on unseen images. We elaborate on the computational challenges and our solutions, including managing memory constraints, optimizing training time, and utilizing Neural Architecture Search tools. The final model outperformed our expectations, achieving over 95% precision and a weighted F-score of more than 90% on our test data set. This paper discusses the development, challenges, and successful outcomes of this significant advancement in the application of AI and ML to materials discovery.

Materials Science↗

NASA NTRS2003

Image Processor Electronics (IPE): The High-Performance Computing System for NASA SWIFT Mission

Gamma Ray Bursts (GRBs) are believed to be the most powerful explosions that have occurred in the Universe since the Big Bang and are a mystery to the scientific community. Swift, a NASA mission that includes international participation, was designed and built in preparation for a 2003 launch to help to determine the origin of Gamma Ray Bursts. Locating the position in the sky where a burst originates requires intensive computing, because the duration of a GRB can range between a few milliseconds up to approximately a minute. The instrument data system must constantly accept multiple images representing large regions of the sky that are generated by sixteen gamma ray detectors operating in parallel. It then must process the received images very quickly in order to determine the existence of possible gamma ray bursts and their locations. The high-performance instrument data computing system that accomplishes this is called the Image Processor Electronics (IPE). The IPE was designed, built and tested by NASA Goddard Space Flight Center (GSFC) in order to meet these challenging requirements. The IPE is a small size, low power and high performing computing system for space applications. This paper addresses the system implementation and the system hardware architecture of the IPE. The paper concludes with the IPE system performance that was measured during end-to-end system testing.

Nguyen, Quang H.↗

DOE OSTI2023

Smart Mobility in the Cloud: Enabling Real-Time Situational Awareness and Cyber-Physical Control Through a Digital Twin for Traffic

This article presents the design, implementation, and use cases of the Chattanooga Digital Twin (CTwin) towards the vision for next-generation smart city applications for urban mobility management. CTwin is an end-to-end web-based platform that incorporates various aspects of the decision-making process for optimizing urban transportation systems in Chattanooga, Tennessee, to reduce traffic congestion, incidents, and vehicle fuel consumption. The platform serves as a cyberinfrastructure to collect and integrate multi-domain urban mobility data from various online repositories and Internet of Things (IoT) sensors, covering multiple urban aspects (e.g., traffic, natural hazards, weather, and safety) that are relevant to urban mobility management. The platform enables advanced capabilities for: (a) real-time situational awareness on traffic and infrastructure conditions on highways and urban roads, (b) cyber-physical control for optimizing traffic signal timing, and (c) interactive visual analytics on big urban mobility data and various metrics for traffic prediction and transportation performance evaluation. The platform is designed using a multi-level componentization paradigm and is implemented using modular and adaptive architecture, rendering it as a generalizable and extendable prototype for other urban management applications. We present several use cases to demonstrate CTwin's core capabilities for supporting decision-making in smart urban mobility management.

33 ADVANCED PROPULSION SYSTEMS↗

DOE OSTI2015

SPR Hydrostatic Column Model Verification and Validation.

A Hydrostatic Column Model (HCM) was developed to help differentiate between normal "tight" well behavior and small-leak behavior under nitrogen for testing the pressure integrity of crude oil storage wells at the U.S. Strategic Petroleum Reserve. This effort was motivated by steady, yet distinct, pressure behavior of a series of Big Hill caverns that have been placed under nitrogen for extended period of time. This report describes the HCM model, its functional requirements, the model structure and the verification and validation process. Different modes of operation are also described, which illustrate how the software can be used to model extended nitrogen monitoring and Mechanical Integrity Tests by predicting wellhead pressures along with nitrogen interface movements. Model verification has shown that the program runs correctly and it is implemented as intended. The cavern BH101 long term nitrogen test was used to validate the model which showed very good agreement with measured data. This supports the claim that the model is, in fact, capturing the relevant physical phenomena and can be used to make accurate predictions of both wellhead pressure and interface movements.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2015

BactoGeNIE: A large-scale comparative genome visualization for big displays

The volume of complete bacterial genome sequence data available to comparative genomics researchers is rapidly increasing. However, visualizations in comparative genomics--which aim to enable analysis tasks across collections of genomes--suffer from visual scalability issues. While large, multi-tiled and high-resolution displays have the potential to address scalability issues, new approaches are needed to take advantage of such environments, in order to enable the effective visual analysis of large genomics datasets. In this paper, we present Bacterial Gene Neighborhood Investigation Environment, or BactoGeNIE, a novel and visually scalable design for comparative gene neighborhood analysis on large display environments. We evaluate BactoGeNIE through a case study on close to 700 draft Escherichia coli genomes, and present lessons learned from our design process. In conclusion, BactoGeNIE accommodates comparative tasks over substantially larger collections of neighborhoods than existing tools and explicitly addresses visual scalability. Given current trends in data generation, scalable designs of this type may inform visualization design for large-scale comparative research problems in genomics.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2020

Multi-Functional Distributed Fiber Sensors for Pipeline Monitoring and Methane Detections. Final Report

As an abundant and cheap fossil energy source, natural gas has become a significant energy supply to support the United States’ economy. However, the large-scale extraction and utilization of natural gas also impose significant challenges on methane leakage. This problem is exacerbated by aging gas utility delivery systems, including interstate high-pressure pipelines, storage, and transmission facilities. This project aims to develop a cost-effective fiber optical sensing method that can perform multi-parameter real-time measurements of natural gas pipelines across long interrogation distances up to 100 km with 1-meter spatial resolution. This sensing tool can evaluate overall pipeline efficiency and reduce methane emissions for mid-stream methane infrastructures. To accomplish this objective, research and development efforts funded by this project have resulted in the following accomplishments: This project successfully has developed new functional sensory polymer materials using Metal-Organic Frameworks (MOFs) that can be coated on optical fiber through the reel-to-reel coating process. Functional polymer-coated optical fibers can perform sensitive methane detection through evanescence wave interaction and strain-based measurements to achieve 1% detection sensitivities. The new sensors fibers support both distributed measurements and multiplexed fiber sensors array for multi-point measurements. This project developed and optimized a new multi-core optical fiber that supports simultaneous and distributed measurements of strain and temperatures with 1-meter spatial resolutions across up to 100-km interrogation distance. This new fiber, combined with sensory polymercoated fiber, could perform both distributed temperature and methane detections. This project developed a new artificial intelligence big data algorithm approach that can effectively analyze high-resolution data harnessed by distributed fiber sensors to protect natural gas pipelines against external threats and detect internal defects induced by corrosion. Working with our industry partner, this project developed new optical fibers that support fiber sensor fabrications through polymer coating after the fibers are drawn. These new fibers eliminate the need for direct sensor fabrication when the fiber is fabricated on a fiber draw tower, which drastically expands fiber sensors' applicability. This research project has significantly advanced the distributed fiber sensing technology. It will dramatically increase the applicability and adaptability of distributed fiber sensors for a wide array of applications in energy, sustainability, and environmental science, including structural health monitoring of natural gas pipelines, oil infrastructures, hydrogen facilities, and environmental monitoring of carbon storage sites, water supply systems, and others.

03 NATURAL GAS↗

DOE OSTI2017

Constraining nuclear data via cosmological observations: Neutrino energy transport and big bang nucleosynthesis

Here, we introduce a new computational capability that moves toward a self-consistent calculation of neutrino transport and nuclear reactions for big bang nucleosynthesis (BBN). Such a self-consistent approach is needed to be able to extract detailed information about nuclear reactions and physics beyond the standard model from precision cosmological observations of primordial nuclides and the cosmic microwave background radiation. We also calculate the evolution of the early universe through the epochs of weak decoupling, weak freeze-out and big bang nucleosynthesis (BBN) by simultaneously coupling a full strong, electromagnetic, and weak nuclear reaction network with a multi-energy group Boltzmann neutrino energy transport scheme. The modular structure of our approach allows the dissection of the relative contributions of each process responsible for evolving the dynamics of the early universe. Such an approach allows a detailed account of the evolution of the active neutrino energy distribution functions alongside and self-consistently with the nuclear reactions and entropy/heat generation and flow between the neutrino and photon/electron/positron/baryon plasma components. Our calculations reveal nonlinear feedback in the time evolution of neutrino distribution functions and plasma thermodynamic conditions. We discuss the time development of neutrino spectral distortions and concomitant entropy production and extraction from the plasma. These effects result in changes in the computed values of the BBN deuterium and helium-4 yields that are on the order of a half-percent relative to a baseline standard BBN calculation with no neutrino transport. This is an order of magnitude larger effect than in previous estimates. For particular implementations of quantum corrections in plasma thermodynamics, our calculations show a 0.4% increase in deuterium and a 0.6% decrease in 4 He over our baseline. The magnitude of these changes are on the order of uncertainties in the nuclear physics for the case of deuterium and are potentially significant for the error budget of helium in upcoming cosmological observations.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2020

Adaptive Anomaly Detection for Dynamic Clinical Event Sequences

Over the past decade, health information technology (IT) has enabled the amount of digital information stored in electronic health records (EHRs) to expand greatly. However, according to some studies, hazards in health IT can lead to changes in clinical decisions, care processes, and care outcomes, as well as other issues. Thus, the effects of health IT hazards on patient safety have been at the forefront of recent patient safety research. Nonetheless, hazard detection in health IT remains a challenge. In this paper, the authors assume that safety-related issues in health IT would exhibit anomalous characteristics in EHR data. Although all hazards will exhibit some anomalous characteristics, not all anomalies can be regarded as hazards. The authors hypothesize that errors in health IT could lead to interruptions in the sequence of clinical actions. To this end, the problem of detecting anomalous sequences in big EHR data is considered. This paper focuses on dynamic event sequences, which are a series of clinical actions in motion. The authors propose an adaptive anomaly detection approach that uses higher-order network representation to detect anomalous sequences. Furthermore, the authors propose a contiguous subsequence anomaly detection approach that identifies abnormal subsequences in the detected anomalous sequences. The proposed approaches are tested by using synthetic and real-world EHR data. The proposed methods outperform existing state of the art anomaly detection techniques. To reduce the computational complexity associated with the operational implementation of the proposed approaches, the Apache Spark environment was leveraged, and a much shorter run time together with improved performance were achieved, especially for data with more than 60,000 sequences.

Niu, Haoran↗

DOE OSTI2025

Barge Site - Avian Radar System / Derived Data

This is a combined data set of 67,410 bird/bat tracks from an avian radar system deployed on a research barge (MERLIN True3D, DeTect, Panama City, Florida, USA) and concurrent wind measurements from two scanning lidars (WindCube v2.1, Vaisala, Vantaa, Finland, and Halo XR+, Halo Photonics, Lannion, France). The research barge (16.5 m x 61 m) was deployed as part of the Wind Forecast Improvement Project (WFIP-3) off the northeast coast of the United States south of Massachusetts (40.9 deg N, 70.79 deg W). This data set comprises 5 weeks of data between August 27th 2024 and September 27th 2024. Radar data were provided by DeTect and Lidar data were accessed through the Wind Data Hub (wfip3/barg.WINDPROF.z01.a0) The data have been filtered and sorted into two size groups ("big" and "small") based on a clustering approach. See Snortland, A., Clerc, J., Hein, C., & Cotter, E. (2025). Wind as Driver of Bird and Bat Abundance, Flight Direction, Altitude, and Speed on the North Atlantic Shelf. arXiv preprint arXiv:2511.14983 for complete details. Data are provided in 2 files: "Birds" and "Birds_hourly" Birds: This file contains information about each of the 67,410 flying animal tracks detected by the radar during the data collection period, including parameters measured by the radar and wind information interpolated from the lidar wind measurements. We note that the raw radar dataset contained 301,618 tracks; tracks in this processed dataset were filtered based on the requirements described in Snortland et al. (2025). Birds_hourly: This file contains timeseries of the number of tracks detected per hour over the course of the data collection period, including wind conditions and sun position for each hour. These data were used for generalized additive modeling in Snortland et al. (2025).

17 WIND ENERGY↗

NASA NTRS1988

Massive parallelism in the future of science

Massive parallelism appears in three domains of action of concern to scientists, where it produces collective action that is not possible from any individual agent's behavior. In the domain of data parallelism, computers comprising very large numbers of processing agents, one for each data item in the result will be designed. These agents collectively can solve problems thousands of times faster than current supercomputers. In the domain of distributed parallelism, computations comprising large numbers of resource attached to the world network will be designed. The network will support computations far beyond the power of any one machine. In the domain of people parallelism collaborations among large groups of scientists around the world who participate in projects that endure well past the sojourns of individuals within them will be designed. Computing and telecommunications technology will support the large, long projects that will characterize big science by the turn of the century. Scientists must become masters in these three domains during the coming decade.

Denning, Peter J.↗

DOE OSTI2021

MindSynchro

This report presents the developments and results of MindSynchro project as part of DOE OE FOA 1861. DOE and Pacific Northwest National Laboratory (PNNL) have made available to FOA awardees datasets containing years of real historical data recorded from various phasor measurement units (PMUs) which are installed in three large US interconnections: Texas (IC A), Western (IC B), and Eastern (IC C). The main goal of the project, which was successfully achieved, was to develop methods for detection and identification of events which are relevant for power grid operation. Tasks performed for achieving the project goals included data exploration and pre-processing, the development and application of physics-based features, data analysis and labeling based on unsupervised learning approaches, training and testing of DSSL models for classification of events which are relevant for power grid operation, and deployment of solutions to cloud environments. The methods developed in the project can potentially provide relevant benefits to power grid asset owners/operators in general in terms of situational awareness. Two main types of outcomes can be provided by these tools: Identification of specific relevant power grid event types: Semi-supervised ML methods developed in the project can adequately employ not only the relatively scarce labeled data but also the large amount of available unlabeled data to train models for detection of specific event types. Such methods enable the application of trained models for the detection of events in a population of PMUs much larger than that associated to the labeled events. Support in data labeling / label validation: Labels are critical for training of models for identification of specific types of events. However, labeling large amounts of data is a manual and tedious process. This means that such process is error prone and is not scalable. Methods developed in the project, based on ensembles of clustering models, have been successfully employed for turning manual labeling into a scalable process. Accurate identification of specific relevant events can provide the operators with immediate situational awareness that could otherwise require hours or days of analysis from domain experts. We envision that such methods could be initially employed in support of post-mortem analysis of events and, as confidence is gained, they could be employed for online/real-time support, providing, among other benefits, insights for avoiding major events which could happen due to a combination of smaller ones. On the longer term, related methods could potentially be employed to improve protection and control.

24 POWER TRANSMISSION AND DISTRIBUTION↗