Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DATA NETWORKS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

Phase Transitions and a Model Order Selection Criterion for Spectral Graph Clustering

One of the longstanding open problems in spectral graph clustering (SGC) is the so-called model order selection problem: automated selection of the correct number of clusters. This is equivalent to the problem of finding the number of connected components or communities in an undirected graph. Here, we propose automated model order selection (AMOS), a solution to the SGC model selection problem under a random interconnection model (RIM) using a novel selection criterion that is based on an asymptotic phase transition analysis. AMOS can more generally be applied to discovering hidden block diagonal structure in symmetric non-negative matrices. Numerical experiments on simulated graphs validate the phase transition analysis, and real-world network data is used to validate the performance of the proposed model selection procedure.

97 MATHEMATICS AND COMPUTING↗

Machine learning-based analysis of COVID-19 pandemic impact on US research networks

Here in this study we explore how fallout from the changing public health policy around COVID-19 has changed how researchers access and process their science experiments. Using a combination of techniques from statistical analysis and machine learning, we conduct a retrospective analysis of historical network data for a period around the stay-at-home orders that took place in March 2020. Our analysis takes data from the entire ESnet infrastructure to explore DOE high-performance computing (HPC) resources at OLCF, ALCF, and NERSC, as well as User sites such as PNNL and JLAB. We look at detecting and quantifying changes in site activity using a combination of t-Distributed Stochastic Neighbor Embedding (t-SNE) and decision tree analysis. Our findings bring insights into the working patterns and impact on data volume movements, particularly during late-night hours and weekends.

97 MATHEMATICS AND COMPUTING↗

FTP 100: An Ultra-High Speed Data Transfer Service Over Next Generation 100 Gigabit Per Second Network

Data-intensive applications, including high energy and nuclear physics, astrophysics, climate modeling, nano-scale materials science, genomics, and financing, are expected to generate exabytes of data over the coming years, which must be transferred, visualized, and analyzed by geographically distributed teams of users. High-performance network capabilities must be available to these users at the application level in a transparent, virtualized manner. Moreover, the application users must have the capability to move large datasets from local and remote locations across network environments to their home institutions. To solve these challenges, the main goal of our project is to design and evaluate high-performance data transfer software to support various data-intensive applications. First, we have designed a middleware software that provides access to Remote Direct Memory Access (RDMA) functionalities. This middleware integrates network access, memory management and multitasking in its core design. We address a number of issues related to its efficient implementation, for instance, explicit buffer management and memory registration, and parallelization of RDMA operations, which are vital to delivering the benefit of RDMA to the applications. Built on top of this middleware, an implementation and experimental evaluation of the RDMA-based FTP software, RFTP, is described and evaluated. This application has been implemented by our team to exploit the full capabilities of advanced RDMA mechanisms for ultra-high speed bulk data transfer applications on Energy Sciences Network (ESnet). Second, we designed our data transfer software to optimize TCP/IP based data transfer performance such that RFTP can be fully compatible with today’s Internet. Our kernel optimization techniques with Linux system calls sendfile and splice, can reduce data copy cost. In this report, we summarize the technical challenges of our project, the primary software design methods, the major project milestones achieved, as well as the testbed evaluation work and demonstrations during our project life time.

97 MATHEMATICS AND COMPUTING↗

Development of a data-driven neural network model for electron thermal transport in NSTX

A data-driven electron thermal transport neural network (ETT-NN) model, trained on TRANSP interpretative analysis results of National Spherical Torus Experiment (NSTX), was developed to enable faster and more accurate ETT computation for spherical tokamaks (STs). The model incorporates both convolutional NNs and recurrent NNs, allowing it to simultaneously account for the spatial and temporal non-localities and multi-scale features of turbulent transport, which have been considered only in a limited manner in conventional models. The model was validated through interpretative analysis and predictive simulations using Tokamak Reactor Integrated Automated Suite for Simulation and Computation, demonstrating relatively high accuracy. Additionally, parameter scans were performed on test discharges known to exhibit specific turbulent modes, such as microtearing mode, trapped electron mode, kinetic ballooning mode, and electron temperature gradient mode. The scanning results revealed that the ETT-NN model exhibits the same trends as those observed in conventional gyrokinetic simulations or theories, while also capturing the global nature of turbulent transport, indicating that the data-driven model accurately reflects the underlying physical characteristics. Furthermore, due to the dimensionless nature of the model, we can feasibly expand its applicability by incorporating data from other devices and uncovering the characteristics of ETT in STs in the future.

NSTX↗

Preliminary Transfer Learning Results on Israel Data

In this preliminary report, we use publicly available data recorded in Israel to test and expand upon existing machine learning models for seismic-phase detection and arrival-time measurement. We downloaded 3-years of waveform data from Geofon, and cross referenced the waveforms to Israel bulletin picks (Schardong et al., 2021). The initial results using existing models directly generated ubiquitous false detections and that obscured detections of signals that are clearly visible in the waveforms. However, after applying transfer learning (tuning parameters in the existing ML models using one year of the Israel-network data), the results are encouraging, i.e. ML picks agree within a few tenths of a second with bulletin picks and the number of false detections is greatly reduced. The bulletin picks are a good starting point, but they cannot be considered ground-truth. To test potential improvement in picking using ML we would like to relocate the events using the ML picks to see if the events cluster more tightly at known mine locations. However, in order to constrain event locations, we need ML picks for the whole Israeli-Jordanian network, which requires waveforms that are not publicly available.

58 GEOSCIENCES↗

Web-Based Reanalysis Intercomparison Tools (WRIT) for Analysis and Comparison of Reanalyses and Other Datasets

While atmospheric reanalysis datasets are widely used in climate science, many technical issues hinder comparing them to each other and to observations. The reanalysis fields are stored in diverse file architectures, data formats, and resolutions. Their metadata, such as variable name and units, can also differ. Individual users have to download the fields, convert them to a common format, store them locally, change variable names, regrid if needed, and convert units. Even if a dataset can be read via the Open-Source Project for a Network Data Access Protocol (commonly known as OPeNDAP) or a similar protocol, most of this work is still needed. All of these tasks take time, effort, and money. Our group at the Cooperative Institute for Research in the Environmental Sciences at the University of Colorado and affiliated colleagues at the NOAA's Earth System Research Laboratory Physical Sciences Division have expertise both in making reanalysis datasets available and in creating web-based climate analysis tools that have been widely used throughout the meteorological community. To overcome some of the obstacles in reanalysis intercomparison, we have created a set of web-based Reanalysis Intercomparison Tools (WRIT) at www.esrl.noaa.gov/psd/data/writ/. WRIT allows users to easily plot and compare reanalysis datasets, and to test hypotheses. For standard pressure-level and surface variables there are tools to plot trajectories, monthly mean maps and vertical cross sections, and monthly mean time series. Some observational datasets are also included. Users can refine date, statistics, and plotting options. WRIT also facilitates the mission of the Reanalyses.org website as a convenient toolkit for studying the reanalysis datasets.

54 ENVIRONMENTAL SCIENCES↗

Inferring microbial co-occurrence networks from amplicon data: a systematic evaluation

Microbes commonly organize into communities consisting of hundreds of species involved in complex interactions with each other. 16S ribosomal RNA (16S rRNA) amplicon profiling provides snapshots that reveal the phylogenies and abundance profiles of these microbial communities. These snapshots, when collected from multiple samples, can reveal the co-occurrence of microbes, providing a glimpse into the network of associations in these communities. However, the inference of networks from 16S data involves numerous steps, each requiring specific tools and parameter choices. Moreover, the extent to which these steps affect the final network is still unclear. In this study, we perform a meticulous analysis of each step of a pipeline that can convert 16S sequencing data into a network of microbial associations. Through this process, we map how different choices of algorithms and parameters affect the co-occurrence network and identify the steps that contribute substantially to the variance. We further determine the tools and parameters that generate robust co-occurrence networks and develop consensus network algorithms based on benchmarks with mock and synthetic data sets. The Microbial Co-occurrence Network Explorer, or MiCoNE (available at https://github.com/segrelab/MiCoNE) follows these default tools and parameters and can help explore the outcome of these combinations of choices on the inferred networks. We envisage that this pipeline could be used for integrating multiple data sets and generating comparative analyses and consensus networks that can guide our understanding of microbial community assembly in different biomes.

16S rRNA↗

Setting the threshold for high throughput detectors: A mathematical approach for ensembles of dynamic, heterogeneous, probabilistic anomaly detectors

Cyber operations now manage a high volume of heterogeneous log data. Anomaly Detection (AD) in such operations involves multiple (e.g., per IP, per data type) ensembles of detectors modeling heterogeneous characteristics (e.g., rate, size, type) often with adaptive online models producing alerts in near real time. Because of the high data volume, setting the threshold for each detector in such a system is an essential yet underdeveloped configuration issue that, if slightly mistuned, can leave the system useless, either producing a myriad of alerts (and flooding downstream systems) or giving none. In this work, we build on the foundations of Ferragut et al. to provide a set of rigorous results for understanding the relationship between threshold values and alert quantities for probabilistic detectors. This informs an algorithm for setting the threshold of multiple, heterogeneous, possibly dynamic detectors completely a priori, in principle. Indeed, if the underlying distribution of the incoming data is known, the algorithm provides provably manageable thresholds. If the distribution is unknown (poorly estimated), our analysis gives insight into how the model distribution differs from the actual distribution, indicating refitting is necessary. We provide empirical experiments, regulating the alert rate of a system with ≈2,500 adaptive detectors scoring over 1.5M events in 5 hours of timestamps. Further, we demonstrate on real network data and detection framework of Harshaw et al. the alternative case, demonstrating that the inability to regulate alerts indicates how the detection model is not a good fit to the data.

Bridges, Robert↗

Predicting and Detecting Emerging Cyberattack Patterns Using StreamWorks

The number and sophistication of cyberattacks on industries and governments have dramatically grown in recent years. To counter this movement, new advanced tools and techniques are needed to detect cyberattacks in their early stages such that defensive actions may be taken to avert or mitigate potential damage. From a cybersecurity analysis perspective, detecting cyberattacks may be cast as a problem of identifying patterns in computer network traffic. Logically and intuitively, these patterns may take on the form of a directed graph that conveys how an attack or intrusion propagates through the computers of a network. Such cyberattack graphs could provide cybersecurity analysts with powerful conceptual representations that are natural to express and analyze. We have been researching and developing graph-centric approaches and algorithms for dynamic cyberattack detection. The advanced dynamic graph algorithms we are developing will be packaged into a streaming network analysis framework known as StreamWorks. With StreamWorks, a scientist or analyst may detect and identify precursor events and patterns as they emerge in complex networks. This analysis framework is intended to be used in a dynamic environment where network data is streamed in and is appended to a large-scale dynamic graph. Specific graphical query patterns are decomposed and collected into a graph query library. The individual decomposed subpatterns in the library are continuously and efficiently matched against the dynamic graph as it evolves to identify and detect early, partial subgraph patterns. The scalable emerging subgraph pattern algorithms will match on both structural and semantic network properties.

Chin, George↗

Evidence for a composite organic–inorganic fabric of belemnite rostra: Implications for palaeoceanography and palaeoecology

Carbonate skeletons of fossil marine organisms are widely used to reconstruct palaeoceanographic parameters. Specifically, the geochemistry of Jurassic and Cretaceous belemnite rostra is traditionally interpreted to represent near sea-surface seawater properties. More recently, an increasing number of workers, have reported significant scatter in geochemical data (e.g., δ 18 O, δ 13 C, element/Ca ratio) when comparing rostra from the same stratigraphic level or within a single belemnite rostrum. This scatter is not explained by differential diagenetic overprint alone. Here we report petrographic evidence on the primary ultrastructure of rostra of Megateuthis (Middle Jurassic) and Belemnitella and Gonioteuthis (Late Cretaceous). The biogenic ultrastructure consists of a filigree framework of triaxial branches and tetrahedrons of variable size forming a honeycomb-like network. Data presented here suggest that these rostra yielded as much as 50 to 90% primary pore space. On the level of a working hypothesis – and in analogy with modern cephalopods – we propose that the pore space was formerly filled with body fluid and/or organic compounds during the life time of these organisms. Intra-rostral porosity was post mortem occluded by earliest diagenetic isopachous calcite cements of a non-biogenic origin. These may have been precipitated due to increased alkalinity related to the decay of organic matter. If this holds true, then the resulting fabric represents a composite biogenic/abiogenic structure. In order to optically separate the two calcite phases forming a single calcite fibre, we employed a wide range of state-of-the-art analytical tools to thin sections and ultra-thin sections of well-preserved specimens. Finally, pending a verification of these well-supported ultrastructural data by means of high-resolution geochemical analyses from biogenic and abiogenic phases, we suggest that these findings have significance for those using belemnite rostra as archives of their palaeoenvironment.

58 GEOSCIENCES↗

Fast katz and commuters : efficient estimation of social relatedness in large networks.

Motivated by social network data mining problems such as link prediction and collaborative filtering, significant research effort has been devoted to computing topological measures including the Katz score and the commute time. Existing approaches typically approximate all pairwise relationships simultaneously. In this paper, we are interested in computing: the score for a single pair of nodes, and the top-k nodes with the best scores from a given source node. For the pairwise problem, we apply an iterative algorithm that computes upper and lower bounds for the measures we seek. This algorithm exploits a relationship between the Lanczos process and a quadrature rule. For the top-k problem, we propose an algorithm that only accesses a small portion of the graph and is related to techniques used in personalized PageRank computing. To test the scalability and accuracy of our algorithms we experiment with three real-world networks and find that these algorithms run in milliseconds to seconds without any preprocessing.

99 GENERAL AND MISCELLANEOUS↗

The Detection of Mesoscale Convective Systems by the GPM Ku-Band Spaceborne Radar

The Global Precipitation Measurement (GPM) core observatory satellite launched in 2014 features more extended latitudinal coverage (65 degree S-65 degree N) than its predecessor Tropical Rainfall Measuring Mission (TRMM, 35 degree S-35 degree N). The Ku-band radar onboard of the GPM is known to be capable of characterizing the 3D structure of deep convection globally. In this study, GPM’s capability for detecting mesoscale convective systems (MCSs) is evaluated. Extreme convective echoes seen by GPM are compared against an MCS database based on tracking convective entities over the contiguous US. The tracking is based on geostationary satellite and ground-based Next Generation Radar (NEXRAD) network data obtained during the 2014-2016 warm seasons. Results show that more than 70% of the GPM-detected Deep-Wide Convective Core (DWC) and Wide Convective Core (WCC) objects are part of NEXRAD identified MCSs, indicating that GPM-classified DWCs and WCCs correlate well with typical MCSs containing large convective features. By applying this method to the rest of the world, a global view of MCS distribution is obtained. This work reveals GPM’s potential in MCS detection at the global scale, particularly over remote regions without dense observation network.

3D reflectivity structure↗

Chemical Sensing Strategies for Real-Time Monitoring of Transformer Oil: A Review

Power transformers are a central component in the field of energy distribution and transmission. The early recognition of incipient faults in operating transformers is substantially cost effective by lessening impromptu blackouts. A standout amongst the most responsive and dependable strategies utilized for assessing the health of oil filled electrical equipment is dissolved gas analysis (DGA). Nowadays, there is an expanding requirement for better nonintrusive diagnostic and online monitoring tools to survey the internal state of the transformers. Chemical sensors are viewed as a key innovation for condition monitoring of transformer health, coordinating the non-invasiveness with typical sensor features, such as cost, usability, portability, and the integration with the data networks. Low-cost chemical sensors-based DGA techniques are expected to drastically augment the diagnostic abilities empowering the deployment on a broader range of oil filled power assets. The recent development involves both specific sensors designed to detect individual dissolved gas in transformer oil and non-specific sensors, operated in near ambient conditions, with the potential to be applied in a DGA system. In this paper, general background and operating guidelines of DGA are presented to address the origin of the gas formation, methods for their detection and the interpretation of the results by data analytics. The recent significant interest and advancements in chemical sensors to DGA applications are reviewed. Lastly, future research perspectives and challenges for the development of novel DGA chemical sensors are also discussed.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Open Power System Datasets and Open Simulation Engines: A Survey Toward Machine Learning Applications

A major factor behind the success of machine learning (ML) models in multiple domains is the availability and accessibility of large, labeled, and well-organized datasets for training and benchmarking. In comparison, power grid datasets face three major challenges: (i) real-world data is often restricted by regulatory constraints, privacy reasons, or security concerns, making it difficult to obtain and work with; (ii) synthetic datasets, which are created to address these limitations, often have incomplete information and are released using specialized tools, making them inaccessible to the broader community; and, (iii) input-output datasets are difficult to generate through simulation for non-experts because open-source simulators are not known outside the power system community. This survey addresses these challenges by serving as an entry point to publicly available datasets and simulators for researchers venturing in this area. We review the current landscape of open-source power network data, machine models, consumer demand profiles, renewable generation data, and inverter models. We also examine open-source power system simulators, which are crucial for generating high-quality, high-fidelity power grid datasets. We aim to provide a foundation for overcoming data scarcity and advance towards a structured web of datasets and simulators to support the development of ML for power systems.

42 ENGINEERING↗

SuperLab 2.0 Showcase: Connecting Five Labs to Tackle Grid Complexity and Unlock Unique Grid Asset Potential

SuperLab 2.0 (5-Lab Demo) is a collaborative, national-scale experiment showcasing the coordination of geographically distributed energy assets in real time. The demonstration integrates 25 physical and digital assets, spanning wind, PV, batteries, electrolyzers, DC fast chargers, microgrid controllers, building automation systems, small modular reactor (SMR), control centers, and gas turbines, across five DOE national laboratories-NLR, INL, NETL, LBNL, and SNL. These assets are unified using Energy Sciences Network (ESnet), a low-latency, high-performance U.S. Department of Energy's (DOE) network, and controlled via a centralized energy controller hosted at NLR's ARIES facility. The demonstration validates the ability to stress-test hybrid energy systems under dynamic scenarios to de-risk advanced control strategies for greater resilience and flexibility. SuperLab 2.0 (5-Lab Demo) showcased a major advancement in federated national laboratory collaboration, enabling real-time, cross-laboratory experimentation to coordinate geographically dispersed distributed energy resources (DERs) using various communication protocols and networks. SuperLab 2.0 (5-Lab Demo) built on previous demonstrations conducted between NLR-PNNL and NLR-INL connecting diverse assets including distant protection devices, a SMR simulator, and a high temperature electrolyzer (HTE). Previous demos were based on a single connection between two labs with minimal coordination challenges. The 5-Lab demo with a centralized controller, distributed testbeds across different geographical locations, and use of protocols-based communication represents a scenario closer to real-world grid operations that coordinate resources across a region to meet system needs. This experiment studied how local DER controllers interact with a centralized energy controller during normal and abnormal events to maintain reliability. The SuperLab team across the five labs implemented a notional power system model equivalent of transmission and distribution lines, represented by the data networks interconnecting the labs. Each lab continuously exchanged local parameters (such as P and Q) from its Hardware-In-Loop (CHIL) and Power Hardware-In-Loop (PHIL) assets through centralized energy controller at NLR, enabling real-time interaction and coordination across sites. By leveraging ESnet as the communication backbone, the team successfully operated the distributed assets as a unified power system, with each bus represented by a different laboratory. This setup mirrors how assets interact in real-world power systems across dispersed locations with various protocols and latencies. At each lab site, assets were operated using their own local controllers which were coordinated through an overarching operation and control layer of centralized energy controller, equivalent to how an energy management system (EMS) orchestrates assets across a regional or national grid. SuperLab's federated connectivity utilized a Digital Real-Time Simulators (DRTS)-type gateway to connect Controller Hardware-In-Loop (CHIL) and PHIL assets between labs. To enable this federated connection through ESnet, a deterministic network was established where latency variations were consistent. This consistency allowed the development of digital filters for the power system assets across CHIL and PHIL interfaces to avoid unstable and unreliable grid conditions. This report provides an overview of the cross-laboratory configuration and offers insights into interconnecting geographically distributed research assets to test them as if they were co-located. This experiment represents a step toward linking nine DOE national laboratories, enabling nation-wide simulations that can address utility-driven challenges with grid resilience, flexibility, and modernization.

24 POWER TRANSMISSION AND DISTRIBUTION↗

NERSC & JGI Seek Elusive Proteins for Gene Editing

A video just released by the Supercomputing 2017 (SC17) conference shows how biologists at UC Berkeley and the Joint Genome Institute (JGI) – a Department of Energy Office of Science User Facility – are using NERSC, the Office of Science’s primary supercomputing center, to identify a class of enzymes capable of editing out genes responsible for disease. This revolutionary work requires petaflops of computing power to sift through billions of DNA sequences in the JGI data portals to identify proteins like Cas9 that, combined with the Clustered Regularly Interspaced Short Palindromic Repeats (CRISPR), can “edit” a genome. First discovered by researchers at UC Berkeley, the CRISPR-Cas9 system has the power to permanently correct disease-causing mutations in cells, animal models, and eventually human patients to cure certain types of cancer, cardiovascular disease, sickle cell disease, and retinal degeneration. Since the CRISPR system was first identified in the early 2000s, it has become the most promising technology for treating genetic disease in humans and enabling plants to survive stress caused by pests and the environment. The annual SC17 conference will take place November 12-17 in Denver, Colorado. The over 12,000 attendees from academia, government and industry are leaders in researching and delivering high performance computing, networking, data storage and analysis in support of scientific discovery. http://sc17.supercomputing.org

Berkeley Lab, LBNL↗

NERSC & JGI Seek Elusive Proteins for Gene Editing

A video just released by the Supercomputing 2017 (SC17) conference shows how biologists at UC Berkeley and the Joint Genome Institute (JGI) – a Department of Energy Office of Science User Facility – are using NERSC, the Office of Science’s primary supercomputing center, to identify a class of enzymes capable of editing out genes responsible for disease. This revolutionary work requires petaflops of computing power to sift through billions of DNA sequences in the JGI data portals to identify proteins like Cas9 that, combined with the Clustered Regularly Interspaced Short Palindromic Repeats (CRISPR), can “edit” a genome. First discovered by researchers at UC Berkeley, the CRISPR-Cas9 system has the power to permanently correct disease-causing mutations in cells, animal models, and eventually human patients to cure certain types of cancer, cardiovascular disease, sickle cell disease, and retinal degeneration. Since the CRISPR system was first identified in the early 2000s, it has become the most promising technology for treating genetic disease in humans and enabling plants to survive stress caused by pests and the environment. The annual SC17 conference will take place November 12-17 in Denver, Colorado. The over 12,000 attendees from academia, government and industry are leaders in researching and delivering high performance computing, networking, data storage and analysis in support of scientific discovery. http://sc17.supercomputing.org Producer: Jennifer Boyd, Boyd Productions DP: Damon Hennessey Audio: Richard K. Pooler Score: Michael Bacon Music

59 BASIC BIOLOGICAL SCIENCES↗

Generalized blockmodeling of multiple valued networks.

This paper presents an extension to generalized blockmodeling where there are more than two types of objects to be clustered based on valued network data. We use the ideas in homogeneity block modeling to develop an optimization model to perform the clustering of the objects and the resulting partitioning of the ties so as to minimize the inconsistency of an empirical block with an ideal block. The ideal block types used in this modeling are null, complete and a new type that is related to that developed in Ziberna (2007). Three case studies are presented, two based on the Southern Women dataset (Davis et al. 1941) and a third based on passenger air travel in the Continental United States.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗