Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data aggregation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,603 records · Page 89

MOSIQS: Persistent Memory Object Storage With Metadata Indexing and Querying for Scientific Computing

Scientific applications often require high-bandwidth shared storage to perform joint simulations and collaborative data analytics. Shared memory pools provide a chance to satisfy such needs. Recently, a high-speed network such as Gen-Z utilizing persistent memory (PM) offers an opportunity to create a shared memory pool connected to compute nodes. However, there are several challenges to use scientific applications on the shared memory pool directly such as scalability, failure-atomicity, and lack of scientific metadata-based search and query. In this paper, we propose MOSIQS, a persistent memory object storage framework with metadata indexing and querying for scientific computing. We design MOSIQS based on the key idea that memory objects on PM pool can live beyond the application lifetime and can become the sharing currency for applications and scientists. MOSIQS provides an aggregate memory pool atop an array of persistent memory devices to store and access memory objects to accelerate scientific computing. MOSIQS uses a lightweight persistent memory key-value store to manage the metadata of memory objects, which enables memory object sharing. To facilitate metadata search and query over millions of memory objects resident on memory pool, we introduce Group Split and Merge (GSM), a novel persistent index data structure designed primarily for scientific datasets. GSM splits and merges dynamically to minimize the query search space and maintains low query processing time while overcoming the index storage overhead. MOSIQS is implemented on top of PMDK. We evaluate the proposed approach on many-core server with an array of real PM devices. Experimental results show that MOSIQS gains a 100% write performance improvement and executes multi-attribute queries efficiently with 2.7× less index storage overhead offering significant potential to speed up scientific computing applications.

97 MATHEMATICS AND COMPUTING↗

Empowered communities: The rise of community choice aggregation in the United States

Community choice aggregation (CCA) is a new model of energy procurement that allows local governmental entities to procure electricity on behalf of retail electricity customers. Through CCA, local governments can control local electricity portfolios while investor-owned utilities remain responsible for transmission and distribution. In this article, we use a combination of publicly-available data, data obtained directly from CCAs, and stakeholder interviews to explore the rise of CCAs, the current and potential future impacts of CCAs on demand for renewable energy, and the factors that will determine future CCA expansion. We believe that CCAs procured about 42 million megawatt-hours of electricity on behalf of about 5 million customers in the United States in 2017. We estimate that CCAs already procure about 8.9 million megawatt-hours more renewable energy than required by state mandates, and that CCAs could procure as much as 28.9 million megawatt-hours of voluntary renewable energy if the CCA model is permitted in more states. The ongoing expansion of CCAs could significantly affect electricity markets, electricity portfolios, and the future role of utilities. We investigate various challenges associated with the further expansion of CCAs.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Multivariate network traffic analysis using clustered patterns

Traffic analysis is a core element in network operations and management for various purposes including change detection, traffic prediction, and anomaly detection. In this paper, we introduce a new approach to online traffic analysis based on a pattern-based representation for high-level summarization of the traffic measurement data. Unlike the past online analysis techniques limited to a single variable to summarize (e.g., sketch), the focus of this study is on capturing the network state from the multivariate attributes under consideration. To this end, we employ clustering with its benefit of the aggregation of multidimensional variables. The clustered result represents the state of the network with regard to the monitored variables, which can also be compared with the observed patterns from previous time windows enabling intuitive analysis. Finally, we demonstrate the proposed method with two popular use cases, one for estimating state changes and the other for identifying anomalous states, to confirm its feasibility. Our extensive experimental results with public traces and collected monitoring measurements from ESnet traffic traces show that our pattern-based approach is effective for multivariate analysis of online network traffic with visual and quantitative tools.

97 MATHEMATICS AND COMPUTING↗

Nanoscale observations of Fe(ii)-induced ferrihydrite transformation

Because of its sorption properties, transformation of the nanomineral ferrihydrite (Fh) into more stable lepidocrocite (Lp) or goethite (Gt) has important impacts on the fate of metals, nutrients, and contaminants in soils/sediments. Although it is well known that the transformation rate is greatly accelerated under suboxic conditions by aqueous Fe(II), the enabling mass transfer process remains an ongoing debate among various mechanisms including dissolution/reprecipitation, solid-state recrystallization, and particle-mediated growth. In this paper, using electron microscopy, we examine the mineralogical evolution of 2-line Fh to Lp/Gt catalyzed by Fe(II) under strict anoxic conditions, including evaluation of Cl–SO 4 –HCO 3 anion effects. Emergence of Lp/Gt crystallites at the nanoscale was observed at ~20 min of reaction, earlier than previously reported. Lp is the first phase to nucleate in Cl-rich solutions without HCO 3 – ; whereas Lp and Gt concomitantly nucleate in SO 4 2– -rich solutions, and also when co-solute HCO 3 – is added. Lp crystallites nucleate as quasi-2D nanosheets one-unit-cell thick that contour the Fh surface; in contrast, rod-shaped (in Cl/SO 4 ) or acicular needle-shaped (in HCO 3 ) Gt crystals nucleate and grow radially outward from the Fh aggregates. Stages of transformation monitored by in situ μ-XRD coupled with aqueous Fe(II) uptake/release measurements are correlated with a short initial sorption stage followed by the onset of Lp/Gt growth that then progresses to Lp loss in favor of Gt. Microscopy data overwhelmingly support dissolution/reprecipitation as the underlying mechanism, including direct evidence for classical ion-by-ion Lp/Gt growth and Lp dissolution. The collective findings imply that the iron mass transfer through solution to distal Lp/Gt growth fronts is a critical enabling process facilitating rapid transformation.

54 ENVIRONMENTAL SCIENCES↗

Online Data Handling and Storage at the CMS Experiment

During the LHC Long Shutdown 1, the CMS Data Acquisition (DAQ) system underwent a partial redesign to replace obsolete network equipment, use more homogeneous switching technologies, and support new detector back-end electronics. The software and hardware infrastructure to provide input, execute the High Level Trigger (HLT) algorithms and deal with output data transport and storage has also been redesigned to be completely file- based. All the metadata needed for bookkeeping are stored in files as well, in the form of small documents using the JSON encoding. The Storage and Transfer System (STS) is responsible for aggregating these files produced by the HLT, storing them temporarily and transferring them to the T0 facility at CERN for subsequent offline processing. The STS merger service aggregates the output files from the HLT from ~62 sources produced with an aggregate rate of ~2GB/s. An estimated bandwidth of 7GB/s in concurrent read/write mode is needed. Furthermore, the STS has to be able to store several days of continuous running, so an estimated of 250TB of total usable disk space is required. In this article we present the various technological and implementation choices of the three components of the STS: the distributed file system, the merger service and the transfer system.

Andre, J. M.↗

Extreme Heterogeneity 2018 - Productive Computational Science in the Era of Extreme Heterogeneity: Report for DOE ASCR Workshop on Extreme Heterogeneity

The 2018 Basic Research Needs Workshop on Extreme Heterogeneity identified five Priority Research Directions for realizing the capabilities needed to address the challenges posed in this era of rapid technological change. This report captures the outcomes of that workshop. In the context of extreme heterogeneity, it defines basic research needs and opportunities in computer science research to develop smart and trainable operating and runtime systems, programming environments, and predictive tools that will make future systems easier to adapt to scientists’ computing needs and easier for facilities to deploy securely.

97 MATHEMATICS AND COMPUTING↗

Reducing Memory Consumption in Calico with Shared Memory

This document details the work to reduce memory consumption in Calico. Calico is SimTools’ Constructive Solid Geometry (CSG) and geometry painting library. It is primarily used to paint material volume fractions in the Eulerian meshes of the physics codes. Calico provides point-in-body checks for the geometry supplied by an Oso model, which are then aggregated by the host codes. In addition, Calico can be used to build Oso models and is used by Ingen for that purpose. Oso models, and thus Calico, provide support for various CSG primitives such as spheres, cylinders, surfaces generated by rotating tabular curve data, and STL files as well as binary combinations of those primitives. Prior to refactoring Calico will run out of memory on CTS-1 machines when 36 MPI ranks are used per node when reading STL models on the order of 1.5 GB. This limitation is a bottleneck in designer workflow. This problem has been alleviated through the use of data structures to both reduce memory consumption and to leverage MPI-3 shared memory. This report details the data structures targeted for refactoring in Calico, the methods and implementation details for reducing memory consumption and leveraging shared memory, and results for one test problem. Results show a memory reduction when loading a 1 GB STL file by a factor of 27.5, from 93.4 to 3.4 GB.

97 MATHEMATICS AND COMPUTING↗

Scalability of Real-time Distribution Models

This work will focus on developing the capabilities and validating the models for a sub transmission network with multiple feeders and microgrids. To achieve this scale of Hardware-in-the-loop (HitL) simulation, it is necessary to federate and collaborate. The work aims to design the large-scale feeder models to allow federation with complementary testbeds in the future. The feeder would be designed to be reconfigurable to put the system into a variety of modes. Aggregators models will be included in each distribution network’s federate to take control actions and interact with the management systems. Lastly, the feeder model will support large scale resilience studies involving complex Distributed Energy Resources (DER) controls, microgrid studies and emulation of complex data flows in future grid architectures.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Resource inventory techniques used in the California Desert Conservation Area

A variety of conventional and remotely sensed data for the 25 million acre California Desert Conservation Area (CDCA) have been integrated and analyzed to estimate range carrying capacity. Multispectral classification was performed on a digital mosaic of ten Landsat frames. Multispectral classes were correlated with low level aerial photography, quantified and aggregated by grazing allotment, land ownership, and slope.

Mcleod, R. G.↗

Experimental Study of Unsupported Nonane fuel Droplet Combustion in Microgravity

Soot formation in droplet flames is the basic component of the particulate emission process that occurs in spray combustion. The complexity of soot formation motivates a one-dimensional transport condition which has obvious advantages in modeling. Recent models of spherically symmetric droplet combustion have made this assumption when incorporating such aspects as detailed chemistry and radiation. Interestingly, spherical symmetry does not necessarily restrict the results because it has been observed that the properties of carbon formed in flames are not strongly affected by the nature of the fuel or flaming configuration. What is affected, however, are the forces acting on the soot aggregates and where they are trapped by a balance of drag and thermophoretic forces. The distribution of these forces depends on the transport conditions of the flame. Prior studies of spherical droplet flames have examined the droplet burning history of alkanes, alcohols and aromatics. Data are typically the evolution of droplet, flame, extinction, and soot shell diameters. These data are only now just beginning to find their way into comprehensive numerical models of droplet combustion to test proposed oxidation schemes for fuels such as methanol and heptane. In the present study, we report new measurements on the burning history of unsupported nonane droplets in a convection-free environment to promote spherical symmetry. The far-field gas is atmospheric pressure air at room temperature. The evolution of droplet diameter was measured using high speed cine photography of a spark-ignited, droplet within a confined volume in a drop tower. The initial droplet diameters varied between 0.5 mm and 0.6 mm. The challenge of unsupported droplets is to form, deploy and ignite them with minimal disturbance, and then to keep them in the camera field of view. Because of the difficulty of this undertaking, more sophisticated diagnostics for studying soot than photographic were not used. Supporting the test droplet by a fiber fixes the droplet position but the fiber can perturb the burning process especially for a sooting fuel. Prior studies on heptane showed little evidence for soot formation due to g-droplets of similar size the relationship between sooting and droplet diameter. For nonane droplets we expect increased sooting due to the greater number of carbon atoms. As a sooting droplet burns and its diameter decreases, proportionally less soot should form. This reduced soot, as well as the influence of soot formed earlier in the burning process which collects in a 'shell', on heat transport to the flame offers the potential for a time-varying burning rate. Such an effect was investigated and revealed in results reported here. Speculation is offered for the cause of this effect and its possible relation to soot formation.

Callahan, B. J.↗

Comparison of Cloud Detection Algorithms for Sentinel-2 Imagery

Accurate, automated cloud and cloud shadow detection is a key component of the processing needed to prepare optical satellite imagery for scientific analysis. Many existing cloud detection algorithms rely on temperature information to identify clouds, making detection difficult for imagers that lack a thermal band, like Sentinel-2. To get maximum benefit from Sentinel-2 products it is critical to understand which algorithms best identify clouds and their shadows in images. We examined the relative performance of five different cloud-masking algorithms (Sen2Cor, MAJA, LaSRC, Fmask and Tmask) in 6 Sentinel-2 scenes (28 total images) distributed across the Eastern Hemisphere. Expanding on these comparisons, we tested ensemble approaches to improve results. We tested three ensemble approaches to cloud and shadow classification based on the outputs of the five initial algorithms using the cloud masks in: (1) a majority prediction model; (2) a random forests model; and (3) a conditional logic model. Accuracy assessments show a trade-off between omission and commission errors in cloud detection for individual algorithms across all sites, and some algorithms are better at detecting either clouds or cloud shadows. No single algorithm outperforms the others for both clouds and shadows. Aggregating the results from multiple algorithms produces fewer undetected clouds and higher overall accuracy than any single algorithm, with as high as 2.7% improvement over the top-performing algorithm, suggesting an ensemble approach may be the most useful for processing of Sentinel-2 data.

Sentinel-2↗

Urban Traveler - Changes and Impacts: Mobility Energy Productivity (MEP) Metric

In response to the need to assess fundamental improvements in mobility as technology provides new modes, methods and opportunities in urban areas to connect people with goods, services and employment, a mobility-energy productivity (MEP) metric has been developed. The MEP metric fundamentally measures the potential of a multi-modal system to connect people to a variety of services and activities that define a high-quality of life. The MEP metric is measured relative to the cost, convenience and energy needed to provide connectivity. Fundamentally derived from accessibility theory, the MEP metric advances practice by using readily available travel time data (either from web-based APIs or outputs from an urban transportation model) combined with established parameters that reflect the cost, and energy intensity of various travel modes, along with relative frequency of activity engagement. The construction of the MEP metric allows for aggregation and disaggregation to the appropriate spatial, modal, and trip purpose resolution, as analysis needs dictate. The MEP metric could be used to compare alternative futures related to technology, infrastructure investment, or policy, providing a much needed tool for planners, researchers, and analysts.

30 DIRECT ENERGY CONVERSION↗

Validation of an Erythema-Weighted UV Model Using Broadband Solar Irradiance Measurements From Eleven U.S. Sites: Preprint

Erythema-weighted UV solar irradiance (UV-E) has a potential impact on human health if the recommended maximum exposure times are exceeded. In spite of this, it is not measured at most sites that measure Global Horizontal Irradiance (GHI). However, since UV-E is highly correlated with GHI and total ozone content it can be estimated from this information with sufficient accuracy to assisst in public health recommendations. The Power Model (PM) provides a simple method to estimate the erythema-weighted UV irradiance (UV-E) from measured GHI, total ozone column and air mass. In this work, the performance of the PM method is assessed using high-quality data from 11 sites in the continental U.S. (part of SURFRAD and SOLRAD networks) and total ozone estimates publicly available from the MERRA-2 re-analysis database. A three year period (2021- 2023) at 1-minute frequency is considered. The results (for time aggregations of 5 and 60 minutes) show a high Pearson's correlations (> 0.99), consistently positive mean bias deviations (below 13%) and dispersions in the 8-19% range at all sites. Relative values are expressed in terms of the corresponding measurement mean. The performance indicators remain consistent across time resolutions (5 or 60 minutes), suggesting that the model's performance is robust and not significantly affected by short-term variability (which is captured by GHI). Spatial patterns reveal higher biases and RMSD in northern and eastern locations. These values represent a significant improvement over widely used satellite-based global UV-E estimates and open the possibility of using the PM with satellite-based GHI estimates for operational UV-E mapping over the contiguous U.S. territory.

14 SOLAR ENERGY↗

Data, model inputs, and analysis scripts associated with a manuscript on stream intermittency controls across spatial scales in Pacific Northwest watersheds

NOTE: The manuscript associated with this data package is currently in review. The data may be revised based on reviewer feedback. Upon manuscript acceptance, this data package will be updated with the final dataset and additional metadata. This data package is associated with the manuscript "Hydroclimatic Memory and Watershed Template Shape Stream Intermittency: Multi-scale Attribution Using Process-based Simulation and Explainable ML" by Niroula et al. (2026), submitted to Water Resources Research (WRR). The study investigates the dominant controls on stream intermittency across local, reach, and watershed scales using a coupled process-based simulation and explainable machine-learning framework. Long-term daily simulations from the Advanced Terrestrial Simulator (ATS) were used to generate wetness states and ponded-depth responses over river-corridor cells. These ATS outputs were then aggregated across scales and used to train XGBoost (eXtreme Gradient Boosting) models. SHAP (SHapley Additive exPlanations) was applied to quantify the relative importance of hydroclimatic forcings, watershed template attributes, and antecedent-memory effects in shaping intermittency behavior. The analysis is carried out for three contrasting Pacific Northwest watersheds: Oak Creek (OCW), American River Watershed (ARW), and H.J. Andrews (HJA). Across these testbeds, the package contains ATS-ready watershed inputs, ATS run configuration and selected output files, model-evaluation data products, intermittency-analysis datasets, machine-learning target-feature tables, SHAP outputs, and notebooks used to organize, analyze, and visualize results. At a high level, the package documents a workflow in which ATS provides the physically based simulation backbone and explainable machine learning is used as a post-processing attribution tool. The contents are intended to support interpretation of the manuscript figures and results, provide context for how intermittency metrics were generated at multiple scales, and preserve the key artifacts needed to understand and reuse the analysis workflow. The package contains a high-level directory summary file (`summary.txt`) and four main content folders (1) `evaluation_plots` contains evaluation figures and supporting evaluation datasets; (2) `intermittency_plots` contains intermittency-focused analysis notebook and prepared datasets; (3) `ml-training-and-shap_values_plots` contains ML training inputs, SHAP outputs, and figure-generation notebooks; and (4) `watershed_mesh_and_ats_input` contains ATS model setup materials, forcing inputs, geometry, and selected run files. More specifically, the `evaluation_plots` folder contains the notebook used for ATS evaluation plotting and site-specific evaluation datasets. These include evapotranspiration and water-balance products for three watersheds, as well as an Oak Creek field-measurement discharge file. The `intermittency_plots` folder contains the notebook used for intermittency analysis and the prepared datasets used to analyze intermittent and non-intermittent wetness behavior across the study watersheds. The `ml-training-and-shap_values_plots` folder contains notebooks and outputs for the machine-learning and explainability workflow. This includes the main XGBoost and SHAP notebook(s), a beeswarm plotting notebook, target-feature tables for machine-learning training, SHAP summary tables, and per-sample SHAP value archives. The `watershed_mesh_and_ats_input` folder contains ATS-related watershed inputs and supporting materials. This includes mesh and shape products, ATS-readable LAI and meteorological forcing inputs, selected ATS spinup and transient-run files, and a watershed workflow example notebook. Subdirectories are organized by watershed where applicable.All files are .cpg (codepage files), .csv (comma-separated values), .dbf (database files), .exo (Exodus mesh format), .h5 (HDF5 format), .ipynb (Jupyter notebooks), .pkl (Python pickle), .prj (projection files), .sh (shell scripts), .shp (shapefile geometry), .shx (shapefile index), .txt (text files), or .xml (markup data).

Advanced Terrestrial Simulator↗

A comprehensive and synthetic dataset for global, regional, and national greenhouse gas emissions by sector 1970–2018 with an extension to 2019

To track progress towards keeping global warming well below 2 °C or even 1.5 °C, as agreed in the Paris Agreement, comprehensive up-to-date and reliable information on anthropogenic emissions and removals of greenhouse gas (GHG) emissions is required. Here we compile a new synthetic dataset on anthropogenic GHG emissions for 1970–2018 with a fast-track extension to 2019. Our dataset is global in coverage and includes CO 2 emissions, CH 4 emissions, N 2 O emissions, as well as those from fluorinated gases (F-gases: HFCs, PFCs, SF 6 , NF 3 ) and provides country and sector details. We build this dataset from the version 6 release of the Emissions Database for Global Atmospheric Research (EDGAR v6) and three bookkeeping models for CO 2 emissions from land use, land-use change, and forestry (LULUCF). We assess the uncertainties of global greenhouse gases at the 90 % confidence interval (5th–95th percentile range) by combining statistical analysis and comparisons of global emissions inventories and top-down atmospheric measurements with an expert judgement informed by the relevant scientific literature. We identify important data gaps for F-gas emissions. The agreement between our bottom-up inventory estimates and top-down atmospheric-based emissions estimates is relatively close for some F-gas species (~ 10 % or less), but estimates can differ by an order of magnitude or more for others. Our aggregated F-gas estimate is about 10 % lower than top-down estimates in recent years. However, emissions from excluded F-gas species such as chlorofluorocarbons (CFCs) or hydrochlorofluorocarbons (HCFCs) are cumulatively larger than the sum of the reported species. Using global warming potential values with a 100-year time horizon from the Sixth Assessment Report by the Intergovernmental Panel on Climate Change (IPCC), global GHG emissions in 2018 amounted to 58 ± 6.1 GtCO 2 eq. consisting of CO 2 from fossil fuel combustion and industry (FFI) 38 ± 3.0 GtCO 2 , CO 2 -LULUCF 5.7 ± 4.0 GtCO 2 , CH 4 10 ± 3.1 GtCO 2 eq., N2O 2.6 ± 1.6 GtCO 2 eq., and F-gases 1.3 ± 0.40 GtCO 2 eq. Initial estimates suggest further growth of 1.3 GtCO 2 eq. in GHG emissions to reach 59 ± 6.6 GtCO 2 eq. by 2019. Our analysis of global trends in anthropogenic GHG emissions over the past 5 decades (1970–2018) highlights a pattern of varied but sustained emissions growth. There is high confidence that global anthropogenic GHG emissions have increased every decade, and emissions growth has been persistent across the different (groups of) gases. There is also high confidence that global anthropogenic GHG emissions levels were higher in 2009–2018 than in any previous decade and that GHG emissions levels grew throughout the most recent decade. While the average annual GHG emissions growth rate slowed between 2009 and 2018 (1.2 % yr –1 ) compared to 2000–2009 (2.4 % yr –1 ), the absolute increase in average annual GHG emissions by decade was never larger than between 2000–2009 and 2009–2018. Our analysis further reveals that there are no global sectors that show sustained reductions in GHG emissions. There are a number of countries that have reduced GHG emissions over the past decade, but these reductions are comparatively modest and outgrown by much larger emissions growth in some developing countries such as China, India, and Indonesia. There is a need to further develop independent, robust, and timely emissions estimates across all gases. As such, tracking progress in climate policy requires substantial investments in independent GHG emissions accounting and monitoring as well as in national and international statistical infrastructures. The data associated with this article (Minx et al., 2021) can be found at https://doi.org/10.5281/zenodo.5566761.

54 ENVIRONMENTAL SCIENCES↗

Morphological changes in the cellulose and lignin components of biomass occur at different stages of steam pretreatment

Morphological changes to the different components of lignocellulosic biomass were observed as they occurred during steam pretreatment by placing a pressure reaction cell in a neutron beam and collecting time-resolved neutron scattering data. Changes to cellulose morphology occurred mainly in the heating phase, whereas changes in lignin morphology occurred mainly in the holding and cooling phases. During the heating stage, water is irreversibly expelled from cellulose microfibrils as the elementary fibrils coalesce. During the holding phase lignin aggregates begin to appear and they increase in size most noticeably during the cooling phase. This experiment demonstrates the unique information that in situ small angle neutron scattering studies of pretreatment can provide. This approach is potentially useful in optimizing the heating, holding and cooling stages of pretreatments to allow the exact size and nature of lignin aggregates to be controlled in order to enhance enzyme accessibility to cellulose and therefore the efficiency of biomass conversion.

09 BIOMASS FUELS↗