Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Preprocessing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 451 records · Page 25
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Janus: A Python Package for Agent-Based Modeling of Land Use and Land Cover Change

Janus is an open source Python package for agent-based modeling (ABM) of land use and land cover change (LULCC). Many ABMs of LULCC have been created across platforms, some of which are not ideal for large scale, high resolution scenarios. This model provides a simple object-oriented framework for creating ABMs specific to LULCC. The organizational philosophy of the modeling framework is to create software objects (agents) that are associated with specific and contextual attributes which are isolated from where those agents exist in the spatial setting of the model, yet provide clear linkages between the agent, their environment, and other agents in the simulation. In this way, the framework allows for assembly of LULCC ABMs with low (programmatic) overhead, making the models extensible and providing clear mechanisms for integrating them with process-oriented biophysical models. Provided with Janus is a suite of geospatial data preprocessing tools that can use arbitrary land cover products as an input. Crop choice decisions are based on potential crop prices, these can be created synthetically, or drawn from integrated human-Earth systems models such as the GCAM. Janus is publicly accessible through GitHub and provides an example dataset for testing.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

NLR HPC Kestrel Jobs Data

Overview: Anonymized job-level records from the Kestrel HPC system at the National Laboratory of the Rockies (NLR). Each record represents a Slurm batch job with scheduling metadata, resource requests, utilization, energy estimates, and efficiency metrics. Sensitive fields (user, account, job name, submit line, working directory, submit script, and job type) are replaced with 7-character cryptographic hashes. System & Timeframe: Kestrel is located at the NLR campus. Standard compute nodes have 104 cores and 256 GB RAM; bigmem nodes have 2,000 GB. GPU nodes (gpu-h100 partition) use NVIDIA H100 GPUs. Data covers jobs submitted August 2023 through December 2025. Funding provided by the U.S. Department of Energy, EERE. Files: esif.hpc.kestrel.job-anon.zip — Anonymized job records (Hive-partitioned Parquet) datacard.md — Full dataset documentation ~11 million rows, 50 variables. Readable with PyArrow, pandas, DuckDB, Apache Spark, or any Parquet-compatible tool. Data Collection: Jobs collected via sacct with timezone-aware export (SLURM_TIME_FORMAT="%Y-%m-%dT%H:%M:%S%z"), loaded into PostgreSQL. Calculated columns updated via database triggers and batch functions. All timestamps use timestamptz and correctly handle DST transitions. Preprocessing: Anonymization of name, user, account, submit_line, work_dir, submit_script, and job_type via 7-char hex hashes Derived columns: queue_wait, cpu_eff, max/min/avg_mem_eff, energy estimates Simplified job state mapping (e.g., "CANCELLED by 132357" → "CANCELLED") Boolean flags: python_job, reframe_job Temporal decomposition: year, month, day, day_of_week, hour, minute from submit_time Shared node tracking: shared_job_count, nodes_shared, jobs_shared Key Variables: Scheduling: job_id, partition, state_simple, submit_time, start_time, end_time, queue_wait Resources: nodes_req/used, processors_req/used, memory_req, wallclock_req/used, gpus_requested Efficiency: cpu_eff, max/min/avg_mem_eff Energy: cpu_energy_tdp_estimated_max/used_watt_hours, consumed_energy_raw_joules, consumed_energy_raw_watt_hours Sharing: shared_job_count, nodes_shared, jobs_shared Partitions: short, standard, debug, gpu-h100 Job States: CANCELLED, COMPLETED, FAILED, PENDING, RUNNING QoS Levels: normal, high Important Notes: Timestamps include timezone offsets; DST transitions are handled correctly, though adding intervals across DST boundaries requires offset adjustment shared_job_count reflects physical node co-residency, not use of the shared partition Job step records and raw Slurm JSONB fields are excluded Do not attempt to re-identify individuals from hashed fields

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

NLR HPC Eagle Jobs Data and Additional Energy Metrics

Overview: Anonymized job-level records from the Eagle high-performance computing (HPC) system at the National Laboratory of the Rockies (NLR). Each record represents a Slurm batch job with scheduling metadata, resource requests, resource utilization, CPU/GPU energy consumption, and efficiency metrics. Sensitive fields (user, account, job name) are replaced with cryptographic hashes. System & Timeframe: Eagle was a 2,000-node, 8-petaflop system operated at NLR from 2019–2024. Data covers the full operational lifetime of the system. Slurm data was processed nightly; timestamps are in Mountain Time. Funding provided by the U.S. Department of Energy, EERE. Files: esif.hpc.eagle.job-anon.zip — Core anonymized job records (Hive-partitioned Parquet) esif.hpc.eagle.job-anon-energy-metrics.zip — Same records with additional iLO and Ganglia energy metrics datacard.md — Full dataset documentation ~13.8 million rows, 62 variables. Readable with PyArrow, pandas, DuckDB, Apache Spark, or any Parquet-compatible tool. Data Collection: Jobs collected via sacct through a pipeline: Eagle Jobs API → Redpanda → StreamSets → HPCMON API → PostgreSQL. Node-level power from iLO (HP Integrated Lights-Out); GPU power from Ganglia monitoring, joined to jobs via node lists and time ranges. Preprocessing: Anonymization of name, user, and account fields via cryptographic hashing Derived columns: queue_wait, cpu_eff, max_mem_eff Simplified job state mapping (e.g., "CANCELLED BY 12345" → "CANCELLED") QoS accounting rules (buy-in, standby, or Slurm QoS value) CPU energy estimated from TDP (200W, Intel Xeon Gold 6154, 18 cores) Timezone-aware columns (_tz) sourced from LEX accounting database to correctly handle DST transitions Key Variables: Scheduling: job_id, partition, state_simple, submit_time_tz, start_time_tz, end_time_tz, queue_waitResources: nodes_req/used, processors_req/used, memory_req, wallclock_req/used, gpus_requested Efficiency: cpu_eff, max_mem_eff Energy: cpu_energy_tdp_estimated_max/used_watt_hours, node_energy_total_watt_hours (iLO), gpu0/1_energy_total_watt_hours (Ganglia) Partitions: bigmem, bigmem-8600, bigscratch, csc, dav, ddn, debug, gpu, haswell, long, mono, short, standard Job States: CANCELLED, COMPLETED, FAILED, NODE_FAIL, OUT_OF_MEMORY, PENDING, RUNNING, TIMEOUT QoS Levels: Unknown, normal, buy-in, debug, penalty, high, standby Important Notes: Non-_tz timestamp columns may be off by one hour across DST boundaries; use _tz columns for time difference calculations Energy fields are null for jobs without monitoring coverage Job step records and raw Slurm JSONB fields are excluded from this extract Do not attempt to re-identify individuals from hashed fields

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

To The Biorefinery: Delivered Forestland and Agricultural Resources

It can be challenging and costly to transport biomass feedstock supplies from the roadside, or farmgate, to a biorefinery. Given the geographic dispersion and lowbulk density of cellulosic feedstocks, cost effective scaling of commercial biorefinery operations requires overcoming many challenges. The Biomass Research and Development Board’s Feedstock Logistics Interagency Working Group identified four primary barriers related to biorefinery commercialization: • Capacity and efficiency of harvest and collection equipment • High-moisture content leading to degradation of biomass • Variable biomass quality upon arrival at the biorefinery • Costly transportation options.1 Further, feedstock supply systems do not currently mitigate risks such as low crop yield, fire, or competition for resource use. Delivery and preprocessing improvements will allow for the development of a commercial-scale bioenergy industry that achieves national production and cost targets.

To The Biorefinery Factsheet↗
DOE OSTI2017

Image processing tool for automatic feature recognition and quantification

A system for defining structures within an image is described. The system includes reading of an input file, preprocessing the input file while preserving metadata such as scale information and then detecting features of the input file. In one version the detection first uses an edge detector followed by identification of features using a Hough transform. The output of the process is identified elements within the image.

36 MATERIALS SCIENCE↗
DOE OSTI2017

Method for discovering relationships in data by dynamic quantum clustering

Data clustering is provided according to a dynamical framework based on quantum mechanical time evolution of states corresponding to data points. To expedite computations, we can approximate the time-dependent Hamiltonian formalism by a truncated calculation within a set of Gaussian wave-functions (coherent states) centered around the original points. This allows for analytic evaluation of the time evolution of all such states, opening up the possibility of exploration of relationships among data-points through observation of varying dynamical-distances among points and convergence of points into clusters. This formalism may be further supplemented by preprocessing, such as dimensional reduction through singular value decomposition and/or feature filtering.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Sandia Engineering Analysis Code Access System v. 2.0.1

The Sandia Engineering Analysis Code Access System (SEACAS) is a suite of preprocessing, post processing, translation, visualization, and utility applications supporting finite element analysis software using the Exodus database file format.

Sjaardema, Gregory D.↗
DOE OSTI2016

Testing Feedstocks: INL Researcher Finds Way to Accurately Test Flowability

Here, cellulosic biofuels plants have been plagued by problems caused by the poor feeding and handling characteristics of biomass feedstocks. A new method has been developed to measure flowability of feedstocks to make sure that they will feed properly before introducing them into processes in which they may plug equipment. The new method uses an instrumented hopper that can be configured to provide real-time, in-line flowability characterization of feedstocks within supply and preprocessing operations for quality assurance and control. There is no other available method that measures flowability that is compatible with real-time, in-line data collection for quality control.

09 BIOMASS FUELS↗
DOE OSTI2018

Assessing Potential Air Pollutant Emissions from Agricultural Feedstock Production using MOVES

Biomass feedstock production is expected to grow as demand for biofuels and bioenergy increases. The change in air pollutant emissions that may result from large-scale biomass supply has implications for local air quality and human health. We developed spatially explicit emissions inventories for corn grain and six cellulosic feedstocks through the extension of the National Renewable Energy Laboratory's Feedstock Production Emissions to Air Model (FPEAM). These inventories include emissions of seven pollutants (nitrogen oxides, ammonia, volatile organic compounds, particulate matter, sulfur oxides, and carbon monoxide) generated from biomass establishment, maintenance, harvest, transportation, and biofuel preprocessing activities. By integrating the EPA's MOtor Vehicle Emissions Simulator (MOVES) into FPEAM, we created a scalable framework to execute county-level runs of the MOVES-Onroad model for representative counties (i.e., those counties with the largest amount of cellulosic feedstock production in each state) on a national scale. We used these results to estimate emissions from the on-road transportation of biomass and combined them with county-level runs of the MOVES-Nonroad model to estimate emissions from agricultural equipment. We also incorporated documented emission factors to estimate emissions from chemical application and the operation of drying equipment for feedstock processing, and used methods developed by the EPA and the California Air Resources Board to estimate fugitive dust emissions. The model developed here could be applied to custom equipment budgets and is extensible to accommodate additional feedstocks and pollutants. Future work will also extend this model to analyze spatial boundaries beyond the county-scale (e.g., regional or sub-county levels).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

URBAN-NET-TOOLKIT

Strengthening and maintaining secure, functioning, and resilient Critical Infrastructure Systems (CISs) is one of our nation's primary goals, and it requires proactive and coo0rdinated efforts among federal, state, local, public and private owners and operators of CISs. Recent worldwide extreme events have shown how interdependencies across CISs (electric grid, telecommunications, transportation, water networks, etc.) can drastically increase system vulnerability. To understand these complex interdependencies, ORNL has instantiated Urban-NET, a novel decision support system for CIS operators who need to identify vulnerabilities for minimizing the cascading impacts - saving utilities, the federal and state government, and private sector companies potentially billions by avoiding a complete CI failure and/or shutdown. The developed system includes a stack of software tools for preprocessing, modeling, interconnecting multiple CIS datasets as an integrated heterogeneous graph (i.e. network), identifying vulnerabilities through topological analysis, and predicting consequences based on simulations along with visualization.

Chinthavali, Supriya↗
DOE OSTI2016

Field-to-Fleet Webinar: How Does Feedstock Type Affect Biofuels Conversion?

The production of renewable motor fuels from lignocellulosic biomass is a key part of the U.S. strategy to ease dependence on imported oil and reduce greenhouse gas emissions. For this strategy to be environmentally and economically sound, biomass resource development and conversion technology development must be coordinated so that cost-competitive biofuels can be produced using sustainable, low-cost, and diverse feedstocks. However, the impacts of different biomass types on conversion performance (process efficiency, product quality, potential waste streams) are not well known. To this end, Idaho National Laboratory (INL), National Renewable Energy Laboratory (NREL), and Pacific Northwest National Laboratory (PNNL) are partnering to understand the ‘field-to-fleet’ implications of using different feedstocks in various thermal conversion processes, including fast pyrolysis, catalytic fast pyrolysis, hydrothermal liquefaction, hydrodeoxygenation, and gasification. We recently published results from an integrated study to produce petroleum refinery blendstocks via pyrolysis and catalytic hydroprocessing using several commercially-relevant feedstocks, including softwoods, hardwoods, herbaceous materials, and blends. The results presented here show that feedstock choice has a significant impact on multiple conversion metrics, including bio-oil yield and composition, hydrotreating yield, H2 consumption, selectivity to fuel products, and biomass carbon-to-fuel efficiency. Technoeconomic analysis was used to estimate conversion costs in a commercial process, based on a published design case, indicating that in-plant fuel production costs are also projected to vary significantly. The overall processing efficiency as a function of feedstock is examined on the basis of mass, carbon, and chemical energy of the inputs and products. It is hoped that a quantitative understanding of how different biomass types behave in various thermal processes, coupled with projected feedstock and preprocessing costs, will allow biomass resources to be well-matched with biofuels conversion processes.

09 BIOMASS FUELS↗
DOE OSTI2019

Systems and methods for determining optimal parameters for dynamic quantum clustering analyses

In the present work, quantum clustering is extended to provide a dynamical approach for data clustering using a time-dependent Schrodinger equation. To expedite computations, we can approximate the time-dependent Hamiltonian formalism by a truncated calculation within a set of Gaussian wave-functions (coherent states) centered around the original points. This allows for analytic evaluation of the time evolution of all such states, opening up the possibility of exploration of relationships among data points through observation of varying dynamical-distances among points and convergence of points into clusters. This formalism may be further supplemented by preprocessing, such as dimensional reduction through singular value decomposition and/or feature filtering. Additionally, the parameters of the analysis can be modified in order to improve the efficiency of the dynamic quantum clustering processes.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2017

Real-time, In-line Assessment of Flow Performance for Compressible and Anisotropic Bulk Solids

Feeding and handling problems have been recognized as a major contributor to low production of renewable cellulosic ethanol in the US, which achieved only 7% of nameplate capacity in 2016. Current methods that assess material feeding and handling properties require multiple hours to complete, making them ill-suited for quality assurance and control (QA/QC) in production environments. This talk presents a method, based upon a custom hopper with adaptable geometry, to assess feeding and handling behavior of materials in a production or receiving line in real-time, so that those properties can be used to either sort material or adjust preprocessing conditions to ensure that incoming material meet feeding and handling specifications. Flow properties are determined by continually adjusting the size of the outlet in a plane-flow hopper to maintain a constant fill level in the hopper. The angle of the hopper walls can also be adjusted to tune the geometry to different materials. Tests conducted at approximately 1,000 lbs per hour indicate that this method readily assesses the flow performance differences of corn stover and pine wood due to changes in particle size from 3 to 13 mm and due to changes in moisture content from 10 to 20%. Comparison of this method with shear tests using an automated Schulze ring shear indicate that this method is more sensitive for determining differences in flowability performance than traditional shear tests for corn stover and pine wood. Test results, including wall pressure measurements, are compared to simulations and indicate that this method shows promise for in-line quantitative estimation of the flow properties of compressible and anisotropic materials, including bulk density, cohesion, and internal angle of friction.

09 BIOMASS FUELS↗
DOE OSTI2018

The CMS Event-Builder System for LHC Run 3 (2021-23)

The data acquisition system (DAQ) of the CMS experiment at the CERN Large Hadron Collider (LHC) assembles events of 2MB at a rate of 100 kHz. The event builder collects event fragments from about 750 sources and assembles them into complete events which are then handed to the highlevel trigger (HLT) processes running on O(1000) computers. The aging eventbuilding hardware will be replaced during the long shutdown 2 of the LHC taking place in 2019/20. The future data networks will be based on 100 Gb/s interconnects using Ethernet and Infiniband technologies. More powerful computers may allow to combine the currently separate functionality of the readout and builder units into a single I/O processor handling simultaneously 100 Gb/s of input and output trac. It might be beneficial to preprocess data originating from specific detector parts or regions before handling it to generic HLT processors. Therefore, we will investigate how specialized coprocessors, e.g. GPUs, could be integrated into the event builder. We will present the envisioned changes to the event-builder compared to today’s system. Initial measurements of the performance of the data networks under the event-building trac pattern will be shown. Implications of a folded network architecture for the event building and corresponding changes to the software implementation will be discussed. Corresponding author: remigius.mommsen@cern.c

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2019

Using Big Data Technologies for HEP Analysis

The HEP community is approaching an era were the excellent performances of the particle accelerators in delivering collision at high rate will force the experiments to record a large amount of information. The growing size of the datasets could potentially become a limiting factor in the capability to produce scientific results timely and efficiently. Recently, new technologies and new approaches have been developed in industry to answer to the necessity to retrieve information as quickly as possible to analyze PB and EB datasets. Providing the scientists with these modern computing tools will lead to rethinking the principles of data analysis in HEP, making the overall scientific process faster and smoother. In this paper, we are presenting the latest developments and the most recent results on the usage of Apache Spark for HEP analysis. The study aims at evaluating the efficiency of the application of the new tools both quantitatively, by measuring the performances, and qualitatively, focusing on the user experience. The first goal is achieved by developing a data reduction facility: working together with CERN Openlab and Intel, CMS replicates a real physics search using Spark-based technologies, with the ambition of reducing 1 PB of public data in 5 hours, collected by the CMS experiment, to 1 TB of data in a format suitable for physics analysis. The second goal is achieved by implementing multiple physics use-cases in Apache Spark using as input preprocessed datasets derived from official CMS data and simulation. By performing different end-analyses up to the publication plots on different hardware, feasibility, usability and portability are compared to the ones of a traditional ROOT-based workflow.

Cremonesi, Matteo↗
DOE OSTI2020

On Performance Prediction of Big Data Transfer in High-performance Networks

Big data generated by large-scale scientific and industrial applications need to be transferred between different geographical locations for remote storage, processing, and analysis. High-speed dedicated connections provisioned in High-performance Networks (HPNs) are increasingly utilized to carry out such big data transfer. HPN management highly relies on an important capability of performance (mainly throughput) prediction to reserve sufficient bandwidth and meanwhile avoid over-provisioning that may result in unnecessary resource waste. This capability is critical to improving the resource (mainly bandwidth) utilization of dedicated connections and meeting various user requests for data transfer. Conventional methods conduct performance prediction by fitting prior observed transfer history with predefined loss functions, without considering unobservable latent factors such as competing loads on end hosts. Such latent factors also have a significant impact on the application-level data transfer performance, which may result in an inaccurate prediction model. In this paper, we first investigate the impact of latent factors and propose a clustering-based method to eliminate their negative impact on performance prediction. We then develop a robust machine learning-based performance predictor by: i) incorporating the proposed latent factor elimination method into data preprocessing, and ii) adopting a customized domain guided loss function. Extensive experimental results show that our predictor achieves significantly higher prediction accuracy than several other state-of-the-art methods.

Liu, Wuji↗
DOE OSTI2020

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗
DOE OSTI2020

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific ap- plications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Net- works (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate band- widths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer perfor- mance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application- level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher pre- diction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗