Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “preprocessed”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Janus: A Python Package for Agent-Based Modeling of Land Use and Land Cover Change

Janus is an open source Python package for agent-based modeling (ABM) of land use and land cover change (LULCC). Many ABMs of LULCC have been created across platforms, some of which are not ideal for large scale, high resolution scenarios. This model provides a simple object-oriented framework for creating ABMs specific to LULCC. The organizational philosophy of the modeling framework is to create software objects (agents) that are associated with specific and contextual attributes which are isolated from where those agents exist in the spatial setting of the model, yet provide clear linkages between the agent, their environment, and other agents in the simulation. In this way, the framework allows for assembly of LULCC ABMs with low (programmatic) overhead, making the models extensible and providing clear mechanisms for integrating them with process-oriented biophysical models. Provided with Janus is a suite of geospatial data preprocessing tools that can use arbitrary land cover products as an input. Crop choice decisions are based on potential crop prices, these can be created synthetically, or drawn from integrated human-Earth systems models such as the GCAM. Janus is publicly accessible through GitHub and provides an example dataset for testing.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

NLR HPC Kestrel Jobs Data

Overview: Anonymized job-level records from the Kestrel HPC system at the National Laboratory of the Rockies (NLR). Each record represents a Slurm batch job with scheduling metadata, resource requests, utilization, energy estimates, and efficiency metrics. Sensitive fields (user, account, job name, submit line, working directory, submit script, and job type) are replaced with 7-character cryptographic hashes. System & Timeframe: Kestrel is located at the NLR campus. Standard compute nodes have 104 cores and 256 GB RAM; bigmem nodes have 2,000 GB. GPU nodes (gpu-h100 partition) use NVIDIA H100 GPUs. Data covers jobs submitted August 2023 through December 2025. Funding provided by the U.S. Department of Energy, EERE. Files: esif.hpc.kestrel.job-anon.zip — Anonymized job records (Hive-partitioned Parquet) datacard.md — Full dataset documentation ~11 million rows, 50 variables. Readable with PyArrow, pandas, DuckDB, Apache Spark, or any Parquet-compatible tool. Data Collection: Jobs collected via sacct with timezone-aware export (SLURM_TIME_FORMAT="%Y-%m-%dT%H:%M:%S%z"), loaded into PostgreSQL. Calculated columns updated via database triggers and batch functions. All timestamps use timestamptz and correctly handle DST transitions. Preprocessing: Anonymization of name, user, account, submit_line, work_dir, submit_script, and job_type via 7-char hex hashes Derived columns: queue_wait, cpu_eff, max/min/avg_mem_eff, energy estimates Simplified job state mapping (e.g., "CANCELLED by 132357" → "CANCELLED") Boolean flags: python_job, reframe_job Temporal decomposition: year, month, day, day_of_week, hour, minute from submit_time Shared node tracking: shared_job_count, nodes_shared, jobs_shared Key Variables: Scheduling: job_id, partition, state_simple, submit_time, start_time, end_time, queue_wait Resources: nodes_req/used, processors_req/used, memory_req, wallclock_req/used, gpus_requested Efficiency: cpu_eff, max/min/avg_mem_eff Energy: cpu_energy_tdp_estimated_max/used_watt_hours, consumed_energy_raw_joules, consumed_energy_raw_watt_hours Sharing: shared_job_count, nodes_shared, jobs_shared Partitions: short, standard, debug, gpu-h100 Job States: CANCELLED, COMPLETED, FAILED, PENDING, RUNNING QoS Levels: normal, high Important Notes: Timestamps include timezone offsets; DST transitions are handled correctly, though adding intervals across DST boundaries requires offset adjustment shared_job_count reflects physical node co-residency, not use of the shared partition Job step records and raw Slurm JSONB fields are excluded Do not attempt to re-identify individuals from hashed fields

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

NLR HPC Eagle Jobs Data and Additional Energy Metrics

Overview: Anonymized job-level records from the Eagle high-performance computing (HPC) system at the National Laboratory of the Rockies (NLR). Each record represents a Slurm batch job with scheduling metadata, resource requests, resource utilization, CPU/GPU energy consumption, and efficiency metrics. Sensitive fields (user, account, job name) are replaced with cryptographic hashes. System & Timeframe: Eagle was a 2,000-node, 8-petaflop system operated at NLR from 2019–2024. Data covers the full operational lifetime of the system. Slurm data was processed nightly; timestamps are in Mountain Time. Funding provided by the U.S. Department of Energy, EERE. Files: esif.hpc.eagle.job-anon.zip — Core anonymized job records (Hive-partitioned Parquet) esif.hpc.eagle.job-anon-energy-metrics.zip — Same records with additional iLO and Ganglia energy metrics datacard.md — Full dataset documentation ~13.8 million rows, 62 variables. Readable with PyArrow, pandas, DuckDB, Apache Spark, or any Parquet-compatible tool. Data Collection: Jobs collected via sacct through a pipeline: Eagle Jobs API → Redpanda → StreamSets → HPCMON API → PostgreSQL. Node-level power from iLO (HP Integrated Lights-Out); GPU power from Ganglia monitoring, joined to jobs via node lists and time ranges. Preprocessing: Anonymization of name, user, and account fields via cryptographic hashing Derived columns: queue_wait, cpu_eff, max_mem_eff Simplified job state mapping (e.g., "CANCELLED BY 12345" → "CANCELLED") QoS accounting rules (buy-in, standby, or Slurm QoS value) CPU energy estimated from TDP (200W, Intel Xeon Gold 6154, 18 cores) Timezone-aware columns (_tz) sourced from LEX accounting database to correctly handle DST transitions Key Variables: Scheduling: job_id, partition, state_simple, submit_time_tz, start_time_tz, end_time_tz, queue_waitResources: nodes_req/used, processors_req/used, memory_req, wallclock_req/used, gpus_requested Efficiency: cpu_eff, max_mem_eff Energy: cpu_energy_tdp_estimated_max/used_watt_hours, node_energy_total_watt_hours (iLO), gpu0/1_energy_total_watt_hours (Ganglia) Partitions: bigmem, bigmem-8600, bigscratch, csc, dav, ddn, debug, gpu, haswell, long, mono, short, standard Job States: CANCELLED, COMPLETED, FAILED, NODE_FAIL, OUT_OF_MEMORY, PENDING, RUNNING, TIMEOUT QoS Levels: Unknown, normal, buy-in, debug, penalty, high, standby Important Notes: Non-_tz timestamp columns may be off by one hour across DST boundaries; use _tz columns for time difference calculations Energy fields are null for jobs without monitoring coverage Job step records and raw Slurm JSONB fields are excluded from this extract Do not attempt to re-identify individuals from hashed fields

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1971

Preparation and evaluation of Apollo 14 composite experiments

An account is given of the work aimed at flight experiments on Apollo 14, in relation to space manufacturing processes. Evaluation of suitable materials, definition of in-flight processing procedures, preparation of preprocessed materials and delivery, and evaluation of the space-processed samples after return from the Apollo 14 flight are presented.

Steurer, W. H.↗
NASA NTRS1972

Data processing 2: Advancements in large scale data processing systems for remote sensing

The development of large scale data processing systems for remote sensing is studied by evaluating: (1) the suitability of several sensor types with regard to producing data required for multispectral machine analysis; (2) various types of data preprocessing necessary to prepare such data for analysis; and (3) transfer of machine processing techniques for earth resources data to user community.

Landgrebe, D. A.↗
NASA NTRS1972

Pattern recognition of native plant communities: Manitou Colorado test site

Optimum channel selection among 12 channels of multispectral scanner imagery identified six as providing the best information about 11 vegetation classes and two nonvegetation classes at the Manitou Experimental Forest. Intensive preprocessing of the scanner signals was required to eliminate a serious scan angle effect. Final processing of the normalized data provided acceptable recognition results of generalized plant community types. Serious errors occurred with attempts to classify specific community types within upland grassland areas. The consideration of the convex mixtures concept (effects of amounts of live plant cover, exposed soil, and plant litter cover on apparent scene radiances) significantly improved the classification of some of the grassland classes.

Driscoll, R. S.↗
NASA NTRS1971

Implicit determination of multispectral scanner data variation over extended areas.

Description of a method by which variations in aerial multispectral scanner data can be accounted for when there is insufficient ground truth information. The data are suitably filtered and smoothed along the flight line, and the reduced results are presented in a form such that the investigator can readily determine the preprocessing that should be used and the spatial extent to which it should be applied. An algorithm is developed for the detection of cloud-shadowed areas.

Kriegler, F. J.↗
NASA NTRS1972

Quantitative simultaneous multi-element microprobe analysis using combined wavelength and energy dispersive systems

A combined WDS-EDS system obviates the severe X-ray peak overlap problems encountered with Na, Mg, Al and Si common to pure EDS systems. By application of easily measured empirical correction factors for pulse pile-up and peak overlaps which are normally observed in the analysis of silicate minerals, the accuracy of analysis is comparable with that expected for WDS electron microprobe analyses. The continuum backgrounds are subtracted for the spectra by a spline fitting technique based on integrated intensities between the peaks. The preprocessed data are then reduced to chemical analyses by existing data reduction programs.

Walter, L. S.↗
NASA NTRS1972

Study to determine cloud motion from meteorological satellite data

Processing techniques were tested for deducing cloud motion vectors from overlapped portions of pairs of pictures made from meteorological satellites. This was accomplished by programming and testing techniques for estimating pattern motion by means of cross correlation analysis with emphasis placed upon identifying and reducing errors resulting from various factors. Techniques were then selected and incorporated into a cloud motion determination program which included a routine which would select and prepare sample array pairs from the preprocessed test data. The program was then subjected to limited testing with data samples selected from the Nimbus 4 THIR data provided by the 11.5 micron channel.

Clark, B. B.↗
NASA NTRS1972

Smoothing of functions of range and range rate measurements from earth orbiting satellites

It is shown that for satellites in circular Earth orbits with altitudes of 500 kilometers to 1500 kilometers, and for satellites in elliptical orbits with an approximate 4000 kilometer height of perigee, a high degree least squares polynomial (e.g., a 9th or 10th degree in some cases) is required to smooth both range and range rate data for purposes of input to orbit determination programs. In order to circumvent this problem, functions of range and range rate are smoothed with lower degree least squares polynomials (e.g., 3rd and 4th degree) and it is shown that under the above geometric constraints the standard deviation of fit can be reduced to levels commensurate with typical S-band tracking system resolution which is 1 to 2 meters in range and 0.005 meters/second in range rate for a 1 per second data rate. Also shown are the effects of Gaussian random noise, biases, and periodic noise. This analysis includes numerous examples applied to the 44 point data smoothing interval currently used in much of the operational preprocessing at the Goddard Space Flight Center.

Grenchik, T. J.↗
NASA NTRS1970

The use of multispectral sensing techniques to detect ponderosa pines trees under stress from insects or diseases

The detection of stress induced by bark beetles in conifers is reviewed in two sections: (1) the analysis of very small scale aerial photographs taken by NASA's RB-57F aircraft on August 10, 1969, and (2) the analysis of multispectral imagery obtained by the optical-mechanical line scanner. Underexposure of all films taken from the RB-57 aircraft and inadequate flight coverage prevented drawing definitive conclusions regarding optimum scales and film combinations to detect the discolored infestations. Preprocessing of the scanner signals by both analog and digital computers improved the accuracy of target recognition. Selection and ranking of the best channels for signature recognition was the greatest contribution of digital processing. Improvements were made in separating hardwoods from conifers and old-kill pine trees from recent discolored trees and from healthy trees, but accuracy of detecting the green infested trees is still not acceptable on either the SPARC or thermal-contouring processor. From six years of experience in processing line scan data it is clear that the greatest gain in previsual detection of stress will occur when registered multispectral data from a single aperture or common instantaneous field of view scanner system can be collected and processed.

Heller, R. C.↗
NASA NTRS1973

Constrained optimization of image restoration filters

A linear shift-invariant preprocessing technique is described which requires no specific knowledge of the image parameters and which is sufficiently general to allow the effective radius of the composite imaging system to be minimized while constraining other system parameters to remain within specified limits.

Riemer, T. E.↗
NASA NTRS1973

The minitrack tracking function description, volume 1

The treatment of tracking data by the Minitrack system is described from the transmission of the nominal 136-MHz radio beacon energy from a satellite and the reception of this signal by the interferometer network through the ultimate derivation of the direction cosines (the angular coordinates of the vector from the tracking station to the spacecraft) as a function of time. Descriptions of some of the lesser-known functions operating on the system, such as the computer preprocessing program, are included. A large part of the report is devoted to the preprocessor, which provides for the data compression, smoothing, calibration correction, and ambiguity resolution of the raw interferometer phase tracking measurements teletyped from each of the worldwide Minitrack tracking stations to the central computer facility at Goddard Space Flight Center. An extensive bibliography of Minitrack hardware and theory is presented.

Englar, T. S., Jr.↗
NASA NTRS1973

Significant techniques in the processing and interpretation of ERTS-1 data

The discipline oriented investigations underway at the Johnson Space Center (JSC) using ERTS-1 data provide an appropriate framework for the systematic evaluation of the various elements comprising a prototype multispectral data processing and analysis system. In particular such a system may be thought of as the integration of: (1) a preprocessing subsystem; (2) a spectral clustering subsystem, (3) a correlation and classification subsystem; (4) mensuration subsystem; and (5) an information management subsystem. Specific elements of this system are already operational at JSC. It is in the context of this system that technique development and application is being pursued at JSC. Aircraft, ERTS and EREP data will be utilized to refine the subsystem elements for each of the data acquisition systems or system combinations that are optimally suited for a specific Earth Resources application. The techniques reported are those that have been developed to date during the utilization of ERTS-1 data in this processing and analysis system.

Cousin, S. B.↗
NASA NTRS1973

Combining human and computer interpretation capabilities to analyze ERTS imagery

The human photointerpreter and the computer have complementary capabilities that are exploited in a computer-based data analysis system developed at the Forestry Remote Sensing Laboratory, University of California. This system is designed to optimize the process of extracting resource information from ERTS images. The human has the ability to quickly delineate gross differences in land classes, such as wildland, urban, and agriculture on appropriate ERTS images, and to further break these gross classes into meaningful subclasses. The computer, however, can more efficiently analyze point-by-point spectral information and localized textural information which can result in a much more detailed agricultural or wildland classification based on species composition and/or plant association. These human and computer capabilities have been integrated through the use of an inexpensive small scale computer dedicated to the interactive preprocessing of the human inputs and the display of raw ERTS images and computer classified images. The small computer is linked to a large scale computer system wherein the bulk of the statistical work and the automatic point-by-point classification is done.

Nichols, J. D.↗