Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “UNIX”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

An Introduction to UNIX, Emacs, Latex, and Python

This document is designed to give a brief introduction to the UNIX Operating System, Emacs, Latex, and Python. Section 1 focuses on UNIX, and presents the basics of the file system, a variety of common commands, and a handful of slightly more in depth topics. Section 2 introduces Emacs and its somewhat unusual keyboard controls. In Section 3, the basics of creating a document in Latex are demonstrated. Finally, Section 4 provides the foundation needed to begin programming in Python.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

UNIX by Examples

Explore the source record for details and available documents.

Nee, F.↗
DOE OSTI2022

Profiling the BLAST bioinformatics application for load balancing on high-performance computing clusters

Abstract Background The Basic Local Alignment Search Tool (BLAST) is a suite of commonly used algorithms for identifying matches between biological sequences. The user supplies a database file and query file of sequences for BLAST to find identical sequences between the two. The typical millions of database and query sequences make BLAST computationally challenging but also well suited for parallelization on high-performance computing clusters. The efficacy of parallelization depends on the data partitioning, where the optimal data partitioning relies on an accurate performance model. In previous studies, a BLAST job was sped up by 27 times by partitioning the database and query among thousands of processor nodes. However, the optimality of the partitioning method was not studied. Unlike BLAST performance models proposed in the literature that usually have problem size and hardware configuration as the only variables, the execution time of a BLAST job is a function of database size, query size, and hardware capability. In this work, the nucleotide BLAST application BLASTN was profiled using three methods: shell-level profiling with the Unix “time” command, code-level profiling with the built-in “profiler” module, and system-level profiling with the Unix “gprof” program. The runtimes were measured for six node types, using six different database files and 15 query files, on a heterogeneous HPC cluster with 500+ nodes. The empirical measurement data were fitted with quadratic functions to develop performance models that were used to guide the data parallelization for BLASTN jobs. Results Profiling results showed that BLASTN contains more than 34,500 different functions, but a single function, RunMTBySplitDB, takes 99.12% of the total runtime. Among its 53 child functions, five core functions were identified to make up 92.12% of the overall BLASTN runtime. Based on the performance models, static load balancing algorithms can be applied to the BLASTN input data to minimize the runtime of the longest job on an HPC cluster. Four test cases being run on homogeneous and heterogeneous clusters were tested. Experiment results showed that the runtime can be reduced by 81% on a homogeneous cluster and by 20% on a heterogeneous cluster by re-distributing the workload. Discussion Optimal data partitioning can improve BLASTN’s overall runtime 5.4-fold in comparison with dividing the database and query into the same number of fragments. The proposed methodology can be used in the other applications in the BLAST+ suite or any other application as long as source code is available.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

DayCent data and results for "Robust paths to net greenhouse gas mitigation and negative emissions via advanced biofuels"

DayCent data and results for:J. L. Field, T. L. Richard, E. A. Smithwick, H. Cai, M. S. Laser, D. S. LeBauer, S. P. Long, K. Paustian, Z. Qin, J. J. Sheehan, P. Smith, M. Q. Wang, L. R. Lynd, Robust paths to net greenhouse gas mitigation and negative emissions via advanced biofuels. Proceedings of the National Academy of Sciences (2020). https://doi.org/10.1073/pnas.1920877117 This zip file contains a UNIX-format DayCent model executable, input files, automation code, and associated directory structure necessary to re-produce the DayCent analysis underlying the manuscript. The main script 'autodaycent.py' (written for Python 2.7) opens an interactive command line routine that facilitates:* Calibrating the DayCent pine growth model.* Initializing DayCent for a set of case studies sites.* Executing an ensemble of model runs representing case study site reforestation, grassland restoration, or conversion to switchgrass cultivation. * Results analysis & generation of manuscript Fig. 3. Note that the interactive analysis code requires that all input files to be contained in the directory structure as uploaded, without modification. Executable versions of the DayCent model (https://www.nrel.colostate.edu/projects/daycent/) compatible with other operating systems are available upon request.Please send questions/comments to John.L.Field@gmail.com

Agro-ecosystem function and prediction↗
DOE OSTI2024

BAD2matrix: Phylogenomic matrix concatenation, indel coding, and more

Common steps in phylogenomic matrix production include biological sequence concatenation, morphological data concatenation, insertion/deletion (indel) coding, gene content (presence/absence) coding, removing uninformative characters for parsimony analysis, recording with reduced amino acid alphabets, and occupancy filtering. Existing software does not accomplish these tasks on a phylogenomic scale using a single program. BAD2matrix is a Python script that performs the above-mentioned steps in phylogenomic matrix construction for DNA or amino acid sequences as well as morphological data. The script works in UNIX-like environments (e.g., LINUX, MacOS, Windows Subsystem for LINUX).

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

High-throughput single-cell transcriptomics of bacteria using combinatorial barcoding

Microbial split-pool ligation transcriptomics (microSPLiT) is a high-throughput single-cell RNA sequencing method for bacteria. With four combinatorial barcoding rounds, microSPLiT can profile transcriptional states in hundreds of thousands of Gram-negative and Gram-positive bacteria in a single experiment without specialized equipment. As bacterial samples are fixed and permeabilized before barcoding, they can be collected and stored ahead of time. During the first barcoding round, the fixed and permeabilized bacteria are distributed into a 96-well plate, where their transcripts are reverse transcribed into cDNA and labeled with the first well-specific barcode inside the cells. The cells are mixed and redistributed two more times into new 96-well plates, where the second and third barcodes are appended to the cDNA via in-cell ligation reactions. Finally, the cells are mixed and divided into aliquot sub-libraries, which can be stored until future use or prepared for sequencing with the addition of a fourth barcode. It takes 4 days to generate sequencing-ready libraries, including 1 day for collection and overnight fixation of samples. Here, the standard plate setup enables single-cell transcriptional profiling of up to 1 million bacterial cells and up to 96 samples in a single barcoding experiment, with the possibility of expansion by adding barcoding rounds. The protocol requires experience in basic molecular biology techniques, handling of bacterial samples and preparation of DNA libraries for next-generation sequencing. It can be performed by experienced undergraduate or graduate students. Data analysis requires access to computing resources, familiarity with Unix command line and basic experience with Python or R.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2019

Spack-Based Packaging and Development for HEP

The art event-based analysis framework is used by multiple Intensity Frontier and other experiments and projects in High Energy Physics (HEP). A system of tools and scripts based around a Fermilab-originated package management system called UNIX Product Support (UPS) has evolved to provide development, packaging, release management, and distribution of the art suite, related experimental software and external dependencies in relocatable binary form. The existing system has limitations in environments such as macOS with System Integrity Protection (SIP) enabled and the various High Performance Computing (HPC) systems that the field of HEP is increasingly looking to leverage. This has led to a search for other ways of providing experiments with a packaging and build system that meets their needs. We describe our efforts to develop a packaging and release management system based on the Spack HPC package management tool to replace the existing UPS-based system. We additionally describe a companion system: SpackDev and cetmodules, allowing simultaneous development of multiple HEP software packages in a consistent environment. A successful implementation of the intended system would have applicability across HEP rather than merely to the users of the art framework.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2015

Software management for the NO ν A Experiment

The NO$\nu$A software (NO$\nu$ASoft) is written in C++, and built on the Fermilab Computing Division's art framework that uses ROOT analysis software. NO$\nu$ASoft makes use of more than 50 external software packages, is developed by more than 50 developers and is used by more than 100 physicists from over 30 universities and laboratories in 3 continents. The software builds are handled by Fermilab's custom version of Software Release Tools (SRT), a UNIX based software management system for large, collaborative projects that is used by several experiments at Fermilab. The system provides software version control with SVN configured in a client-server mode and is based on the code originally developed by the BaBar collaboration. In this paper, we present efforts towards distributing the NO$\nu$A software via the CernVM File System distributed file system. Here, we will also describe our recent work to use a CMake build system and Jenkins, the open source continuous integration system, for NOASoft.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Pseudonymization at Scale: OLCF’s Summit Usage Data Case Study

The analysis of vast amounts of data and the processing of complex computational jobs have traditionally relied upon high performance computing (HPC) systems, which offer reliable and efficient management of large-scale computational and data resources. Understanding these analyses’ needs is paramount for designing solutions that can lead to better science, and similarly, understanding the characteristics of the user behavior on those systems is important for improving user experiences on HPC systems. A common approach to gathering data about user behavior is to extract workload characteristics from system log data available only to system administrators. Recently at Oak Ridge Leadership Computing Facility (OLCF), however, we unveiled user behavior about the Summit supercomputer by collecting data from a user’s point of view with ordinary Unix commands.In this paper, we discuss the process, challenges, and lessons learned while preparing this dataset for publication and submission to an open data challenge. The original dataset contains personal identifiable information (PII) about the users of OLCF which needed be masked prior to publication, and we determined that anonymization, which scrubs PII completely, destroyed too much of the structure of the data to be interesting for the data challenge. We instead chose to pseudonymize the dataset, which reduced the linkability of the dataset to the users’ identities. Pseudonymization is significantly more computationally expensive than anonymization, and the size of our dataset, which is approximately 175 million lines of raw text, necessitated the development of a parallelized workflow that could be reused on different HPC machines. We demonstrate the scaling behavior of the workflow on two leadership class HPC systems at OLCF, and we show that we were able to bring the overall makespan time from an impractical 20+ hours on a single node down to around 2 hours. As a result of this work, we release the entire pseudonymized dataset and make the workflows and source code publicly available.

Maheshwari, Ketan↗
DOE OSTI2014

SU-E-I-66: Radiomics and Image Registration Updates for the Computational Environment for Radiotherapy Research (CERR)

Purpose: To present new tools in CERR for Radiomics, image registration and other software updates and additions. Methods: Radiomics: CERR supports generating 3-D texture metrics based on gray scale co-occurance. Two new ways to calculate texture features were added: (1) Local Texture Averaging: Local texture is calculated around a voxel within the userdefined bounding box. The final texture metrics are the average of local textures for all the voxels. This is useful to detect any local texture patterns within an image. (2) Image Smoothing: A convolution ball of user-defined radius is rolled over an image to smooth out artifacts. The texture metrics are then computed on the smooth image. Image Registration: (1) Support was added to import deformation vector fields as well as non-deformable transformation matrices generated by vendor software and stored in standard DICOM format. (2) Support was added to use image within masks while computing image deformations. CT to MR registration is supported. This registration uses morphological edge information within the images to guide the deformation process. In addition to these features, other noteworthy additions to CERR include (1) Irregularly shaped ROI: This is done by taking intersection between infinitely extended irregular polygons drawn on any of the two views. Such an ROI is more conformal and useful in avoiding any unwanted parts of images that cannot be avoided with the conventional cubic box. The ROI is useful to generate Radiomics metrics. (2) Ability to insert RTDOSE in DICOM format to existing CERR plans. (3) Ability to import multi-frame PET-CT and SPECT-CT while maintaining spatial registration between the two modalities. (4) Ability to compile CERR on Unix-like systems. Results: The new features and updates are available via https://www.github.com/adityaapte/cerr . Conclusion: Features added to CERR increase its utility in Radiomics, Image-Registration and Outcomes modeling.

07 ISOTOPE AND RADIATION SOURCES↗
DOE OSTI2016

SU-F-T-94: Plan2pdf - a Software Tool for Automatic Plan Report for Philips Pinnacle TPS

Purpose: To implement an automatic electronic PDF plan reporting tool for Philips Pinnacle treatment planning system (TPS) Methods: An electronic treatment plan reporting software is developed by us to enable fully automatic PDF report from Pinnacle TPS to external EMR programs such as MOSAIQ. The tool is named “plan2pdf”. plan2pdf is implemented using Pinnacle scripts, Java and UNIX shell scripts, without any external program needed. plan2pdf supports full auto-mode and manual mode reporting. In full auto-mode, with a single mouse click, plan2pdf will generate a detailed Pinnacle plan report in PDF format, which includes customizable cover page, Pinnacle plan summary, orthogonal views through each plan POI and maximum dose point, DRR for each beam, serial transverse views captured throughout the dose grid at a user specified interval, DVH and scorecard windows. The final PDF report is also automatically bookmarked for each section above for convenient plan review. The final PDF report can either be saved on a user specified folder on Pinnacle, or it can be automatically exported to an EMR import folder via a user configured FTP service. In manual capture mode, plan2pdf allows users to capture any Pinnacle plan by full screen, individual window or rectangular ROI drawn on screen. Furthermore, to avoid possible patients’ plan mix-up during auto-mode reporting, a user conflict check feature is included in plan2pdf: it prompts user to wait if another patient is being exported by plan2pdf by another user. Results: plan2pdf is tested extensively and successfully at our institution consists of 5 centers, 15 dosimetrists and 10 physicists, running Pinnacle version 9.10 on Enterprise servers. Conclusion: plan2pdf provides a highly efficient, user friendly and clinical proven platform for all Philips Pinnacle users, to generate a detailed plan report in PDF format for external EMR systems.

60 APPLIED LIFE SCIENCES↗