Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “File Systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

Improving the Effectiveness of Burst Buffers for Big Data Processing in HPC Systems with Eley

Here, burst buffer is an effective solution for reducing the data transfer time and the I/O interference in HPC systems. Extending Burst Buffers (BBs) to handle Big Data applications is challenging because BBs must account for the large data inputs of Big Data applications and the Quality-of-Service (QoS) of HPC applications which are considered as first-class citizens in HPC systems. Existing BBs focus on only intermediate data of Big Data applications and incur a high performance degradation of both Big Data and HPC applications. We present Eley , a burst buffer solution that helps to accelerate the performance of Big Data applications while guaranteeing the QoS of HPC applications. To achieve this goal, Eley embraces interference-aware prefetching technique that makes reading data input faster while introducing low interference for HPC applications. Evaluations using a wide range of Big Data and HPC applications demonstrate that Eley improves the performance of Big Data applications by up to 30% compared to existing BBs while maintaining the QoS of HPC applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

AMIGO

The code provides methods for perturbing binaries without breaking functionality. It is useful for evaluating the security/robustness of existing detection systems. The files generated by AMIGO can also be used to retrain a vulnerable detector.

Koch, Lucas↗
DOE OSTI2024

Fox Trails

1. This software utilizes python pandas to pull data from P6 databases or XER files. The software transforms the datasets into multiple main tables by joining, filtering, iteratively flattening hierarchical structured data, and pivoting datasets to give simple flat output tables. The activity table includes all of the information related to an activity including activity codes, global, EPS, and project codes, UDFs, and WBS information as separate columns. This includes the code id, code value and sequence number for all levels in hierarchical codes. The resource table is similar to the activity table and includes all of the information related to resources on activities including UPFs and resource codes. The resource time phased table takes the resource information and time phases it for the budget, forecast, late, and actual dates/units/costs that closely matches P6's user interface's values as it implements the resource curve and calendars. The wbs table contains the WBS structure broken out by levels and includes UDFs, codes, and notebook topics. The final P6 data table is the relationships table which simply contains the relationships. 2. When a user updates the tool with data (via giving it P6 project names with database username/password information or XER files) the system creates the data in #1, then creates a networkx graph with the activity data imbedded in the node data and the relationships added as edges. Each edge also has it's float calculated (working time distance between the predecessor and successor) and attached to the edge. Activities are also tagged as a potential start of a path based on their constraints, constraint dates, remaining start date, and activity status. When a user enters an activity ID into the UI, it runs a shortest path calculation on the network graph between each node tagged as potential start to the entered activity id based on the float tagged on the edge. Each path returned by the algorithm contains all of the nodes on the path in order, as well as the total float of the edges that make the path. This data is then collected and returned to the user in the form of a gantt chart with groupings for each path that includes the total float for each group. 3. Similar to 2, if the user passes through a reference dataset each activity set in the path is checked to see if it had a path in the reference dataset, if that path was the primary path between the start and end activities, and what has changed regarding logic and durations. These changes are color coded and summarized before sent to the user to be displayed by the UI for simple discovery. 4. Utilizing the data from #1, the user can submit desired grouping code(s) and filters to the system. The system will then pull the activities, resources, and relationships and create a gantt chart based on the groupings sent and filtered based on the filters sent. 5. The system will produce a gantt chart in a similar method to #4, but allows interactivity with the data. As the user interacts with the gantt chart, the software captures the changes and stores it with the user making the change so that project controls and implement those changes in P6.

Fox, Ben↗
DOE OSTI2026

DOE EV Data Collection - Vehicle Data

Vehicle data consist of electric vehicle performance data collected directly from the vehicle during standard operations. Data were collected using onboard data loggers that were either installed by the project team or preinstalled by the original equipment manufacturer. Data recorded by the data loggers were made accessible via an online web portal or an application programming interface. Different data loggers were used (HEM, ViriCiti, and Geotab), and the method for each vehicle is defined in the vehicle attributes file. Some systems collected data on a “trip-level” basis, in which each row of a table represents a single trip (the period between a key-on and key-off event), whereas other data were collected on a per-day basis, in which each row represents a single day of operation. Data were collected over a range of data collection periods, depending on the project. Data have been anonymized by removing information or decreasing information resolution as necessary so that fleets are not identifiable. Due to the wide range of vehicle types represented and variation in data collection, data parameters and frequencies differ between vehicles and fleets The **Performance Data Daily/Trip Data Dictionaries** contain definitions for each available parameter associated with a vehicle’s operations, aggregated at either a daily or trip level. The parameters available will vary from vehicle to vehicle, but every possible parameter will be defined. The **Vehicle Attributes Data Dictionary** contains definitions for each available parameter associated with a vehicle’s physical and functional attributes and fleet context. The **Vehicle Attributes** table contains specific vehicle characteristics, coded to an anonymous Vehicle ID. This Vehicle ID can be used as a key between vehicle data and vehicle attribute tables. The **Vehicle Data** tables contain the data from each vehicle’s operations, aggregated at either a daily or trip level, coded to an anonymous Vehicle ID. This Vehicle ID can be used as a key between vehicle data and vehicle attribute tables. Data is being uploaded quarterly through 2023 and subject to change until the conclusion of the project.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Data, figures, animations, and scripts associated with the manuscript "Integrated Effects of Site Hydrology and Vegetation on Exchange Fluxes and Carbon Cycling at the Coastal Terrestrial Aquatic Interface"

This package contains the data, figures, animations, and scripts used in "Integrated Effects of Site Hydrology and Vegetation on Exchange Fluxes and Carbon Cycling at the Coastal Terrestrial Aquatic Interface" (Li et al., 2023). This study examines the interactions between soil, vegetation and hydrologic conditions in coastal areas, with a focus on the Chesapeake Bay region. The researchers used a mechanistic model called ATS-PFLOTRAN to explore how these interactions affect exchange process in different regions and the carbon/nitrogen cycle across the terrestrial-aquatic interface (TAI). Simulation scenarios isolate the effects of control factors and reaction constants derived from laboratory experiments. The results show a carbon cycle "hot zone" in coastal wetlands and in the transition zone between wetlands and uplands. Transpiration enhances fluxes between the surface and subsurface domains and increases dissolved oxygen in TAI. The decomposition of leaf-derived organic carbon provides an additional source of carbon for aerobic respiration and denitrification in the TAI. Microbial activity plays a key role in controlling redox conditions and their variability. This modeling study improves the understanding of complex TAI interactions and facilitates the representation of coastal ecosystems in larger-scale Earth system models.Several files can be found from this data package.1. Readme,md: This file describe the Title, Target Journal, Target submission date, Co-author, Science Questions, Hypotheses, Key words, Key message, Model and Data, Repo Structure. The user can read this file first and then go to details.2. mesh.zip: This file contains the mesh file for simulation cases. 3. Figure.zip: This file contains figures used in the manuscript . 4. animation.zip: This file contains animations used in the manuscript . 5. Data.zip: This file contains all required input data, such as concentration and flow boundaries. Also, it contains the DEM and processes results on area fraction. 6. Simulation_setup_and_results.zip: This file contains setup of cases used in the manuscript . The result files are too large, if you need that data, please contact to the Author. 7. notebooks.zip: This file contains the Jupyter notebooks for performing sensitivity analysis and other pre-process and post-process analyses.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Data from: "Reply to ‘The challenge of defining effectively-no-snow’"

This repository contains the data and code associated with the paper titled "Reply to ‘The challenge of defining effectively-no-snow’" published in Nature Reviews Earth and Environment, 2026. In this reply, we argue that the 10th percentile of peak SWE (Snow Water Equivalent), which we propose in the original article, can be used as intended given it's a standardized, impact-based benchmark for comparing snow conditions across regions, not as a literal measure of snow absence. We present new evidence with SNOwpack TELemetry (SNOTEL) data showing that years meeting the threshold are overwhelmingly associated with subsequent drought (given United States Drought Monitor conditions), supporting its hydrologic and societal relevance. We conclude that while the distinction between "effectively no snow" and "zero snow" should be clearly communicated, the original definition remains appropriate for assessing impacts on snow-dependent water systems. The file code_nree_ML_reply_2026.Rmd contains the main processing scripts which analyze the SNOTEL data. Data from the US Drought Monitor was downloaded at: https://usdmdataservices using the Get Drought Severity Statistics By Area Percent' option, saved to the *_HUC4_delineated.csv files (Hydrologic Unit Code), which are labeled accordingly. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.

EARTH SCIENCE > TERRESTRIAL HYDROSPHERE > SNOW/ICE↗
DOE OSTI2019

Universal Utility Data Exchange Workflow Design

This workflow design document describes the process of establishing a Universal Utility Data Exchange (UUDEX) Connection between two or more UUDEX Endpoints. The existing processes required to establish a data link using Inter Control Center Communications Protocol (ICCP) are very time consuming, from both the perspectives of effort and calendar time. The intent of UUDEX is to provide a more streamlined alternative. The UUDEX Workflow is also used to establish UUDEX Connections to exchange data other than that found in traditional ICCP data exchanges such as exchanges of power system model files, security events and mitigations, disturbance reports, and market data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Universal Utility Data Exchange (UUDEX) - Workflow Design - Rev 1

This workflow design document describes the process of establishing a Universal Utility Data Exchange (UUDEX) Connection between two or more UUDEX Endpoints. The existing processes required to establish a data link using Inter Control Center Communications Protocol (ICCP) are very time consuming, from both the perspectives of effort and calendar time. The intent of UUDEX is to provide a more streamlined alternative. The UUDEX Workflow is also used to establish UUDEX Connections to exchange data other than that found in traditional ICCP data exchanges such as exchanges of power system model files, security events and mitigations, disturbance reports, and market data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Moving small files in a networked environment

Globally distributed computing infrastructures, such as clouds and supercomputers, are currently used to manage data that is generated with an unprecedented speed from a variety of resources. Coping with this trend, the volume of data exchanged across distant sites increases substantially. To accelerate data transfer, high-speed networks are provided to connect remote sites. Most existing data movement solutions are optimized for moving large files. However, it is still challenging to transfer a large number of small files across networks. This disadvantage not only lowers data transfer performance, but also decreases overall system utilization. Here, we identify that moving small files is mainly constrained by degraded file system throughput, not just network performance as might be suspected. We have built a data transfer pipeline model to analyze the impact of small network I/O and storage I/O on data movement. Extending one of the widely used open source data movement solutions, GridFTP, we demonstrate several appropriate engineering approaches that mitigate the bottleneck and increase data transfer efficiency. We show optimizations that improve data transfer performance more than 5 times. In comparison to existing solutions, our approaches can save a significant amount of system resources for moving lots of small files.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Toward the Detection of Polyglot Files

Standardized file types play a key role in the development and use of computer software. However, it is possible to confound standardized file type processing by creating a file that is valid in multiple file types. The resulting polyglot (many languages) file can confuse file type identification, allowing elements of the file to evade analysis. This is especially problematic for malware detection systems that rely on file type identification for feature extraction. Although work has been done to identify file types using more comprehensive methods than file signatures, accurate identification of polyglot files remains an open problem. Since malware detection systems routinely perform file type-specific feature extraction, polyglot files need to be filtered out prior to ingestion by these systems. Otherwise, malicious content could pass through undetected. To address the problem of polyglot detection we assembled a data set using the mitra tool. We then evaluated the performance of the most commonly used file identification tools, including file, polydet, binwalk, and TrID. Our analysis demonstrates that existing file type detection tools fail to provide reliable polyglot detection. We then evaluated the ability of a range of machine and deep learning models to detect polyglot files. The most performant models were MalConv2 and Catboost, which demonstrated the highest recall on our data set with 95.16% and 95.45%, respectively. These models outperformed existing methods and could be incorporated into a malware detector’s file processing pipeline to filter out potentially malicious polyglots before file type-dependent feature extraction takes place.

Koch, Luke↗
DOE OSTI2022

Toward the Detection of Polyglot Files

Standardized file types play a key role in the development and use of computer software. However, it is possible to confound standardized file type processing by creating a file that is valid in multiple file types. The resulting polyglot (many languages) file can confuse file type identification, allowing elements of the file to evade analysis. This is especially problematic for malware detection systems that rely on file type identification for feature extraction. Although work has been done to identify file types using more comprehensive methods than file signatures, accurate identification of polyglot files remains an open problem. Since malware detection systems routinely perform file type-specific feature extraction, polyglot files need to be filtered out prior to ingestion by these systems. Otherwise, malicious content could pass through undetected. To address the problem of polyglot detection we assembled a data set using the mitra tool. We then evaluated the performance of the most commonly used file identification tools, including file, polydet, binwalk, and TrID. Our analysis demonstrates that existing file type detection tools fail to provide reliable polyglot detection. We then evaluated the ability of a range of machine and deep learning models to detect polyglot files. The most performant models were MalConv2 and Catboost, which demonstrated the highest recall on our data set with 95.16% and 95.45%, respectively. These models outperformed existing methods and could be incorporated into a malware detector’s file processing pipeline to filter out potentially malicious polyglots before file type-dependent feature extraction takes place.

Koch, Lucas↗
DOE OSTI2020

Uncovering Access, Reuse, and Sharing Characteristics of I/O-Intensive Files on Large-Scale Production HPC Systems

Large-scale high-performance computing (HPC) applications running on supercomputers produce large amounts of data routinely and store it in files on multi-PB shared parallel storage systems. Unfortunately, storage community has a limited understanding of the access and reuse patterns of these files. This paper investigates the access and reuse patterns of I/O-intensive files on a production-scale supercomputer.

Patel, Tirthak↗
DOE OSTI2015

Information system of rice planting calendar based on ten-day (Dasarian) rainfall prediction

This paper describes information system of rice planting calendar to help farmers in determining the time for rice planting. The information includes rainfall prediction in ten days (dasarian) scale overlaid to map of rice field to produce map of rice planting in village level. The rainfall prediction was produced by stochastic modeling using Fast Fourier Transform (FFT) and Non-Linier Least Squares methods to fit the curve of function to the rainfall data. In this research, the Fourier series has been modified become non-linear function to follow the recent characteristics of rainfall that is non stationary. The results have been also validated in 4 steps, including R-Square, RMSE, R-Skill, and comparison with field data. The development of information system (cyber extension) provides information such as rainfall prediction, prediction of the planting time, and interactive space for farmers to respond to the information submitted. Interfaces for interactive response will be critical to the improvement of prediction accuracy of information, both rainfall and planting time. The method used to get this information system includes mapping on rice planting prediction, converting the format file, developing database system, developing website, and posting website. Because of this map was overlaid with the Google map, the map files must be converted to the .kml file format.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2015

SU-E-T-651: Quantification of Dosimetric Accuracy of Respiratory Gated Stereotactic Body Radiation Therapy

Purpose: To quantify the dosimetric accuracy of respiratory gated stereotactic body radiation therapy delivery using dynamic thorax phantom. Methods: Three patients with mobile target (2 lung, 1liver) were chosen. Retrospective 4DCT image sets were acquired for using Varian RPM system. An in-house MATLAB program was designed for MIP, MinIP and AvgIP generation. ITV was contoured on MIP image set for lung patients and on MinIP for liver patient. Dynamic IMRT plans were generated on selected phase bin image set in Eclipse (v10.0) planning system. CIRS dynamic thorax phantom was used to perform the dosimetric quality assurance. Patient breathing pattern file from RPM system was converted to phantom compatible file by an in-house MATLAB program. This respiratory pattern fed to the CIRS dynamic thorax phantom. 4DCT image set was acquired for this phantom using patient breathing pattern. Verification plans were generated using patient gating window and delivered on the phantom. Measurements were carried out using with ion chamber and EBT2 film. Exposed films were analyzed and evaluated in FilmQA software. Results: The stability of gated output in comparison with un-gated output was within 0.5%. The Ion chamber measured and TPS calculated dose compared for all the patients. The difference observed was 0.45%, −0.52% and −0.54 for Patient 1, Patient2 and Patient 3 respectively.Gamma value evaluated from EBT film shows pass rates from 92.41% to 99.93% for 3% dose difference and 3mm distance to agreement criteria. Conclusion: Dosimetric accuracy of respiratory gated SBRT delivery for lung and liver was dosimetrically acceptable. The Ion chamber measured dose was within 0.203±0.5659% of the expected dose. Gamma pass rates were within 96.63±3.84% of the expected dose.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2018

Mission possible: Unify HPC and Big Data stacks towards application-defined blobs at the storage layer

HPC and Big Data stacks are completely separated today. The storage layer offers opportunities for convergence, as the challenges associated with HPC and Big Data storage are similar: trading versatility for performance. This motivates a global move towards dropping file-based, POSIX-IO compliance systems. However, on HPC platforms this is made difficult by the centralized storage architecture using file-based storage. In this paper we advocate that the growing trend of equipping HPC compute nodes with local storage redistributes the cards by enabling object storage to be deployed alongside the application on the compute nodes. Furthermore, such integration of application and storage not only allows fine-grained configuration of the storage system, but also improves application portability across platforms. In addition, the single-user nature of such application-specific storage obviates the need for resource-consuming storage features like permissions or file hierarchies offered by traditional file systems. In this article we propose and evaluate Blobs (Binary Large Objects) as an alternative to distributed file systems. Finally, we factually demonstrate that it offers drop-in compatibility with a variety of existing applications while improving storage throughput by up to 28%.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Learning concave-convex profiles of data transport over dedicated connections

Dedicated data transport infrastructures are increasingly being deployed to support distributed big-data and high-performance computing scenarios. These infrastructures employ data transfer nodes that use sophisticated software stacks to support network transport among sites, which often house distributed file and storage systems. Throughput measurements collected over such infrastructures for a range of round trip times (RTTs) reflect the underlying complex end-to-end connections, and have revealed dichotomous throughput profiles as functions of RTT. In particular, concave regions of throughput profiles at lower RTTs indicate near-optimal performance, and convex regions at higher RTTs indicate bottlenecks due to factors such as buffer or credit limits. We present a machine learning method that explicitly infers these concave and convex regions and transitions between them using sigmoid functions. We also provide distribution-free confidence estimates for the generalization error of these concave-convex profile estimates. Throughput profiles for data transfers over 10 Gbps connections with 0–366 ms RTT provide important performance insights, including the near optimality of transfers performed with the XDD tool between XFS filesystems, and the performance limits of wide-area Lustre extensions using LNet routers. A direct application of generic machine learning packages does not adequately highlight these critical performance regions or provide as precise confidence estimates.

Rao, Nageswara↗