Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Distributed File Systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

fp-tools

The output of a molecular dynamics simulation is a data file containing properties of the motion of the simulated particles as a function of time (e.g., positions, velocities). From this output, scientists can extract various properties which give insight into the physics of the system; for example, the radial distribution function can be computed from the particle positions, which gives insight into how close the system is to melting or freezing. Many publicly available codes to perform molecular dynamics simulations exist and have been well adopted by the scientific community (e.g., LAMMPS, VASP). However, it remains standard practice for scientists to write their own post-processing scripts to extract various properties of interest from the molecular dynamics simulation data. Since in many cases scientists are interested in the same set of “textbook” properties, there is a large duplication of effort in writing these post-processing scripts. Our proposed code, FP-Tools, is a C++ toolkit designed to help scientists extract and analyze quantities of interest from the output of a molecular dynamics simulation, eliminating the need for them to write these codes themselves and thus reducing this duplication of effort. The quantities which are computed by our code are well documented in the literature (either in publications or textbooks), and the algorithms we implement to compute these properties are also well known in the field. We are not introducing new science or methods here; rather, our primary goal is to provide a useful tool to the community in the form of a well-documented, easy to use package.

Hartman, Leah↗

A methodology for customizing implantation profiles of light ions using a single thin foil energy degrader

We report a method was developed to quantify the spatial distribution and implantation depth of energy-degraded light ions with a thin foil rotating energy degrader for use during multiple ion beam irradiation. The methodology covers three physical phenomena: ions passing through the thin foil, ions travelling through the vacuum to the target, and ion implantation into the target, and accounts for the distribution of ions both in depth and in plane. The processes of energy straggling and scattering were calculated using SRIM. The effects of raster-scanning, and the geometry of the system were implemented in scripts handling the SRIM output files. Elastic backscattering (EBS) using 2.38 MeV H+ protons was used to measure the helium depth profiles after implantation with and without thin foil energy degradation. Defect analysis with transmission electron microscopy confirmed the implantation profiles measured with EBS and calculated with SRIM.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

RLC4CLR (Reinforcement Learning Controller for Critical Load Restoration Problems)

RLC4CLR demonstrates using a reinforcement learning controller (RLC) to solve a critical load restoration (CLR) problem, which improves the grid resilience after a substation outage event. RLC4CLR consists of two parts. (1) RL environment: This environment encapsulates the CLR problem to be solved and provides interfacing functions to follow the standard OpenAI Gym format. A power system simulator, i.e., OpenDSS, is included to provide the power flow solution. Controller inputs and outputs (RL state and action) as well as the reward are defined in this environment as well. In summary, the RL environment is the problem formulation from which the RL agent can learn. (2) RL training script: The training script enables the RL agent to learn its control policy by interacting with the RL environment. For RL training, an open-sourced RL library, i.e., RLlib, is leveraged which is based on a distributed computing framework (Ray). The training script is designed to be able to be run on both local machine or the NREL HPC system. Other components of RLC4CLR include input data, e.g., grid model (standard IEEE test feeders), and other files used for results analysis.

Zhang, Xiangyu↗

A data distribution strategy for the 1990s (files are not enough)

Virtually all of the data distribution strategies being contemplated for the EOSDIS era revolve around the use of files. Most, if not all, mass storage technologies are based around the file model. However, files may be the wrong primary abstraction for supporting scientific users in the 1990s and beyond. Other abstractions more closely matching the respective scientific discipline of the end user may be more appropriate. JPL has built a unique multimission data distribution system based on a strategy of telemetry stream emulation to match the responsibilities of spacecraft team and ground data system operators supporting our nations suite of planetary probes. The current system, operational since 1989 and the launch of the Magellan spacecraft, is supporting over 200 users at 15 remote sites. This stream-oriented data distribution model can provide important lessons learned to builders of future data systems.

Tankenson, Mike↗

DIstributed VIRtual System (DIVIRS) Project

The development of Prospero moved from the University of Washington to ISI and several new versions of the software were released from ISI during the contract period. Changes in the first release from ISI included bug fixes and extensions to support the needs of specific users. Among these changes was a new option to directory queries that allows attributes to be returned for all files in a directory together with the directory listing. This change greatly improves the performance of their server and reduces the number of packets sent across their trans-pacific connection to the rest of the internet. Several new access method were added to the Prospero file method. The Prospero Data Access Protocol was designed, to support secure retrieval of data from systems running Prospero.

Schorr, Herbert↗

funcX: Federated Function as a Service for Science

Here, funcX is a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. Unlike centralized FaaS systems, funcX decouples the cloud-hosted management functionality from the edge-hosted execution functionality. funcX's endpoint software can be deployed, by users or administrators, on arbitrary laptops, clouds, clusters, and supercomputers, in effect turning them into function serving systems. funcX's cloud-hosted service provides a single location for registering, sharing, and managing both functions and endpoints. It allows for transparent, secure, and reliable function execution across the federated ecosystem of endpoints-enabling users to route functions to endpoints based on specific needs. funcX uses containers (e.g., Docker, Singularity, and Shifter) to provide common execution environments across endpoints. funcX implements various container management strategies to execute functions with high performance and efficiency on diverse funcX endpoints. funcX also integrates with an in-memory data store and Globus for managing data that may span endpoints. We motivate the need for funcX, present our prototype design and implementation, and demonstrate, via experiments on two supercomputers, that funcX can scale to more than 130000 concurrent workers. We show that funcX's container warming-aware routing algorithm can reduce the completion time for 3,000 functions by up to 61% compared to a randomized algorithm and the in-memory data store can speed up data transfers by up to 3x compared to a shared file system.

97 MATHEMATICS AND COMPUTING↗

Data-link alternatives for the NASA pilot data systems

There are several preliminary, or pilot, efforts for developing data systems for supporting NASA science research. The Pilot Data Systems consist of the Pilot Land Data System (PLDS), the Pilot Ocean Data System (PODS), the Pilot Climate Data System (PCDS), and the Pilot Planetary Data System (PPDS). The PLDS is likely to have the greatest communication needs of the various pilots. Communications issues and requirements are examined in the context of the mature PLDS as it may exist by 1990. The PLDS is seen as a distributed processing system linking resources at a number of NASA research centers and outside universities. Large image data sets, including LANDSAT scenes, are a major data type to be moved along the PLDS communications network. The unusually large size of these data files requires examining new technologies that may allow efficient and affordable communication at rates of up to 60 megabits per second. Potentially useful developments include portable satellite ground stations, and time division multiple access for sharing high-speed satellite links. Further pooling of communications for the various data systems into a single network would reduce costs and may make economic justification of access to the required data rates possible.

Likens, W. C.↗

Data processing pipeline with transaction-oriented data sharing

This paper makes three contributions to the area of modern science data processing systems. First, the paper describes the science data processing pipeline, developed at the Multi-mission Image Processing Lab of JPL, for transforming raw space data into high quality image data and automating the distribution of data using a high-performance file transaction service. File Exchange Interface is the file transaction service developed MIPL. Second, it presents the FEI component architecture in the are of file transaction management, security,a nd file integrity verfication. Finally, the paper presents the federated model for the FEI service to demonstrate how to create a pool of file trasaction services to support load balancing and service fallover, and simplify service management.

science data processing↗

NASA Tech Briefs, July 2010

Topics covered include: Wirelessly Interrogated Wear or Temperature Sensors; Processing Nanostructured Sensors Using Microfabrication Techniques; Optical Pointing Sensor; Radio-Frequency Tank Eigenmode Sensor for Propellant Quantity Gauging; High-Temperature Optical Sensor; Integral Battery Power Limiting Circuit for Intrinsically Safe Applications; Configurable Multi-Purpose Processor; Squeezing Alters Frequency Tuning of WGM Optical Resonator; Automated Computer Access Request System; Range Safety for an Autonomous Flight Safety System; Fast and Easy Searching of Files in Unisys 2200 Computers; Parachute Drag Model; Evolutionary Scheduler for the Deep Space Network; Modular Habitats Comprising Rigid and Inflatable Modules; More About N2O-Based Propulsion and Breathable-Gas Systems; Ultrasonic/Sonic Rotary-Hammer Drills; Miniature Piezoelectric Shaker for Distribution of Unconsolidated Samples to Instrument Cells; Lunar Soil Particle Separator; Advanced Aerobots for Scientific Exploration; Miniature Bioreactor System for Long-Term Cell Culture; Electrochemical Detection of Multiple Bioprocess Analytes; Fabrication and Modification of Nanoporous Silicon Particles; High-Altitude Hydration System; Photon Counting Using Edge-Detection Algorithm; Holographic Vortex Coronagraph; Optical Structural Health Monitoring Device; Fuel-Cell Power Source Based on Onboard Rocket Propellants; Polar Lunar Regions: Exploiting Natural and Augmented Thermal Environments; Simultaneous Spectral Temporal Adaptive Raman Spectrometer - SSTARS; Improved Speed and Functionality of a 580-GHz Imaging Radar; Bolometric Device Based on Fluxoid Quantization; Algorithms for Learning Preferences for Sets of Objects; Model for Simulating a Spiral Software-Development Process; Algorithm That Synthesizes Other Algorithms for Hashing; Algorithms for High-Speed Noninvasive Eye-Tracking System; and Adapting ASPEN for Orbital Express.

Source record↗

Software for Checking Statecharts

HiVy is a software tool set that enables verification through model checking of designs represented as finite-state machines or statecharts. HiVy provides automated translation of (1) statecharts created by use of the MathWorks Stateflow program to (2) Promela, the input language of the Spin model checker, which can then be used to verify, or trace logical errors in, distributed software systems. HiVy can operate directly on Stateflow models, or its abstract syntax of hierarchical sequential automata (HSA) can be used independently as an intermediate format for translation to Promela. In a typical design application, HiVy parses and reformats Stateflow model file data using the programs SfParse and sf2hsa, respectively. If the parsing effort is successful, an abstract syntax tree is delivered into a file named with the extension .hsa. If the design comprises several model files, they may be merged into one .hsa file before translation into Promela. Stateflow scope is preserved, and name clashes are avoided in the merge process. The HiVy program hsa2pr translates the model from the intermediate HSA format into Promela. Additionally, HiVy provides through translation a list of all statechart model propositions that are the means for formalizing linear temporal logic (LTL) properties about the model for Spin verification.

Pingree, Paula↗

Content Platforms Meet Data Storage, Retrieval Needs

Earth is under a constant barrage of information from space. Whether from satellites orbiting our planet, spacecraft circling Mars, or probes streaking toward the far reaches of the Solar System, NASA collects massive amounts of data from its spacefaring missions each day. NASA s Earth Observing System (EOS) satellites, for example, provide daily imagery and measurements of Earth s atmosphere, oceans, vegetation, and more. The Earth Observing System Data and Information System (EOSDIS) collects all of that science data and processes, archives, and distributes it to researchers around the globe; EOSDIS recently reached a total archive volume of 4.5 petabytes. Try to store that amount of information in your standard, four-drawer file cabinet, and you would need 90 million to get the job done. To manage the flood of information, NASA has explored technologies to efficiently collect, archive, and provide access to EOS data for scientists today and for years to come. One such technology is now providing similar capabilities to businesses and organizations worldwide.

Source record↗

Data Albums: An Event Driven Search, Aggregation and Curation Tool for Earth Science

Approaches used in Earth science research such as case study analysis and climatology studies involve discovering and gathering diverse data sets and information to support the research goals. To gather relevant data and information for case studies and climatology analysis is both tedious and time consuming. Current Earth science data systems are designed with the assumption that researchers access data primarily by instrument or geophysical parameter. In cases where researchers are interested in studying a significant event, they have to manually assemble a variety of datasets relevant to it by searching the different distributed data systems. This paper presents a specialized search, aggregation and curation tool for Earth science to address these challenges. The search rool automatically creates curated 'Data Albums', aggregated collections of information related to a specific event, containing links to relevant data files [granules] from different instruments, tools and services for visualization and analysis, and information about the event contained in news reports, images or videos to supplement research analysis. Curation in the tool is driven via an ontology based relevancy ranking algorithm to filter out non relevant information and data.

Ramachandran, Rahul↗

Enabling machine learning-ready HPC ensembles with Merlin

With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensemble data. With complexities such as multi-component workflows, heterogeneous machine architectures, parallel file systems, and batch scheduling, care must be taken to facilitate this analysis in a high performance computing (HPC) environment. Here, we present Merlin, a workflow framework to enable large ML-friendly ensembles of scientific HPC simulations. By augmenting traditional HPC with distributed compute technologies, Merlin aims to lower the barrier for scientific subject matter experts to incorporate ML into their analysis. As a producer–consumer workflow model, Merlin enables multi-machine, cross-batch job, dynamically allocated yet persistent workflows capable of utilizing surge-compute resources. Key features of Merlin are a flexible HPC-centric interface, low per-task overhead, multi-tiered fault recovery, and a hierarchical sampling algorithm that allows for $\mathscr{O}$(N) task execution and $\mathscr{O}$(N ln N) task queuing to ensembles of millions of tasks. In addition to Merlin’s design, we test the algorithm’s performance in an HPC center and demonstrate the ability to enqueue 40 million simulations in 100 s, with a 30 millisecond per-task overhead that is independent of ensemble size. Finally, we describe some example applications that Merlin has enabled on leadership-class HPC resources, such as the ML-augmented optimization of nuclear fusion experiments and the calibration of infectious disease models to study the progression of and possible mitigation strategies for COVID-19.

97 MATHEMATICS AND COMPUTING↗

Current and Projected Future Distribution of Plant Community Types for the Southern Seward Peninsula, Alaska

Landscape scale maps of plant community distribution at 5 m resolution were developed for southern Seward Peninsula. A Random Forest based environmental niche model was developed using topographic and climate data sets. Models were trained for the 2010-2019 period and applied to develop plant community distributions for contemporary (2010-2019) period. Trained model was applied to project the plant community distributions under future climate. Using downscaled projections for RCP8.5 scenario from five climate models (CCSM4, GFDL-CM3, GISS-E2-R, IPSL-CM5A-LR, MRI-CGCM3) and multi-model mean, a six member ensemble of plant community distribution were developed for current decade (2010-2019) and for future decades (2020-2029, 2030-2039, 2040-2049, 2050-2059). Contains *.tif and *.sh files. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

Engineering description of the ascent/descent bet product

The Ascent/Descent output product is produced in the OPIP routine from three files which constitute its input. One of these, OPIP.IN, contains mission specific parameters. Meteorological data, such as atmospheric wind velocities, temperatures, and density, are obtained from the second file, the Corrected Meteorological Data File (METDATA). The third file is the TRJATTDATA file which contains the time-tagged state vectors that combine trajectory information from the Best Estimate of Trajectory (BET) filter, LBRET5, and Best Estimate of Attitude (BEA) derived from IMU telemetry. Each term in the two output data files (BETDATA and the Navigation Block, or NAVBLK) are defined. The description of the BETDATA file includes an outline of the algorithm used to calculate each term. To facilitate describing the algorithms, a nomenclature is defined. The description of the nomenclature includes a definition of the coordinate systems used. The NAVBLK file contains navigation input parameters. Each term in NAVBLK is defined and its source is listed. The production of NAVBLK requires only two computational algorithms. These two algorithms, which compute the terms DELTA and RSUBO, are described. Finally, the distribution of data in the NAVBLK records is listed.

Seacord, A. W., II↗

Science network resources: Distributed systems

The Master Directory, which is overview information about whole data sets, is outlined. The data system environment is depicted. The question is explored of what is a prototype international directory including purpose and features. Advantages of on-line directories are listed. Interconnected directory assumptions are given. A description of given of DIF (Directory Interchange Format), which is an exchange file for directory information, along with information content of DIF and directories. The directory population status is given in a percentage viewgraph. The present and future directory interconnections status at GSFC is also listed.

Cline, Neal↗

Simple, Script-Based Science Processing Archive

The Simple, Scalable, Script-based Science Processing (S4P) Archive (S4PA) is a disk-based archival system for remote sensing data. It is based on the data-driven framework of S4P and is used for data transfer, data preprocessing, metadata generation, data archive, and data distribution. New data are automatically detected by the system. S4P provides services such as data access control, data subscription, metadata publication, data replication, and data recovery. It comprises scripts that control the data flow. The system detects the availability of data on an FTP (file transfer protocol) server, initiates data transfer, preprocesses data if necessary, and archives it on readily available disk drives with FTP and HTTP (Hypertext Transfer Protocol) access, allowing instantaneous data access. There are options for plug-ins for data preprocessing before storage. Publication of metadata to external applications such as the Earth Observing System Clearinghouse (ECHO) is also supported. S4PA includes a graphical user interface for monitoring the system operation and a tool for deploying the system. To ensure reliability, S4P continuously checks stored data for integrity, Further reliability is provided by tape backups of disks made once a disk partition is full and closed. The system is designed for low maintenance, requiring minimal operator oversight.

Lynnes, Christopher↗

Computational Support for Technology- Investment Decisions

Strategic Assessment of Risk and Technology (START) is a user-friendly computer program that assists human managers in making decisions regarding research-and-development investment portfolios in the presence of uncertainties and of non-technological constraints that include budgetary and time limits, restrictions related to infrastructure, and programmatic and institutional priorities. START facilitates quantitative analysis of technologies, capabilities, missions, scenarios and programs, and thereby enables the selection and scheduling of value-optimal development efforts. START incorporates features that, variously, perform or support a unique combination of functions, most of which are not systematically performed or supported by prior decision- support software. These functions include the following: Optimal portfolio selection using an expected-utility-based assessment of capabilities and technologies; Temporal investment recommendations; Distinctions between enhancing and enabling capabilities; Analysis of partial funding for enhancing capabilities; and Sensitivity and uncertainty analysis. START can run on almost any computing hardware, within Linux and related operating systems that include Mac OS X versions 10.3 and later, and can run in Windows under the Cygwin environment. START can be distributed in binary code form. START calls, as external libraries, several open-source software packages. Output is in Excel (.xls) file format.

Adumitroaie, Virgil↗