Software-Defined Network Solutions for Science Scenarios: Performance Testing Framework and Measurements
Explore the source record for details and available documents.
Computer Science↗
SEARCH · Engineering Papers
Results for “Scientific software”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Tools for 3D scientific visualization in computational aerodynamics at NASA Ames Research Center
Hardware, software, and techniques used by the Fluid Dynamics Division (NASA) for performing visualization of computational aerodynamics, which can be applied to the visualization of flow fields from computer simulations of fluid dynamics about the Space Shuttle, are discussed. Three visualization techniques applied, post-processing, tracking, and steering, are described, as well as the post-processing software packages used, PLOT3D, SURF (Surface Modeller), GAS (Graphical Animation System), and FAST (Flow Analysis software Toolkit). Using post-processing methods a flow simulation was executed on a supercomputer and, after the simulation was complete, the results were processed for viewing. It is shown that the high-resolution, high-performance three-dimensional workstation combined with specially developed display and animation software provides a good tool for analyzing flow field solutions obtained from supercomputers.
Bancroft, Gordon↗
Frameworks Coordinate Scientific Data Management
Jet Propulsion Laboratory computer scientists developed a unique software framework to help NASA manage its massive amounts of science data. Through a partnership with the Apache Software Foundation of Forest Hill, Maryland, the technology is now available as an open-source solution and is in use by cancer researchers and pediatric hospitals.
Source record↗
Louisiana Immersive Technologies Enterprise (LITE): Enabling Coastal Stewardship and Reliable Energy
This project will focus on research and application development in the area of large scale data computing and visualization that are critical to the DoE BER program areas of climate change research, environmental sciences, life sciences, and medical sciences. Most commercial applications available today are unable to handle the sizes of data in these domains, usually in the order of terabytes and petabytes. This project will address this problem, by taking advantage of the advanced scientific computing infrastructure of the LITE and LONI to conduct research and develop software applications that run natively on these 64-bit supercomputing platforms. The project is consistent with the goals of Scientific Discovery through Advanced Computing (SciDAC), a DoE initiative.
54 ENVIRONMENTAL SCIENCES↗
Combinatorial Algorithms in Scientific Computing
We provide the final report for this grant, detailing the publications, software produced, students trained who have joined the DOE workforce, and the impact our work has had on computational mathematics and related disciplines.
97 MATHEMATICS AND COMPUTING↗
Python-Based Scientific Analysis and Visualization of Precipitation Systems at NASA Marshall Space Flight Center
At NASA Marshall Space Flight Center (MSFC), Python is used several different ways to analyze and visualize precipitating weather systems. A number of different Python‐based software packages have been developed, which are available to the larger scientific community. The approach in all these packages is to utilize pre‐existing Python modules as well as to be object‐oriented and scalable. The first package that will be described and demonstrated is the Python Advanced Microwave Precipitation Radiometer (AMPR) Data Toolkit, or PyAMPR for short. PyAMPR reads geolocated brightness temperature data from any flight of the AMPR airborne instrument over its 25‐year history into a common data structure suitable for user‐defined analyses. It features rapid, simplified (i.e., one line of code) production of quick‐look imagery, including Google Earth overlays, swath plots of individual channels, and strip charts showing multiple channels at once. These plotting routines are also capable of significant customization for detailed, publication‐ready figures. Deconvolution of the polarization‐varying channels to static horizontally and vertically polarized scenes is also available. Examples will be given of PyAMPR's contribution toward real‐time AMPR data display during the Integrated Precipitation and Hydrology Experiment (IPHEx), which took place in the Carolinas during May‐June 2014. The second software package is the Marshall Multi‐Radar/Multi‐Sensor (MRMS) Mosaic Python Toolkit, or MMM‐Py for short. MMM‐Py was designed to read, analyze, and display three‐dimensional national mosaicked reflectivity data produced by the NOAA National Severe Storms Laboratory (NSSL). MMM‐Py can read MRMS mosaics from either their unique binary format or their converted NetCDF format. It can also read and properly interpret the current mosaic design (4 regional tiles) as well as mosaics produced prior to late July 2013 (8 tiles). MMM‐Py can easily stitch multiple tiles together to provide a larger regional or national picture of precipitating weather systems. Composites, horizontal and vertical crosssections, and combinations thereof are easily displayed using as little as one line of code. MMM‐Py can also write to the native MRMS binary format, and sub‐sectioning of tiles (or multiple stitched tiles) is anticipated to be in place by the time of this meeting. Thus, MMM‐Py also can be used to power the creation of custom mosaics for targeted regional studies. Overlays of other data (e.g., lightning observations) are easily accomplished. Demonstrations of MMM‐Py, including the creation of animations, will be shown. Finally, Marshall has done significant work to interface Python‐based analysis routines with the U.S. Department of Energy's Py‐ART software package for radar data ingest, processing, and analysis. One example of this is the Python Turbulence Detection Algorithm (PyTDA), an MSFC‐based implementation of the National Center for Atmospheric Research (NCAR) Turbulence Detection Algorithm (NTDA) for the purposes of convective‐scale analysis, situational awareness, and forensic meteorology. PyTDA exploits Py‐ART's radar data ingest routines and data model to rapidly produce aviation‐relevant turbulence estimates from Doppler radar data. Work toward processing speed optimization and better integration within the Py‐ART framework will be highlighted. Python‐based analysis within the Py‐ART framework is also being done for new research related to intercomparison of ground‐based radar data with satellite estimates of ocean winds, as well as research on the electrification of pyrocumulus clouds.
Lang, Timothy J.↗
Issues in knowledge representation to support maintainability: A case study in scientific data preparation
Scientific data preparation is the process of extracting usable scientific data from raw instrument data. This task involves noise detection (and subsequent noise classification and flagging or removal), extracting data from compressed forms, and construction of derivative or aggregate data (e.g. spectral densities or running averages). A software system called PIPE provides intelligent assistance to users developing scientific data preparation plans using a programming language called Master Plumber. PIPE provides this assistance capability by using a process description to create a dependency model of the scientific data preparation plan. This dependency model can then be used to verify syntactic and semantic constraints on processing steps to perform limited plan validation. PIPE also provides capabilities for using this model to assist in debugging faulty data preparation plans. In this case, the process model is used to focus the developer's attention upon those processing steps and data elements that were used in computing the faulty output values. Finally, the dependency model of a plan can be used to perform plan optimization and runtime estimation. These capabilities allow scientists to spend less time developing data preparation procedures and more time on scientific analysis tasks. Because the scientific data processing modules (called fittings) evolve to match scientists' needs, issues regarding maintainability are of prime importance in PIPE. This paper describes the PIPE system and describes how issues in maintainability affected the knowledge representation used in PIPE to capture knowledge about the behavior of fittings.
Chien, Steve↗
Intelligent assistance in scientific data preparation
Scientific data preparation is the process of extracting usable scientific data from raw instrument data. This task involves noise detection (and subsequent noise classification and flagging or removal), extracting data from compressed forms, and construction of derivative or aggregate data (e.g. spectral densities or running averages). A software system called PIPE provides intelligent assistance to users developing scientific data preparation plans using a programming language called Master Plumber. PIPE provides this assistance capability by using a process description to create a dependency model of the scientific data preparation plan. This dependency model can then be used to verify syntactic and semantic constraints on processing steps to perform limited plan validation. PIPE also provides capabilities for using this model to assist in debugging faulty data preparation plans. In this case, the process model is used to focus the developer's attention upon those processing steps and data elements that were used in computing the faulty output values. Finally, the dependency model of a plan can be used to perform plan optimization and run time estimation. These capabilities allow scientists to spend less time developing data preparation procedures and more time on scientific analysis tasks.
Chien, Steve↗
A database of refractive indices and dielectric constants auto-generated using ChemDataExtractor
The ability to auto-generate databases of optical properties holds great potential for advancing optical research, especially with regards to the data-driven discovery of optical materials. An optical property database of refractive indices and dielectric constants is presented, which comprises a total of 49,076 refractive index and 60,804 dielectric constant data records on 11,054 unique chemicals. The database was auto-generated using the state-of-the-art natural language processing software, ChemDataExtractor, using a corpus of 388,461 scientific papers. The data repository offers a representative overview of the information on linear optical properties that resides in scientific papers from the past 30 years. Public availability of these data will enable a quick search for the optical property of certain materials. The large size of this repository will accelerate data-driven research on the design and prediction of optical materials and their properties. To the best of our knowledge, this is the first auto-generated database of optical properties from a large number of scientific papers. We provide a web interface to aid the use of our database.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs
Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth make it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple and mathematically sound operations. However, the effective use of such representations has been hindered until now by the relatively high costs of creating, accessing, reducing, and otherwise operating on such representations. We describe here highly optimized data refactoring kernels for GPU accelerators that enable efficient creation and manipulation of data in multigrid-based hierarchical forms. We demonstrate that our optimized design can achieve up to 250 TB/s aggregated data refactoring throughput—83% of theoretical peak—on 1024 nodes of the Summit supercomputer. We showcase our optimized design by applying it to a large-scale scientific visualization workflow and the MGARD lossy compression software.
Data refactoring↗
Research Software Engineering at Oak Ridge National Laboratory
Research software engineers (RSE) play a vital role in scientific discoveries worldwide. They lead the core development of the applications, libraries, and tools that enable today’s supercomputers to process the vast volumes of data generated from the world’s most extensive and complicated scientific instruments and facilities. This article describes the mission, culture, and practices of RSE teams at Oak Ridge National Laboratory (ORNL), including their team dynamics and composition, work ethics, standard practices, and ever-evolving skill sets vital to pursuing scientific innovations and implementing novel ideas. We describe the lessons learned from specific activities that contribute to shaping the identity and growth of RSE roles at ORNL. Lastly, we provide our view for the near future on effective strategies for establishing, leading, and nurturing RSE teams and building a thriving community in collaboration with science stakeholders.
42 ENGINEERING↗
A domain-specific compiler for a parallel multiresolution adaptive numerical simulation environment
This paper describes the design and implementation of a layered domain-specific compiler to support MADNESS---Multiresolution ADaptive Numerical Environment for Scientific Simulation. MADNESS is a high-level software environment for the solution of integral and differential equations in many dimensions, using adaptive and fast harmonic analysis methods with guaranteed precision. MADNESS uses k-d trees to represent spatial functions and implements operators like addition, multiplication, differentiation, and integration on the numerical representation of functions. The MADNESS runtime system provides global namespace support and a task-based execution model including futures. MADNESS is currently deployed on massively parallel supercomputers and has enabled many science advances. Due to the highly irregular and statically unpredictable structure of the k-d trees representing the spatial functions encountered in MADNESS applications, only purely runtime approaches to optimization have previously been implemented in the MADNESS framework. This paper describes a layered domain-specific compiler developed to address some performance bottlenecks in MADNESS. The newly developed static compile-time optimizations, in conjunction with the MADNESS runtime support, enable significant performance improvement for the MADNESS framework.
Rajbhandari, Samyam↗
How does the connectivity of open-framework conglomerates within multi-scale hierarchical fluvial architecture affect oil-sweep efficiency in waterflooding?
Understanding multi-phase fluid flow and transport processes within aquifers, candidate reservoirs for CO 2 sequestration, and petroleum reservoirs requires understanding a diverse set of geologic properties of the aquifer or reservoir, over a wide range of spatial and temporal scales. We focus on multiphase flow dynamics with wetting (e.g., water) and non-wetting (e.g., gas or oil) fluids, with one invading another. This problem is of general interest in a number of fields and is illustrated here by considering the sweep efficiency of oil during a waterflood. Using a relatively fine-resolution grid throughout a relatively large domain in these simulations and probing the results with advanced scientific visualization tools (Reservoir Visualization Analysis [RVA]/ ParaView software) promote a better understanding of how smaller-scale features affect the aggregate behavior at larger scales. We studied the effects on oil-sweep efficiency of the proportion, hierarchical organization, and connectivity of high-permeability open-framework conglomerate (OFC) cross-sets within the multi-scale stratal architecture found in fluvial deposits. We further analyzed oil production rate, water breakthrough time, and spatial and temporal distribution of residual oil saturation. As expected, the effective permeability of the reservoir exhibits large-scale anisotropy created by the organization of OFC cross-sets within unit bars, and the organization of unit bars within compound- bars. As a result, oil-sweep efficiency critically depends on the direction of the pressure gradient. However, contrary to expectations, the total amount of trapped oil due to the effect of capillary trapping does not depend on the magnitude of the pressure gradient within the examined range. Hence the pressure difference between production and injection wells does not affect sweep efficiency; although the spatial distribution of oil remaining in the reservoir depends on this value. Whether or not clusters of connected OFC span the domain affects only the absolute rate of oil production—not sweep efficiency.
58 GEOSCIENCES↗
Provenance Environment Services Software
The ProvEn Software is designed to analyze and monitor distributed scientific workflow application behavior in the context of its environment.
Source record↗
HPCGraph v. 0.1
Graphs occur in several places in real world from road networks, social networks and scientific simulations. HPC Graph is a graph analysis software to analyze the properties such as connectivity, communities etc in extreme scale graphs.
Slota, George↗
Exascale models of stellar explosions: Quintessential multi-physics simulation
The ExaStar project aims to deliver an efficient, versatile, and portable software ecosystem for multi-physics astrophysics simulations run on exascale machines. The code suite is a component-based multi-physics toolkit, built on the capabilities of current simulation codes (in particular Flash-X and Castro), and based on the massively parallel adaptive mesh refinement framework AMReX. It includes modules for hydrodynamics, advanced radiation transport, thermonuclear kinetics, and nuclear microphysics. The code will reach exascale efficiency by building upon current multi- and many-core packages integrated into an orchestration system that uses a combination of configuration tools, code translators, and a domain-specific asynchronous runtime to manage performance across a range of platform architectures. The target science includes multi-physics simulations of astrophysical explosions (such as supernovae and neutron star mergers) to understand the cosmic origin of the elements and the fundamental physics of matter and neutrinos under extreme conditions.
79 ASTRONOMY AND ASTROPHYSICS↗
Final Report: Super Instruction Architecture for Scalable Parallel Computations
The most advanced methods for reliable and accurate computation of the electronic structure of molecular and nano systems are the coupled-cluster techniques. These high-accuracy methods help us to understand, for example, how biological enzymes operate and contribute to the design of new organic explosives. The ACES III software provides a modern, high-performance implementation of these methods optimized for high performance parallel computer systems, ranging from small clusters typical in individual research groups, through larger clusters available in campus and regional computer centers, all the way to high-end petascale systems at national labs, including exploiting GPUs if available. This project enhanced the ACESIII software package and used it to study interesting scientific problems.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Open Source Platform for Live Processing of High Data Rate Electron Microscopy (SBIR Phase I Award Topic 15, Subtopic B, Final Scientific / Technical Report)
4D scanning transmission electron microscopy (STEM) is experiencing a revolution both in terms of data size, and data veracity, as are many other areas of experimental data acquisition. This presents enormous opportunities to make new scientific discoveries, but is also hamstrung by aging software that is no longer fit for purpose. Experimental data acquisition platforms must make the same investments in software infrastructure that the simulation community made decades ago to take advantage of high-performance computing (HPC) if they hope to realize the full potential of the latest advances made in detector technology. This requires rethinking the design and deployment of hardware, moving away from small, fixed commodity computing capabilities with closed interfaces to open, modular, scalable systems designed for parallelism and throughput. The technical approach will leverage best-in-class open source technologies, and develop new approaches strategically to build an open source platform for experimental data acquisition targeting high data rate electron microscopy. The project will develop a software platform that uses the power and performance of modern HPC resources to ingest the large volume of data produced by electron microscopes using direct electron detectors. A set of configurable pipelines will be used to enable the processing to be tailored to the problem being investigated with builtin visualization of data at interactive rates. This project will leverage investments made by the DOE and other agencies to provide a powerful software platform for the experimental data community working with most powerful electron microscopes, such as 4D STEM detectors. It will create a platform that serves as a reference for the community. Permissive open source licensing will make the platform available to all, with a simple deployment strategy offering the ability to keep data close to where it is generated. This advanced software platform will require customization for each deployment, creating opportunities for software services that are typically offered to extend open source software platforms.
97 MATHEMATICS AND COMPUTING↗