Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “software design parallel computation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

394 records · Page 22

Batched matrix computations on hardware accelerators based on GPUs

Scientific applications require solvers that work on many small size problems that are independent from each other. At the same time, the high-end hardware evolves rapidly and becomes ever more throughput-oriented and thus there is an increasing need for an effective approach to develop energy-efficient, high-performance codes for these small matrix problems that we call batched factorizations. The many applications that need this functionality could especially benefit from the use of GPUs, which currently are four to five times more energy efficient than multicore CPUs on important scientific workloads. This study, consequently, describes the development of the most common, one-sided factorizations, Cholesky, LU, and QR, for a set of small dense matrices. The algorithms we present together with their implementations are, by design, inherently parallel. In particular, our approach is based on representing the process as a sequence of batched BLAS routines that are executed entirely on a GPU. Importantly, this is unlike the LAPACK and the hybrid MAGMA factorization algorithms that work under drastically different assumptions of hardware design and efficiency of execution of the various computational kernels involved in the implementation. Thus, our approach is more efficient than what works for a combination of multicore CPUs and GPUs for the problems sizes of interest of the application use cases. The paradigm where upon a single chip (a GPU or a CPU) factorizes a single problem at a time is not at all efficient in our applications’ context. We illustrate all of these claims through a detailed performance analysis. With the help of profiling and tracing tools, we guide our development of batched factorizations to achieve up to two-fold speedup and three-fold better energy efficiency as compared against our highly optimized batched CPU implementations based on MKL library. Finally, the tested system featured two sockets of Intel Sandy Bridge CPUs and we compared with a batched LU factorizations featured in the CUBLAS library for GPUs, we achieve as high as 2.5× speedup on the NVIDIA K40 GPU.

97 MATHEMATICS AND COMPUTING↗

SU-D-213-05: Design, Evaluation and First Applications of a Off-Site State-Of-The-Art 3D Dosimetry System

Purpose: To design, construct and commission a prototype in-house three dimensional (3D) dose verification system for stereotatic body radiotherapy (SBRT) verification at an off-site partner institution. To investigate the potential of this system to achieve sufficient performance (1mm resolution, 3% noise, within 3% of true dose reading) for SBRT verification. Methods: The system was designed utilizing a parallel ray geometry instigated by precision telecentric lenses and an LED 630nm light source. Using a radiochromic dosimeter, a 3D dosimetric comparison with our gold-standard system and treatment planning software (Eclipse) was done for a four-field box treatment, under gamma passing criteria of 3%/3mm/10% dose threshold. Post off-site installation, deviations in the system’s dose readout performance was assessed by rescanning the four-field box irradiated dosimeter and using line-profiles to compare on-site and off-site mean and noise levels in four distinct dose regions. As a final step, an end-to-end test of the system was completed at the off-site location, including CT-simulation, irradiation of the dosimeter and a 3D dosimetric comparison of the planned (Pinnacle{sup 3}) to delivered dose for a spinal SBRT treatment(12 Gy per fraction). Results: The noise level in the high and medium dose regions of the four field box treatment was relatively 5% pre and post installation. This reflects the reduction in positional uncertainty through the new design. This At 1mm dose voxels, the gamma pass rates(3%,3mm) for our in-house gold standard system and the off-site system were comparable at 95.8% and 93.2% respectively. Conclusion: This work will describe the end-to-end process and results of designing, installing, and commissioning a state-of-the-art 3D dosimetry system created for verification of advanced radiation treatments including spinal radiosurgery.

60 APPLIED LIFE SCIENCES↗

Treatment planning of a skin-sparing conical breast brachytherapy applicator using conventional brachytherapy software

Purpose: AccuBoost is a noninvasive image-guided technique for the delivery of partial breast irradiation to the tumor bed and currently serves as an alternate to conventional electron beam boost. To irradiate the target volume while providing dose sparing to the skin, the round applicator design was augmented through the addition of an internally truncated conical shield and the reduction of the source to skin distance. Methods: Brachytherapy dose distributions for two types of conical applicators were simulated and estimated using Monte Carlo (MC) methods for radiation transport and a conventional treatment planning system (TPS). MC-derived and TPS-generated dose volume histograms (DVHs) and dose distribution data were compared for both the conical and round applicators for benchmarking purposes. Results: Agreement using the gamma-index test was {>=}99.95% for distance to agreement and dose accuracy criteria of 2 mm and 2%, respectively. After observing good agreement, TPS DVHs and dose distributions for the conical and round applicators were obtained and compared. Brachytherapy dose distributions generated using Pinnacle{sup 3} for ten CT data sets showed that the parallel-opposed beams of the conical applicators provided similar PTV coverage to the round applicators and reduced the maximum dose to skin, chest wall, and lung by up to 27%, 42%, and 43%, respectively. Conclusions: Brachytherapy dose distributions for the conical applicators have been generated using MC methods and entered into the Pinnacle{sup 3} TPS via the Tufts technique. Treatment planning metrics for the conical AccuBoost applicators were significantly improved in comparison to those for conventional electron beam breast boost.

60 APPLIED LIFE SCIENCES↗

HIPLZ: Enabling performance portability for exascale systems

While heterogeneous computing has emerged as a dominant trend in current and future High-Performance Computing (HPC) systems, it is also widely recognized that this shift has led to increased software complexity due to a proliferation of programming systems for different heterogeneous processors. One such example is the Heterogeneous-Compute Interface for Portability from AMD (HIP ), which is composed of a C Runtime API and C++ Kernel Language. Many HPC applications will likely use HIP on future exascale systems (e.g., Frontier and El Capitan), but HIP currently only targets AMD and NVIDIA processors. This limitation creates challenges for users who would also like to run their applications on exascale systems based on other architectures (e.g., Aurora, which is based on Intel hardware) that are currently not targeted by HIP . In this paper, we introduce the design and implementation of HIPLZ , a compiler and runtime system that uses the Intel Level Zero API to support HIP on Intel GPU architectures. We discuss the design of HIPLZ , derived from HIPCL (an implementation of HIP on top of OpenCL ), and portability issues that occur from using the Level Zero runtime as a backend. We evaluate our implementation by running several performance benchmarks and mini-apps written in HIP on Intel architectures using HIPLZ . Our results show that this approach provides competitive performance relative to Intel's OpenCL implementations on Intel Gen9 and UHD Graphics 770 GPUs, while providing good coverage of features needed by HPC applications. Overall, this approach is a promising demonstration of enabling performance portability for exascale systems.

97 MATHEMATICS AND COMPUTING↗

Advanced Test Reactor Core Modeling Update Project Annual Report for Fiscal Year 2011

Legacy computational reactor physics software tools and protocols currently used for support of Advanced Test Reactor (ATR) core fuel management and safety assurance and, to some extent, experiment management are obsolete, inconsistent with the state of modern nuclear engineering practice, and are becoming increasingly difficult to properly verify and validate (V&V). Furthermore, the legacy staff knowledge required for application of these tools and protocols from the 1960s and 1970s is rapidly being lost due to staff turnover and retirements. In 2009 the Idaho National Laboratory (INL) initiated a focused effort to address this situation through the introduction of modern high-fidelity computational software and protocols, with appropriate V&V, within the next 3-4 years via the ATR Core Modeling and Simulation and V&V Update (or 'Core Modeling Update') Project. This aggressive computational and experimental campaign will have a broad strategic impact on the operation of the ATR, both in terms of improved computational efficiency and accuracy for support of ongoing DOE programs as well as in terms of national and international recognition of the ATR National Scientific User Facility (NSUF). The ATR Core Modeling Update Project, targeted for full implementation in phase with the anticipated ATR Core Internals Changeout (CIC) in the 2014 time frame, began during the last quarter of Fiscal Year 2009, and has just completed its first full year. Key accomplishments so far have encompassed both computational as well as experimental work. A new suite of stochastic and deterministic transport theory based reactor physics codes and their supporting nuclear data libraries (SCALE, KENO-6, HELIOS, NEWT, and ATTILA) have been installed at the INL under various permanent sitewide license agreements and corresponding baseline models of the ATR and ATRC are now operational, demonstrating the basic feasibility of these code packages for their intended purpose. Furthermore, a capability for rigorous sensitivity analysis and uncertainty quantification based on the TSUNAMI system is being implemented and initial computational results have been obtained. This capability will have many applications in 2011 and beyond as a tool for understanding the margins of uncertainty in the new models as well as for validation experiment design and interpretation. Finally we note that although full implementation of the new computational models and protocols will extend over a period 3-4 years as noted above, interim applications in the much nearer term have already been demonstrated. In particular, these demonstrations included an analysis that was useful for understanding the cause of some issues in December 2009 that were triggered by a larger than acceptable discrepancy between the measured excess core reactivity and a calculated value that was based on the legacy computational methods. As the Modeling Update project proceeds we anticipate further such interim, informal, applications in parallel with formal qualification of the system under the applicable INL Quality Assurance procedures and standards.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Designing the Cloud-based DOE Systems Biology Knowledgebase

Systems Biology research, even more than many other scientific domains, is becoming increasingly data-intensive. Not only have advances in experimental and computational technologies lead to an exponential increase in scientific data volumes and their complexity, but increasingly such databases themselves are providing the basis for new scientific discoveries. To engage effectively with these community resources, integrated analyses, synthesis and simulation software is needed, regularly supported by scientific workflows. In order to provide a more collaborative, community driven research environment for this heterogeneous setting, the Department of Energy (DOE) has decided to develop a federated, cloud based cyber infrastructure - the Systems Biology Knowledgebase (Kbase). Pacific Northwest National Laboratory (PNNL) with its long tradition in data intensive science lead two of the five initial pilot projects, these two focusing on defining and testing the basic federated cloud-based system architecture and develop a prototype implementation. Hereby the community wide accessibility of biological data and the capability to integrate and analyze this data within its changing research context were seen as key technical functionalities the Kbase needed to enable. In this paper we describe the results of our investigations into the design of a cloud based federated infrastructure for: (1) Semantics driven data discovery, access and integration; (2) Data annotation, publication and sharing; (3) Workflow enabled data analysis; and (4) Project based collaborative working. We describe our approach, exemplary use cases and our prototype implementation that demonstrates the feasibility of this approach.

99 GENERAL AND MISCELLANEOUS↗

Non-equilibrium reaction and relaxation dynamics in a strongly interacting explicit solvent: F + CD{sub 3}CN treated with a parallel multi-state EVB model

We describe a parallelized linear-scaling computational framework developed to implement arbitrarily large multi-state empirical valence bond (MS-EVB) calculations within CHARMM and TINKER. Forces are obtained using the Hellmann-Feynman relationship, giving continuous gradients, and good energy conservation. Utilizing multi-dimensional Gaussian coupling elements fit to explicitly correlated coupled cluster theory, we built a 64-state MS-EVB model designed to study the F + CD{sub 3}CN → DF + CD{sub 2}CN reaction in CD{sub 3}CN solvent (recently reported in Dunning et al. [Science 347(6221), 530 (2015)]). This approach allows us to build a reactive potential energy surface whose balanced accuracy and efficiency considerably surpass what we could achieve otherwise. We ran molecular dynamics simulations to examine a range of observables which follow in the wake of the reactive event: energy deposition in the nascent reaction products, vibrational relaxation rates of excited DF in CD{sub 3}CN solvent, equilibrium power spectra of DF in CD{sub 3}CN, and time dependent spectral shifts associated with relaxation of the nascent DF. Many of our results are in good agreement with time-resolved experimental observations, providing evidence for the accuracy of our MS-EVB framework in treating both the solute and solute/solvent interactions. The simulations provide additional insight into the dynamics at sub-picosecond time scales that are difficult to resolve experimentally. In particular, the simulations show that (immediately following deuterium abstraction) the nascent DF finds itself in a non-equilibrium regime in two different respects: (1) it is highly vibrationally excited, with ∼23 kcal mol{sup −1} localized in the stretch and (2) its post-reaction solvation environment, in which it is not yet hydrogen-bonded to CD{sub 3}CN solvent molecules, is intermediate between the non-interacting gas-phase limit and the solution-phase equilibrium limit. Vibrational relaxation of the nascent DF results in a spectral blue shift, while relaxation of the post-reaction solvation environment results in a red shift. These two competing effects mean that the post-reaction relaxation profile is distinct from what is observed when Franck-Condon vibrational excitation of DF occurs within a microsolvation environment initially at equilibrium. Our conclusions, along with the theoretical and parallel software framework presented in this paper, should be more broadly applicable to a range of complex reactive systems.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

3D tomodosimetry using long scintillating fibers: A feasibility study

Purpose: 3D dosimetry is recognized as an ideal for patient-specific quality assurance (QA) of highly conformal radiotherapy treatments. However, existing 3D dosimeters are not straightforward to implement in the clinic, as their read-out procedure is often tedious and their accuracy, precision, and/or sample size exhibit limitations. The purpose of this work is to develop a 3D dosimeter based on the concept of tomodosimetry inside concentric cylindrical planes using long scintillating fibers for the QA of modern radiotherapy techniques such as intensity-modulated radiation therapy (IMRT) or intensity-modulated arc therapy (IMAT).Methods: Using a model-based simulation, scintillating fibers were modeled on three concentric cylindrical planes of radii 2.5, 5.0, and 7.5 cm, inside a 10 cm radius water-equivalent cylinder phantom. The phantom was set to rotate around its central axis, made parallel to the linac gantry axis of rotation. Light acquisitions were simulated using the calculated dose from the treatment planning software and reconstructed in each cylindrical plane at a resolution of 1 mm{sup 2} using a total-variation minimization iterative reconstruction algorithm. The 3D dose was then interpolated from the reconstructed cylindrical plane doses at a resolution of 1 mm{sup 3}. Different scintillating fiber patterns were compared by varying the angle of each fiber in its cylindrical plane and introducing a light-tight cut in each fiber. The precision of the reconstructed cylindrical dose distribution was evaluated using a Poisson modeling of the acquired light signals and the accuracy of the interpolated 3D dose was evaluated using an IMRT clinical plan for a prostate case.Results: Straight scintillating fiber patterns with light-tight cuts were the most accurate in cylindrical dose reconstruction, showing less than 0.5 mm distance-to-agreement in dose gradients and a mean local dose difference of less than 0.2% in the high dose region for a 10 × 10 cm{sup 2} field. The precision attained with this fiber configuration was less than 0.9% in the high dose, low gradient region of an IMRT segment for light acquisitions of 0.1 MU over a 360 degree rotation of the cylinder phantom. 3D dose interpolation for the IMRT clinical plan yielded an overall dose difference with the reference input of less than 1%, except in high dose gradients.Conclusions: Using long scintillating fibers inside rotating, concentric cylindrical planes, the authors demonstrate that their tomodosimetry method has the potential for high resolution, precise, and accurate 3D dosimetry. Moreover, because of its water-equivalence and rotational symmetry, this design should find interesting application for both treatment QA and machine commissioning.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Award ER25844: Minimizing System Noise Effects for Extreme-Scale Scientific Simulation Through Function Delegation

In software running on distributed computing clusters, time spent on communication between nodes in the cluster can be a significant portion of the overall computation time; background operating system tasks and other computational noise on the nodes of the system can have a significant impact on the amount of time this communication takes, especially on large systems. The research completed in this period has improved understanding of when such noise will have a significant impact. Specifically, it was demonstrated that not just noise on the nodes, but also noise on the network between nodes can have a significant impact on computation time. It was also demonstrated that noise patterns matter more than noise intensity: very regular noise can cause less disruption than lighter (on average) but less regular noise. It was also demonstrated that the effect of noise is more prominent as the speed of the network between nodes is increased. Furthermore, a tracing tool, Netgauge, was improved via our work, and a system simulator, LogGOPSim, was developed; they can be used by application developers to improve performance of their program and by system designers to mitigate the effects of noise by adjusting the noise characteristics of the operating system. Both have been made freely available as open source programs. In the course of developing these tools, we demonstrated weaknesses in existing methodologies for modeling communication, and we introduced a more detailed model, LogGOPS, for simulating systems. Not only were the deleterious effects of noise explored but we have also offered solutions. Our studies of simulations of system noise have led to specific recommendations on tuning systems to mitigate noise. We have also improved existing approaches to mitigating noise. Non-blocking collective communication avoids the effects of noise by letting communication continue simultaneously with computation (thus being non-blocking), so that the delays in communication introduced by noise have a smaller impact on overall computation time. Potentially, noise can be reduced much further by offloading communication tasks to a separate processing element than the operating system is using. We have improved our library LibNBC, which provides an implementation of non-blocking collectives, via this work. During this research, our proposal to include non-blocking collectives (which used LibNBC as a reference implementation) in the upcoming MPI-3 standard was accepted. As MPI is a ubiquitous and important standard for communication in parallel computing, this demonstrates a certain acceptance of the practicality and desirability of non-blocking collectives. Now that non-blocking collectives are a part of the standard we can expect to see optimized platform-specific implementations of non-blocking collectives. Also as part of this work we have also developed a language GOAL (Global Operation Assembly Language) that can be used as a starting point for defining languages to express optimized communication algorithms.

97 MATHEMATICS AND COMPUTING↗

A Public-Private-Acadmic Partnership to Advance Solar Power Forecasting

The National Center for Atmospheric Research (NCAR) is pleased to have led a partnership to advance the state-of-the-science of solar power forecasting by designing, developing, building, deploying, testing, and assessing the SunCast™ Solar Power Forecasting System. The project has included cutting edge research, testing in several geographically- and climatologically-diverse high penetration solar utilities and Independent System Operators (ISOs), and wide dissemination of the research results to raise the bar on solar power forecasting technology. The partners include three other national laboratories, six universities, and industry partners. This public-private-academic team has worked in concert to perform use-inspired research to advance solar power forecasting through cutting-edge research to advance both the necessary forecasting technologies and the metrics for evaluating them. The project has culminated in a year-long, full-scale demonstration of provide irradiance and power forecasts to utilities and ISOs to use in their operations. The project focused on providing elements of a value chain, beginning with the weather that causes a deviation from clear sky irradiance and progresses through monitoring and observations, modeling, forecasting, dissemination and communication of the forecasts, interpretation of the forecast, and through decision-making, which produces outcomes that have an economic value. The system has been evaluated using metrics developed specifically for this project, which has provided rich information on model and system performance. Research was accomplished on the very short range (0-6 hours) Nowcasting system as well as on the longer term (6-72 hour) forecasting system. The shortest range forecasts are based on observations in the field. The shortest range system, built by Brookhaven National Laboratory (BNL) and based on Total Sky Imagers (TSIs) is TSICast, which operates on the shortest time scale with a latency of only a few minutes and forecasts that currently go out to about 15 min. This project has facilitated research in improving the hardware and software so that the new high definition cameras deployed at multiple nearby locations allow discernment of the clouds at varying levels and advection according to the winds observed at those levels. Improvements over “smart persistence” are about 29% for even these very short forecasts. StatCast is based on pyranometer data measured at the site as well as concurrent meteorological observations and forecasts. StatCast is based on regime-dependent artificial intelligence forecasting techniques and has been shown to improve on “smart persistence” forecasts by 15-50%. A second category of short-range forecasting systems employ satellite imagery and use that information to discern clouds and their motion, allowing them to project the clouds, and the resulting blockage of irradiance, in time. CIRACast (the system produced by the Cooperative Institute for Atmospheric Research [CIRA] at Colorado State University) was already one of the more advanced cloud motion systems, which is the reason that team was brought to this project. During the project timeframe, the CIRA team was able to advance cloud shadowing, parallax removal, and implementation of better advecting winds at different altitudes. CIRACast shows generally a 25-40% improvement over Smart Persistence between sunrise and approximately 1600 UTC (Coordinated Universal Time) . A second satellite-based system, MADCast (Multi-sensor Advective Diffusive foreCast system), assimilates data from multiple satellite imagers and profilers to assimilate a fully three-dimensional picture of the cloud into the dynamic core of WRF. During 2015, MADCast (provided at least 70% improvement over Smart Persistence, with most of that skill being derived during partly cloudy conditions. That allows advection of the clouds via the Weather Research and Forecasting (WRF) model dynamics directly. After WRF-Solar™ showed initial success, it was also deployed in nowcasting mode with coarser runs out to 6 hours made hourly. It provided improvements on the order of 50-60% over Smart Persistence for forecasts up to 1600 UTC. The advantages of WRF-Solar-Nowcasting and MADCast were then blended to develop the new MAD-WRF model that incorporates the most important features of each of those models, both assimilating satellite cloud fields and using WRF-So far physics to develop and dissipate clouds. MAE improvements for MAD-WRF for forecasts from 3-6 hours are improved over WRF-Solar-Now by 20%. While all the Nowcasting system components by themselves provide improvement over Smart Persistence, the largest benefit is derived when they are smartly blended together by the Nowcasting Integrator to produce an integrated forecast. The development of WRF-Solar™ under this project has provided the first numerical weather prediction (NWP) model specifically designed to meet the needs of irradiance forecasting. The first augmentation improved the solar tracking algorithm to account for deviations associated with the eccentricity of the Earth’s orbit and the obliquity of the Earth. Second, WRF-Solar™ added the direct normal irradiance (DNI) and diffuse (DIF) components from the radiation parameterization to the model output. Third, efficient parameterizations were implemented to either interpolate the irradiance in between calls to the expensive radiative transfer parameterization, or to use a fast radiative transfer code that avoids computing three-dimensional heating rates but provides the surface irradiance. Fourth, a new parameterization was developed to improve the representation of absorption and scattering of radiation by aerosols (aerosol direct effect). A fifth advance is that the aerosols now interact with the cloud microphysics, altering the cloud evolution and radiative properties, an effect that has been traditionally only implemented in atmospheric computationally costly chemistry models. A sixth development accounts for the feedbacks that sub-grid scale clouds produce in shortwave irradiance as implemented in a shallow cumulus parameterization Finally, WRF-Solar™ also allows assimilation of infrared irradiances from satellites to determine the three dimensional cloud field, allowing for an improved initialization of the cloud field that increases the performance of short-range forecasts. We find that WRF-Solar™ can improve clear sky irradiance prediction by 15-80% over a standard version of WRF, depending on location and cloud conditions. In a formal comparison to the NAM baseline, WRF-Solar™ showed improvements in the Day-Ahead forecast of 22-42%. The SunCast™ system requires substantial software engineering to blend all of the new model components as well as existing publically available NWP model runs. To do this we use an expert system for the Nowcasting blender and the Dynamic Integrated foreCast (DICast®) system for the NWP models. These two systems are then blended, we use an empirical power conversion method to convert the irradiance predictions to power, then apply an analog ensemble (AnEn) approach to further tune the forecast as well as to estimate its uncertainty. The AnEn module decreased RMSE (root mean squared error) by 17% over the blended SunCast™ power forecasts and provided skill in the probabilistic forecast with a Brier Skill Score of 0.55. In addition, we have also developed a Gridded Atmospheric Forecast System (GRAFS) in parallel, leveraging cost share funds. An economic evaluation based on Production Cost Modeling in the Public Service Company of Colorado showed that the observed 50% improvement in forecast accuracy will save their customers $819,200 with the projected MW deployment for 2024. Using econometrics, NCAR has scaled this savings to a national level and shown that an annual expected savings for this 50% forecast error reduction ranges from $11M in 2015 to $43M expected in 2040 with increased solar deployment. This amounts to a $455M discounted savings over the 26 year period of analysis.

14 SOLAR ENERGY↗

TU-H-BRA-04: A Novel Superconducting Magnet Design for Optimized Patient Access and Minimal SSD for Use in a Linac-MR Hybrid

Purpose: A prototype rotating hybrid MR imaging system and linac has been developed to allow for simultaneous imaging and radiation delivery parallel to B{sub 0}. However, the design of a compact magnet capable of rotation in a small vault with sufficient patient access and a typical clinical source-to-surface distance (SSD) is challenging. This work presents a novel superconducting magnet design that allows for a reduced SSD and ample patient access by moving the superconducting coils to the side of the yoke. The yoke and pole-plate structures are shaped to direct the magnetic flux appropriately. Methods: The surface of the pole plate for the magnet assembly is optimized. The magnetic field calculations required in this work are performed with the 3D finite element method software package Opera-3D. Each tentative design strategy is virtually modeled in this software package and externally controlled by MATLAB, with its key geometries defined as variables. The particle swarm optimization algorithm is used to optimize the variables subject to the minimization of a cost function. At each iteration, Opera-3D will solve the magnetic field solution over a field-of-view suitable for MR imaging and the degree of field uniformity will be assessed to calculate the value of the cost function associated with that iteration. Results: An optimized magnet assembly that generates a homogenous 0.2T magnetic field over an ellipsoid with large axis of 30 cm and small axes of 20 cm is obtained. Conclusion: The distinct features of this model are the minimal distance between the yoke’s top and the isocentre and the improved patient access. On the other hand, having homogeneity over an ellipsoid give us a larger field-of-view, essential for geometric accuracy of the MRI system. The increase of B{sub 0} from 0.2T in the present model to 0.5T is the subject of future work. Funding Sources: Alberta Innovates - Health Solutions (AIHS)| Disclosure and Conflict of Interest: B. Gino Fallone is a co-founder and CEO of MagnetTx Oncology Solutions (under discussions to license Alberta biplanar linac MR for commercialization).

60 APPLIED LIFE SCIENCES↗

Large-scale Nanostructure Simulations from X-ray Scattering Data On Graphics Processor Clusters

X-ray scattering is a valuable tool for measuring the structural properties of materialsused in the design and fabrication of energy-relevant nanodevices (e.g., photovoltaic, energy storage, battery, fuel, and carbon capture andsequestration devices) that are key to the reduction of carbon emissions. Although today's ultra-fast X-ray scattering detectors can provide tremendousinformation on the structural properties of materials, a primary challenge remains in the analyses of the resulting data. We are developing novelhigh-performance computing algorithms, codes, and software tools for the analyses of X-ray scattering data. In this paper we describe two such HPCalgorithm advances. Firstly, we have implemented a flexible and highly efficient Grazing Incidence Small Angle Scattering (GISAXS) simulation code based on theDistorted Wave Born Approximation (DWBA) theory with C++/CUDA/MPI on a cluster of GPUs. Our code can compute the scattered light intensity from any givensample in all directions of space; thus allowing full construction of the GISAXS pattern. Preliminary tests on a single GPU show speedups over 125x compared tothe sequential code, and almost linear speedup when executing across a GPU cluster with 42 nodes, resulting in an additional 40x speedup compared to usingone GPU node. Secondly, for the structural fitting problems in inverse modeling, we have implemented a Reverse Monte Carlo simulation algorithm with C++/CUDAusing one GPU. Since there are large numbers of parameters for fitting in the in X-ray scattering simulation model, the earlier single CPU code required weeks ofruntime. Deploying the AccelerEyes Jacket/Matlab wrapper to use GPU gave around 100x speedup over the pure CPU code. Our further C++/CUDA optimization deliveredan additional 9x speedup.

36 MATERIALS SCIENCE↗

Data Locality Enhancement of Dynamic Simulations for Exascale Computing (Final Report)

The development of modern processors exhibits two trends that complicate the optimizations of modern software. The first is the increasing sensitivity of processors' throughput to irregularities in computation. With more processors produced through a massive integration of simple cores, future systems will increasingly favor regular data-level parallel computations, but deviate from the needs of applications with complex patterns. Some evidences are already shown on Graphic Processing Units (GPU): Irregular data accesses (e.g., indirect references A[D[i]]) and conditional branches are limiting many GPU applications' performance at a level an order of magnitude lower than the peak of GPU. The second hardware trend is the growing gap between memory bandwidth and the aggregate speed—that is, the sum of all cores' computing power—of a Chip Multiprocessor (CMP). Despite the capped growth of the peak CPU speed, the aggregate speed of a CMP keeps increasing as more cores get into a single chip. It is expected that by 2018, node concurrency in an exascale system will increase by hundreds of times, whereas, memory bandwidth will expand by only 10 to 20 times. Consequently, data movement and storage is expected to consume more than 70% of the total system power. Bridging this gap is difficult; the complexities of modern CMP memory hierarchy make it even harder: Data cache becomes shared among computing units, and the sharing is often non-uniform—whether two computing units share a cache depends on their proximity and the level of the cache. On the recent IBM Power7 architecture, for instance, four hardware contexts (or SMT threads) in a core share the entire memory hierarchy, all cores in one chip share an on-chip L3 cache, and cores across chips share L3 and main memory through off-chip connections. These two trends complicate the translation of computing power into performance, especially for a program with either intensive data accesses or complex patterns in data accesses or control flow paths. Unfortunately, both attributes present and will persist in a class of important applications. For instance, many scientific simulations deal with a large volume of data. And meanwhile, as most real-world processes are non-uniform and evolving (e.g., the evolution of a galaxy or the process of a drug injection), both the computations and data accesses of these programs tend to be irregular and dynamically changing. Currently, the lack of support to these applications on modern CMP severely limits their performance. On GPU, as our recent study shows and other studies echo, performance enhancement of a factor of integers is possible when memory accesses or control flows are streamlined for a set of GPU applications. On multicore CPU, our studies show that traditional locality enhancement, for being oblivious to the new features of multicore memory hierarchy, may even cause large slowdown to data-intensive dynamic applications. The severity of the issues is expected to worsen as the two hardware trends continue. Some recent studies try to match software with the trends, but in a limited scope or manner. For irregularities on GPU, most studies focus on irregularities analyzable through static analysis (e.g., data accesses in regular loops). Dynamic irregularities are harder to address because the needed analysis and transformations typically have to happen at run time. Some other research resorts to hardware extensions, an actual adoption of which is unclear for the entailed space cost and complexity. For data locality, recent years have seen some exploitations of the new memory hierarchy on multicore for performance, but most of them are on process or thread scheduling, rather than program transformations. Our recent study reveals that program-level transformations may magnify the scheduling benefits by a factor of seven, concluding that program-level transformation should play a central role for data locality enhancement on modern CMP. But research in this direction has been sparse, and most have focused on data layout or cache performance modeling, rather than program transformations to match with the new memory hierarchy features. Overall, it is still an open question how to bridge the gap between dynamic computations and the two prominent properties of modern processors. The goal of this project is to develop a set of techniques and software tools to enhance the matching between memory accesses in dynamic simulations and the prominent features of modern and future CMP, alleviating the memory performance issues for petascale and exascale computing. This report summarizes the discoveries and products produced throughout this project. It includes free launch, a new software approach to overcoming the shortcomings of both methods; coherence-free multiview, an approach that allows multiple views of a single data object to co-exist on GPU memory during a GPU kernel execution; algorithmic optimizations to data analytics problems, especially those that involve lots of distance calculations. Supporting dynamic parallelism is important for GPU to benefit a broad range of applications. There are currently two fundamental ways for programs to exploit dynamic parallelism on GPU: a software-based approach with software-managed worklists, and a hardware-based approach through dynamic subkernel launches. Neither is satisfactory. The former is complicated to program and is often subject to some load imbalance; the latter suffers large runtime overhead. In this work, we propose free launch, a new software approach to overcoming the shortcomings of both methods. It allows programmers to use subkernel launches to express dynamic parallelism. It employs a novel compiler-based code transformation named subkernel launch removal to replace the subkernel launches with the reuse of parent threads. Coupled with an adaptive task assignment mechanism, the transformation reassigns the tasks in the subkernels to the parent threads with a good load balance. The technique requires no hardware extensions, immediately deployable on existing GPUs. It keeps the programming convenience of the subkernel launch-based approach while avoiding its large runtime overhead. Meanwhile, its superior load balancing makes it outperform manual worklist-based techniques by 3X on average. The work was published at the 48th Annual IEEE/ACM International Symposium on Microarchitecture (Micro'2015). A Graphic Processing Unit (GPU) system is typically equipped with many types of memory (e.g., global, constant, texture, shared, cache). Data placement determines what data are placed on which type of memory, essential for GPU memory performance. Prior optimizations of data placement always require a single view of a data object on memory, which limits the optimization effectiveness. In this work, we propose coherence-free multiview, an approach that allows multiple views of a single data object to co-exist on GPU memory during a GPU kernel execution. We demonstrate that under certain conditions, the multiple views can remain incoherent while facilitating enhanced data placement. We present a theorem and some compiler support to ensure the soundness of the usage of coherence-free multiview. We further develop reference-discerning data placement, a new way to enhance data placements on GPU. It enables more flexible data placements by using coherence-free multiview to leverage the slack in coherence requirement of some GPU programs. Experiments on three types of GPU systems show that, with less than 200KB space cost, the new data placement technique can provide a 1.6X average (up to 4.27X) speedup. In addition, we have examined algorithmic optimizations to data analytics problems, especially those that involve lots of distance calculations. Computing distances among data points is an essential part of many important algorithms in data analytics, graph analysis, and other domains. In each of these domains, developers have spent significant manual effort optimizing algorithms, often through novel applications of the triangle equality, in order to minimize the number of distance computations in the algorithms. In this work, we observe that many algorithms across these domains can be generalized as an instance of a generic distance-related abstraction. Based on this abstraction, we derive seven principles for correctly applying the triangular inequality to optimize distance-related algorithms. Guided by the findings, we develop Triangular OPtimizer (TOP), the first software framework that is able to automatically produce optimized algorithms that either matches or outperforms manually designed algorithms for solving distance-related problems. TOP achieves up to 237x speedups and 2.5X on average. The work has been published at the 32nd International Conference on Machine Learning (ICML'15) and the 41st International Conference on Very Large Data Bases (VLDB'15).

42 ENGINEERING↗

Coastal Ohio Wind Project

The Coastal Ohio Wind Project intends to address problems that impede deployment of wind turbines in the coastal and offshore regions of Northern Ohio. The project evaluates different wind turbine designs and the potential impact of offshore turbines on migratory and resident birds by developing multidisciplinary research, which involves wildlife biology, electrical and mechanical engineering, and geospatial science. Firstly, the project conducts cost and performance studies of two- and three-blade wind turbines using a turbine design suited for the Great Lakes. The numerical studies comprised an analysis and evaluation of the annual energy production of two- and three-blade wind turbines to determine the levelized cost of energy. This task also involved wind tunnel studies of model wind turbines to quantify the wake flow field of upwind and downwind wind turbine-tower arrangements. The experimental work included a study of a scaled model of an offshore wind turbine platform in a water tunnel. The levelized cost of energy work consisted of the development and application of a cost model to predict the cost of energy produced by a wind turbine system placed offshore. The analysis found that a floating two-blade wind turbine presents the most cost effective alternative for the Great Lakes. The load effects studies showed that the two-blade wind turbine model experiences less torque under all IEC Standard design load cases considered. Other load effects did not show this trend and depending on the design load cases, the two-bladed wind turbine showed higher or lower load effects. The experimental studies of the wake were conducted using smoke flow visualization and hot wire anemometry. Flow visualization studies showed that in the downwind turbine configuration the wake flow was insensitive to the presence of the blade and was very similar to that of the tower alone. On the other hand, in the upwind turbine configuration, increasing the rotor blade angle of attack reduced the wake size and enhanced the vortices in the flow downstream of the turbine-tower compared with the tower alone case. Mean and rms velocity distributions from hot wire anemometer data confirmed that in a downwind configuration, the wake of the tower dominates the flow, thus the flow fields of a tower alone and tower-turbine combinations are nearly the same. For the upwind configuration, the mean velocity shows a narrowing of the wake compared with the tower alone case. The downwind configuration wake persisted longer than that of an upwind configuration; however, it was not possible to quantify this difference because of the size limitation of the wind tunnel downstream of the test section. The water tunnel studies demonstrated that the scale model studies could be used to adequately produce accurate motions to model the motions of a wind turbine platform subject to large waves. It was found that the important factors that affect the platform is whether the platform is submerged or surface piercing. In the former, the loads on the platform will be relatively reduced whereas in the latter case, the structure pierces the wave free surface and gains stiffness and stability. The other important element that affects the movement of the platform is depth of the sea in which the wind turbine will be installed. Furthermore, the wildlife biology component evaluated migratory patterns by different monitoring systems consisting of marine radar, thermal IR camera and acoustic recorders. The types of radar used in the project are weather surveillance radar and marine radar. The weather surveillance radar (1988 Doppler), also known as Next Generation Radar (NEXRAD), provides a network of weather stations in the US. Data generated from this network were used to understand general migratory patterns, migratory stopover habitats, and other patterns caused by the effects of weather conditions. At a local scale our marine radar was used to complement the datasets from NEXRAD and to collect additional monitoring parameters such as passage rates, flight paths, flight directions, and flight altitudes of nocturnal migrating species. Our work focused on the design and development of custom built marine radar that used t-bar and parabolic dish antennas. The marine radar used in the project was Furuno (XANK250) which was coupled with a XIR3000B digitizing card from Russell Technologies for collection of the radar data. The radar data was processed by open source radR processing software using different computational techniques and methods. Additional data from thermal IR imaging cameras were collected to detect heat emitted from objects and provide information on movements of birds and bats, data which we used for different animal flight behavior analysis. Lastly, the data from the acoustic recorders were used to provide the number of bird calls for assessing patterns and peak passage rates during migration. The development of the geospatial database included collection of different data sources that are used to support offshore wind turbine development. Many different data sets were collected and organized using initial version of web-based repository software tools that can accommodate distribution of rectified pertinent data sets such as the lake depth, lake bottom engineering parameters, extent of ice, navigation pathways, wind speed, important bird habitats, fish efforts and other layers that are relevant for supporting robust offshore wind turbine developments. Additional geospatial products developed during the project included few different prototypes for offshore wind farm suitability which can involve different stakeholders and participants for solving complex planning problems and building consensus. Some of the prototypes include spatial decision support system (SDSS) for collaborative decision making, a web-based Participatory Geographic Information System (PGIS) framework for evaluating importance of different decision alternatives using different evaluation criteria, and an Android application for collection of field data using mobile and tablet devices . In summary, the simulations of two- and three-blade wind turbines suggested that two-bladed machines could produce comparable annual energy as the three-blade wind turbines but have a lighter tower top weight, which leads to lower cost of energy. In addition, the two-blade rotor configuration potentially costs 20% less than a three blade configuration that produces the same power at the same site. The cost model analysis predicted a potential cost savings of approximately 15% for offshore two-blade wind turbines. The foundation design for a wind turbine in Lake Erie is likely to be driven by ice loads based on the currently available ice data and ice mechanics models. Hence, for Lake Eire, the cost savings will be somewhat smaller than the other lakes in the Great Lakes. Considering the size of cranes and vessels currently available in the Great Lakes, the cost optimal wind turbine size should be 3 MW, not larger. The surveillance data from different monitoring systems suggested that bird and bat passage rates per hour were comparable during heavy migrations in both spring and fall seasons while passage rates were significantly correlated to wind directions and wind speeds. The altitude of migration was higher during heavy migrations and higher over water relative to over land. Notable portions of migration on some spring nights occurred parallel the shoreline, often moving perpendicular to southern winds. The birds approaching the Western basin have a higher propensity to cross than birds approaching the Central basin of Lake Erie and as such offshore turbine development might be a better option further east towards Cleveland than in the Western basin. The high stopover density was more strongly associated with migration volume the following night rather than the preceding night. The processed mean scalar wind speeds with temporal resolutions as fine as 10-minute intervals near turbine height showed that August is the month with the weakest winds while December is the month, which typically has the strongest winds. The ice data suggests that shallow western basin of Lake Erie has higher ice cover duration many times exceeding 90 days during some winters.

17 WIND ENERGY↗

TH-E-9A-01: Medical Physics 1.0 to 2.0, Session 4: Computed Tomography, Ultrasound and Nuclear Medicine

Medical Physics 2.0 is a bold vision for an existential transition of clinical imaging physics in face of the new realities of value-based and evidencebased medicine, comparative effectiveness, and meaningful use. It speaks to how clinical imaging physics can expand beyond traditional insular models of inspection and acceptance testing, oriented toward compliance, towards team-based models of operational engagement, prospective definition and assurance of effective use, and retrospective evaluation of clinical performance. Organized into four sessions of the AAPM, this particular session focuses on three specific modalities as outlined below. CT 2.0: CT has been undergoing a dramatic transition in the last few decades. While the changes in the technology merits discussions of their own, an important question is how clinical medical physicists are expected to effectively engage with the new realities of CT technology and practice. Consistent with the upcoming paradigm of Medical Physics 2.0, this CT presentation aims to provide definitions and demonstration of the components of the new clinical medical physics practice pertaining CT. The topics covered include physics metrics and analytics that aim to provide higher order clinicallyrelevant quantification of system performance as pertains to new (and not so new) technologies. That will include the new radiation and dose metrics (SSDE, organ dose, risk indices), image quality metrology (MTF/NPS/d’), task-based phantoms, and the effect of patient size. That will follow with a discussion of the testing implication of new CT hardware (detectors, tubes), acquisition methods (innovative helical geometries, AEC, wide beam CT, dual energy, inverse geometry, application specialties), and image processing and analysis (iterative reconstructions, quantitative CT, advanced renditions). The presentation will conclude with a discussion of clinical and operational aspects of Medical Physics 2.0 including training and communication, use optimization (dose and technique factors), automated analysis and data management (automated QC methods, protocol tracking, dose monitoring, issue tracking), and meaningful QC considerations. US 2.0: Ultrasound imaging is evolving at a rapid pace, adding new imaging functions and modes that continue to enhance its clinical utility and benefits to patients. The ultrasound talk will look ahead 10–15 years and consider how medical physicists can bring maximal value to the clinical ultrasound practices of the future. The roles of physics in accreditation and regulatory compliance, image quality and exam optimization, clinical innovation, and education of staff and trainees will all be considered. A detailed examination of expected technology evolution and impact on image quality metrics will be presented. Clinical implementation of comprehensive physics services will also be discussed. Nuclear Medicine 2.0: Although the basic science of nuclear imaging has remained relatively unchanged since its inception, advances in instrumentation continue to advance the field into new territories. With a great number of these advances occurring over the past decade, the role and testing strategies of clinical nuclear medicine physicists must evolve in parallel. The Nuclear Medicine 2.0 presentation is designed to highlight some of the recent advances from a clinical medical physicist perspective and provide ideas and motivation for designing better evaluation strategies. Topics include improvement of traditional physics metrics and analytics, testing implications of hybrid imaging and advanced detector technologies, and strategies for effective implementation into the clinic. Learning Objectives: Become familiar with new physics metrics and analytics in nuclear medicine, CT, and ultrasound. To become familiar with the major new developments of clinical physics support. To understand the physics testing implications of new technologies, hardware, software, and applications. Identify approaches for implementing comprehensive medical physics services in future imaging practices.

60 APPLIED LIFE SCIENCES↗

Automated Cache Performance Analysis And Optimization

While there is no lack of performance counter tools for coarse-grained measurement of cache activity, there is a critical lack of tools for relating data layout to cache behavior to application performance. Generally, any nontrivial optimizations are either not done at all, or are done ”by hand” requiring significant time and expertise. To the best of our knowledge no tool available to users measures the latency of memory reference instructions for particular addresses and makes this information available to users in an easy-to-use and intuitive way. In this project, we worked to enable the Open|SpeedShop performance analysis tool to gather memory reference latency information for specific instructions and memory addresses, and to gather and display this information in an easy-to-use and intuitive way to aid performance analysts in identifying problematic data structures in their codes. This tool was primarily designed for use in the supercomputer domain as well as grid, cluster, cloud-based parallel e-commerce, and engineering systems and middleware. Ultimately, we envision a tool to automate optimization of application cache layout and utilization in the Open|SpeedShop performance analysis tool. To commercialize this software, we worked to develop core capabilities for gathering enhanced memory usage performance data from applications and create and apply novel methods for automatic data structure layout optimizations, tailoring the overall approach to support existing supercomputer and cluster programming models and constraints. In this Phase I project, we focused on infrastructure necessary to gather performance data and present it in an intuitive way to users. With the advent of enhanced Precise Event-Based Sampling (PEBS) counters on recent Intel processor architectures and equivalent technology on AMD processors, we are now in a position to access memory reference information for particular addresses. Prior to the introduction of PEBS counters, cache behavior could only be measured reliably in the aggregate across tens or hundreds of thousands of instructions. With the newest iteration of PEBS technology, cache events can be tied to a tuple of instruction pointer, target address (for both loads and stores), memory hierarchy, and observed latency. With this information we can now begin asking questions regarding the efficiency of not only regions of code, but how these regions interact with particular data structures and how these interactions evolve over time. In the short term, this information will be vital for performance analysts understanding and optimizing the behavior of their codes for the memory hierarchy. In the future, we can begin to ask how data layouts might be changed to improve performance and, for a particular application, what the theoretical optimal performance might be. The overall benefit to be produced by this effort was a commercial quality easy-to- use and scalable performance tool that will allow both beginner and experienced parallel programmers to automatically tune their applications for optimal cache usage. Effective use of such a tool can literally save weeks of performance tuning effort. Easy to use. With the proposed innovations, finding and fixing memory performance issues would be more automated and hide most to all of the performance engineer expertise ”under the hood” of the Open|SpeedShop performance tool. One of the biggest public benefits from the proposed innovations is that it makes performance analysis more usable to a larger group of application developers. Intuitive reporting of results. The Open|SpeedShop performance analysis tool has a rich set of intuitive, yet detailed reports for presenting performance results to application developers. Our goal was to leverage this existing technology to present the results from our memory performance addition to Open|SpeedShop. Suitable for experts as well as novices. Application performance is getting more difficult to measure as the hardware platforms they run on become more complicated. This makes life difficult for the application developer, in that they need to know more about the hardware platform, including the memory system hierarchy, in order to understand the performance of their application. Some application developers are comfortable in that sceario, while others want to do their scientific research and not have to understand all the nuances in the hardware platform they are running their application on. Our proposed innovations were aimed to support both experts and novice performance analysts. Useful in many markets. The enhancement to Open|SpeedShop would appeal to a broader market space, as it will be useful in scientific, commercial, and cloud computing environments. Our goal was to use technology developed initially at the and Lawrence Livermore National Laboratory combined with the development and commercial software experience of the Argo Navis Technologies, LLC (ANT) to form a powerful combination to delivery these objectives.

97 MATHEMATICS AND COMPUTING↗