Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “algorithm selection”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,405 records · Page 78

SU-G-TeP1-14: SRS Dose Calculation Accuracy Comparison Between Pencil Beam and Monte Carlo Algorithms

Purpose: Stereotactic radiosurgery (SRS) outcomes are related to the delivered dose to the target and to surrounding tissue. We have commissioned a Monte Carlo based dose calculation algorithm to recalculated the delivered dose planned using pencil beam calculation dose engine. Methods: Twenty consecutive previously treated patients have been selected for this study. All plans were generated using the iPlan treatment planning system (TPS) and calculated using the pencil beam algorithm. Each patient plan consisted of 1 to 3 targets and treated using dynamically conformal arcs or intensity modulated beams. Multi-target treatments were delivered using multiple isocenters, one for each target. These plans were recalculated for the purpose of this study using a single isocenter. The CT image sets along with the plan, doses and structures were DICOM exported to Monaco TPS and the dose was recalculated using the same voxel resolution and monitor units. Benchmark data was also generated prior to patient calculations to assess the accuracy of the two TPS against measurements using a micro ionization chamber in solid water. Results: Good agreement, within −0.4% for Monaco and +2.2% for iPlan were observed for measurements in water phantom. Doses in patient geometry revealed up to 9.6% differences for single target plans and 9.3% for multiple-target-multiple-isocenter plans. The average dose differences for multi-target-single-isocenter plans were approximately 1.4%. Similar differences were observed for the OARs and integral dose. Conclusion: Accuracy of the beam is crucial for the dose calculation especially in the case of small fields such as those used in SRS treatments. A superior dose calculation algorithm such as Monte Carlo, with properly commissioned beam models, which is unaffected by the lack of electronic equilibrium should be preferred for the calculation of small fields to improve accuracy.

60 APPLIED LIFE SCIENCES↗

Evaluating the Assumptions of Surface Reflectance and Aerosol Type Selection Within the MODIS Aerosol Retrieval Over Land: The Problem of Dust Type Selection

Aerosol Optical Depth (AOD) and Angstrom exponent (AE) values derived with the MODIS retrieval algorithm over land (Collection 5) are compared with ground based sun photometer measurements at eleven sites spanning the globe. Although, in general, total AOD compares well at these sites (R2 values generally over 0.8), there are cases (from 2 to 67% of the measurements depending on the site) where MODIS clearly retrieves the wrong spectral dependence, and hence, an unrealistic AE value. Some of these poor AE retrievals are due to the aerosol signal being too small (total AOD<0.3) but in other cases the AOD should have been high enough to derive accurate AE. However, in these cases, MODIS indicates AE values close to 0.6 and zero fine model weighting (FMW), i.e. dust model provides the best fitting to the MODIS observed reflectance. Yet, according to evidence from the collocated sun photometer measurements and back-trajectory analyses, there should be no dust present. This indicates that the assumptions about aerosol model and surface properties made by the MODIS algorithm may have been incorrect. Here we focus on problems related to parameterization of the land-surface optical properties in the algorithm, in particular the relationship between the surface reflectance at 660 and 2130 nm.

Mielonen, T.↗

Multiangle Imaging Spectroradiometer (MISR) Global Aerosol Optical Depth Validation Based on 2 Years of Coincident Aerosol Robotic Network (AERONET) Observations

Performance of the Multiangle Imaging Spectroradiometer (MISR) early postlaunch aerosol optical thickness (AOT) retrieval algorithm is assessed quantitatively over land and ocean by comparison with a 2-year measurement record of globally distributed AERONET Sun photometers. There are sufficient coincident observations to stratify the data set by season and expected aerosol type. In addition to reporting uncertainty envelopes, we identify trends and outliers, and investigate their likely causes, with the aim of refining algorithm performance. Overall, about 2/3 of the MISR-retrieved AOT values fall within [0.05 or 20% x AOT] of Aerosol Robotic Network (AERONET). More than a third are within [0.03 or 10% x AOT]. Correlation coefficients are highest for maritime stations (approx.0.9), and lowest for dusty sites (more than approx.0.7). Retrieved spectral slopes closely match Sun photometer values for Biomass burning and continental aerosol types. Detailed comparisons suggest that adding to the algorithm climatology more absorbing spherical particles, more realistic dust analogs, and a richer selection of multimodal aerosol mixtures would reduce the remaining discrepancies for MISR retrievals over land; in addition, refining instrument low-light-level calibration could reduce or eliminate a small but systematic offset in maritime AOT values. On the basis of cases for which current particle models are representative, a second-generation MISR aerosol retrieval algorithm incorporating these improvements could provide AOT accuracy unprecedented for a spaceborne technique.

global meaurement↗

Autonomous Vegetation Cover Scene Classification of EO-1 Hyperion Hyperspectral Data

The Autonomous Sciencecraft Experiment (ASE) is a JPL-led, New Millennium Program mission containing new technology in the form of software to be flown on the Earth Observer-1 (EO-1) satellite in early 2004. This new technology will facilitate an artificially intelligent machine with autonomous science-driven capabilities. Among the ASE flight software is a set of onboard science algorithms designed for autonomous data processing, primarily based on change detection from observation to observation. Using the output from these algorithms, ASE has the ability to autonomously modify the EO-1 observation plan, retargeting itself for a more in-depth observation of a scientific event in progress. Furthermore, intelligent and selective information down-linking will maximize return of the most valuable scientific data. Among the algorithms developed for use on ASE is a Lava-Vegetation (L-V) detection algorithm. This algorithm can effectively identify the initial location and extent of lava and vegetation coverage based on spectral shape. Comparison of several different observations, all classified via this algorithm, can make change detection possible.

Lee, R. J.↗

HACC: Simulating sky surveys on state-of-the-art supercomputing architectures

Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the ‘Dark Universe’, dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers that enable these simulations. In this paper, we report on HACC (Hardware/Hybrid Accelerated Cosmology Code), a recently developed and evolving cosmology N-body code framework, designed to run efficiently on diverse computing architectures and to scale to millions of cores and beyond. HACC can run on all current supercomputer architectures and supports a variety of programming models and algorithms. It has been demonstrated at scale on Cell- and GPU-accelerated systems, standard multi-core node clusters, and Blue Gene systems. HACC’s design allows for ease of portability, and at the same time, high levels of sustained performance on the fastest supercomputers available. We present a description of the design philosophy of HACC, the underlying algorithms and code structure, and outline implementation details for several specific architectures. We show selected accuracy and performance results from some of the largest high resolution cosmological simulations so far performed, including benchmarks evolving more than 3.6 trillion particles.

79 ASTRONOMY AND ASTROPHYSICS↗

Predicting diagnostic error in radiology via eye-tracking and image analytics: Preliminary investigation in mammography

Purpose: The primary aim of the present study was to test the feasibility of predicting diagnostic errors in mammography by merging radiologists’ gaze behavior and image characteristics. A secondary aim was to investigate group-based and personalized predictive models for radiologists of variable experience levels.Methods: The study was performed for the clinical task of assessing the likelihood of malignancy of mammographic masses. Eye-tracking data and diagnostic decisions for 40 cases were acquired from four Radiology residents and two breast imaging experts as part of an IRB-approved pilot study. Gaze behavior features were extracted from the eye-tracking data. Computer-generated and BIRADS images features were extracted from the images. Finally, machine learning algorithms were used to merge gaze and image features for predicting human error. Feature selection was thoroughly explored to determine the relative contribution of the various features. Group-based and personalized user modeling was also investigated.Results: Machine learning can be used to predict diagnostic error by merging gaze behavior characteristics from the radiologist and textural characteristics from the image under review. Leveraging data collected from multiple readers produced a reasonable group model [area under the ROC curve (AUC) = 0.792 ± 0.030]. Personalized user modeling was far more accurate for the more experienced readers (AUC = 0.837 ± 0.029) than for the less experienced ones (AUC = 0.667 ± 0.099). The best performing group-based and personalized predictive models involved combinations of both gaze and image features.Conclusions: Diagnostic errors in mammography can be predicted to a good extent by leveraging the radiologists’ gaze behavior and image content.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Efficient Application of the Factorized form of the Unitary Coupled-Cluster Ansatz for the Variational Quantum Eigensolver Algorithm by Using Linear Combination of Unitaries

The variational quantum eigensolver is one of the most promising algorithms for near-term quantum computers. It has the potential to solve quantum chemistry problems involving strongly correlated electrons with relatively low-depth circuits, which are otherwise difficult to solve on classical computers. The variational eigenstate is constructed from a number of factorized unitary coupled-cluster terms applied onto an initial (single-reference) state. Current algorithms for applying one of these operators to a quantum state require a number of operations that scale exponentially with the rank of the operator. We exploit a hidden SU(2) symmetry to allow us to employ the linear combination of unitaries approach, Our Prepare subroutine uses n+2 ancilla qubits for a rank-n operator. Our Select (U) scheme uses O(n) Cnot gates. This results in a full algorithm that scales like the cube of the rank of the operator n 3 , a significant reduction in complexity for rank five or higher operators. This approach, when combined with other algorithms for lower-rank operators (when compared to the standard implementation), will make the factorized form of the unitary coupled-cluster approach much more efficient to implement on all types of quantum computers.

97 MATHEMATICS AND COMPUTING↗

A Testbed of Parallel Kernels for Computer Science Research

For several decades, computer scientists have sought guidance on how to evolve architectures, languages, and programming models for optimal performance, efficiency, and productivity. Unfortunately, this guidance is most often taken from the existing software/hardware ecosystem. Architects attempt to provide micro-architectural solutions to improve performance on fixed binaries. Researchers tweak compilers to improve code generation for existing architectures and implementations, and they may invent new programming models for fixed processor and memory architectures and computational algorithms. In today's rapidly evolving world of on-chip parallelism, these isolated and iterative improvements to performance may miss superior solutions in the same way gradient descent optimization techniques may get stuck in local minima. In an initial study, we have developed an alternate approach that, rather than starting with an existing hardware/software solution laced with hidden assumptions, defines the computational problems of interest and invites architects, researchers and programmers to implement novel hardware/ software co-designed solutions. Our work builds on the previous ideas of computational dwarfs, motifs, and parallel patterns by selecting a representative set of essential problems for which we provide: An algorithmic description; scalable problem definition; illustrative reference implementations; verification schemes. For simplicity, we focus initially on the computational problems of interest to the scientific computing community but proclaim the methodology (and perhaps a subset of the problems) as applicable to other communities. We intend to broaden the coverage of this problem space through stronger community involvement. Previous work has established a broad categorization of numerical methods of interest to the scientific computing, in the spirit of the NAS Benchmarks, which pioneered the basic idea of a 'pencil and paper benchmark' in the 1990s. The initial result of the more modern study was the seven dwarfs, which was subsequently extended to 13 motifs. These motifs have already been useful in defining classes of applications for architecture-software studies. However, these broad-brush problem statements often miss the nuance seen in individual kernels. For example, the computational requirements of particle methods vary greatly between the naive (but more accurate) direct calculations and the particle-mesh and particle-tree codes. Thus we commenced our study with an enumeration of problems, but then proceeded by providing not only reference implementations for each problem, but more importantly a mathematical definition that allows one to escape iterative approaches to software/hardware optimization. To ensure long term value, we have augmented each of our reference implementations with both a scalable problem generator and a verification scheme. In a paper we have prepared that documents our efforts, we describe in detail this process of problem definition, scalable input creation, verification, and implementation of reference codes for the scientific computing domain. Table 1 enumerates and describes the level of support we've developed for each kernel. We group these important kernels using the Berkeley dwarfs/motifs taxonomy using a red box in the appropriate column. As kernels become progressively complex, they build upon other, simpler computational methods. We note this dependency via orange boxes. After enumeration of the important numerical problems, we created a domain-appropriate high-level definition of each problem. To ensure future endeavors are not tainted by existing implementations, we specified the problem definition to be independent of both computer architecture and existing programming languages, models, and data types. Then, to provide context as to how such kernels productively map to existing architectures, languages and programming models, we produced reference implementations for most of the kernel (see table). These sample codes should be viewed as 'hints,' designed to show how other designers have mapped a problem's operands and operators to existing hardware and software. Since we wanted such implementations to be illustrative, we tried to ensure they were the most straightforward implementation in the easiest to understand languages using familiar architectures. To that end, most of the linear algebra-oriented computations are written in MATLAB using array indexing to process matrices, rather than one-line library calls to compute the same kernel. This ensures that the kernel's computation is explicit and readable in the implementation and not hidden behind a library. For other problems, such as the Barnes-Hut n-Body solver, the implementations were written in pure C, without any supporting library computations. Along this line, we have also created a scalable problem generator to accompany each computation.

97 MATHEMATICS AND COMPUTING↗

Photoinduced Dynamics with Constrained Vibrational Motion: FrozeNM Algorithm

Ab initio molecular dynamics (AIMD) simulation, analyzed in terms of vibrational normal modes, is a widely used technique that facilitates understanding of complex structural motions and coupling between electronic and nuclear degrees of freedom. Usually, only a subset of vibrations is directly involved in the process of interest. The impact of these vibrations can be evaluated by performing AIMD simulations by selectively freezing certain motions. In this paper, we present frozen normal mode (FrozeNM), a new algorithm to apply normal-mode constraints in AIMD simulations, as implemented in the nonadiabatic excited state molecular dynamics code. We further illustrate its capacity by analyzing the impact of normal-mode constraints on the photoinduced energy transfer between polyphenylene ethynylene dendrimer building blocks. Our results show that the electronic relaxation can be significantly slowed down by freezing a well-selected small subset of active normal modes characterized by their contributions in the direction of energy transfer. The application of these constraints reduces the nonadiabatic coupling between electronic excited states during the entire dynamical simulations. Furthermore, we validate reduced dimensionality models by freezing all the vibrations, except a few active modes. Altogether, we consider FrozeNM as a useful tool that can be broadly used to underpin the role of vibrational motion in a studied process and to formulate reduced models that describe essential physical phenomena.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Optimizing wind farm siting to reduce power system impacts of wind variability

We introduce and solve two variants of a biobjective optimization model to reduce the negative impact of wind variability on the power system by strategically locating wind farms. The first model variant considers average changes in wind power over time; the second captures extreme fluctuations in wind power. A complementary set of wind sites is selected with the aim of minimizing both residual demand and the variability in residual demand. Because exact optimization is computationally intensive, we develop two heuristics—forward and backward greedy algorithms—to find approximate solutions. The results are compared with the exact optimization results for a well-selected subset of the data as well as to the results from selecting sites based on average wind alone. The two models are solved using demand data and potential wind sites for the Southwest Power Pool. Though both objectives can be improved by adding more sites, for a fixed number of sites, minimizing residual demand and variability in residual demand are competing objectives. Here, we find an approximate efficient frontier to compare trade-offs between the two objectives. We also vary the parameter in the heuristic that controls how the two objectives are prioritized. For the case study, the backward greedy algorithm is more effective at reducing the wind power variability than the forward greedy algorithm. Furthermore, using the backward algorithm for the full dataset is more effective than solving the exact optimization on a subset of the data when the results are evaluated using the full dataset.

17 WIND ENERGY↗

Robust and Simple ADMM Penalty Parameter Selection

We present a new method for online selection of the penalty parameter for the alternating direction method of multipliers (ADMM) algorithm. ADMM is a widely used method for solving a range of optimization problems, including those that arise in signal and image processing. In its standard form, ADMM includes a scalar hyperparameter, known as the penalty parameter, which usually has to be tuned to achieve satisfactory empirical convergence. In this work, we develop a framework for analyzing the ADMM algorithm applied to a quadratic problem as an affine fixed point iteration. Using this framework, we develop a new method for automatically tuning the penalty parameter by detecting when it has become too large or small. We analyze this and several other methods with respect to their theoretical properties, i.e., robustness to problem transformations, and empirical performance on several optimization problems. Our proposed algorithm is based on a theoretical framework with clear, explicit assumptions and approximations, is theoretically covariant/invariant to problem transformations, is simple to implement, and exhibits competitive empirical performance.

42 ENGINEERING↗

SNAD transient miner: Finding missed transient events in ZTF DR4 using k-D trees

Here we report the automatic detection of 11 transients (7 possible supernovae and 4 active galactic nuclei candidates) within the Zwicky Transient Facility fourth data release (ZTF DR4), all of them observed in 2018 and absent from public catalogs. Among these, three were not part of the ZTF alert stream. Our transient mining strategy employs 41 physically motivated features extracted from both real light curves and four simulated light curve models (SN Ia, SN II, TDE, SLSN-I). These features are input to a k-D tree algorithm, from which we calculate the 15 nearest neighbors. After pre-processing and selection cuts, our dataset contained approximately a million objects among which we visually inspected the 105 closest neighbors from seven of our brightest, most well-sampled simulations, comprising 89 unique ZTF DR4 sources. Our result illustrates the potential of coherently incorporating domain knowledge and automatic learning algorithms, which is one of the guiding principles directing the SNAD team. It also demonstrates that the ZTF DR is a suitable testing ground for data mining algorithms aiming to prepare for the next generation of astronomical data.

79 ASTRONOMY AND ASTROPHYSICS↗

High redshift LBGs from deep broadband imaging for future spectroscopic surveys

Lyman break galaxies (LBGs) are promising probes for clustering measurements at high redshift, z > 2, a region only covered so far by Lyman-α forest measurements. Here, in this paper, we investigate the feasibility of selecting LBGs by exploiting the existence of a strong deficit of flux shortward of the Lyman limit, due to various absorption processes along the line of sight. The target selection relies on deep imaging data from the HSC and CLAUDS surveys in the g, r, z and u bands, respectively, with median depths reaching 27 AB in all bands. The selections were validated by several dedicated spectroscopic observation campaigns with DESI. Visual inspection of spectra has enabled us to develop an automated spectroscopic typing and redshift estimation algorithm specific to LBGs. Based on these data and tools, we assess the efficiency and purity of target selections optimised for different purposes. Selections providing a wide redshift coverage retain 57% of the observed targets after spectroscopic confirmation with DESI, and provide an efficiency for LBGs of 83 ± 3%, for a purity of the selected LBG sample of 90 ± 2%. This would deliver a confirmed LBG density of ~ 620 deg$^{-2}$ in the range 2.3 < z < 3.5 for a r-band limiting magnitude r < 24.2. Selections optimised for high redshift efficiency retain 73% of the observed targets after spectroscopic confirmation, with 89 ± 4% efficiency for 97 ± 2% purity. This would provide a confirmed LBG density of ~ 470 deg$^{-2}$ in the range 2.8 < z < 3.5 for a r-band limiting magnitude r < 24.5.A preliminary study of the LBG sample 3d-clustering properties is also presented and used to estimate the LBG linear bias. A value of b$_{LBG}$ = 3.3 ± 0.2 (stat.) is obtained for a mean redshift of 2.9 and a limiting magnitude in r of 24.2, in agreement with results reported in the literature.

79 ASTRONOMY AND ASTROPHYSICS↗

Adaptive local learning in sampling based motion planning for protein folding

Background: Simulating protein folding motions is an important problem in computational biology. Motion planning algorithms, such as Probabilistic Roadmap Methods, have been successful in modeling the folding landscape. Probabilistic Roadmap Methods and variants contain several phases (i.e., sampling, connection, and path extraction). Most of the time is spent in the connection phase and selecting which variant to employ is a difficult task. Global machine learning has been applied to the connection phase but is inefficient in situations with varying topology, such as those typical of folding landscapes. Results: We develop a local learning algorithm that exploits the past performance of methods within the neighborhood of the current connection attempts as a basis for learning. It is sensitive not only to different types of landscapes but also to differing regions in the landscape itself, removing the need to explicitly partition the landscape. We perform experiments on 23 proteins of varying secondary structure makeup with 52–114 residues. We compare the success rate when using our methods and other methods. We demonstrate a clear need for learning (i.e., only learning methods were able to validate against all available experimental data) and show that local learning is superior to global learning producing, in many cases, significantly higher quality results than the other methods. Conclusions: We present an algorithm that uses local learning to select appropriate connection methods in the context of roadmap construction for protein folding. Our method removes the burden of deciding which method to use, leverages the strengths of the individual input methods, and it is extendable to include other future connection methods.

Mathematical & Computational Biology↗

A technique for pole-zero placement for dual-input control systems

A technique is presented for determining state variable feedback gains that will place both the poles and zeros of a selected transfer function of a dual-input control system at pre-determined locations in the s-plane. Leverrier's algorithm is used to determine the numerator and denominator coefficients of the closed-loop transfer function as functions of the feedback gains. The values of gain that match these coefficients to those of a pre-selected model are found by solving two systems of linear simultaneous equations. The algorithm has been used in a computer simulation of the CH-47 helicopter to control longitudinal dynamics.

Reid, G. F.↗

An evaluation of low-cost heuristics for matrix bandwidth and profile reductions

Hundreds of heuristics have been proposed to resolve the problems of bandwidth and profile reductions since the 1960s. We found 132 heuristics that have been applied to these problems in reviews of the literature. Among them, 14 were selected for which no other simulation or comparison revealed that the heuristic could be superseded by any other algorithm in the analyzed articles with respect to bandwidth or profile reduction. We also considered the computational costs of the heuristics during this process. Therefore, these 14 heuristics were selected as potentially being the best low-cost methods to solve the bandwidth and/or profile reduction problems. Results of the 14 selected heuristics are evaluated in this work. For evaluation on the set of test problems, a metric based on the relative percentage distance to the best possible bandwidth or profile is proposed. The most promising heuristics for several application areas are identified. Moreover, it was found that the FNCHC and GPS heuristics showed the best overall results in reducing the bandwidth of symmetric and asymmetric matrices among the evaluated heuristics, respectively. In addition, the NSloan and MPG heuristics showed the best overall results in reducing the profile of symmetric and asymmetric matrices among the heuristics among the evaluated heuristics, respectively.

97 MATHEMATICS AND COMPUTING↗

A Framework Including Recombination for Analyzing the Dynamics of Within-Host HIV Genetic Diversity

This paper presents a novel population genetic model and a computationally and statistically tractable framework for analyzing within-host HIV diversity based on serial samples of HIV DNA sequences. This model considers within-host HIV evolution during the chronic phase of infection and assumes that the HIV population is homogeneous at the beginning, corresponding to the time of seroconversion, and evolves according to the Wright-Fisher reproduction model with recombination and variable mutation rate across nucleotide sites. In addition, the population size and generation time vary over time as piecewise constant functions of time. Under this model I approximate the genealogical and mutational processes for serial samples of DNA sequences by a continuous coalescent-recombination process and an inhomogeneous Poisson process, respectively. Based on these derivations, an efficient algorithm is described for generating polymorphisms in serial samples of DNA sequences under the model including various substitution models. Extensions of the algorithm are also described for other demographic scenarios that can be more suitable for analyzing the dynamics of genetic diversity of other pathogens in vitro and in vivo . For the case of the infinite-sites model, I derive analytical formulas for the expected number of polymorphic sites in sample of DNA sequences, and apply the developed simulation and analytical methods to explore the fit of the model to HIV genetic diversity based on serial samples of HIV DNA sequences from 9 HIV-infected individuals. The results particularly show that the estimates of the ratio of recombination rate over mutation rate can vary over time between very high and low values, which can be considered as a consequence of the impact of selection forces.

60 APPLIED LIFE SCIENCES↗

Improved learning from data competitions through strategic design of training and test data sets

Leveraging the depth and breadth of solutions generated through crowdsourcing can be a powerful accelerator to method development for high consequence problems. While data science competitions have become quite popular and prevalent, particularly in supervised learning formats, their implementations by the host are highly variable. Without careful planning, a supervised learning competition is vulnerable to overfitting, where the winning solutions are so closely tuned to the particular set of provided data that they cannot generalize to the general underlying problem of interest to the host. This paper outlines important considerations for strategically designing relevant and informative data sets to maximize the learning outcome from hosting a competition. These include: precisely defining the scope of the problem, encouraging participation by competitors from diverse technical backgrounds, specifying the most interesting solution space in order to encourage improvement and distinguish between competitors, strategically generating data sets that enable testing for interpolation and extrapolation to new scenarios of interest, leveraging design of experiment principles for strategic data design while preventing unintentional artifacts in the competition data set that an algorithm could exploit without addressing the real problem of interest, and carefully designing the leaderboard scoring metric to select top solutions that exactly match the overall competition goals. The methods are illustrated with a recently completed competition in the context of urban radiological search to evaluate algorithms capable of detecting, identifying, and locating radioactive materials. Simulated data were used in the urban search competition. Ideas for using measured real data in competitions are also suggested

97 MATHEMATICS AND COMPUTING↗