Improved learning from data competitions through strategic design of training and test data sets
Leveraging the depth and breadth of solutions generated through crowdsourcing can be a powerful accelerator to method development for high consequence problems. While data science competitions have become quite popular and prevalent, particularly in supervised learning formats, their implementations by the host are highly variable. Without careful planning, a supervised learning competition is vulnerable to overfitting, where the winning solutions are so closely tuned to the particular set of provided data that they cannot generalize to the general underlying problem of interest to the host. This paper outlines important considerations for strategically designing relevant and informative data sets to maximize the learning outcome from hosting a competition. These include: precisely defining the scope of the problem, encouraging participation by competitors from diverse technical backgrounds, specifying the most interesting solution space in order to encourage improvement and distinguish between competitors, strategically generating data sets that enable testing for interpolation and extrapolation to new scenarios of interest, leveraging design of experiment principles for strategic data design while preventing unintentional artifacts in the competition data set that an algorithm could exploit without addressing the real problem of interest, and carefully designing the leaderboard scoring metric to select top solutions that exactly match the overall competition goals. The methods are illustrated with a recently completed competition in the context of urban radiological search to evaluate algorithms capable of detecting, identifying, and locating radioactive materials. Simulated data were used in the urban search competition. Ideas for using measured real data in competitions are also suggested