Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training
Deep ReLU networks have exponential expressive potential but behave identically to their shallow counterparts under random initialization. This paper introduces a novel training paradigm that constrains weights into mathematically optimized patterns. This enables the resulting network to make exponential use of its depth.