RandomOut: Using a convolutional gradient norm to rescue convolutional filters
Abstract
Filters in convolutional neural networks are sensitive to their initialization. The random numbers used to initialize filters are a bias and determine if you will "win" and converge to a satisfactory local minimum so we call this The Filter Lottery. We observe that the 28x28 InceptionV3 model without Batch Normalization fails to train 26% of the time when varying the random seed alone. This is a problem that affects the trial and error process of designing a network. Because random seeds have a large impact it makes it hard to evaluate a network design without trying many different random starting weights. This work aims to reduce the bias imposed by the initial weights so a network converges more consistently. We propose to evaluate and replace specific convolutional filters that have little impact on the prediction. We use the gradient norm to evaluate the impact of a filter on error, and reinitialize filters when the gradient norm of its weights falls below a specific threshold. This consistently improves accuracy on the 28x28 InceptionV3 with a median increase of +3.3%. In effect our method RandomOut increases the number of filters explored without increasing the size of the network. We observe that the RandomOut method has more consistent generalization performance, having a standard deviation of 1.3% instead of 2% when varying random seeds, and does so faster and with fewer parameters.
 Publication:

arXiv eprints
 Pub Date:
 February 2016
 arXiv:
 arXiv:1602.05931
 Bibcode:
 2016arXiv160205931C
 Keywords:

 Computer Science  Computer Vision and Pattern Recognition
 EPrint:
 Extended version of the ICLR 2016 workshop track paper