Title: Exponential convergence rates for Batch Normalization

Abstract: Batch Normalization is a normalization technique that has been used in training deep Neural Networks since 2015. In spite of its empirical benefits, there exists little theoretical understanding as to why this normalization technique speeds up learning. From a classical optimization perspective, we will discuss specific problem instances in which we can prove that Batch Normalization can accelerate learning, and how this acceleration is due to the fact that Batch Normalization splits the optimization task into optimizing length and direction of parameters separately.