Layer Normalization

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton submitted Layer Normalization on July 21, 2016. The paper proposed computing normalization statistics across the summed inputs to the neurons in a layer for a single training case, rather than across a mini-batch, and described the method as straightforward to apply to recurrent neural networks.