Batch and Layer normalization

Layer Normalization and Group Normalization

Layer Normalization

Layer normalization is a technique used to normalize the activations of a neural network across all channels for each individual observation. This method is particularly useful for stabilizing the learning process and improving the convergence speed of the network. Unlike batch normalization, which normalizes across the batch dimension, layer normalization operates on each observation independently, making it more suitable for recurrent neural networks or scenarios where batch sizes are small.

In layer normalization, the mean and variance are computed across the features of each observation, and the normalized activations are scaled and shifted using learnable parameters $\gamma$ (scale) and $\beta$ (shift). This process ensures that the normalized activations have a mean of zero and a variance of one, which helps in maintaining the stability of the network during training.

Group Normalization

Group normalization (GN) is another normalization technique that divides the channels of the input data into groups and normalizes the features within each group independently. This method is designed to address the limitations of batch normalization, especially when the batch size is small. Unlike batch normalization, GN does not rely on the batch dimension and is thus invariant to batch sizes, making it more stable and effective in various training scenarios.

The process of group normalization involves the following steps:

1. Grouping Channels: The channels of the input data are divided into predefined groups.
2. Computing Statistics: For each group, the mean and variance are computed.
3. Normalization: The activations within each group are normalized by subtracting the group mean and dividing by the group standard deviation.
4. Scaling and Shifting: The normalized activations are then scaled and shifted using learnable parameters $\gamma$ and $\beta$.

Group normalization can be seen as a generalization of layer normalization and instance normalization:

• Layer Normalization: When all channels are grouped together into a single group.
• Instance Normalization: When each channel is treated as a separate group.

Applications and Benefits

• Layer Normalization: Often used in recurrent neural networks and transformers, where the independence from batch size is crucial.
• Group Normalization: Particularly useful in convolutional neural networks (CNNs) when training with small batch sizes, as it provides more stable training and faster convergence compared to batch normalization.

Example of Group Normalization in MATLAB

% Create a group normalization layer that normalizes incoming data across three groups of channels
layer = groupNormalizationLayer(3, 'Name', 'groupnorm');

% Example of including a group normalization layer in a layer array
layers = [
    imageInputLayer([28 28 3])
    convolution2dLayer(5, 20)
    groupNormalizationLayer(4)
    reluLayer
    maxPooling2dLayer(2, 'Stride', 2)
    fullyConnectedLayer(10)
    softmaxLayer
];

In summary, both layer normalization and group normalization are effective techniques for normalizing activations in neural networks, each with its specific use cases and advantages. Layer normalization is ideal for scenarios where batch size independence is required, while group normalization excels in stabilizing training for CNNs with small batch sizes[1][2].

Citations: [1] https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.groupnormalizationlayer.html [2] https://paperswithcode.com/method/group-normalization [3] https://www.baeldung.com/cs/group-normalization [4] https://towardsdatascience.com/different-normalization-layers-in-deep-learning-1a7214ff71d6?gi=b8778cb8e35c [5] https://www.pinecone.io/learn/batch-layer-normalization/ [6] https://www.linkedin.com/pulse/understanding-batch-normalization-layer-group-implementing-pasha-s [7] https://towardsdatascience.com/what-is-group-normalization-45fe27307be7 [8] https://www.youtube.com/watch?v=1JmZ5idFcVI