=================================================================================

Non-saturating, non-linear activation functions such as ReLUs can be used to fix the successive reduction of signal vs. noise caused by each additional layer in the network during the training process.

The ReLU activation function is defined as follows:

          --------------------------------- [4082a]

In other words, the output is zero for negative input values and equal to the input for non-negative values.

Figure 4082. ReLU (Code).

This script below creates an example model that includes convolutional layers with ReLU activations, showing how ReLU operates within the encoder part of a U-net (including first convolutional block, second convolutional block, and max pooling), and then let us visualize the feature maps before and after applying the ReLU activation:

Input:

Output (feature maps before ReLU activation):

Output (feature maps after ReLU activation):

By comparing the feature maps before and after ReLU activation, we can see how ReLU modifies the activations, helping the model to focus on important features while discarding negative activations. This process is crucial for enabling the model to learn and generalize from the data.

Other output:

In the script above, a max-pooling layer is added after the second ReLU activation to simulate downsampling. It passes the input image through the model to obtain the feature maps at each layer, and then the feature maps before and after applying ReLU activation are visualized using matplotlib.

The output feature maps before and after ReLU activation represent the intermediate processed data at different stages of the neural network:

• Convolutional Layers:
  • Convolutional layers in a neural network apply multiple filters (kernels) to the input image or feature maps from the previous layer.
  • Each filter detects different features such as edges, textures, or more complex patterns.
  • The number of filters in each convolutional layer determines the number of output feature maps.
  • For example, if a convolutional layer has 16 filters, it will produce 16 feature maps. In fact, in the case above, we indeed have 16 feature maps before ReLU activation, which means that the convolutional layer used 16 different filters. Each filter is a small matrix (kernel) that is applied to the input image or previous layer's feature maps to detect specific patterns.
    • Filters (Kernels):
      • Filters are small matrices, typically of size 3x3 or 5x5, that slide over the input feature maps.
      • Each filter is designed to detect specific features such as edges, textures, or more complex patterns.
      • During training, the network learns the values of these filters to best detect the relevant features in the data.
    • Feature Maps:
      • Each filter produces one feature map by convolving (sliding and applying) the filter across the input and performing element-wise multiplication and summation.
      • he resulting feature maps highlight the presence of the feature detected by the corresponding filter at different spatial locations in the input.
    • Example of Filters and Their Function:
      • Edge Detection Filters: Some filters may detect horizontal, vertical, or diagonal edges in an image.
      • Texture Filters: Others may highlight textures or repeated patterns.
      • Complex Patterns: In deeper layers of the network, filters may detect more complex structures like shapes, parts of objects, or even whole objects.

• Capturing Different Features:
  • Each feature map is the result of applying a specific filter to the input. This means each feature map highlights different aspects of the input image.
  • Early layers might capture basic features like edges and colors, while deeper layers capture more complex patterns and structures.

• Before ReLU Activation:
  • The feature maps before ReLU activation are the raw output of the convolutional layers.
  • These values can be positive, negative, or zero. Negative values indicate that the features detected by the filter are not strongly present in the input at that location.
  • The range of values can be quite large, depending on the filter weights and the input image.

• After ReLU Activation:
  • ReLU (Rectified Linear Unit) activation function applies a non-linearity to the feature maps.
  • It sets all negative values to zero while keeping positive values unchanged. This helps introduce non-linearity into the model, allowing it to learn more complex patterns.
  • The feature maps after ReLU activation contain only non-negative values, making the model more robust to various inputs and aiding in faster convergence during training.

• Selection and visualization of feature maps before and after the ReLU activation:
  • To select and visualize feature maps before and after the ReLU activation in a convolutional neural network, we need to create a model that outputs the intermediate layers' feature maps.

The output table of the script above shows the summary of the model architecture and the number of parameters in each layer:

• Input Layer:
  • Layer Name: input_1
  • Type: InputLayer
  • Output Shape: (None, 128, 128, 3) (This indicates that the input is expected to be a batch of RGB images with dimensions 128x128 pixels.)
  • Param #: 0 (There are no parameters associated with the input layer.)

• First Convolutional Layer:
  • Layer Name: conv1
  • Type: Conv2D
  • Output Shape: (None, 128, 128, 32) (The output is a batch of feature maps with 32 channels, each of size 128x128.)
  • Param #: 896 (This is calculated as (3*3*3*32) + 32 where 3*3 is the kernel size, 3 is the number of input channels, 32 is the number of filters, and 32 is the bias term for each filter.)

• First ReLU Activation:
  • Layer Name: tf.nn.relu
  • Type: TFOpLambda
  • Output Shape: (None, 128, 128, 32) (The shape remains the same as the input to ReLU.)
  • Param #: 0 (ReLU has no trainable parameters.)

• Second Convolutional Layer:
  • Layer Name: conv2
  • Type: Conv2D
  • Output Shape: (None, 128, 128, 64) (The output is a batch of feature maps with 64 channels, each of size 128x128.)
  • Param #: 18496 (This is calculated as (3*3*32*64) + 64 where 3*3 is the kernel size, 32 is the number of input channels, 64 is the number of filters, and 64 is the bias term for each filter.)

• Second ReLU Activation:
  • Layer Name: tf.nn.relu_1
  • Type: TFOpLambda
  • Output Shape: (None, 128, 128, 64) (The shape remains the same as the input to ReLU.)
  • Param #: 0 (ReLU has no trainable parameters.)

• Max Pooling Layer:
  • Layer Name: pool1
  • Type: MaxPooling2D
  • Output Shape: (None, 64, 64, 64) (The spatial dimensions are halved to 64x64, but the number of channels remains 64.)
  • Param #: 0 (Max pooling has no trainable parameters.)

• Total Parameters
  • Total params: 19392 (This is the total number of trainable parameters in the model.)
  • Trainable params: 19392 (All parameters are trainable in this model.)
  • Non-trainable params: 0 (There are no non-trainable parameters in this model.)

• Interpretation of Feature Maps
  The feature maps generated at each stage in the model can be interpreted as follows:
  • Feature Maps Before ReLU Activation:
    • These show the raw outputs of the convolutional layers (conv1 and conv2) before any non-linearity is applied.
    • The values in these feature maps can be positive or negative, and they represent the result of applying convolutional filters to the input image or the previous layer's output.
  • Feature Maps After ReLU Activation:
    • These show the outputs of the ReLU activations applied to the convolutional layer outputs.
    • ReLU sets all negative values to zero, introducing non-linearity into the model. This helps in learning complex patterns.
    • The values in these feature maps are either zero or positive.

============================================