Gradient Descent

Gradient descent is an optimization algorithm used to train neural networks, the purpose of this algorithm is to find the best values for the parameters W and b that minimize the loss function.

With the code below we change the value of W:

W = W - learning_rate * dW

We need the derivative of W dW and a hyperparameter called learning rate.

When we are training a neural network we compute the gradient descent algorithm multiple times to minimize the loss function.

If the loss function value increases this algorithm will decrease the value of W, whereas if the loss function value decreases the algorithm will increase the value of W.

if the loss function value increased then dW is positive as a result when we multiple dW by the learning rate which is negative we end up with a negative value that decreases the value of W:

W = W - 0.003 * 0.04

if the loss function value decreased then dW is negative as a result when we multiple dW by the learning rate we end up with a positive value that increases the value of W:

W = W - 0.003 * -0.06

The learning rate is a hyper parameter that says how much the value of W will increment or decrement. We compute the derivate of W to know how the change in W affects the final value of the loss function.

We can illustrate the gradient descent algorithm as follows:

The yellow star indicates the position where the loss function minimizes its value and the color points are the path that W has to follow to reach this position.

We repeat this process for the parameter b as well.

Backpropagation

As we have seen in the previous section, we need the derivatives of W and b to perform the gradient descent algorithm.

In python we use the code below to compute the derivatives of a neural network with two hidden layers and the sigmoid activation function. Usually we call this the backpropagation step:

dZ2 = A2 - Y

dW2 = (dZ2 * A1)

db2 = dZ2

dZ1 = dW2 * dZ2 * primesigmoid(Z1)

dW1 = dZ1 * X

db1 = dZ1

In the following section I will explain the maths behind this algorithm.

Derivatives

Before getting to the algorithm we need to know some concepts about derivatives:

Derivative

A derivative measures the change of the function value (in this case the loss function) with respect to a change in its argument (in this case W2, W1 and b2, b2).

Partial derivative

We compute a derivative of a function with respect to its variable, usually is x, however if the function has two or more variables we compute its derivative with respect to one of those variables and the remaining variables held constant, we call this a partial derivative.

Chain rule

This rule helps us when we want to compute a derivative of a function with respect to one variable but the function and the variable are far from each other. In this case if we want the derivatives of the loss function with respect to W1, W2, b1, b2 we need to chain the functions L, a2, z2, a1, z1 to create a path between W1, W2, b1, b2 and the loss function, therefore we need to compute the derivatives of these functions.

To represent a derivate we will use the syntaxis dx, d_L_/d_a_: this means the derivative of L with respect to a.

The neural network

Z1 = W1 * x + b1 

a1 = sigmoid(Z1)

Z2 = W2 * a1 + b2

a2 = sigmoid(Z2)

pred_y = L(a2, y)

Loss function derivative (L)

As the name suggest in the backpropagation algorithm we start computing the derivative of the last function, in this case the loss function:

L = yIn(a2) + (1 - y)In(1 - a2)

Here we have a partial derivative since the function has two variables (a2, y), we want the derivative of this function with respect to a2 that is the same than: d_L_/d_a2_.

First we compute the derivative of the first term yIn(a2)

The derivative of In(u) is 1/u

1y/a2

How the y variable multiples In(u) we keep y and we know than 1 * y is the same than just y so at the end we have:

y/a2

The second term is (1 - y)In(1 - a2), as we can see is the same, the derivative of In(u) is 1/u so:

(1-y) / (1-a2)

The final result is:

dL/da2 = y/a2 + (1 - y)/(1 - a2)

Second sigmoid function derivative

This is the hardest derivative of the neural network, we know the sigmoid function:

1/1 + e^-z

The derivative of this function is:

da2/dZ2 = a2(1 - a2)

if you want a more complete explanation you can read this post.

we know that:

a2 = sig(Z2)

In the backpropagation code this derivative is:

primesigmoid(Z1)

Second lineal function derivative (Z2)

Here we need to apply the chain rule and join the two previous derivatives:

dL/dZ2 = dL/da2 * da2/dZ2

dL/dZ2 = (y/a2 + (1 - y)/(1 - a2)) * (a2(1 - a2))

The result is:

A2 - Y

Now we have the first derivative that appears in the backpropagation code:

dZ2 = A2 - Y

W2 Derivative

Here we have another partial derivative d_Z2_/d_W2_ but in this case we have three variables (W2, b2, a2).

Z2 = W2 * a1 + b2

We will compute the derivative of z2 with respect to W2 then a1 and b2 will held constant:

The derivative of a constant is 0, b2 = 0

W2 * a1 + 0

The derivative of a multiplication between a constant and a variable is just the constant:

dZ2/dW2 = a1

Now we want to know how the change in W2 affects the loss function L final value, therefore we need d_L_/d_W2_, we can apply the chain rule to compute this derivative.

since d_L_/d_Z2_ is already connected to the loss function, we only need to multiply d_Z2_/d_W2_ by d_L_/d_Z2_:

dL/dW2 = dZ2/dW2 * dL/dZ2

dL/dW2 = a1 * (a2 - y)

In the backpropagation code we have:

dW2 = (dZ2 * A1)

And we know that:

dZ2 = A2 - Y

b2 Derivative

We have the same function:

Z2 = W2 * a1 + b2

But in this case we want the derivative of Z2 with respect to b2 dZ2/db2, therefore now W2 is a constant:

The derivative of a constant is 0, W2 = 0

0 * a1 + b1

0 * a1 = 0

0 + b1

The derivative of a variable is its constant:

dZ2/db2 = 1

Now we want to know how the change in b2 affects the loss function value, therefore we compute d_L_/d_b2_. Again we need the chain rule:

dL/db2 = dZ2/db2 * dL/dZ2

dL/db2 = 1 * (a2 - y)

dL/db2 = (a2 - y)

In the backpropagation code it appears like:

db2 = dZ2

And we know that:

dZ2 = A2 - y

First sigmoid function derivative

This derivative is the same than the second sigmoid function but in this case the variable is a1

da1/dZ1 = a1(1 - a1)

First lineal function derivative (Z1)

The first lineal function is:

Z1 = W1 * x + b1

We use the chain rule to compute d_L_/d_Z1_

dL/dZ1 = dZ2/dW2 * dZ2/db2 * da1/dZ1

dL/dZ1 =  a1 * (a2 - y) * a1(1 - a1)

In the backpropagation code it is:

dZ1 = dW2 * dZ2 * primesigmoid(Z1)]

W1 Derivative

The process is the same than dW2 but now the function is:

Z1 = W1 * x + b1

First we compute d_Z1_/d_W1_

b1 = 0

W1 * x + 0

dZ1/dW1 = x

Now we need to know how the change in W1 affects the loss function value, then we compute d_L_/d_W1_:

dL/dW1 = dL/dZ1 * dZ1/dW1

dL/dW1 = (a1 * (a2 - y) * (a2 - y) * a1(1 - a1)) * x

In the backpropagation code:

dW1 = dZ1 * X

b1 Derivative

Again the process is the same than db2:

Z1 = W1 * x + b1

dZ1/db1 = 1

We apply the chain rule to compute dL/db1

dL/db1 = dZ1/db1 * dL/dZ1

dL/db1 = 1 * ((a1 * (a2 - y) * (a2 - y))

In the backpropagation code:

db1 = dZ1