Straightforward as it may seem, the idea developed and refined in the previous chapters has serious practical difficulties. We shall look into them in this chapter.

C3.1 Let's Start Dissecting the Cost vs. Weight Equation

The equation that opened the gate of Sesame (Simsim) for us has all the potential to lead us astray and fruitless. As innocent and self-explanatory this equation may look, it has a lot of mischief up its sleeve. Let's look at it again.

$$\boxed{{\displaystyle \begin{array}{rl}& \text{Rate of change of }C\text{ w.r.t. any }w\\ & =\text{Output of the }\mathit{\text{in}}\text{ node}\\ & \times \text{Rate of change of }C\text{ w.r.t. output of the }\mathit{\text{out}}\text{ node}\\ & \times \text{how rapidly }\sigma \text{ is changing at the }\mathit{\text{out}}\text{ node}\end{array}}}$$

$$\begin{array}{}\text{(1)}& \text{Or, }\frac{\partial C}{\partial w}& =a.\frac{\partial C}{\partial \sigma }.\frac{\partial \sigma }{\partial z}=a.\frac{\partial C}{\partial \sigma }.{\sigma }^{\prime }(z)\end{array}$$

Before we even look at the potential issues this equation is hinting at, there is something that needs to be clarified.

C3.2 Reconsidering Our Choice of the Cost Function

In the first chapter, we saw how something like a mean square error or a quadratic distance function is an acceptable choice for the cost function. These generally are and do the job nicely in some cases, but there are quite better candidates available. E.g. the quadratic cost function becomes non-convex when used with the sigmoid activation function, which means there are multiple minima which can be a real issue for our gradient descent function. Also, it does not punish the the misclassifications as much as we would want to. The nail in the coffin comes with the amazing discovery that by having it replaced by something carefully chosen, we can get rid of another unrelated unwanted effect - the slow learning because of ${\sigma }^{\prime }(z^L)$. See below. And after that, if you want to look at a formal reasoning of settling on to the Cross Entropy cost function, check the supplemental section The Cross Entropy Cost Function^↕.

C3.3 Slow Learning Because of ${\sigma }^{\prime }(z^L)$, or $\partial \sigma /\partial z^L$

Summarily, we get very slow learning for weights connected to such nodes, i.e. for nodes whose output are close to 0 or 1. And, since in classification problems, the correct outputs are, in fact, close to 0 or 1, it means that we will have very slow learning for sure.

This problem is not simply solved by changing $\sigma$ for the choice of activation function - many other functions will cause the same effect. Instead we go about changing the $C$ function from quadratic to something else! Yes, counter-intuitive as it may seem, changing $C$ is not that a big deal (especially when we have other reasons as well, see C3.2 Reconsidering Our Choice of the Cost Function^↕), and there exist beautiful functions out there using which for $C$ simply cancels out the ${\sigma }^{\prime }(z)$ factor in eqn. $(3)$^↕! So, by changing $C$, we may be able to kill two birds with a stone. Let's see how.

If you want to be able to make sense of the new cost function I'm going to present below, you need to be familiar with the Information Theory and the corresponding Entropy equation. In any case, I'll highly recommend a reading of this once when you have time - Information Theory - Rationale Behind Using Logarithm for Entropy, and Other Explanations.

For the sake of avoiding clutter, I've placed the reasoning for settling on the function shown below in the supplemental section The Cross Entropy Cost Function^↕.

$$C(y,a)=-\frac{1}{n}\sum _x\left[y\ln (a)+(1-y)\ln (1-a)\right]$$

Ok, a little peek into this function shows that it is convex (has one minimum), and that it is good in punishing misclassifications, but what about the slowness of learning being caused by the very small values of ${\sigma }^{\prime }(z)$?

$$\begin{array}{rl}{\delta }^L& ={\mathrm{\nabla }}_{a}C\odot {\sigma }^{\prime }(z^L)\\ & =\frac{\partial }{\partial a}(-\frac{1}{n}\sum _x\left[y\ln (a)+(1-y)\ln (1-a)\right]).{\sigma }^{\prime }(z)& & \dots \text{omitting the L superscript}\\ & =-\frac{1}{n}\frac{\partial }{\partial \sigma }\sum _x\left[y\ln (\sigma )+(1-y)\ln (1-\sigma )\right].{\sigma }^{\prime }(z)& & \dots \text{using }a=\sigma =\sigma (z)\\ & =-\frac{1}{n}\sum _x\left[\frac{y}{\sigma }-\frac{(1-y)}{(1-\sigma )}\right].{\sigma }^{\prime }(z)& & \\ \text{(4)}& & =\frac{1}{n}\sum _x\frac{(\sigma -y)}{\sigma (1-\sigma )}.{\sigma }^{\prime }(z)& & \text{Now, }{\sigma }^{\prime }(z)& =\frac{\partial }{\partial z}\left(\frac{1}{1+e^{-z}}\right)& & \\ & =\frac{e^{-z}}{(1+e^{-z}{)}^2}=\sigma .\frac{e^{-z}}{(1+e^{-z})}=\sigma .\frac{1+e^{-z}-1}{(1+e^{-z})}& & \\ & =\sigma .(1-\sigma )& & \\ & \text{Substituting this value of }{\sigma }^{\prime }(z)\text{ in eqn. }(4)\text{ above:}\\ {\delta }^L& =\frac{1}{n}\sum _x\frac{(\sigma -y)}{\sigma (1-\sigma )}.{\sigma }^{\prime }(z)& & \\ & =\frac{1}{n}\sum _x\frac{(\sigma -y)}{\cancel{\sigma (1-\sigma )}}.\cancel{\sigma (1-\sigma )}& & \\ & =\frac{1}{n}\sum _x(\sigma (z)-y)\end{array}$$

This is magic! The error term ($\delta$) now depends only on $\sigma (z)-y=a-y$. $a-y$ is nothing else but the error in the output, and so, ($\delta$), and hence $\frac{\partial C}{\partial w}$ is free of ${\sigma }^{\prime }(z)$, and of the curse of it being small for extreme values of $\sigma$. Our learning doesn't need to slowdown.

So, we have solved at least one problem with a suitable choice of a cost function.

C3.4 $C$ Still Haunts... Let's Summon 'Stochastic Gradient Descent'

So much for the choice of cost function. But a significant issue still remains that is related to $C$ - it is the calculation of $\mathrm{\nabla }C$, which is needed for calculating ${\delta }^L$. More specifically, it is the $\sum _x$ (where the range of $x$ is the complete training set) that appears in $\mathrm{\nabla }C$. In order to understand this, if you may, it will help us to revise the difference between features and training inputs.

$C^p$ - for an output feature $y^p$ - is calculated using the predicted and the provided values for that output feature for all the training examples. E.g. it can be a sum or average of $(y_j^p-{\hat{y}}_j^p{)}^2$ where $j$ runs from $1$ to $R$. Let's, for simplicity, assume our output has only one feature. Now, $C$ becomes average (say) of $(y_j-{\hat{y}}_j{)}^2$ for $1<j<=R$. We actually don't need $C$ as such, we need $\partial C_j/\partial a_j^L$ - denoted by (where $j$ signifies one example at a time), but the average (or summation) part is still there.

Now, for getting better and better results, we want the training data as big as possible, but that immediately has a detrimental effect on the calculation of $C$. In every pass (iteration), we need to feed all the features present in the input of all the training examples, and use the predicted results along with the given output in all the training examples in order to calculate $C$. This can and does cause serious slowness.

To combat this issue, we bring in stochastic gradient descent. In this, instead of running $j$ from $1$ to $R$, we run it from $1$ to $s$ where $s$ is a randomly chosen subset from $R$ entities - hoping that this sample will do some justice in representing the complete training set. The sample cannot, of course, represent the complete set exactly. We make up for this by running more iterations of the complete learning process. Thus, we have, roughly speaking, turned spatial complexity of our formulae into a temporal one - in each iteration there are less inputs to be considered for calculating $\mathrm{\nabla }C$ and other things, but there are more iterations. We keep on repeating the process until the samples collectively exhaust the training set - and then we say that an epoch of the training is complete.

Once again, in normal gradient descent, we would feed all the inputs to our neural network at once, calculate $\mathrm{\nabla }C$, use that to improve our $w$'s and so on. In stochastic gradient descent, we feed inputs from a small randomly chosen subset of the training set to the network, learn the $w$', and move on to next subset until we exhaust the subset.

C3.5 Vanishing and Exploding Gradients

Coming soon...