2_Neural_Network_Tutorial_Matrix_Representations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Matrix representations\n",
    "\n",
    "## Matrix representations - input to the network\n",
    "\n",
    "Suppose an input has $d_{i}$ dimensions. (Remember that the input has been normalized to range between 0 and 1.)\n",
    "\n",
    "Then each input would be:\n",
    "\n",
    "$$X \\; (without bias) _{1{\\times}d_{i}} = \\left[ \\begin{array}{c} x_{0} & x_{1} & \\cdots & x_{(d_{i}-1)} \\end{array} \\right] _{1{\\times}d_{i}}$$\n",
    "\n",
    "After adding the bias term,\n",
    "\n",
    "$$X_{1{\\times}(d_{i}+1)} = \\left[ \\begin{array}{c} 1 & X_{1{\\times}d_{i}} \\end{array} \\right] _{1{\\times}(d_{i}+1)}$$\n",
    "\n",
    "For example, one of the data points given above to make a logic gate was $(0,1)$. Here, $X = \\left[ \\begin{array}{c} 1 & 0 & 1 \\end{array} \\right]_{1{\\times}(2+1)}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we provide $n$ $d_{i}$-dimensional data points. For the first layer of neurons, we can make an input matrix of $n{\\times}d_{i}$ dimension.\n",
    "\n",
    "$$X^{(1)}_{n{\\times}(d_{i}+1)} = \n",
    "\\left[ \\begin{array}{c} 1 & _{(0)}X \\\\ 1 & _{(1)}X \\\\ \\vdots & \\vdots \\\\ 1 & _{(n-1)}X \\end{array} \\right] _{n{\\times}(d_{i}+1)}\n",
    "=\n",
    "\\left[ \\begin{array}{c} \n",
    "1 & _{(0)}x_{0} & _{(0)}x_{1} & _{(0)}x_{2} & \\cdots & _{(0)}x_{(d_{i}-1)} \\\\ \n",
    "1 & _{(1)}x_{0} & _{(1)}x_{1} & _{(1)}x_{2} & \\cdots & _{(1)}x_{(d_{i}-1)} \\\\ \n",
    "\\vdots & \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\ \n",
    "1 & _{(n-1)}x_{0} & _{(n-1)}x_{1} & _{(n-1)}x_{2} & \\cdots & _{(n-1)}x_{(d_{i}-1)} \n",
    "\\end{array} \\right] _{n{\\times}(d_{i}+1)}$$\n",
    "\n",
    "For example, for logic gates, the input matrix was $X = \\left[ \\begin{array}{c} 1 & 0 & 0 \\\\ 1 & 0 & 1 \\\\ 1 & 1 & 0 \\\\ 1 & 1 & 1 \\end{array} \\right] _{4{\\times}3} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix representations - output of a layer\n",
    "\n",
    "Suppose the output of the $l^{th}$ layer has $o_{l}$ dimensions, meaning there are $o_{l}$ neurons in the layer.\n",
    "\n",
    "In the above example, the output of the 1st Layer of 2 neurons is $o_{1} = 2$, and the output of the 2nd layer of 1 neuron is $o_{2} = 1$.\n",
    "\n",
    "For each input, the output is an $o_{l}$-dimensional vector:\n",
    "\n",
    "$$Y^{(l)} = \\left[ \\begin{array}{c} y_{[0]}^{(l)} & y_{[1]}^{(l)} & \\cdots & y_{[o_{l}-1]}^{(l)} \\end{array} \\right] _{1{\\times}o_{l}}$$\n",
    "\n",
    "\n",
    "For example, for an AND gate, the output of $(0,1)$ is $Y = \\left[ \\begin{array}{c} 0 \\end{array} \\right] _{1{\\times}1}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, for $n$ data points, the output is:\n",
    "\n",
    "$$Y^{(l)} = \\left[ \\begin{array}{c} \n",
    "{_{(0)}}Y^{(l)} \\\\ {_{(1)}}Y^{(l)} \\\\ \\vdots \\\\ _{(n-1)}Y^{(l)} \\end{array} \\right] _{n{\\times}o_{l}} \n",
    "= \\left[ \\begin{array}{c} \n",
    "{_{(0)}}y_{[0]}^{(l)} & \\cdots & {_{(0)}}y_{[o_{l}-1]}^{(l)} \\\\ \n",
    "{_{(1)}}y_{[0]}^{(l)} & \\cdots & {_{(1)}}y_{[o_{l}-1]}^{(l)} \\\\ \n",
    "\\vdots & \\ddots & \\vdots \\\\ \n",
    "_{(n-1)}y_{[0]}^{(l)} & \\cdots & _{(n-1)}y_{[o_{l}-1]}^{(l)} \n",
    "\\end{array} \\right] _{n{\\times}o_{l}}$$\n",
    "\n",
    "For example, for an AND gate, the output matrix is $Y = \\left[ \\begin{array}{c} 0 \\\\ 0 \\\\ 0 \\\\ 1 \\end{array} \\right] _{4{\\times}1}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix representations - input to a layer\n",
    "\n",
    "Suppose at the $l^{th}$ layer, the input has $i_{l}$ dimensions.\n",
    "\n",
    "(The number of inputs to the layer) = (1 bias term) + (the number of outputs from the previous layer):\n",
    "$$i_{l} = 1 + o_{(l-1)}$$\n",
    "\n",
    "In the above example, the input to the first layer of 2 neurons has $i_{1} = d_{i}+1 = 3$, and the second layer of 1 neuron has $i_{2} = o_{1} + 1 = 3$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there are $n$ data points given, the input to the $l^{th}$ layer would be an $n{\\times}i_{l} = n{\\times}(o_{(l-1)}+1)$ matrix:\n",
    "\n",
    "$$X^{(l)}_{n{\\times}i_{l}} \n",
    "= \\left[ \\begin{array}{c} \n",
    "1 & _{(0)}Y^{(l-1)} \\\\ \n",
    "1 & _{(1)}Y^{(l-1)} \\\\ \n",
    "\\vdots & \\vdots \\\\  \n",
    "1 & _{(n-1)}Y^{(l-1)} \n",
    "\\end{array} \\right] _{n{\\times}i_{l}}\n",
    "= \\left[ \\begin{array}{c} \n",
    "1 & _{(0)}y^{(l-1)}_{[0]} & \\cdots & _{(0)}y^{(l-1)}_{[o_{l-1}-1]} \\\\ \n",
    "1 & _{(1)}y^{(l-1)}_{[0]} & \\cdots & _{(1)}y^{(l-1)}_{[o_{l-1}-1]} \\\\ \n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\ \n",
    "1 & _{(n-1)}y^{(l-1)}_{[0]} & \\cdots & _{(n-1)}y^{(l-1)}_{[o_{l-1}-1]} \n",
    "\\end{array} \\right] _{n{\\times}i_{l}}$$\n",
    "\n",
    "For example, in the 3-neurons neural network above, input matrix to the first layer is $\\left[ \\begin{array}{c} 1 & x_0 & x_1 \\end{array} \\right] _{1{\\times}3}$, and the input matrix to the second layer is $\\left[ \\begin{array}{c} 1 & y_0 & y_1 \\end{array} \\right] _{1{\\times}3}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix representations - weight matrix of one neuron\n",
    "\n",
    "For a neuron, the weight matrix multiplies a weight with each input in every dimension, and sums them. This can be represented by a dot product.\n",
    "\n",
    "Assuming the input to the $k^{th}$ neuron in the $l^{th}$ layer has $i_{l}$ dimensions,\n",
    "\n",
    "$$W^{(l)}_{[k]} {_{1{\\times}i_{l}}} = \\left[ \\begin{array}{c} w^{(l)}_{[k],0} & w^{(l)}_{[k],1} & \\cdots & w^{(l)}_{[k],i_{l}-1} \\end{array} \\right] _{1{\\times}i_{l}}$$\n",
    "\n",
    "(Remember $i_{l} = 1 + o_{(l-1)}$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then the output of that neuron for one data point is x < dot product \\> weights.\n",
    "\n",
    "$$y^{(l)}_{[k]} {_{1{\\times}1}} = Sigmoid( x^{(l)} {_{1{\\times}i_{l}}} \\; .* \\; W^{(l)}_{[k]}{^T}{_{i_{l}{\\times}1}} )$$\n",
    "\n",
    "$$\n",
    "=\n",
    "Sigmoid \\left(\n",
    "x^{(l)}_{[k]}\n",
    "\\left[ \\begin{array}{c} 1 & y^{(l-1)}_{0} & \\cdots & y^{(l-1)}_{(o_{l-1}-1)}\n",
    "\\end{array} \\right] _{1{\\times}i_{l}}\n",
    "\\;\\;\\; .* \\;\\;\\;\n",
    "W^{(l)}_{[k]} {^{T}}\n",
    "\\left[ \\begin{array}{c} w^{(l)}_{[k],0} \\\\ w^{(l)}_{[k],1} \\\\ \\vdots \\\\ w^{(l)}_{[k],i_{l}-1} \\end{array} \\right] _{i_{l}{\\times}1}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "$$\n",
    "= Sigmoid(1*w^{(l)}_{[k],0} \\;\\;+\\;\\; y^{(l-1)}_{0}*w^{(l)}_{[k],1} \\;\\;+\\;\\; ... \\;\\;+\\;\\; y^{(l-1)}_{o_{l-1}-1}*w^{(l)}_{[k],i_{l}-1})\n",
    "$$\n",
    "\n",
    "(We can see that the dot product of the $x$ and $W$ matrices does indeed give the output of the neuron)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For $n$ data points, the output of the $k^{th}$ neuron in the $l^{th}$ layer is:\n",
    "$$Y^{(l)}_{[k]} {_{n{\\times}1}}\n",
    "=\n",
    "Sigmoid \\left(\n",
    "X^{(l)}_{[k]}\n",
    "\\left[ \\begin{array}{c} \n",
    "1 & _{(0)}y^{(l-1)}_{0} & \\cdots & _{(0)}y^{(l-1)}_{(o_{l-1}-1)} \\\\\n",
    "1 & _{(1)}y^{(l-1)}_{0} & \\cdots & _{(1)}y^{(l-1)}_{(o_{l-1}-1)} \\\\\n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "1 & _{(n-1)}y^{(l-1)}_{0} & \\cdots & _{(n-1)}y^{(l-1)}_{(o_{l-1}-1)}\n",
    "\\end{array} \\right] _{n{\\times}i_{l}}\n",
    "\\; .* \\;\n",
    "W^{(l)}_{[k]} {^{T}}\n",
    "\\left[ \\begin{array}{c} w^{(l)}_{[k],0} \\\\ w^{(l)}_{[k],1} \\\\ \\vdots \\\\ w^{(l)}_{[k],i_{l}-1} \\end{array} \\right] _{i_{l}{\\times}1}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "$$\n",
    "=\n",
    "Sigmoid \\left(\n",
    "\\left[ \\begin{array}{c} \n",
    "1*w^{(l)}_{[k],0} \\;\\;+\\;\\; _{(0)}y^{(l-1)}_{(0)}*w^{(l)}_{[k],1} \\;\\;+\\;\\; ... \\;\\;+\\;\\; _{(0)}y^{(l-1)}_{o_{l-1}-1}*w^{(l)}_{[k],i_{l}-1} \\\\\n",
    "1*w^{(l)}_{[k],0} \\;\\;+\\;\\; _{(1)}y^{(l-1)}_{0}*w^{(l)}_{[k],1} \\;\\;+\\;\\; ... \\;\\;+\\;\\; _{(1)}y^{(l-1)}_{o_{l-1}-1}*w^{(l)}_{[k],i_{l}-1} \\\\\n",
    "\\vdots \\\\\n",
    "1*w^{(l)}_{[k],0} \\;\\;+\\;\\; _{(n-1)}y^{(l-1)}_{0}*w^{(l)}_{[k],1} \\;\\;+\\;\\; ... \\;\\;+\\;\\; _{(n-1)}y^{(l-1)}_{o_{l-1}-1}*w^{(l)}_{[k],i_{l}-1} \\\\\n",
    "\\end{array} \\right] _{n{\\times}1}\n",
    "\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix representations - weight of a layer of neurons\n",
    "\n",
    "Suppose the $l^{th}$ layer in a neural network has $o_{l}$ neurons.\n",
    "\n",
    "Each neuron would produce one number as its output - the dot product of its weights, and the inputs.\n",
    "\n",
    "In matrix form, the weight matrix of the layer is:\n",
    "\n",
    "$$\n",
    "W^{(l)}_{o_{l}{\\times}i_{l}} = \\left[ \\begin{array}{c} W^{(l)}_{[0]} \\\\ W^{(l)}_{[1]} \\\\ \\cdots \\\\ W^{(l)}_{[o_{l}-1]} \\end{array} \\right] _{o_{l}{\\times}i_{l}} \n",
    "= \n",
    "\\left[ \\begin{array}{c} \n",
    "w^{(l)}_{[0],0} & w^{(l)}_{[0],1} & w^{(l)}_{[0],2} & \\cdots & w^{(l)}_{[0],i_{l}-1} \\\\ \n",
    "w^{(l)}_{[1],0} & w^{(l)}_{[1],1} & w^{(l)}_{[1],2} & \\cdots & w^{(l)}_{[1],i_{l}-1} \\\\ \n",
    "\\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\ \n",
    "w^{(l)}_{[o_{l}-1],0} & w^{(l)}_{[o_{l}-1],1} & w^{(l)}_{[o_{l}-1],2} & \\cdots & w^{(l)}_{[o_{l}-1],i_{l}-1} \n",
    "\\end{array} \\right] _{o_{l}{\\times}i_{l}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of this layer of neurons is:\n",
    "\n",
    "$$ Y^{(l)}_{n{\\times}o_{l}} = Sigmoid\\;(\\;X^{(l)}_{n{\\times}i_{l}} \\; .* \\; W^{(l)}{^{T}}_{i_{l}{\\times}o_{l}} \\;)\\; $$\n",
    "\n",
    "$$\n",
    "Y^{(l)}_{n{\\times}o_{l}} \\left[ \\begin{array}{c} \n",
    "{_{(0)}}y_{0}^{(l)} & \\cdots & {_{(0)}}y_{o_{l}-1}^{(l)} \\\\ \n",
    "{_{(1)}}y_{0}^{(l)} & \\cdots & {_{(1)}}y_{o_{l}-1}^{(l)} \\\\ \n",
    "\\vdots & \\ddots & \\vdots \\\\ \n",
    "_{(n-1)}y_{0}^{(l)} & \\cdots & _{(n-1)}y_{o_{l}-1}^{(l)} \n",
    "\\end{array} \\right] _{n{\\times}o_{l}}\n",
    "=\n",
    "Sigmoid \\left(\n",
    "X^{(l)}_{n{\\times}i_{l}} \\left[ \\begin{array}{c} \n",
    "1 & _{(0)}y^{(l-1)}_{0} & \\cdots & _{(0)}y^{(l-1)}_{(o_{l-1}-1)} \\\\ \n",
    "1 & _{(1)}y^{(l-1)}_{0} & \\cdots & _{(1)}y^{(l-1)}_{(o_{l-1}-1)} \\\\ \n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\ \n",
    "1 & _{(n-1)}y^{(l-1)}_{0} & \\cdots & _{(n-1)}y^{(l-1)}_{(o_{l-1}-1)} \n",
    "\\end{array} \\right] _{n{\\times}i_{l}}\n",
    "\\; .* \\;\n",
    "W^{(l)}{^{T}}_{i_{l}{\\times}o_{l}} \\left[ \\begin{array}{c} \n",
    "w^{(l)}_{[0],0} & w^{(l)}_{[1],1} & \\cdots & w^{(l)}_{[o_{l}-1],0} \\\\ \n",
    "w^{(l)}_{[0],1} & w^{(l)}_{[1],1} & \\cdots & w^{(l)}_{[o_{l}-1],1} \\\\ \n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\ \n",
    "w^{(l)}_{[0],i_{l}-1} & w^{(l)}_{[1],1} & \\cdots & w^{(l)}_{[o_{l}-1],i_{l}-1} \n",
    "\\end{array} \\right] _{i_{l}{\\times}o_{l}}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "$$\n",
    "=\n",
    "Sigmoid \\left(\n",
    "\\left[ \\begin{array}{c} \n",
    "1*w^{(l)}_{[0],0} + \\cdots + _{(0)}y^{(l-1)}_{(i_{l-1}-1)}*w^{(l)}_{[0],i_{l-1}-1}\n",
    "&\n",
    "\\cdots\n",
    "&\n",
    "1*w^{(l)}_{[(o_{l}-1)],0} + \\cdots + _{(0)}y^{(l-1)}_{(i_{l-1}-1)}*w^{(l)}_{[(o_{l}-1)],i_{l-1}-1}\n",
    "\\\\\n",
    "\\vdots & \\ddots & \\vdots\n",
    "\\\\\n",
    "1*w^{(l)}_{[0],0} + \\cdots + _{(n-1)}y^{(l-1)}_{(i_{l-1}-1)}*w^{(l)}_{[0],i_{l-1}-1}\n",
    "&\n",
    "\\cdots\n",
    "&\n",
    "1*w^{(l)}_{[(o_{l}-1)],0} + \\cdots + _{(n-1)}y^{(l-1)}_{(i_{l-1}-1)}*w^{(l)}_{[(o_{l}-1)],i_{l-1}-1}\n",
    "\\end{array} \\right] _{n{\\times}o_{l}}\n",
    "\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We have seen that the action of a layer of a neural network can be written as the following matrix operation:\n",
    "\n",
    "$$ Y^{(l)}_{n{\\times}o_{l}} = Sigmoid\\;(\\;X^{(l)}_{n{\\times}i_{l}} \\; .* \\; W^{(l)}{^{T}}_{i_{l}{\\times}o_{l}} \\;)\\; $$\n",
    "\n",
    "So, a neural network can be defined as the set of weights $W^{(l)}_{i_{l}{\\times}o_{l}}$ for all its layers, where $l$ is the index of the layer we are considering, $i_{l}$ and $o_{l}$ are its input and output dimensions.\n",
    "\n",
    "Also, because of adding a bias term at every layer,\n",
    "\n",
    "$$i_{l} = 1 + o_{(l-1)}$$\n",
    "\n",
    "The utility of neural networks can be exploited only once the weight matrices $W^{(l)}_{i_{l}{\\times}o_{l}}$ for all $l$ have ben set according to need."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}