15.1 The Neuron Model and Architectures

Biological Inspiration

Artificial Neural Networks (ANNs) are inspired by the structure and function of the human brain. The brain is a complex network of billions of neurons, each connected to thousands of others. These neurons transmit signals to each other, and this collective activity gives rise to thought and consciousness. ANNs are a simplified mathematical model of this process.

The Artificial Neuron

The fundamental building block of an ANN is the artificial neuron, or "unit." It receives one or more inputs, performs a computation, and produces an output.

A neuron's operation can be broken down into two parts:

1. Weighted Sum: The neuron calculates a weighted sum of its inputs. Each input \(x_i\) is multiplied by a weight \(w_i\), which represents the strength of the connection. A bias term \(b\) is also added. The result is often called the "net input" or "pre-activation" \(z\).

   \(z = (w_1 x_1 + w_2 x_2 + \dots + w_n x_n) + b = \mathbf{w} \cdot \mathbf{x} + b\)

2. Activation Function: The net input \(z\) is then passed through a non-linear function called an activation function, \(g(z)\), to produce the neuron's final output, \(a\).

   \(a = g(z) = g(\mathbf{w} \cdot \mathbf{x} + b)\)

The non-linearity introduced by the activation function is crucial. Without it, a multi-layer network would be equivalent to a single-layer network, and it would only be able to learn linear relationships.

Common Activation Functions

• Sigmoid: \(g(z) = \frac{1}{1 + e^{-z}}\). This function squashes its input into the range (0, 1). It was historically popular but can suffer from the "vanishing gradient" problem.
• Hyperbolic Tangent (Tanh): \(g(z) = \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}\). Similar to sigmoid but squashes the input into the range (-1, 1). It is zero-centered, which can be advantageous.
• Rectified Linear Unit (ReLU): \(g(z) = \max(0, z)\). This is the most popular activation function in deep learning today. It is computationally efficient and helps to mitigate the vanishing gradient problem.

Neural Network Architectures

Neurons are organized into layers to form a network.

• Input Layer: Receives the raw input features.
• Hidden Layers: Layers between the input and output layers. These are where the network learns complex representations of the data. A network with one or more hidden layers is a "deep" neural network.
• Output Layer: Produces the final result (e.g., a classification score or a regression value).

The simplest architecture is the Feedforward Neural Network (FNN), where connections do not form cycles. Information flows in one direction, from the input layer, through the hidden layers, to the output layer. A multi-layer perceptron (MLP) is a classic example of an FNN.