Activation Functions: Sigmoid and ReLU
Explain why an activation function is applied to a neuron's weighted sum and distinguish sigmoid from ReLU.
Sigmoid
The sigmoid function is: σ(z) = 1 / (1 + e^(-z))
It maps a real-valued input to a value strictly between 0 and 1.
Preserved worked examples:
- sigmoid(1.5) ≈ 0.818
- sigmoid(0.6) ≈ 0.646
- sigmoid(-0.5) ≈ 0.378
- sigmoid(1.0) ≈ 0.731
A sigmoid output may be interpreted as a score in the range (0,1). It is not automatically a calibrated probability. Calibration is a separate property that must be evaluated.
ReLU
ReLU is defined as: ReLU(z) = max(0, z)
It returns zero for negative inputs and returns z for positive inputs. ReLU is commonly used in hidden layers of neural networks.
Why nonlinearity matters
If a network stacks only linear transformations, the composition is still a linear transformation. Nonlinear activation functions allow multi-layer networks to represent more complex nonlinear relationships.
Common misconceptions
- Sigmoid output is not automatically a calibrated probability.
- ReLU is not the same as sigmoid.
- An activation value is not automatically a business decision.
Knowledge check
1. What range does sigmoid return? Strictly between 0 and 1. 2. What does ReLU return for z = -2? 0. 3. Why do neural networks use nonlinear activations? They allow stacked layers to represent nonlinear relationships; without nonlinearity, stacked linear transformations collapse to a single linear transformation.
Recovered from the available teaching record; not a verbatim transcript.
Introduced; learner mastery not verified.
Help improve this learning path.
Suggest a concept we should teach, or tell us what is unclear. No account is required. Email is optional and only for a reply. Do not include confidential or sensitive information.