An activation function in a machine learning model is used to add Non-linearity to the output. This is needed, because otherwise, our model will learn to act as a linear least squares model. We usually want our models to solve complex, non-linear problems.

Examples

• ReLU - $f(x) = max(0, x)$
• Sigmoid - $f(x) = \frac{1}{1+e^{-x}}$
• Identity - $f(x) = x$

References

• https://www.wikiwand.com/en/Activation_function
• https://www.quora.com/What-do-you-mean-by-introducing-non-linearity-in-a-neural-network