A cost function (also known as a loss function) measures how wrong a machine learning model is.
It’s different from the error of a neural network in that the error simply measures how far off a single guess was from the real answer. The loss function on the other hand tells you how bad it is that it failed to recognize something and you average the result over the entire data set (or batch).
It is an important distinction because in some contexts, you might want to punish certain results more, e.g getting a false positive might be worse than getting a false negative. In other cases a simple squared error will do just fine.
It is used for optimising the model by computing its derivative and using that for adjusting the weights and biases.