Implementation of Optimizater

In the previous section, we have discussed the loss function. By using the loss function, we know how bad the model is. But we still don't know how to improve the model. This is where the optimization algorithms come in. It will using the loss computed by the loss function to update the model.

Although the optimization algorithms are a group of algorithms, they are still have some common features.

• They are all based on the gradient of the loss function.
• They are all iterative algorithms.
• They are all used to update the model.