Optimizers in tensorflow from scratch. See example.py and optimizers.py.
Optimizers
- Stochastic Gradient descent
- Stochastic Gradient descent with gradient clipping
- Momentum
- Nesterov momentum
- Adagrad
- Adadelta
- RMSProp
- Adam
- Adamax
- SMORMS3
- An overview of gradient descent optimization algorithms: http://sebastianruder.com/optimizing-gradient-descent/
- Chainer Optimizers: http://docs.chainer.org/en/stable/reference/optimizers.html
- SMORMS3: http://sifter.org/~simon/journal/20150420.html
- AdaGrad, RMSProp, AdaDelta, Adam, SMORMS3 (in japanese): http://qiita.com/skitaoka/items/e6afbe238cd69c899b2a