Neural Network

• Neural Network
  • Perceptron
  • Backpropagation
• Loss Functions
  • L2 & L1 Norm (regularization)
  • Regression Loss Functions
  • Binary Classification Loss Functions
  • Multi-Class Classification Loss Functions
  • CNN Loss
  • Detection
  • Face
  • Divergence loss (on probability)
• Optimizer
  • Gradient Descent/ SGD
  • Momentum
  • AdaGrad
  • AdaDelta
  • RMSprop (Root Mean Square Prop)
  • Adam
  • EVE
  • Cyclical Learning Rates
  • SGDR
• Activation Functions
  • Step
  • Signum
  • Sigmoid
  • Tanh
  • ReLU
  • Softmax
• Normalization
  • Local Response Normalization
  • Batch Normalization
  • Layer Normalization (NIPS 2016)
  • Weight Normalization (NIPS 2016)
  • Instance normalization (2016)
  • SELU (NIPS 2017)
  • Group Normalization (ECCV 2018)
  • Conditional BatchNorm (NIPS 2017)
  • Conditional Instance Normalizatoin (ICLR 2017)
  • Adaptive Instance Normalization (ICCV 2017)
  • Batch Renormalization (NIPS 2017)
  • SPADE
  • Summary and use cases
• Weight Initialization
  • Unsupervised pre-training
  • Xavier Initialization
  • Kaiming Initialization
• Tricks
  • Data
  • Dropout
  • Regarlization
  • Normalization
  • Skip Connection
  • Residual scaling
  • visual gradient vanishing
  • Batch size