Attention Mechanism¶
Attention mechanism m ostly used on RNN for NLP, but SAGAN using self-attention on CNN.
Align and Translate¶
Neural Machine Translation by Jointly Learning to Align and Translate (ICLR 2015)
In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.
Show, Attend and Tell¶
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (ICML 2015)
Generate word by word based on image
Self-Attention¶
Non-local Neural Networks¶
Non-local Neural Networks (CVPR 2018)
Caffe code
spacetime Non-local block
CBAM¶
CBAM: Convolutional block attention module(ECCV 2018)
Channel attention module (CAM)¶
Spatial Attention Module (SAM)¶
An Empirical Study of Spatial Attention Mechanisms in Deep Networks¶
An Empirical Study of Spatial Attention Mechanisms in Deep Networks (ICCV 2019) from MSRA