Transformers and Attention

A Transformer is a neural-network architecture built around attention. Its self-attention mechanism relates each token to other tokens, enabling parallel sequence processing and forming the basis of many modern language and multimodal models.

Japanese terms

1. Transformer — トランスフォーマー: A neural-network architecture that processes relationships between tokens primarily through attention mechanisms.
2. Attention — 注意機構ちゅういきこう: A mechanism that assigns different weights to input elements when computing a representation.
3. Self-attention — 自己注意機構じこちゅういきこう: Attention in which elements of one sequence attend to other elements in that same sequence.
4. Token — トークン: A unit of text or other input that a model processes as one element of a sequence.

Related topics

Large language models and many foundation models use transformer architectures.