Feed-Forward Network (FFN)

A Feed-Forward Network (FFN), in the context of Transformer models, is a sequential arrangement of layers that processes information in one direction. It typically consists of two linear transformations separated by a non-linearity, such as ReLU. Each position in the input sequence passes through this FFN independently. After the attention computation, the FFN refines the position-specific representations by applying complex non-linear transformations to these isolated data points, without exchanging information between positions. This enables deeper feature extraction for each token.

For marketing managers and CTOs, the FFN is relevant because it significantly influences the ability of AI models to process complex data. It contributes to the performance of language models used in marketing for content generation, customer communication, and data analysis. An efficient FFN enables more precise and relevant results, directly increasing the effectiveness of AI-powered marketing campaigns and supporting strategic decisions.

When automatically generating marketing texts, a Transformer model analyzes a product description. After the attention calculation, which identifies relevant words, the FFN processes each word independently. It transforms the vector representation of 'innovative' and 'sustainable' into more specific features, which then contribute to generating a compelling slogan that highlights these attributes.

Position-wise FFN was part of the original Transformer (2017). GPT and BERT used GELU instead of ReLU. LLaMA (2023) introduced SwiGLU activation which became the norm in modern LLMs. MoE models (Mixtral, GPT-4) make FFN sparse.

• Attention Is All You NeedPaper
• SwiGLU Activation (Shazeer)Paper