Meme detail
About this meme
Text detected in the image
(J(J() 'yoav))) @yoavgo modern day alchemy Training Recipes. All models used the AdamW optimizer with • gradient clip value 1.o • weight decay 0.1 . no dropout . linear learning rate warmup with cosine decay By default, the peak learning rate is the GPT3 specification. We give several models an "improved recipe", inspired by changes adopted by popular large language models such as PaLM (Chowdhery et al. 2023) and LLaMa (Touvron et al. 2023). These include: . linear learning rate warmup with cosine decay to 1e - 5, with a peak value of 5x the GPT3 value . no linear bias terms . RMSNorm instead of LayerNorm • AdamW hyperparameter β = (.9,.95) (the GPT3 value) instead of the PyTorch default of β = (.9,.999) Architecture and Training Details. Our models are: . Transformer: The standard Transformer based on GPT3 (Table 12), . Transformer++: A Transformer with an improved architecture, namely rotary positional encodings (Su et al. 2021) and SwiGLU MLP (Shazeer 2020), and the improved training recipe above. . Hyena: Interleaving a Hyena block (the H3 block with S4 replaced by a global convolution parameterized by an MLP) with standard MLP blocks. The MLP blocks have expansion factor 2 instead of 4 and the number of layers is correspondingly increased by 1.5x to preserve parameter count.
