공대생 도전 일지

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts - 논문 리뷰

https://arxiv.org/abs/2112.06905 GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsScaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these laarxiv.org MoE는 파라미터를 늘리면서도 추론 속도나 전력 사용을 줄였..

Using Degeneracy in the Loss Landscape for Mechanistic Interpretability - 논문리뷰

https://arxiv.org/abs/2405.10927 Using Degeneracy in the Loss Landscape for Mechanistic InterpretabilityMechanistic Interpretability aims to reverse engineer the algorithms implemented by neural networks by studying their weights and activations. An obstacle to reverse engineering neural networks is that many of the parameters inside a network are not involvarxiv.org 신경망의 해석 가능성을 방해하는 퇴행적 구조를 해결..