공대생 도전 일지

https://aclanthology.org/2024.acl-long.536/ Dodo: Dynamic Contextual Compression for Decoder-only LMsGuanghui Qin, Corby Rosset, Ethan Chau, Nikhil Rao, Benjamin Van Durme. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.aclanthology.orgacl 2024 long에 붙은 논문입니다. 기존 방법들(sparse attention, 커널 등)은 nlp에서 일관적인 효과가 나지 않거나, 대형 llm에 적용이..

https://arxiv.org/abs/2310.01732 Nugget: Neural Agglomerative Embeddings of TextEmbedding text sequences is a widespread requirement in modern language understanding. Existing approaches focus largely on constant-size representations. This is problematic, as the amount of information contained in text often varies with the length of tarxiv.org고정 길이 임베딩은 문장 길이와 정보량이 달라도 동일한 크기로 압축해야 해서 긴 텍스트에서 정보..

https://arxiv.org/abs/2006.03236 Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language ProcessingWith the success of language pretraining, it is highly desirable to develop more efficient architectures of good scalability that can exploit the abundant unlabeled data at a lower cost. To improve the efficiency, we examine the much-overlooked redundancyarxiv.orgTransformer가..