공대생 도전 일지

class로 된 python이라 self나 다른 것 들이 붙어있긴 한데 적당히 보면 될 것 같습니다.기록 용이라....from datasets import load_from_disk, DatasetDictimport argparse, os, json, torch, itertools, math, refrom typing import List, Dict, Tuplefrom scipy.special import digammafrom vllm import LLM, SamplingParamsfrom collections import defaultdict, Counterfrom transformers import AutoTokenizerfrom setproctitle import setproctitle 일단 전부 ..

지금 데이터를 늘리는 작업을 진행해서... 문서와 질문 그리고 정답을 통해 정답이 추론되는 과정을 만들려고 합니다.import jsonlinesimport jsonimport timefrom typing import Any, Optionalimport torchfrom transformers import AutoTokenizer # AutoModelForCausalLM 사용 안 함# vllm 임포트 (원래 주석처리 되어있던 부분을 활성화)from vllm import LLM, SamplingParams# from huggingface_hub import login# login("만약 access 필요한 모델이면, 토큰 발급받고 여기에 입력하삼!")import base64import timefrom ty..

https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro TIGER-Lab/MMLU-Pro · Datasets at Hugging Face[ "Boycotts, Buyalls, Blockchain technology, Increased Sales", "Buycotts, Boycotts, Digital technology, Decreased Sales", "Boycotts, Buycotts, Digital technology, Decreased Sales", "Buycotts, Boycotts, Blockchain technology, Charitable donations", "Boycotthuggingface.co 이번에 만드는 모델을 평가하기 위해 벤치마크 중 하나인 ..

https://huggingface.co/google/gemma-3-12b-it google/gemma-3-12b-it · Hugging FaceThis repository is publicly accessible, but you have to accept the conditions to access its files and content. To access Gemma on Hugging Face, you’re required to review and agree to Google’s usage license. To do this, please ensure you’re logged inhuggingface.co 일단 저는 12b모델을 사용했습니다.from transformers import pipeline..

https://arxiv.org/abs/2005.14165 Language Models are Few-Shot LearnersRecent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fiarxiv.orghttps://arxiv.org/abs/2201.11903 Chain-of-Thought Prompting Eli..

https://github.com/jina-ai/late-chunking GitHub - jina-ai/late-chunking: Code for explaining and evaluating late chunking (chunked pooling)Code for explaining and evaluating late chunking (chunked pooling) - jina-ai/late-chunkinggithub.com 일단 코드는 여기서 나왔습니다.코드에 익숙해지기 위해 조금 제맘대로 파 해치기도 했습니다.청크 풀링 (Chunked Pooling)그 다음으로, 우리가 임베딩에 사용할 모델을 로드합니다. 여기에서는 jinaai/jina-embeddings-v2-base-en을 선택했지만, 평균 풀링..

일단 RAG에 좋은 사이트를 발견해서 기록https://openrag.notion.site/Open-RAG-c41b2a4dcdea4527a7c1cd998e763595#6d4997a734a24a658fafcabb16684abe Open RAG | NotionAn open-source and open-access RAG platformopenrag.notion.site https://arxiv.org/abs/2410.13070 Is Semantic Chunking Worth the Computational Cost?Recent advances in Retrieval-Augmented Generation (RAG) systems have popularized semantic chunking, which aim..

모델을 굽기 위해 데이터를 수집하면서 토큰 수 확인은 필수기에 한번 가지고 왔습니다.from datasets import list_datasets, load_dataset# 데이터셋 불러오기dataset = load_dataset("nvidia/ChatQA-Training-Data","synthetic_convqa")# 데이터셋 분할 정보 확인print(dataset)일단 데이터 불러오기!import pandas as pdimport tiktokendf = pd.DataFrame(dataset["train"])df이제 DataFrame으로 변경하고 데이터 형식 확인하기여기선 다른 이름이 많은데 저는 특정 column만 골라서 사용할 겁니다.import mathdef tokenize_in_batches(d..

2024.12.23 - [인공지능/논문 리뷰 or 진행] - ChatQA: Surpassing GPT-4 on Conversational QA and RAG - 논문 리뷰 ChatQA: Surpassing GPT-4 on Conversational QA and RAG - 논문 리뷰https://arxiv.org/abs/2401.10225 ChatQA: Surpassing GPT-4 on Conversational QA and RAGIn this work, we introduce ChatQA, a suite of models that outperform GPT-4 on retrieval-augmented generation (RAG) and conversational question answering (Q..

2024.12.23 - [인공지능/논문 리뷰 or 진행] - ChatQA: Surpassing GPT-4 on Conversational QA and RAG - 논문 리뷰 ChatQA: Surpassing GPT-4 on Conversational QA and RAG - 논문 리뷰https://arxiv.org/abs/2401.10225 ChatQA: Surpassing GPT-4 on Conversational QA and RAGIn this work, we introduce ChatQA, a suite of models that outperform GPT-4 on retrieval-augmented generation (RAG) and conversational question answering (Q..