Few-Shot, CoT(Chain-of-Thought)와 ReAct 하나 하나 뜯어보기

https://arxiv.org/abs/2005.14165

Language Models are Few-Shot Learners

https://arxiv.org/abs/2201.11903

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

https://arxiv.org/abs/2210.03629

ReAct: Synergizing Reasoning and Acting in Language Models

 

ReAct는 Agent 논문에 빠지면 안되는 존재라고 생각하기에 한번 쭉 정리해보려고 합니다.

Prompt, In-Context-Learning 위주의 논문들이 나오네요

Few-Shot

Few-Shot은 단순합니다.

LLM이 진행할 Task에 대한 예시를 던져주고, 그에 맞게 적절한 Output을 추출합니다.

Prompt에서 모델이 학습하고, 그에 따라서 정답을 내는 것으로 거대 모델이 아니면 좋은 성과를 보여주기 힘들었습니다.

ReAct까지 공통점은 거대 모델이 아니면 따라하기 힘들다는 것 이네요...

Few-Shot의 성능

 

Few-Shot도 논리적 추론 문제나 복잡한 Task에서는 성능이 저하되며 한계를 보여줍니다.

 

그리하여 발전된 Few-Shot인 CoT가 등장합니다.

CoT도 단순합니다.

Few-Shot 예시에서 정답만 주는 것이 아닌 그 정답이 나오기까지의 사고 과정을 자연어로 예시를 전달하여 모델의 성능을 높입니다.

이를 통해 다양한 추론 Task에서 높은 성능을 보여줍니다.

수학 문제에서 SOTA 달성

 

여기서도 거대 모델이 아니면 Long-Context를 잘 따르지 못하는 문제가 있어 성능 향상에 문제가 되었지만 Fine-tuning 없이 Prompt 만으로 높은 성능을 내는 것을 볼 수 있습니다.

이 Ablation Study를 통해 자연어를 통한 사고 과정 예시가 중요하다는 것을 볼 수 있습니다.

 

 

Reasoning 단계에서의 환각 문제, Acting 만 진행하면 논리적 사고 과정 없이 단순 정보 탐생만 수행하여 생기는 문제들을 해결하기 위해 독립적으로 진행했던 Reasoning과 Acting을 묶어서 ReAct를 만듭니다.

문제가 들어오면 생각하는 과정을 거쳐 행동을 진행하고, 그에 따른 관측값을 다시 얻어 Reasoning 을 다시 진행하는 과정을 반복합니다.

 

https://github.com/ysymyth/ReAct

GitHub - ysymyth/ReAct: [ICLR 2023] ReAct: Synergizing Reasoning and Acting in Language Models

이제 코드를 통해 ReAct 한번 활용해보기...

 

"react_put_0": "You are in the middle of a room. Looking quickly around you, 
you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, 
a handtowelholder 2, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, 
a toiletpaperhanger 1, and a towelholder 1.\nYour task is to: put some spraybottle on toilet.
> think: First I need to find a spraybottle. 
A spraybottle is more likely to appear in cabinet (1-4), countertop (1), toilet (1), 
sinkbasin (1-2), garbagecan (1). I can check one by one, starting with cabinet 1.
OK.
> go to cabinet 1
On the cabinet 1, you see a cloth 1, a soapbar 1, a soapbottle 1.\n> go to cabinet 2
The cabinet 2 is closed.
> open cabinet 2
You open the cabinet 2. The cabinet 2 is open. In it, you see a candle 1, and a spraybottle 2.
> take spraybottle 2 from cabinet 2
You pick up the spraybottle 2 from the cabinet 2.
> think: Now I find a spraybottle (2). Next, I need to put it in/on toilet 1.
OK.
> go to toilet 1
On the toilet 1, you see a soapbottle 2.
> put spraybottle 2 in/on toilet 1
You put the spraybottle 2 in/on the toilet 1.\n",

React Few-shot 중 하나. - ALF World

 

"webact_simple3": "
Determine if there is Observation that SUPPORTS or REFUTES a Claim, 
or if there is NOT ENOUGH INFORMATION. 
Claim: Nikolaj Coster-Waldau worked with the Fox Broadcasting Company.
Action 1: Search[Nikolaj Coster-Waldau]
Observation 1: Nikolaj William Coster-Waldau (born 27 July 1970) is a Danish actor and producer. 
He graduated from the Danish National School of Performing Arts in Copenhagen in 1993,
[1] and had his breakthrough role in Denmark with the film Nightwatch (1994). 
He played Jaime Lannister in the HBO fantasy drama series Game of Thrones, 
for which he received two Primetime Emmy Award nominations for Outstanding Supporting Actor 
in a Drama Series.. Coster-Waldau has appeared in numerous films in his native Denmark and 
Scandinavia, including Headhunters (2011) and A Thousand Times Good Night (2013). 
In the U.S, his debut film role was in the war film Black Hawk Down (2001), 
playing Medal of Honor recipient Gary Gordon.
[2] He then played a detective in the short-lived Fox television series New Amsterdam (2008), 
and appeared in the 2009 Fox television film Virtuality, originally intended as a pilot.
Action 2: Finish[SUPPORTS]
Claim: Stranger Things is set in Bloomington, Indiana.
Action 1: Search[Stranger Things]
Observation 1: Stranger Things is an American science fiction horror drama television series created by the Duffer Brothers. Set in the 1980s, primarily in the fictional town of Hawkins, Indiana, the series centers on a number of mysteries and supernatural events occurring around the town and their impact on an ensemble of child and adult characters. 
Action 2: Finish[REFUTES]
Claim: Beautiful reached number two on the Billboard Hot 100 in 2003.?
Action 1: Search[Beautiful]
Observation 1: Could not find [Beautiful]. Similar: ['Beautiful', 'Beautiful, Beautiful', 'A Beautiful Mind (film)', 'Beautiful (Christina Aguilera song)', 'Life Is Beautiful'].
Action 2: Search[Beautiful (Christina Aguilera song)]
Observation 2: \"Beautiful\" is a song recorded by American singer Christina Aguilera for her fourth studio album, Stripped (2002).
Action 3: Lookup[Billboard Hot 100]
Observation 3: (Result 1 / 3) The song peaked at number two on the Billboard Hot 100 in the United States, where it was certified Gold for 500,000 units shipped.
Action 4: Finish[NOT ENOUGH INFO]
"

이 Prompt는 단순한 Act네요

 

"webthink_simple3": "
Determine if there is Observation that SUPPORTS or REFUTES a Claim, or if there is NOT ENOUGH INFORMATION. 
Claim: Nikolaj Coster-Waldau worked with the Fox Broadcasting Company.
Thought 1: I need to search Nikolaj Coster-Waldau and find if he has worked with the Fox Broadcasting Company.
Action 1: Search[Nikolaj Coster-Waldau]
Observation 1: Nikolaj William Coster-Waldau (born 27 July 1970) is a Danish actor and producer. He graduated from the Danish National School of Performing Arts in Copenhagen in 1993,[1] and had his breakthrough role in Denmark with the film Nightwatch (1994). He played Jaime Lannister in the HBO fantasy drama series Game of Thrones, for which he received two Primetime Emmy Award nominations for Outstanding Supporting Actor in a Drama Series.. Coster-Waldau has appeared in numerous films in his native Denmark and Scandinavia, including Headhunters (2011) and A Thousand Times Good Night (2013). In the U.S, his debut film role was in the war film Black Hawk Down (2001), playing Medal of Honor recipient Gary Gordon.[2] He then played a detective in the short-lived Fox television series New Amsterdam (2008), and appeared in the 2009 Fox television film Virtuality, originally intended as a pilot.
Thought 2: Because he \"appeared in the 2009 Fox television film Virtuality\", he should have worked with the Fox Broadcasting Company.
Action 2: Finish[SUPPORTS]

Claim: Stranger Things is set in Bloomington, Indiana.
Thought 1: I should search for Stranger Things, and see if it is set in Bloomington, Indiana.
Action 1: Search[Stranger Things]
Observation 1: Stranger Things is an American science fiction horror drama television series created by the Duffer Brothers. Set in the 1980s, primarily in the fictional town of Hawkins, Indiana, the series centers on a number of mysteries and supernatural events occurring around the town and their impact on an ensemble of child and adult characters. 
Thought 2: The observation says that it is set in a \"fictional town of Hawkins, Indiana\", so it is not set in Bloomington.
Action 2: Finish[REFUTES]

Claim: Beautiful reached number two on the Billboard Hot 100 in 2003.?
Thought 1: I need to search the song Beautiful and find if it reached number two on the Billboard Hot 100 in 2003.
Action 1: Search[Beautiful]
Observation 1: Could not find [Beautiful]. Similar: ['Beautiful', 'Beautiful, Beautiful', 'A Beautiful Mind (film)', 'Beautiful (Christina Aguilera song)', 'Life Is Beautiful'].
Thought 2: From suggestions, I should search \"Beautiful (Christina Aguilera song)\" to find the song.
Action 2: Search[Beautiful (Christina Aguilera song)]
Observation 2: \"Beautiful\" is a song recorded by American singer Christina Aguilera for her fourth studio album, Stripped (2002).
Thought 3: It does not mention Billboard, so I need to look up \"Billboard Hot 100\" to find if it reached number two on it in 2003.
Action 3: Lookup[Billboard Hot 100]
Observation 3: (Result 1 / 3) The song peaked at number two on the Billboard Hot 100 in the United States, where it was certified Gold for 500,000 units shipped.
Thought 4: It only says the song peaked at number two on the Billboard Hot 100, but not if it was in 2003. I am not sure if this claim is true or not.
Action 4: Finish[NOT ENOUGH INFO]\n\n"

이게 진짜 ReAct Few-shot이었네요