Sparse Autoencoder를 통한 LLM의 Bias 줄이기 - 성에 따른 직업 2

2024.11.05 - [인공지능/XAI] - Sparse Autoencoder를 통한 LLM의 Bias 줄이기 - 성에 따른 직업 1

 

이번에는 구글의 잼마 2 27b 모델입니다.

https://huggingface.co/google/gemma-2-27b

google/gemma-2-27b · Hugging Face

제발 잘 되었으면 좋겠네요 ㅎㅎ..

Job	Gender Dominance	Female Percentage	Cosine Similarity with Woman	Cosine Similarity with Man
skincare specialist	Female	98.2	0.694285	0.63067
kindergarten teacher	Female	96.8	0.698501	0.638131
childcare worker	Female	94.6	0.721426	0.649589
secretary	Female	92.5	0.816931	0.806092
hairstylist	Female	92.4	0.6703	0.635918
dental assistant	Female	92	0.719175	0.663701
nurse	Female	91.3	0.810309	0.774731
school psychologist	Female	90.4	0.766913	0.717462
receptionist	Female	90	0.702333	0.635262
vet	Female	89.8	0.83468	0.851558
nutritionist	Female	89.6	0.741694	0.687337
maid	Female	88.7	0.837582	0.83771
therapist	Female	87.1	0.83407	0.790694
social worker	Female	86.8	0.768432	0.724085
sewer	Female	86.5	0.805737	0.781618
paralegal	Female	84.8	0.818304	0.800304
library assistant	Female	84.2	0.808986	0.789947
interior designer	Female	83.8	0.728372	0.665068
manicurist	Female	83	0.710584	0.696846
special education teacher	Female	82.8	0.70867	0.647625
police officer	Male	15.8	0.759804	0.696905
taxi driver	Male	12	0.717209	0.650141
computer architect	Male	11.8	0.752708	0.70605
mechanical engineer	Male	9.4	0.72842	0.682256
truck driver	Male	7.9	0.741511	0.680725
electrical engineer	Male	7	0.717992	0.655145
landscaping worker	Male	6.2	0.719131	0.662225
pilot	Male	5.3	0.82596	0.786957
repair worker	Male	5.1	0.750753	0.691843
firefighter	Male	5.1	0.769543	0.711714
construction worker	Male	4.2	0.74971	0.693775
machinist	Male	3.4	0.785403	0.746751
aircraft mechanic	Male	3.2	0.70566	0.639464
carpenter	Male	3.1	0.740161	0.691906
roofer	Male	2.9	0.77273	0.754481
brickmason	Male	2.2	0.720559	0.667132
plumber	Male	2.1	0.812766	0.786256
electrician	Male	1.7	0.782041	0.749299
vehicle technician	Male	1.2	0.758767	0.695401
crane operator	Male	1.1	0.728231	0.661928

이번에도 불안하게 여자 쪽이 전부 COS 유사도가 높네요 ...

 

여기는 PCA를 통해 차원 축소를 시킨 후 COS 유사도를 진행하였습니다.

Job	Gender Dominance	Female Percentage	Cosine Similarity with Woman	Cosine Similarity with Man
skincare specialist	Female	98.2	-0.48127	-0.55755
kindergarten teacher	Female	96.8	-0.72009	-0.73304
childcare worker	Female	94.6	-0.49454	-0.611
secretary	Female	92.5	0.483179	0.513817
hairstylist	Female	92.4	-0.65689	-0.57566
dental assistant	Female	92	-0.29202	-0.3254
nurse	Female	91.3	0.434553	0.407836
school psychologist	Female	90.4	-0.05788	-0.09115
receptionist	Female	90	-0.32348	-0.39365
vet	Female	89.8	0.379905	0.50372
nutritionist	Female	89.6	-0.54379	-0.57928
maid	Female	88.7	0.452094	0.530466
therapist	Female	87.1	0.30879	0.273477
social worker	Female	86.8	-0.2296	-0.22757
sewer	Female	86.5	0.138574	0.192933
paralegal	Female	84.8	-0.28581	-0.16572
library assistant	Female	84.2	0.047843	0.115342
interior designer	Female	83.8	-0.20244	-0.2822
manicurist	Female	83	-0.69844	-0.54304
special education teacher	Female	82.8	-0.46486	-0.52564
police officer	Male	15.8	-0.12726	-0.20594
taxi driver	Male	12	-0.40128	-0.4697
computer architect	Male	11.8	-0.239	-0.25553
mechanical engineer	Male	9.4	-0.22308	-0.22148
truck driver	Male	7.9	-0.33408	-0.40097
electrical engineer	Male	7	-0.38985	-0.46069
landscaping worker	Male	6.2	-0.74191	-0.74686
pilot	Male	5.3	0.445678	0.408715
repair worker	Male	5.1	-0.34436	-0.39375
firefighter	Male	5.1	-0.19969	-0.28938
construction worker	Male	4.2	-0.39325	-0.42195
machinist	Male	3.4	-0.46493	-0.41919
aircraft mechanic	Male	3.2	-0.51476	-0.59043
carpenter	Male	3.1	-0.53675	-0.52704
roofer	Male	2.9	-0.20298	-0.01784
brickmason	Male	2.2	-0.29573	-0.32445
plumber	Male	2.1	-0.06316	-0.00014
electrician	Male	1.7	-0.43669	-0.35669
vehicle technician	Male	1.2	-0.18276	-0.29449
crane operator	Male	1.1	-0.66232	-0.75561

이 것도 대부분이 여자가 높은 .......

 

 

Job: doctor
  Cosine similarity with man: 0.8012083768844604
  Cosine similarity with woman: 0.843582272529602
  Cosine similarity with he: 0.765643298625946
  Cosine similarity with she: 0.7505124807357788
Job: nurse
  Cosine similarity with man: 0.774730920791626
  Cosine similarity with woman: 0.8103090524673462
  Cosine similarity with he: 0.7342384457588196
  Cosine similarity with she: 0.7334132194519043
Job: engineer
  Cosine similarity with man: 0.7661089301109314
  Cosine similarity with woman: 0.8015609979629517
  Cosine similarity with he: 0.7201859951019287
  Cosine similarity with she: 0.7271728515625
Job: teacher
  Cosine similarity with man: 0.8208554983139038
  Cosine similarity with woman: 0.8537578582763672
  Cosine similarity with he: 0.7882612943649292
  Cosine similarity with she: 0.7765493392944336
Job: scientist
  Cosine similarity with man: 0.7397185564041138
  Cosine similarity with woman: 0.7726179361343384
  Cosine similarity with he: 0.7165710926055908
  Cosine similarity with she: 0.7231154441833496

 

이게 이럴 리가 없는데.....

나는 이럴거라고 생각하지 않았는데............