ivarfresh / Interaction_LLMs

[Personalize@EACL 2024] LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models.

Evaluating personality consistency and language use in Interactive LLMs

This repository contains the code for the paper "LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models."

Please make sure to:

1. set your working directory to: “coding/py”

2. Enter your OpenAI API key in following the data generation files:

• “Control_groups.py”, “ANACREA_Experimental_groups.py”, “CREAANA_Experimental_groups.py”

3. Intall Python version and packages**

Required Libraries:

Python                    3.11.4
liwc                      0.5.0
langchain                 0.0.268
numpy                     1.25.2
openai                    0.27.8 
matplotlib-base           3.7.1             
matplotlib-inline         0.1.6
pandas                    2.0.3           
pandas-stubs              1.5.3.230203    
mpmath                    1.3.0
scipy                     1.11.1
seaborn                   0.12.2
scikit-learn				 1.3.2           
pillow                    9.4.0
requests                  2.31.0
regex                     2023.8.8

4. Data

• Our data can be found in “output/Data”

• In order to generate your own data and perform our analyses, continue below

5. Data generation:

Main files:

• py/Control_groups.py
• py/CREAANA_Experimental_groups.py
• py/ANACREA_Experimental_groups.py

5.1 Make sure that the folders: “output/Control”, “output/ANACREA”, “output/CREAANA” exist.

5.2 Errors:

• When runnning the files in 4. you might get a ”ValueError” or “IndexError”. We tried to prevent this by adding error handling, however this is not complete error proof because the GPT model temperature is set to 0.7 (this is needed for variable responses).

• If this happens, just remove the corresponding rows from the csv files (if something has been saved already) and run it again.

6. Data Processing:

Combining the data files into one file: “BFI_Story_data.csv”. This is necessary in order to be able to run the statistical analyses

6.1 For the Experimental groups

• Run: merge_subject_groups.py

• Run: combine_csv_files.py (this will also generate the “BFI_Story_data.csv” file for the control group.)

• Remove all csv files except for “BFI_Story_data.csv”

6.2 For the Control group

• Remove all csv files except for “BFI_Story_data.csv”

6.3 Pre-processing for Point Biserial correlation:

• Run: PB_encodings.py

7. LIWC

• to run the LIWC test

• Run: LIWC.py

• note: You must run this before you can perform the correlations

8. Statistical analyses

• Run: all the files in ”plot_stats”