I got different triggers with the one in fact_retrieval_bert_prompts.jsonl

[Spike Xiong]

`class FactRetriveArgs():

train = Path("/path/P108/train.jsonl")
dev = Path('/path/P108/dev.jsonl')
template = '[CLS] {sub_label} [T] [T] [T] [T] [T] [P]. [SEP]'
num_cand = 10
accumulation_steps = 1
model_name = '/var/www/pre_models/bert-base-cased'
bsz = 32
eval_size = 32
iters = 1000
label_field = 'obj_label'
tokenize_labels = True
filter = True
print_lama = True
use_ctx = False
seed = 0
label_map = None
initial_trigger = None
perturbed = False
patience = 5
debug = False
limit = None`

I have trained P108 from TREx by the parameters above, the result is down below.

INFO:autoprompt.create_trigger:No improvement detected. Skipping evaluation. INFO:autoprompt.create_trigger:Iteration: 999 INFO:autoprompt.create_trigger:Accumulating Gradient 100%|██████████| 1/1 [00:00<00:00, 36.45it/s] INFO:autoprompt.create_trigger:Evaluating Candidates 100%|██████████| 1/1 [00:00<00:00, 10.50it/s] INFO:autoprompt.create_trigger:No improvement detected. Skipping evaluation. INFO:autoprompt.create_trigger:Best tokens: ['helped', 'survives', 'computer', 'lawsuit', 'against'] INFO:autoprompt.create_trigger:Best dev metric: -3.59419067382812 {"relation": "P108", "template": "[ x ] helped survives computer lawsuit against [Y]."}

The final template is "template": "[ x ] helped survives computer lawsuit against [Y]." which is different with {"relation": "P108", "template": "[X] 1987adeNBC computing succeeded [Y]."} in fact_retrieval_bert_prompts.jsonl.

So is there anything wrong with my training?

[Robert L. Logan IV]

Hi @CaptXiong,

The prompt search procedure is typically unstable from run to run; even differences in how the data is shuffled could produce an entirely different prompt. While setting the random seed should consistently produce the same output, it is possible that changes between your and @taylorshin's development environments could affect the rng.

So, is there anything wrong with my training?

If the prompt is roughly as accurate as ours on that relation, then training is working fine. If not, you may get better results by increasing the number of candidates/batch size.

Best,

Rob