Not able to bring lambada templates

[zoeyki]

Hello, I am running bigscience-workshop/bigscience-lm-evaluation-harness.
I wanted to add new task: LAMBADA.
I respected and followed the "new_prompt_source_task" to create one.
But I am keep getting this kind of error message:
raise ValueError(f"No original task templates found for {task_name}")
ValueError: No original task templates found for lambada

[zoeyki]

task > lambada.py is defined as:

class LAMBADA(PromptSourceTask):
    VERSION = 0
    DATASET_PATH = inspect.getfile(lm_eval.datasets.lambada.lambada)
    DATASET_NAME = None

    def has_training_docs(self):
        return False

    def has_validation_docs(self):
        return True

    def has_test_docs(self):
        return False

    def training_docs(self):
        pass

    def validation_docs(self):
        return self.dataset["validation"]

    def test_docs(self):
        pass

    def doc_to_text(self, doc):
        return doc["text"].rsplit(" ", 1)[0]

    def should_decontaminate(self):
        return True

    def doc_to_decontamination_query(self, doc):
        return doc["text"]

    def doc_to_target(self, doc):
        return " " + doc["text"].rsplit(" ", 1)[1]

    def construct_requests(self, doc, ctx):
        ll, is_greedy = rf.loglikelihood(ctx, self.doc_to_target(doc))

        return ll, is_greedy

    def process_results(self, doc, results):
        ll, is_greedy = results
        return {"ppl": ll, "acc": int(is_greedy)}

    def aggregation(self):
        return {"ppl": perplexity, "acc": mean}

    def higher_is_better(self):
        return {"ppl": False, "acc": True}

[Stephen Bach]

Hi @dayeonki, this might be better posted in bigscience-lm-evaluation-harness, but I can try to help with debugging. Can you post the entire backtrace? You could also try to load LAMBADA in promptsource following the example in the promptsource README.