Giters

xiaoyu2018 / TextSum

2021CCF BDCI 新闻摘要自动生成

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

pretrained-modelspytorchsummarizer

TextSum

0 使用说明

  1. 项目相关依赖已写入requirements.txt文件 pip install -r requirements.txt
  2. 项目使用了transformers提供的预训练模型，相关模型、配置文件、词典文件等于此处下载
  3. 运行项目前，于/source/settings.py中修改路径设置为本地实际绝对路径
  4. 项目结构：
    TextSum
    --dataset（数据集、词典、词频表）
    --params（预训练模型、模型参数保存文件）
    --source（源代码）
    ----go.py（主控函数）
    ----pretrained_models.py（预训练模型）
    ----models.py（自定义模型）
    ----settings.py（项目设置）
    ----utils.py（工具函数）
  5. python go.py 运行项目，可选命令行参数如下： 
    -h, --help            show this help message and exit
-p, --preprocess      预处理数据
-b, --build           建立词频表
-m, --make            建立词典
-t 模型名, --train           训练
                        
-f 模型名, --fine_tune       微调
                        
-g 模型名 参数路径, --gen             生成submission

1 数据处理

本项目数据处理共分为部分：数据清洗与划分、词典生成、张量转换

  • 数据清洗与划分
    • 使用正则表达式清洗原始数据，去除文本中与任务无关的信息
    • 从原始训练集中划分出验证集
    • 将原始CSV文件转换为逐条文本的JSON文件
  • 词典生成
    统计数据集中出现过的所有单词的词频，取一定数目的高频词生成字典
  • 张量转换
    读取预处理完毕的json文件，进一步处理后将文本数据集转换为成batch的Tensor

2 模型结构

本项目使用pytorch实现了模型基础结构、自定义损失函数、优化器以及模型训练、验证过程；
本项目还使用transformers提供的预训练模型（bart、t5、pegasus）及函数接口实现了模型的微调与推断
以下给出部分模型的网络结构

  1. GRU编码器-解码器架构网络结构如下： 
    EncoderDecoder(
(encoder): GruEncoder(
    (embdding): Embedding(10004, 512)
    (rnn): GRU(512, 256, num_layers=2)
)
(decoder): GruDecoder(
    (embdding): Embedding(10004, 512)
    (rnn): GRU(768, 256, num_layers=2)
    (dense): Linear(in_features=256, out_features=10004, bias=True)
)
)
  2. t5(small) 
    T5ForConditionalGeneration(
    (shared): Embedding(32128, 512)
    (encoder): T5Stack(
        (embed_tokens): Embedding(32128, 512)
        (block): ModuleList(
        (0): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                (relative_attention_bias): Embedding(32, 8)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        (1): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        (2): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        (3): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        (4): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        (5): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        )
        (final_layer_norm): T5LayerNorm()
        (dropout): Dropout(p=0.1, inplace=False)
    )
    (decoder): T5Stack(
        (embed_tokens): Embedding(32128, 512)
        (block): ModuleList(
        (0): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                (relative_attention_bias): Embedding(32, 8)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerCrossAttention(
                (EncDecAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (2): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        (1): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerCrossAttention(
                (EncDecAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (2): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        (2): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerCrossAttention(
                (EncDecAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (2): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        (3): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerCrossAttention(
                (EncDecAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (2): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        (4): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerCrossAttention(
                (EncDecAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (2): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        (5): T5Block(
            (layer): ModuleList(
            (0): T5LayerSelfAttention(
                (SelfAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (1): T5LayerCrossAttention(
                (EncDecAttention): T5Attention(
                (q): Linear(in_features=512, out_features=512, bias=False)
                (k): Linear(in_features=512, out_features=512, bias=False)
                (v): Linear(in_features=512, out_features=512, bias=False)
                (o): Linear(in_features=512, out_features=512, bias=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            (2): T5LayerFF(
                (DenseReluDense): T5DenseReluDense(
                (wi): Linear(in_features=512, out_features=2048, bias=False)
                (wo): Linear(in_features=2048, out_features=512, bias=False)
                (dropout): Dropout(p=0.1, inplace=False)
                )
                (layer_norm): T5LayerNorm()
                (dropout): Dropout(p=0.1, inplace=False)
            )
            )
        )
        )
        (final_layer_norm): T5LayerNorm()
        (dropout): Dropout(p=0.1, inplace=False)
    )
    (lm_head): Linear(in_features=512, out_features=32128, bias=False)
    )

3 最终成绩

本项目最终成绩为0.32107609

参数设置如下：

  • 模型：bart-large-cnn
  • 搜索束个数：2
  • 最大序列长度：1024
  • 激活函数：gelu
  • 预测序列最短长度：30
  • 预测序列最长长度：590
  • 是否允许提前停止（预测出<EOS>即停止）：是

About

2021CCF BDCI 新闻摘要自动生成

pretrained-modelspytorchsummarizer

License:Apache License 2.0

Languages

Language:Python 100.0%