预训练

[pythonla]

您好，想问一个热别简单的问题。bert提供的预训练好的模型都训练好哪些参数呀？ 难道和代码中初始化的参数不一样么？如果一样的话，随机初始化的话不相当于白训练了？谢谢！

`class BertPreTrainedModel(nn.Module):
""" An abstract class to handle weights initialization and
a simple interface for downloading and loading pretrained models.
"""

def __init__(self, config, *inputs, **kwargs):
    super(BertPreTrainedModel, self).__init__()
    if not isinstance(config, BertConfig):
        raise ValueError(
            "Parameter config in `{}(config)` should be an instance of class `BertConfig`. "
            "To create a model from a Google pretrained model use "
            "`model = {}.from_pretrained(PRETRAINED_MODEL_NAME)`".format(
                self.__class__.__name__, self.__class__.__name__
            ))
    self.config = config

def init_bert_weights(self, module):
    """ Initialize the weights.
    """
    if isinstance(module, (nn.Linear)):
        # 初始线性映射层的参数为正态分布
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
    elif isinstance(module, BertLayerNorm):
        # 初始化LayerNorm中的alpha为全1, beta为全0
        module.beta.data.zero_()
        module.gamma.data.fill_(1.0)
    if isinstance(module, nn.Linear) and module.bias is not None:
        # 初始化偏置为0
        module.bias.data.zero_()

`

[zhaohu xing]

bert 只是做了一个编码而已 要想做下游任务 还得加点别的层。

[pythonla]

原来如此，谢谢了