shpotes / ELECTRA-lightning

An unofficial reimplementation of ELECTRA on pytorch (lightning)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[WIP] ELECTRA Lightning

This is just another ELECTRA re-implementation, but on pytorch-lightning. Currently, there are several ELECTRA implementation, which have helped me in considerably to make my own, in particular, the implementations made by Richard Wang, Phil Wang and of course the official implementation by Google.

Motivation

Although all the implementation mentioned above have a high quality. I want to do some multi-modal experiments using this model. So I need a bit more of control. In addition, I also want to play a bit training language models and ELECTRA is the only one (for now) that allows me to train interesting models using achievable computational power.

About

An unofficial reimplementation of ELECTRA on pytorch (lightning)

License:MIT License


Languages

Language:Python 100.0%