Albert-tiny English support for NLU tasks

[Rishabbh]

Is there a way to get an equivalent albert-tiny english language model to perform downstream tasks like intent and entity classification. I'm afraid there is no albert-tiny model present hence any lead on this regards or guide to create one from scratch, would be highly appreciated.
Thanks

[Mu Yang]

It seems there's no pre-trained english albert-tiny model.
Maybe you can try bert-tiny models instead.

[Rishabbh]

Thanks for your response however I'm aware of bert-small/tiny and used them already. This is more to understand, have you created the Chinese version of albert-tiny which you've used in this repo ? if yes, how can I do the same for English language.

[Mu Yang]

Here is the Chinese ALBERT tiny model of this repo: https://huggingface.co/ckiplab/albert-tiny-chinese

This model is trained by the legacy LM training script with HuggingFace v3.5.1
https://github.com/huggingface/transformers/tree/v3.5.1/examples/contrib/legacy

To train LM model by scratch, I created a config.json file like https://huggingface.co/ckiplab/albert-tiny-chinese/blob/main/config.json.
Since ALBERT does not provide Chinese tokenizer, I use the bert-base-chinese tokenizer instead. (However, you may use the ALBERT tokenizer since you are using English)

[Rishabbh]

@emfomy , thanks for your response. I will go thru the files you'd referred. thanks a lot for your time again.