How to add new control code into vocabulary?

Question

How to add new control code into vocabulary?

leejason opened this issue 5 years ago · comments

Is it possible or is there any code for adding new control code into the vocabulary file?

parser.add_argument('--control_code', type=str, required=True,
                                        help='control code to use for this file. must be in the vocabulary, else it will error out.')

Nitish Shirish Keskar · Answer 1 · Wed Oct 16 2019 01:10:25 GMT+0800 (China Standard Time)

You can't add a control code that isn't in the vocabulary. Typically, exact specification of the token is not needed, a synonym (or really, any unused code) should work. What did you have in mind?

leejason · Answer 2 · Thu Oct 17 2019 17:29:06 GMT+0800 (China Standard Time)

Patent classification codes which may have 600+ codes. Assuming that such codes are not in vocabulary, would CTRL model work?

Nitish Shirish Keskar · Answer 3 · Sat Oct 19 2019 00:59:10 GMT+0800 (China Standard Time)

No, they certainly need to be in the vocabulary at the moment. But, maybe what can work is that you can just start using word2idx[0], word2idx[1], ..., word2idx[600] as the control codes? It's convenient for the word used for the control code to have some relevance to the domain but that's not important to the model.

leejason · Answer 4 · Sat Oct 19 2019 17:29:01 GMT+0800 (China Standard Time)

Interesting ideas & thanks. Let me try later.