Implement a sane token scheme to make sure the context is filled with the balance of room for answer

[Dion Almaer]

NOTES:

"DEFAULT_CONTEXT_TOKEN_COUNT = 1500
Is this a “safeish” number?

I’m looking this from:

• We have 4000
• Embeddings have token counts in the results.

When I grab a bunch of context I think I want to:

• sort based on similarity
• grab enough to fit 4000 - how much the rest of the prompt takes - how much we want for the answer
• then go through the context adding the token count until I hit that amount

Can use this to get the token count of text: GPT Encoder

[Dion Almaer]

Implemented in #b8eeb19