Potential solution for the issue of it knowing relative time differences
chrish-slingshot opened this issue · comments
Hey. Sorry, couldn't see the discussions section so I thought this would be the next best place. I saw in the memory recall code you mentioned you need to get it to adjust for relative time differences. One idea I've had for this is to always send the current time through when you send a prompt to the AI, so you prepend the message with something like [2023-04-03 11:05:23]. Then you give it some instructions to tell it that you'll always send the current time like this, and it can use that to work out the relative elapsed time between messages (or in your case, the memories when it goes to look them up). It takes some prompting to get it to not repeat writing out this time difference every time in it's response, but I've had some success with this in getting it to recognise a long or short amount of time has passed between certain events.
There's a prompt which the app sends every 5 messages which gives it the current date and time, but it might work better if it was included in every prompt.
The example I couldn't get to work was something like this:
- 1st April - tell it "yesterday I went to the shops" so it records 'memory stored' as 1st April
- 2nd April - ask it when I went to the shops
If you want to reproduce it you could add an entry to the memory file.
It reliably tells me I went to the shops yesterday which is untrue. Even with further prompting it still insists it's right. I think only once or twice I got it to understand that I actually went to the shops two days ago, but not consistently, and all my attempts ended up being wasted tokens.
If you can find a variation of the prompt which works properly that'd be awesome, but I don't know if I'll have much time to play around with this bit of the code for a while.
I'm attempting to reproduce something similar to your code in C# (only because I know the language a lot better). If I find a solution for it in there I'll let you know.
One thing I have found is that you can pretty reliably use the 3.5 model for generating code, so you can save some calls to the v4 model there. It's a lot faster too.
Apologies for the formatting, I'm on my mobile. I had some success, even in the 3.5 model! See below. Needs more testing to see how it goes longer term through the conversation.
///
User:
I am going to prepend all my messages with the current timestamp in the format:
[2023-04-04 16:07:00]
You can use this timestamp to give you context of when a message was sent and how much time has passed between messages. When a date, time or similar (such as "yesterday", "last Monday", "tomorrow") is mentioned in my messages, you should refer to the timestamp of the original message to provide context on the relative date from the current date. For example, if I send a message like this:
"[2023-04-03 11:56:34]
Tomorrow I will go to the shops"
And then send the following message:
[2023-04-04 09:32:55]"
Then your response should be "today", because the first message was sent a day before saying I would go to the shops tomorrow, and now a day had passed and I will go to the shops today.
Please ask me the follow up questions now to clear up your understanding. Once I have answered your questions I will reply "GO" and we can implement these rules.
///
Thank you for letting me know about your new message format. Just to clarify, when you say "relative date from the current date," you mean the date that corresponds to the timestamp of the original message, correct? Also, do you want me to always use the same time zone for the timestamps, or should I convert them to the time zone of the user?
///
User:
You should always assume a timezone of GMT. The relative date from the current date should be taken from the timestamp of the message that has just been received.
///
Thank you for clarifying. Just to confirm, I should use the timestamp of the message that I am currently responding to, rather than the timestamp of the initial message, to determine the relative date, correct?
///
User:
Correct
///
Great, I understand now. In that case, I am ready to implement these rules. Please start sending messages with the timestamp format "[YYYY-MM-DD HH:MM:SS]" and I will use it to provide context for our conversation.
///
User:
GO
[2023-04-04 13:55:00]
Yesterday I went to a party
///
Thanks for letting me know. So today is April 4th, which means that you went to the party on April 3rd. Was it a fun party?
///
User:
[2023-04-06 09:44:23]
How many days ago did I go to the party?
///
Based on the timestamp of your previous message, you went to the party two days ago, on April 4th.
///
User:
[2023-04-09 14:43:11]
How many days will it be tomorrow since I went to the party?
///
Based on the timestamp of your previous message, tomorrow will be April 10th, which means that you went to the party six days ago, on April 4th
That looks like a great improvement! I'll update the memory prompt when I get a chance.
Thanks @chrish-slingshot 🙏
@chrish-slingshot @ian-kent Whoa there, guys. Let's take a closer look at this. I've done a decent amount of work with GPT-4 text adventure games and I noticed that tokens get chewed up by ISO-8601 timestamps. The main thing to keep in mind with any piece of context that's going to be repeated is 1 short english word = 1 special character.
Head over to https://platform.openai.com/tokenizer
Input: [2023-04-09 14:43:11]
Tokens: 13
Input: April 9 14:43:11
Tokens: 7
So I would expect these to be about equally readable to the AI but one's twice as long. You could get rid of the brackets, etc but that's still 11 tokens as opposed to 7 (multiplied by how many timestamps in the context?)
Furthermore, consider the fact that while special characters often take up a token on their own, spaces are almost always grouped with the word ahead of them. So if you're creating a list:
- cherry: 2 tokens
cherry: 1 token
You can think of this as a "one space indent for a list of items". But the space doesn't add a token per item.
Getting back to timestamps, though...
April 9: 2 tokens
April 9th: 3 tokens
I'd also recommend AM/PM indicators because otherwise a time like 08:30 is ambiguous due to some people using it in circumstances where the am/pm is implied.
Just like the months, am and pm are each 1 token.
Btw, you can use the full month name because all 12 of them are singular tokens. Also,
12 30pm: 3 tokens
12:30pm: 4 tokens
12:30:56pm: 6 tokens
April 9 12:30:56pm: 8 tokens
April 9 12:30:56.001pm: 10 tokens
2023 April 9 12:30:56.001pm: 12 tokens
1685710607: 4 tokens (unix timestamp; seconds)
1685710655538: 5 tokens (javascript timestamp; milliseconds)
(these are just a variety of options that each will be suited for various situations... for example, I doubt the AI could tell you the hour of the unix timestamp... but it's a very concise way to ensure knowledge of the relative order of things)
It seems to me that a reasonable way to go with this is to use one of the bottom two options for tagging messages or whatever and apply a reference point every so often:
1685710655538 = 9:00am EST (timezone indicators are also a single token)
By prepending a space to each timestamp, you can denote it as separate from the rest of the text without using additional tokens:
1685710655538: 5 tokens (still)
The space prepending trick is very useful for lists and for tags/metadata. If you're concerned about ambiguity, you can use a few of the tokens you saved with it: "/^ / =
There's another interesting thing that happens with space prepending.... it sometimes reduces the number of tokens:
Astronaut: 3 tokens
Astronaut: 2 tokens
My guess on this is "when it comes to English, there's a lot more words with spaces before them than spaces after them".
Oh, and markdown is very token-efficient and well-understood by LLMs because of how ubiquitous they are while offering a structured but flexible framework for data that doesn't very often conflict with other content.
OH! And the dot language for graphviz graphs is very well-understood by LLMs and fairly token efficient. It's an excellent way to discuss relationships between entities.
Some context for my game might look something like:
# Map
digraph G {
A -> B
B -> C
A -> C
C -> D
}
# Locations
## [A] The Castle
Stuff about the castle...
## [B] The Woods
Stuff about the woods...
...
# Current State
Location:A
# User Input
inventory@timhuff Thanks, that's actually really interesting seeing how it compresses down the tokens in various formats. For the context of what I'm using it for (separate to this project), the token size isn't such a major issue as generally the accompanying message size will far outweigh it anyway (typically with the initialisation prompt I'm using, messages come in at about 700 - 1,000 tokens each). But I can definitely take away some of the points you made about how to structure it in order to make it convey the time more accurately.
So for the digraph stuff, you're basically providing it with a state machine and specifying how it can move between the various states?
yea, it's graphviz code. Useful for all sorts of stuff. It's common enough that GPT-4 is well-aware of how to read it. Very compact and full-featured. You can use it to draw a finite state machine. But in my game, I relied on the LLM to take on the task of interpreting it. The game was made such that the "cartridges", let's call them, were basically rough blueprints for the structure, thematic elements, etc of the game. I guess it's a bit like D&D in that way. The LLM is tasked with filling in all the detail and doing improv as needed.
Naturally, that game was a "send a 5 word message every 10 seconds" so token management and context management was extremely important to keep cost down. So I had to do a bit of experimentation in "prompt compression". That's right; my LLM could fold into a small ball at a moment's notice! (jk)
https://en.wikipedia.org/wiki/DOT_(graph_description_language)