1·
8 months ago15 seconds per reply with just 1 token/s?! How short are they? What’s the context size to be processed? I get like 5 tokens per second on my GPU and need 1-2 minutes per reply on 4k context size.
- 0 Posts
- 2 Comments
Joined 1 year ago
Cake day: March 9th, 2024
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
I mean the actual context size to be processed for the message, based on chat history, character cards, world info, etc. And which model?