I think doing the calculations at full precision (FP16) is a waste. You should try somewhere between the Q4_K_M version to Q6_K (or at least Q8_0, that’s supposed to the same quality as FP16). That way it should be considerably faster… At least twice as fast.

(The GGUF page of that model has a list of recommended quantization levels.)

Nice! KoboldCpp is also my software of choice. It’s easy to install, all-in-one and has a good amount of features.

What kind of model size do you use to arrive at 1token/s? I’m in the same ballpark. Though my old desktop PC is a bit faster than my laptop. Probably because it has dual-channel memory and doesn’t throttle.

I think that’s the point where it gets usable. At least for consecutive chat. If I feed in longer text, or KoboldCpp decides to recalculate large portions of the context, it’ll be several minutes for me until I get a reply. And that’s less fun for use-cases like dialougue.

Nice, thanks for sharing and posting something in this low-frequency community 😅

Is there a way to pull out the character cards / prompts from poe? Bc I use a different service and I’m more used to downloading characters from characterhub.org which lets you see the full text.

• L3-8B-Stheno ( GGUF ) seems quite nice, worthy successor to Fimbulvetr
• also L3-8B-Lunaris seems really good