I can reaffirm what they said with slightly more proof.

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard

I am using this: https://github.com/oobabooga/text-generation-webui … It is running great with my AMD 7900XT. It also ran great with my 5700xt. It sets up itself within a conda virtual environment so it takes all mess out of getting the packages to work correctly. It can use NVIDIA cards too.

Once you get it installed you can then get your models from huggingface.co

I’m on arch, btw. ;)

Edit: I just went and reinstalled it and saw it supports these gpus

Open-webui is the best self hosted LLM chat interface IMO. It works seamlessly with Ollama, but also supports other openAI-API compatible APIs AFAIK.

I’m using both in combination with each other and both downloading and using models is super easy. Also integrates well with VSCode extension “Continue”, an open source Copilot alternative (setup might require editing the extension’s config file).

it has in fact been a delightful creative aid for brainstorming fiction, actually!

The things are fantastic at “yes-and” improvisation and extrapolating from a premise.

If I want to build a world and populate it with loosely defined ‘impressionistic’ background info that doesn’t necessarily require fully fledged lore that interconnects, it can do a great job at showing where the lore could go if i decided to explore there. It’s great at suggesting character names, place names, and ways to fill in blanks that make it easier for me to pick or reject individual elements.

In a story idea I’ve been marinating for a while, one character possesses advanced medical knowledge in a world where germ theory, medicine, and surgery never developed because people had access to ‘healing magic’. The problem is, healing magic works on all organisms - including parasites, bacteria, and cancer, which means trying to ‘heal’ someone with an infection makes the infection worse because the pathogens benefited from the healing magic.

I asked AI to extrapolate more detail about this character’s background and it suggested that his father was the village healer and simply didn’t mention his mother at all.

Those two little details exploded in my imagination as an entire history of emotional conflict:
His mother fell ill with a bacterial infection that magic couldn’t fix when he was too little to do anything about it even though he knew what was wrong and how to help her, and so he blamed himself.
His ‘strange ideas’ about physiology, epidemiology, and concepts like hygiene and medicine put him at odds with the traditional teachings his father, and made the other people in his village view him as a ‘problem child’.
This led him to be quiet and withdrawn until he befriends the protagonist, and it is her falling ill when the same disease that killed his mother that motivates him to try again with the rudimentary resources he was able to secretly scrape together since.

(this is an ‘isekai inversion’ where all the reincarnators are disillusioned and discouraged, and the protagonist is a native of that world who travels around finding them, putting them in touch with one another, and motivating them to pursue their specializations again. A nuclear engineer, for instance, won’t be able to get much done in a world where the scientific method hasn’t been codified, manufacturing doesn’t exist let alone precision machining, and chemistry has not clawed its way to distinction out of the vague, secretive, formless depths of alchemy)

Lotsa processing power behind it all!