I use mixtral8x7b locally and it’s been great. I am genuinely excited to see ddg offering it and the service in general. Now I can use this service when not on my network.
What GPU are you using to run it? And what UI are you using to interface with it? (I know of gpt4all and the generic sounding ui-text-generation program or something)
I am using this: https://github.com/oobabooga/text-generation-webui … It is running great with my AMD 7900XT. It also ran great with my 5700xt. It sets up itself within a conda virtual environment so it takes all mess out of getting the packages to work correctly. It can use NVIDIA cards too.
Once you get it installed you can then get your models from huggingface.co
I’m on arch, btw. ;)
Edit: I just went and reinstalled it and saw it supports these gpus
Open-webui is the best self hosted LLM chat interface IMO.
It works seamlessly with Ollama, but also supports other openAI-API compatible APIs AFAIK.
I’m using both in combination with each other and both downloading and using models is super easy.
Also integrates well with VSCode extension “Continue”, an open source Copilot alternative (setup might require editing the extension’s config file).
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !privacyguides@lemmy.one
In the digital age, protecting your personal information might seem like an impossible task. We’re here to help.
This is a community for sharing news about privacy, posting information about cool privacy tools and services, and getting advice about your privacy journey.
You can subscribe to this community from any Kbin or Lemmy instance:
Check out our website at privacyguides.org before asking your questions here. We’ve tried answering the common questions and recommendations there!
Want to get involved? The website is open-source on GitHub, and your help would be appreciated!
This community is the “official” Privacy Guides community on Lemmy, which can be verified here. Other “Privacy Guides” communities on other Lemmy servers are not moderated by this team or associated with the website.
Moderation Rules:
We prefer posting about open-source software whenever possible.
No soliciting engagement: Don’t ask for upvotes, follows, etc.
Surveys, Fundraising, and Petitions must be pre-approved by the mod team.
Be civil, no violence, hate speech. Assume people here are posting in good faith.
Don’t repost topics which have already been covered here.
News posts must be related to privacy and security, and your post title must match the article headline exactly. Do not editorialize titles, you can post your opinions in the post body or a comment.
Memes/images/video posts that could be summarized as text explanations should not be posted. Infographics and conference talks from reputable sources are acceptable.
No help vampires: This is not a tech support subreddit, don’t abuse our community’s willingness to help. Questions related to privacy, security or privacy/security related software and their configurations are acceptable.
No misinformation: Extraordinary claims must be matched with evidence.
Do not post about VPNs or cryptocurrencies which are not listed on privacyguides.org. See Rule 2 for info on adding new recommendations to the website.
General guides or software lists are not permitted. Original sources and research about specific topics are allowed as long as they are high quality and factual. We are not providing a platform for poorly-vetted, out-of-date or conflicting recommendations.
I use mixtral8x7b locally and it’s been great. I am genuinely excited to see ddg offering it and the service in general. Now I can use this service when not on my network.
What GPU are you using to run it? And what UI are you using to interface with it? (I know of gpt4all and the generic sounding ui-text-generation program or something)
I am using this: https://github.com/oobabooga/text-generation-webui … It is running great with my AMD 7900XT. It also ran great with my 5700xt. It sets up itself within a conda virtual environment so it takes all mess out of getting the packages to work correctly. It can use NVIDIA cards too.
Once you get it installed you can then get your models from huggingface.co
I’m on arch, btw. ;)
Edit: I just went and reinstalled it and saw it supports these gpus
That’s right, “text-generation-webui”. At least its unambiguous lol. Thanks for sharing.
Open-webui is the best self hosted LLM chat interface IMO. It works seamlessly with Ollama, but also supports other openAI-API compatible APIs AFAIK.
I’m using both in combination with each other and both downloading and using models is super easy. Also integrates well with VSCode extension “Continue”, an open source Copilot alternative (setup might require editing the extension’s config file).