Instructions to use mykor/bge-m3.gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- llama-cpp-python
How to use mykor/bge-m3.gguf with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="mykor/bge-m3.gguf", filename="Bge-M3-567M-BF16.gguf", )
output = llm( "Once upon a time,", max_tokens=512, echo=True ) print(output)
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use mykor/bge-m3.gguf with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf mykor/bge-m3.gguf:Q4_K_M # Run inference directly in the terminal: llama-cli -hf mykor/bge-m3.gguf:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf mykor/bge-m3.gguf:Q4_K_M # Run inference directly in the terminal: llama-cli -hf mykor/bge-m3.gguf:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf mykor/bge-m3.gguf:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf mykor/bge-m3.gguf:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf mykor/bge-m3.gguf:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf mykor/bge-m3.gguf:Q4_K_M
Use Docker
docker model run hf.co/mykor/bge-m3.gguf:Q4_K_M
- LM Studio
- Jan
- Ollama
How to use mykor/bge-m3.gguf with Ollama:
ollama run hf.co/mykor/bge-m3.gguf:Q4_K_M
- Unsloth Studio
How to use mykor/bge-m3.gguf with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for mykor/bge-m3.gguf to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for mykor/bge-m3.gguf to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for mykor/bge-m3.gguf to start chatting
- Atomic Chat new
- Docker Model Runner
How to use mykor/bge-m3.gguf with Docker Model Runner:
docker model run hf.co/mykor/bge-m3.gguf:Q4_K_M
- Lemonade
How to use mykor/bge-m3.gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull mykor/bge-m3.gguf:Q4_K_M
Run and chat with the model
lemonade run user.bge-m3.gguf-Q4_K_M
List all available models
lemonade list
How to use from
LemonadeRun and chat with the model
lemonade run user.bge-m3.gguf-List all available models
lemonade listQuick Links
bge-m3.gguf
import torch
from llama_cpp import Llama
from sentence_transformers import SentenceTransformer
from scipy.spatial.distance import cosine
model = SentenceTransformer(
"BAAI/bge-m3",
model_kwargs={"torch_dtype": torch.float16}
)
llm = Llama.from_pretrained(
"mykor/bge-m3.gguf",
filename="bge-m3-567M-F16.gguf",
embedding=True,
verbose=False,
)
text = """μμΈ λ¬λ μ΄κΉ¨λ₯Ό λ°λΌμ λ€μ μ λ¬Όμ΄κ°λ μ€λμ λ
λ°€μ΄ μ‘°μ©ν λλ₯Ό μμΌλ©΄ 무λμ Έκ°λ λ μμ΄λ²λ¦΄ μ μμ΄
μ λ°λ μ€λμ ν¬λ§ μμ λ΄μΌμ ꡬλ¦μ λ리μ°κ³
λ€μ κΉμ μ μ λΉ μ Έλ€μ΄, κ·Έλ μ μν μ°μ΅μΈ κ²μ²λΌ
μ§λ¦¬μ§λ μκ³ λλ₯Ό μ²λ°©νλ λ§μ½μ΄λΌλ λ§
νμ λκ°μ λ§€μΌμ λ΄μ±μ΄ λμ΄ λ΄μΌμ μ΄μ§λ¬μ΄ 무λλ¨λ €
μ°λΌλ¦° λ μ μ°λΌλ¦° λλ₯Ό μΌν€μ§ λͺ»ν΄ λ±μ΄λ΄κ³ μΆμλ λ°€
μλ―Έλ μμ΄ κ±΄λ¨ μμ μ λ§, μΆλ½μ ν₯ν΄ μ¬λΌκ°λ λ λ§λ€μ΄
κ·Έλ¦¬μ΄ λ μ λλ¦¬μ΄ λ§μ΄ μλ¦λ€μ λ λ λ€μ λ§μΉ ν κΉ λ΄
μμ΄λ²λ¦΄κ², λμ κ°κ³
ν©μ΄μ Έ μ¬λΌμ§ λ―ν κ·Έλ ν무νκ³ μ λ¬ν κ½λ§μΈ
λͺ¨μ§κ² λ΄λ¦° λλ¬Όμ μ 겨 νΌμ§ λͺ»νκ³ λ©μΆ°μμ§λ§
μ°¨λμ°¬ μ² κΈΈ μμ λμ¬ λμκ° λ°©ν₯μ λͺ¨λ₯Ό λΏμ΄μΌ
λ΄κ° κ·Έλ λ μμ κ·Έλ¬λͺ¨μ λ μΌμ κ½ ν λ΄μΌμ λΉμΆκ² ν΄μ€
λ©λ§λ₯Έ κ½μμ΄ μ½μ§ λͺ»ν μ€λμ κ°νΌλ₯Ό κ½μμ
λ μ΄μ κ·Έλ μ½μ§ λͺ»νλ λλ κ·Έμ μ€λμ λμ λ§€λ¬λ¦΄ λΏ
μ°¬λν λ μ μ°¬λν κ·Έλ μ°¨λ§ λΉμΆμ§ λͺ»νκ³ μ€λ¬μ Έκ°λ λ―
μ¬μ₯μ λμ§μ΄λ΄ νκ» μ리μ³λ κ²°λ§μ ν₯ν΄ μΆλ½νλ μ°λ¦¬κ° μμ΄
κ·Έλ¦¬μ΄ λ μ λλ¦¬μ΄ λ§μ΄ λ΄μΌμ‘°μ°¨ νλ½νμ§ μλλ€ ν΄λ
μμ§ μμκ², λ λ κ°λ λ κΉμ§
νΌμ΄λκ³ νΌμ΄λλ μλ€μ΄λ²λ¦¬λ μ¬νμ΄λ κ½
μ§μ΄μ Έλ§ κ°λ κ·Έλμ μνμ΄ λ§μ§λ§μ ν₯ν΄ κ½μ νΌμλ΄κ³ μμ΄
κ³ λ§μ μ΄, λ―Έμνμ΄, μμμ κ°λ νμ κ½λ€λ°κ³Ό
λλ₯Ό λ λκ°λ κ±Έ
μ¬μ€μ λλ μμμ, μ΄μκ°κ³ μΆμ΄, λ°λ €λλ μ λ§μ 묻ν μ¬λΌμ§λ
μν μ€λκ³Ό λλ €μ΄ λ΄μΌ κ·Έ μ¬μ΄μ μ΄λμ λ€κ° λ€μ΄μμ΄
μ°λΌλ¦° λ μ μ°¬λν λ€κ° λ΄κ² μ΄μμμ΄μ€μ κ·Έμ κ³ λ§λ€κ³
μμ§ μμκ² μμν"""
embed1 = model.encode(text)
embed2 = llm.embed(text)
print(cosine(embed1, embed2))
2.879617546280855e-05
- Downloads last month
- 249
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
16-bit
32-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. π Ask for provider support
Model tree for mykor/bge-m3.gguf
Base model
BAAI/bge-m3
Pull the model
# Download Lemonade from https://lemonade-server.ai/