--- license: apache-2.0 language: - en - zh tags: - long-context - infllm - sparse-attention - llama-cpp - gguf-my-repo base_model: openbmb/InfLLM-V2-Long-Sparse-Base --- # AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF This model was converted to GGUF format from [`openbmb/InfLLM-V2-Long-Sparse-Base`](https://huggingface.co/openbmb/InfLLM-V2-Long-Sparse-Base) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space. Refer to the [original model card](https://huggingface.co/openbmb/InfLLM-V2-Long-Sparse-Base) for more details on the model. ## Use with Ollama ```bash root@90dd7d73d62b:/# ollama pull hf.co/AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF:Q8_0 pulling manifest pulling 5bdae6814d6a: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 8.7 GB pulling a47b02e00552: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 106 B pulling b78301c0df4d: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 38 B pulling d03c38a6a78c: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 557 B verifying sha256 digest writing manifest success root@90dd7d73d62b:/# ollama run hf.co/AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF:Q8_0 >>> hello Hello! How can I assist you today? >>> Write an article about Artificial Intelligence. Artificial Intelligence (AI) has revolutionized the way we live and work in recent years. AI refers to the ability of machines to simulate human intelligence and perform tasks that typically require human intelligence, such as learning, reasoning, problem-solving, decision making, perception, and language understanding. With the increasing demand for automation and efficiency, AI has become an essential tool across various industries, including healthcare, finance, manufacturing, transportation, and education. AI systems are designed to analyze vast amounts of data and learn from it to make predictions or decisions. They use algorithms that can identify patterns and relationships in data, and then apply these insights to real-world problems. AI systems can be classified into three types: narrow AI, general AI, and superintelligence. Narrow AI is designed to perform a specific task, such as recognizing faces or translating languages. General AI refers to machines that have the ability to understand and learn any intellectual task that humans can do. Superintelligence refers to machines that are much smarter than humans in every aspect, including creativity, wisdom, and emotional intelligence. AI has already made significant progress in several fields, with applications ranging from self-driving cars to virtual assistants like Siri and Alexa. In healthcare, AI can help doctors diagnose diseases more accurately and provide personalized treatment plans for patients. In finance, AI can detect fraudulent transactions and predict market trends. In manufacturing, AI can optimize supply chain management and improve production efficiency. In transportation, AI can enable autonomous vehicles to navigate safely on roads. In education, AI can personalize learning experiences for students based on their individual needs and preferences. However, AI also raises some ethical concerns and challenges. For instance, there are concerns about job displacement as machines replace human workers. There are also issues related to privacy and security, as AI systems collect and analyze vast amounts of personal data. Additionally, there is a risk of bias in AI algorithms that can perpetuate discrimination against certain groups of people. In conclusion, AI has the potential to transform various aspects of our lives, making them more efficient, effective, and personalized. However, it also presents challenges and risks that need to be addressed through responsible development and deployment of AI systems. >>> Send a message (/? for help) ``` ## Chat Template ```bash {% for message in messages %}{{'<|im_start|>' + message['role'] + ' ' + message['content'] + '<|im_end|>' + ' '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant ' }}{% if enable_thinking is defined and enable_thinking is false %}{{ ' ' }}{% endif %}{% endif %} ``` ## Use with llama.cpp Install llama.cpp through brew (works on Mac and Linux) ```bash brew install llama.cpp ``` Invoke the llama.cpp server or the CLI. ### CLI: ```bash llama-cli --hf-repo AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF --hf-file infllm-v2-long-sparse-base-q8_0.gguf -p "The meaning to life and the universe is" ``` ### Server: ```bash llama-server --hf-repo AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF --hf-file infllm-v2-long-sparse-base-q8_0.gguf -c 2048 ``` Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well. Step 1: Clone llama.cpp from GitHub. ``` git clone https://github.com/ggerganov/llama.cpp ``` Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux). ``` cd llama.cpp && LLAMA_CURL=1 make ``` Step 3: Run inference through the main binary. ``` ./llama-cli --hf-repo AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF --hf-file infllm-v2-long-sparse-base-q8_0.gguf -p "The meaning to life and the universe is" ``` or ``` ./llama-server --hf-repo AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF --hf-file infllm-v2-long-sparse-base-q8_0.gguf -c 2048 ```