---
license: cc-by-nc-sa-4.0
language: en
tags:
- splade
- conversational-search
- multi-turn retrieval
- query-expansion
- document-expansion
- passage-retrieval
- knowledge-distillation
pipeline_tag: fill-mask
---

## [SIGIR 2025] DiSCo: LLM Knowledge Distillation for Efficient Sparse Retrieval in Conversational Search


This model is a conversational search adaptation of the original [SPLADE++ (CoCondenser-EnsembleDistil)](https://huggingface.co/naver/splade-cocondenser-ensembledistil) model. It retains the original document encoder and **finetunes the query encoder on QReCC**, a dataset designed for multi-turn conversational search.

Training is performed via **distillation from multiple teachers: human and [Mistral](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) rewrites**, allowing the model to better capture the semantics of conversational queries. For more details, see the original paper:

* DiSCo SPLADE - SIGIR 2025 full paper: https://arxiv.org/abs/2410.14609

> **Note:** This is the **query encoder**. For inference, you also need the corresponding [document encoder](https://huggingface.co/naver/splade-cocondenser-ensembledistil), which remains unchanged from the original SPLADE++ checkpoint. SPLADE can use asymmetric architecture: separate models for query and document representation.

## Evaluation:

(new!) Added evaluation of DiSCo on [CORAL](https://aclanthology.org/2025.findings-naacl.72.pdf).

| Model | Dataset    | Setting        | R@100 | R@10 | MRR | nDCG@3 |
|------------ |------------|----------------|------:|-----------:|-------:|-------:|
DiSCo multi-teach | QReCC      | In-domain      | 0.928 | 0.754  | 0.498    | 0.487  |
DiSCo multi-teach | CORAL      | Zero-shot OOD  | 0.833 | 0.642  | 0.509    | 0.428 |
DiSCo multi-teach | CAsT 2020  | Zero-shot OOD  | 0.531 | - | 0.483      | 0.353  |
DiSCo multi-teach | CAsT 2022  | Zero-shot OOD  | 0.334 | - | 0.512      | 0.322  |
DiSCo multi-teach | iKAT 2023  | Zero-shot OOD  | 0.147 | - | 0.192      | 0.125  |


## Usage:
Please refer to the DiSCo github for complete usage [[github]](https://github.com/SimonLupart/disco-conv-splade).

Below is also an example script for encoding a conversation:

> The input format is a flattened version of the conversational history.
q_n [SEP] a_{n-1} [SEP] q_{n-1} [SEP] ... [SEP] a_0 [SEP] q_0

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch.nn.functional as F
import torch


model = AutoModelForMaskedLM.from_pretrained("slupart/splade-disco-human-mistral")
tokenizer = AutoTokenizer.from_pretrained("slupart/splade-disco-human-mistral")
model.eval()

conv = [
    ("what's the weather like today?", "it's sunny."),
    ("should I wear sunscreen?", "yes, UV index is high."),
    ("do I need sunglasses?", "definitely."),
    ("where can I buy sunglasses?", "try the optician nearby."),
    ("how much do they cost?", None)
]

parts = [conv[-1][0]] + [x for q, a in reversed(conv[:-1]) for x in (a, q) if x]
text = " [SEP] ".join(parts)

inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
sparse = F.relu(logits).max(1).values.squeeze(0)

scores = [(tokenizer.convert_ids_to_tokens([i.item()])[0], sparse[i].item())
          for i in torch.nonzero(sparse).squeeze(1)]
for token, score in sorted(scores, key=lambda x: -x[1]):
    print(f"Token: {token:15} | Score: {score:.4f}")
```


## Citation
If you use our checkpoint, please cite our work:
```
@inproceedings{lupart2025disco,
  title={Disco: Llm knowledge distillation for efficient sparse retrieval in conversational search},
  author={Lupart, Simon and Aliannejadi, Mohammad and Kanoulas, Evangelos},
  booktitle={Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  pages={9--19},
  year={2025}
}
```