--- language: - en license: apache-2.0 tags: - sentence-transformers - sentence-similarity - feature-extraction - dense - generated_from_trainer - dataset_size:5600 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss base_model: Qwen/Qwen3-Embedding-0.6B widget: - source_sentence: What were the total assets at fair value on December 31, 2023? sentences: - In addition to its contractual cash requirements, the Company has an authorized share repurchase program. The program does not obligate the Company to acquire a minimum amount of shares. As of September 30, 2023, the Company’s quarterly cash dividend was $0.24 per share. - Effective January 1, 2023, we prospectively adopted new guidance that eliminated the recognition and measurement of TDRs. We evaluate all loans and receivables restructurings according to accounting guidance for loan refinancing and restructuring. Modifications to loans and receivables primarily include temporary interest rate reductions and placing the customer on a fixed payment plan not to exceed 60 months. - Total assets at fair value on December 31, 2023 were reported to be $71,921 million. - source_sentence: What were the key factors affecting the company's cash flow from operations in fiscal 2023? sentences: - General and administrative | $ | 950 | | $ | 2,025 | 113 | % Percentage of revenue | 11 | % | 20 | % | General and administrative expense increased $1.1 billion, or 113%, in 2023, compared to 2022. - Within two months after submission of each annual execution proposal, the Macao government will decide on their approval, and may request adjustments to specific projects, to the investment amount and to the execution schedule. - The company's cash flow from operations in fiscal 2023 was affected by various factors including changes in working capital components like accounts payable, inventories, and accounts receivable. - source_sentence: What percentage of the total U.S. dialysis patient service revenues were generated from government-based programs in 2023? sentences: - The document includes a 'Glossary of Terms and Acronyms' that provides definitions and explanations of financial terms used. - Profit before taxes for 2022 was $8,752 million and rose to $13,050 million in 2023. - In 2023, approximately 67% of the total U.S. dialysis patient service revenues were generated from government-based programs. - source_sentence: What was the effective income tax rate for the Company in 2023? sentences: - Chevron's oil-equivalent production in the UK has increased by approximately 4 percent from 2022 to 2023, as indicated in the production summary tables. - The Company’s effective income tax rate decreased to 25.1% in 2023 compared to 25.9% in the prior year. - In 2023, 45% of our consolidated Gross Merchandise Sales was generated when a seller or buyer, or both, were located outside of the United States. - source_sentence: Which section of the financial document addresses Financial Statements and Supplementary Data? sentences: - The gift card liability was $145,014 in 2022 and increased to $164,930 in 2023. - The 7% Notes due 2029 are scheduled to mature on February 15, 2029. - Financial Statements and Supplementary Data are addressed in Item 8 of the financial document. pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 model-index: - name: 'Qwen3 base Financial ' results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 1024 type: dim_1024 metrics: - type: cosine_accuracy@1 value: 0.7507142857142857 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8707142857142857 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8985714285714286 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9364285714285714 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.7507142857142857 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.29023809523809524 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.1797142857142857 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09364285714285712 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.7507142857142857 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8707142857142857 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8985714285714286 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9364285714285714 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.846090041345316 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.8169348072562356 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.8197317550291238 name: Cosine Map@100 --- # Qwen3 base Financial This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) - **Maximum Sequence Length:** 32768 tokens - **Output Dimensionality:** 1024 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - json - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False, 'architecture': 'Qwen3Model'}) (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("PhilipCisco/qwen3-base-financial2") # Run inference queries = [ "Which section of the financial document addresses Financial Statements and Supplementary Data?", ] documents = [ 'Financial Statements and Supplementary Data are addressed in Item 8 of the financial document.', 'The 7% Notes due 2029 are scheduled to mature on February 15, 2029.', 'The gift card liability was $145,014 in 2022 and increased to $164,930 in 2023.', ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) print(query_embeddings.shape, document_embeddings.shape) # [1, 1024] [3, 1024] # Get the similarity scores for the embeddings similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) # tensor([[0.7375, 0.1121, 0.0035]]) ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `dim_1024` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 1024 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.7507 | | cosine_accuracy@3 | 0.8707 | | cosine_accuracy@5 | 0.8986 | | cosine_accuracy@10 | 0.9364 | | cosine_precision@1 | 0.7507 | | cosine_precision@3 | 0.2902 | | cosine_precision@5 | 0.1797 | | cosine_precision@10 | 0.0936 | | cosine_recall@1 | 0.7507 | | cosine_recall@3 | 0.8707 | | cosine_recall@5 | 0.8986 | | cosine_recall@10 | 0.9364 | | **cosine_ndcg@10** | **0.8461** | | cosine_mrr@10 | 0.8169 | | cosine_map@100 | 0.8197 | ## Training Details ### Training Dataset #### json * Dataset: json * Size: 5,600 training samples * Columns: anchor and positive * Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | anchor | positive | |:--------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | What was the increase in sales and marketing expenses for the year ended December 31, 2023 compared to 2022? | Sales and marketing expenses increased by $42.5 million, or 6%, for the year ended December 31, 2023 compared to 2022. | | What method is used to provide information about legal proceedings in the Annual Report on Form 10-K? | Information about legal proceedings in the Annual Report on Form 10-K is incorporated by reference under several notes and sections. | | How did selling, distribution, and administration expenses change in 2023 compared to previous years? | In 2023, the decline in Selling, distribution and administration expense was driven by lower compensation expense associated with workforce reductions, lower costs for professional services and lower freight and warehousing expenses as a result of lower shipments during 2023. Additionally, Selling, distribution and administration expense in 2023 included $116.0 million of intangible asset impairment charges as compared to $281.0 million of intangible asset impairment charges in 2022. | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 1024 ], "matryoshka_weights": [ 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 4 - `per_device_eval_batch_size`: 4 - `gradient_accumulation_steps`: 8 - `learning_rate`: 2e-05 - `num_train_epochs`: 4 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `bf16`: True - `tf32`: True - `load_best_model_at_end`: True - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 4 - `per_device_eval_batch_size`: 4 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 8 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 4 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: True - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `parallelism_config`: None - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `hub_revision`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `liger_kernel_config`: None - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | dim_1024_cosine_ndcg@10 | |:-------:|:-------:|:-------------:|:-----------------------:| | -1 | -1 | - | 0.7762 | | 0.1713 | 10 | 0.0243 | - | | 0.3426 | 20 | 0.0269 | - | | 0.5139 | 30 | 0.0171 | - | | 0.6852 | 40 | 0.0224 | - | | 0.8565 | 50 | 0.0376 | - | | 1.0 | 59 | - | 0.8200 | | 1.0171 | 60 | 0.0221 | - | | 1.1884 | 70 | 0.0089 | - | | 1.3597 | 80 | 0.0127 | - | | 1.5310 | 90 | 0.0116 | - | | 1.7024 | 100 | 0.0086 | - | | 1.8737 | 110 | 0.0113 | - | | 2.0 | 118 | - | 0.8280 | | 2.0343 | 120 | 0.0074 | - | | 2.2056 | 130 | 0.0077 | - | | 2.3769 | 140 | 0.0107 | - | | 2.5482 | 150 | 0.0089 | - | | 2.7195 | 160 | 0.0098 | - | | 2.8908 | 170 | 0.006 | - | | **3.0** | **177** | **-** | **0.8448** | | 3.0514 | 180 | 0.0111 | - | | 3.2227 | 190 | 0.0074 | - | | 3.3940 | 200 | 0.0082 | - | | 3.5653 | 210 | 0.0047 | - | | 3.7366 | 220 | 0.0076 | - | | 3.9079 | 230 | 0.0085 | - | | 4.0 | 236 | - | 0.8461 | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.14 - Sentence Transformers: 5.1.0 - Transformers: 4.56.1 - PyTorch: 2.8.0+cu128 - Accelerate: 1.10.1 - Datasets: 2.19.1 - Tokenizers: 0.22.0 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```