how can we obtain the embedding model after fitting the classifier?

[fyears]

https://minish.ai/packages/model2vec/training

After classifier.fit , can we obtain the StaticModel again? So that we can generate the new embedding (after being biased to the classification task)

[Pringled]

Hi @fyears, this is unfortunately not possible since the classifier is not an embedding model, it's fine-tuned to predict the classes that were used during training (the embedding layer after training is likely not usable anymore for making general embeddings). For fine-tuning a general embedding model, I suggest using https://github.com/MinishLab/tokenlearn, which is also what we used for creating our own pre-trained embedding models.

[fyears]

I see.

Initially , I want to do this, because StaticEmbedding classification score (f1 on test set) is not good enough in my specific task. So I would like to export the finetunee embedding for down stream task (for example, as the input for a xgb model!).

So my best bet is using the original model embedding, or the predict_proba result after the fitting?

[Pringled]

I think for that it's probably better to directly use embeddings from either https://huggingface.co/minishlab/potion-base-32M for English, or https://huggingface.co/minishlab/potion-multilingual-128M for multilingual data. Fine-tuning the model as a classifier will likely make the embeddings themselves meaningless except for the classification task itself; though you can ofcourse try it. In that case you should indeed take the StaticModel.embedding from your fine-tuned model. It's hard to say what will work best here so I'd suggest just trying both if that's feasible.