Training 0.6B 1TT models with different Multisynth mixes

[vitiugin]

Train 2B 0.5/1TT models with different proportions of Multisynth data:
Option 1: Aim for near-uniform mix of languages other than English, include up to 50% language total translated data when the amount of native data is insufficient
Option 2: Aim for near-uniform mix of languages other than English, include 50% translated data even when sufficient native data is available
Option 3: Aim for near-uniform mix of languages other than English, including as much translated data as necessary to get close to uniform
Option 4: Aim for a balance between "natural" and uniform mix of languages other than English (Jörg Tiedemann when should we use translated data in this option?)

[vitiugin]

• 1TT sample datasets are available on LUMI and LEONARDO
• datamix tool is available here

[vitiugin]

We trained 4 0.6B models with described mixes on 100BT.

HellaSwag results demonstrates similar performance on English, but non-English averaged results demonstrate the best performance of option 4 mix (near natural non-En distribution).

Belebele (cf) results are quite noisy and lack a clear signal. We assume 100BT isn't enough to make conclusion.

Results of MultiBLiMP evaluations over four models shows similar results, with small leading of option 4 mix (near natural non-En distribution).