[WIP][New Model] PerpetualBoosting

[LennartPurucker]

This code adds the PerpetualBooster (https://github.com/perpetual-ml/perpetual).

Benchmark TabArena-Full Results

All results: perpetual_boosting.zip
Raw results: https://data.lennart-purucker.com/tabarena/data_PerpetualBoosting.zip

Notes

• I only evaluated the one default config, which already has a lot of open TODOs, before we should try it with HPO.
• The code has a few problems with memory management. This goes so far that I was not able to get one dataset with many categoricals to run at all. So I had to impute one dataset (see the LB impute column)
• Another big problem is that one cannot add a callback for early stopping on external validation data.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[LennartPurucker]

The authors of PerpetualBoosting have closed the integration issue on their side.
So it is unlikely that the model will receive the updates or integration (with verification by the authors) that is required.

Moreover, as the initial results do not seem promising enough, I am not willing to invest more time into fixing the problems for the integration myself in the near future.

How should we proceed? @Innixma @dholzmueller
Should we still include these results and add some kind of additional tag or column to indicate clearly that this is even more experimental/unverified than our other unverified results?

[Innixma]

@LennartPurucker Up to you. IMO we could keep a list somewhere of models that were not finished being integrated, with a link to the GitHub PR/Issue? That way we don't need to get the actual results themselves in the codebase. I'd prefer the results we have available to see in the code-base to pass some level of quality bar.

[LennartPurucker]

I have updated the PR to the current state of TabArena and integrated the feedback from the author in their issue.

Some of the bugs have not been addressed but maybe it is fine now. I will rerun the benchmark and then we can merge this model IMO.

[deadsoul44]

Hi,

I am the author of PerpetualBooster. I have a couple of comments:

1. You can rename PerpetualBoosting to PerpetualBooster.
2. budget parameter is not meant to be tuned but for the purpose of getting the best result, you can treat it as a hyperparameter to be tuned. Try running the benchmark with 0.5, 1.0, 1.5, 2.0 and get the best result on validation set.
3. PerpetualBooster doesn't need a separate set for early stopping. The algorithm stops itself when it doesn't see any performance gain. You can use all data at the end for the final performance.
4. Increase iteration_limit to 10,000.

Let me know if you need any detail about the algorithm.

[LennartPurucker]

Heyho @deadsoul44,

Great to see you here!

To clarify: the code already implements (2) and (3).
I will add (4) again, thanks! I will rename the model as well.

[LennartPurucker]

It seems my CPU compute was put on hold for ~two weeks, but afterward, we will have new and better hardware. Sorry for the delay!

[deadsoul44]

v1.1.2 is released with improved performance and numerical stability. Benchmarked version can be updated.

[LennartPurucker]

@deadsoul44 sounds great, I will check also for the newest version once I start the benchmarks, thank you!

[deadsoul44]

Do you have any plans to add a "Large" option to the All Dataset section? In this option, datasets with more than 1,000,000 samples can be included.

[LennartPurucker]

@deadsoul44, yes, very much! The next version of TabArena will include datasets with more than 250k samples.
However, so far, we have no datasets larger than 250k, so we can only have a medium tab.

Note, we did not filter large datasets, but our first curation round for TabArena-v0.1 simply did not find any larger datasets following our curation rules, sadly.