In this post, I present a very cool tool for comparing multiple machine learning algorithm.
Suppose that you have multiple learning algorithms and that you have to find which one performs better. By performing better, I mean that one algorithm would have more chances to produce a better estimator than another. This comparison is crucial because you only can use your test set once, so you don’t wan’t to choose an algorithm that would return a poor estimator. How can you make an more educated guess about the algorithm to select? The answer to this question could surely be to use the binomial poisson test!
The binomial poisson test was proven to be a very effective test. Alexandre Lacoste, from Laval University, posted a web tool for doing this test for you and I. Basically, this will help you determine if your new algorithm is better than some state of the art algorithms. Also, most importantly, it will tell you if you have enough data to produce such a conclusion.
This project is definitely not mine and I do not intend to get any credit out of this post. Nevertheless, since I’ve used this tool and that it has proven itself very efficient, I wanted to contribute to its sharing to the world.
If you want to see the code, just go there: https://code.google.com/p/mleval/
If you want to use the web tool directly, just go here instead: http://graal.ift.ulaval.ca/mleval/.
On both pages, you can get access to the paper that demonstrate the efficiency of the method. Enjoy!