I realized today after googling for a bit that a few others were also unable to reproduce the results.
Is there a list of such papers?
It will save people a lot of time and effort.
Easier to compile a list of reproducible ones…, one user responded.
Probably 50%-75% of all papers are unreproducible.
Its sad, but its true, another user wrote.
Think about it, most papers are optimized to get into a conference.
So they dont have to worry about reproducibility because nobody will attempt to reproduce them.
40% off TNW Conference!
This will give the authors a chance to either release their code, provide pointers or rescind the paper.
My hope is that this incentivizes a healthier ML research culture around not publishing unreproducible work.
But this is not a requirement for machine learning conferences.
As a result, many students and researchers who read these papers struggle with reproducing their results.
But ContributionSecure14 also acknowledges that there are sometimes legitimate reasons for machine learning researchers not to release their code.
For example, some authors may train their models on internal infrastructure or use large internal datasets for pretraining.
This can be due to various reasons.
For instance, the authors might cherry-pick the best results from several experiments and present them as state-of-the-art achievements.
[The] absence of sufficiently documented methods and computer code underlying the study effectively undermines its scientific value.
Recent years have seen a growing focus on AIs reproducibility crisis.
Better reproducibility means its much easier to build on a paper.
Her efforts have resulted in an increase in code and data submission at NeurIPS.
Papers With Code currently hosts the implementation of more than 40,000 machine learning research papers.
PapersWithCode plays an important role in highlighting papers that are reproducible.
However, it does not address the problem of unreproducible papers, ContributionSecure14 said.
The hope is that Papers Without Code will help establish a culture that incentivizes reproducibility in machine learning research.
Another good resource is professor Pineaus Machine Learning Reproducibility Checklist.
ContributionSecure14 believes that machine learning researchers can play a crucial role in promoting a culture of reproducibility.
you could read the original articlehere.
