Troubling Trends in Machine Learning Scholarship - Zachary Lipton

From Youtube: The machine learning community is struggling to deal with several well-documented crises in scholarship: (i) a blurring of fact and fancy (ii) experiments divorced from falsifiability (iii) math that cannot, should not, and often isn’t meant to be followed, and (iv) exposition that sows confusion and distorts the public discourse. However, in other ways, the field is healthier than ever: (a) vibrant economy supports careers in machine learning, (b) mature tooling makes algorithms easier to run and experiments easier to reproduce, and (c) the field is far more welcoming and accessible to new talent. While, at an individual level, clear steps can be improve the quality of research and the resulting papers, what steps can be taken at a community-level is a far more challenging question. What levers can influence community practices? Who should pull them? And which interventions can curb flawed scholarship without undermining the community’s strengths? This talk will aim to present a balanced picture, both of the status quo, the ecosystem that supports it, and the difficulty of improving upon it.