Adversarial Examples are not Bugs, They are Features: A Discussion
On May 6th, Andrew Ilyas and colleagues published a paper outlining two sets of experiments. Firstly, they showed that models trained on adversarial examples can transfer to real data, and secondly that models trained on a dataset derived from the representations of robust neural networks seem to inherit non-trivial robustness. They proposed an intriguing interpretation for their results: adversarial examples are due to “non-robust features” which are highly predictive but imperceptible to humans.
The paper was received with intense interest and discussion on social media, mailing lists, and reading groups around the world. How should we interpret these experiments? Would they replicate? Adversarial example research is particularly vulnerable to a certain kind of non-replication among disciplines of machine learning, because it requires researchers to play both attack and defense. It’s easy for even very rigorous researchers to accidentally use a weak attack. However, as we’ll see, Ilyas et al’s results have held up to initial scrutiny. And if non-robust features exist… what are they?
To explore these questions, Distill decided to run an experimental “discussion article.” Running a discussion article is something Distill has wanted to try for several years. It was originally suggested to us by Ferenc Huszár, who writes many lovely discussions of papers on his blog.
Why not just have everyone write private blog posts like Ferenc? Distill hopes that providing a more organized forum for many people to participate can give more researchers license to invest energy in discussing other’s work and make sure there’s an opportunity for all parties to comment and respond before the final version is published. We invited a number of researchers to write comments on the paper and organized discussion and responses from the original authors.
The Machine Learning community sometimes worries that peer review isn’t thorough enough. In contrast to this, we were struck by how deeply respondents engaged. Some respondents literally invested weeks in replicating results, running new experiments, and thinking deeply about the original paper. We also saw respondents update their views on non-robust features as they ran experiments — sometimes back and forth! The original authors similarly deeply engaged in discussing their results, clarifying misunderstandings, and even running new experiments in response to comments.
We think this deep engagement and discussion is really exciting, and hope to experiment with more such discussion articles in the future.