The bitter lesson by Rich Sutton
Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation.
ICLR2019 Keynote by Leon Bottou
- The statistical problem is only a proxy.
- Nature does not shuffle the examples. We shouldn’t.
- Invariance across environments buys extrapolation powers.
- Invariance across environments is related to causation.
- Invariant representations enable invariance.
- Realizable problems bring different challenges.