Neural networks underlie state-of-the-art algorithms in a large and growing range of applied domains. That they work so well challenges long-held principles in statistics and optimization that have often guided applied work. This talk identifies and discusses several such points of tension between theory and practice and provide some heuristic explanations for how to think about them.