Ard Louis

Abstract:

© 7th International Conference on Learning Representations, ICLR 2019. All Rights Reserved. Deep neural networks (DNNs) generalize remarkably well without explicit regularization even in the strongly over-parametrized regime where classical learning theory would instead predict that they would severely overfit. While many proposals for some kind of implicit regularization have been made to rationalise this success, there is no consensus for the fundamental reason why DNNs do not strongly overfit. In this paper, we provide a new explanation. By applying a very general probability-complexity bound recently derived from algorithmic information theory (AIT), we argue that the parameter-function map of many DNNs should be exponentially biased towards simple functions. We then provide clear evidence for this strong bias in a model DNN for Boolean functions, as well as in much larger fully conected and convolutional networks trained on CIFAR10 and MNIST. As the target functions in many real problems are expected to be highly structured, this intrinsic simplicity bias helps explain why deep networks generalize well on real world problems. This picture also facilitates a novel PAC-Bayes approach where the prior is taken over the DNN input-output function space, rather than the more conventional prior over parameter space. If we assume that the training algorithm samples parameters close to uniformly within the zero-error region then the PAC-Bayes theorem can be used to guarantee good expected generalization for target functions producing high-likelihood training sets. By exploiting recently discovered connections between DNNs and Gaussian processes to estimate the marginal likelihood, we produce relatively tight generalization PAC-Bayes error bounds which correlate well with the true error on realistic datasets such as MNIST and CIFAR10and for architectures including convolutional and fully connected networks.

Abstract:

From the birth of galaxies to the self-organizing dynamics of our planet to the ongoing expansion of the universe, the more we discover about the evolution of the cosmos, the more acutely we realize the enormity of what remains to be known. Just this year astrophysicists at the University of Nottingham confirmed that there are at least two trillion galaxies in the cosmos, 10 times more than had been previously thought. What guidance or wisdom can the study of cosmology and astrophysics offer us in our search for meaning and purpose? In conversation with Steve Paulson, executive producer and host of To the Best of Our Knowledge, theoretical physicists Paul Davies and Ard Louis, and astrophysicist Lucianne Walkowicz share their perceptions based on years of gazing upward and beyond our own intimate planet.

Abstract:

The mechanical properties of DNA nanostructures are of widespread interest as applications that exploit their stability under constant or intermittent external forces become increasingly common. We explore the force response of DNA origami in comprehensive detail by combining AFM single molecule force spectroscopy experiments with simulations using oxDNA, a coarse-grained model of DNA at the nucleotide level, to study the unravelling of an iconic origami system: the Rothemund tile. We contrast the force-induced melting of the tile with simulations of an origami 10-helix bundle. Finally, we simulate a recently proposed origami biosensor, whose function takes advantage of origami behavior under tension. We observe characteristic stick–slip unfolding dynamics in our force–extension curves for both the Rothemund tile and the helix bundle and reasonable agreement with experimentally observed rupture forces for these systems. Our results highlight the effect of design on force response: we observe regular, modular unfolding for the Rothemund tile that contrasts with strain-softening of the 10-helix bundle which leads to catastropic failure under monotonically increasing force. Further, unravelling occurs straightforwardly from the scaffold ends inward for the Rothemund tile, while the helix bundle unfolds more nonlinearly. The detailed visualization of the yielding events provided by simulation allows preferred pathways through the complex unfolding free-energy landscape to be mapped, as a key factor in determining relative barrier heights is the extensional release per base pair broken. We shed light on two important questions: how stable DNA nanostructures are under external forces and what design principles can be applied to enhance stability.