Having heard Geoffrey Hinton’s somewhat dismissive account of the contribution by physicists to machine learning in his online MOOC, it was interesting to listen to one of those physicists, Naftali Tishby, here at PI:
The surprising success of learning with deep neural networks poses two fundamental challenges: understanding why these networks work so well and what this success tells us about the nature of intelligence and our biological brain. Our recent Information Theory of Deep Learning shows that large deep networks achieve the optimal tradeoff between training size and accuracy, and that this optimality is achieved through the noise in the learning process.
In this talk, I will focus on the statistical physics aspects of our theory and the interaction between the stochastic dynamics of the training algorithm (Stochastic Gradient Descent) and the phase structure of the Information Bottleneck problem. Specifically, I will describe the connections between the phase transition and the final location and representation of the hidden layers, and the role of these phase transitions in determining the weights of the network.
Based partly on joint works with Ravid Shwartz-Ziv, Noga Zaslavsky, and Shlomi Agmon.
(See also Steve Hsu’s discussion of a similar talk Tishby gave in Berlin, plus other notes on history.)
I was familiar with the general concept of over-fitting, but I hadn’t realized you could talk about it quantitatively by looking at the mutual information between the output of a network and all the information in the training data that isn’t the target label.… [continue reading]
A lot of people sound worried that new and improving techniques for creating very convincing videos of anyone saying and doing anything will lead to widespread misinformation and even a break down of trust in society.
I’m not very worried. Two hundred years ago, essentially all communication, other than in-person conversation, was done through written word, which is easy to fake and impersonate. In particular, legal contracts were (and are) typeset, and so are trivially fakeable. But although there were (and are) cases of fraud and deception through foraged documents, society has straightforward mechanisms for correctly attributing such communication to individuals. Note, for instance, that multi-billion-dollar contracts between companies are written in text, and we have never felt it necessary to record difficult-to-fake videos of the CEOs reciting them.
The 20th century was roughly a technological Goldilocks period where technology existed to capture images and video but not to fake them. Images, of course, have been fakeable at modest cost for many years. Even in 1992, Michael Crichton’s Rising Sun used high-tech fraudulent security footage as a realistic plot point in then-current day. Although we may see some transition costs as folks are tricked into believing fraudulent videos because the ease of faking them has not yet entered the conventional wisdom, eventually people will learn that video can’t be trusted much more than the written word.Which is to say, most of the time you can trust both text and video because most people aren’t trying to defraud you, but extra confirmatory steps are taken for important cases.a This will not be catastrophic because our trust networks are not critically dependent on faithful videos and images.… [continue reading]
We recently hosted a conference at Perimeter Institute on “Open Science”. Video from all the talks is available here. I spokeIt might be more accurate to say that I occasionally mumbled something intelligible in between long stretches of the words “um” and “ah”. Luckily, you can watch the video at high speed by using using a browser plugin like Video Speed Controller for Chrome. Unfortunately, I don’t know a simple way to embed playback speed controls directly into the HTML rather than forcing you to install a plugin or download the video and watch it with a player featuring such controls.a on the importance of “knowledge ratchets”, i.e., pedagogical documents (textbooks, monographs, and review papers) that allow for continuous improvement by anyone. After starting off with my new favorite example of how basic physics textbooks, and physicists, are egregiously uninformed about central elementary things, I ranted about how important it is to allow for people who are not the original author to contribute easily to the documents composing our educational pipeline (broadly construed to include the training researchers on recent developments).
(I forgot to put on the microphone for the first minute and a half; the sound quality improves after that.)
Luckily, when I wanted to illustrate the idea of in-PDF commenting on articles that generated feedback for the authors, I didn’t have to just use mock-ups. Luis Batalha from Fermat’s Library took the mic for the second half of the talk to show off their Chrome plugin “Librarian” and talk about their strategy for gaining users.… [continue reading]
Extrapolating my current trajectory, I will combine more and more links posts into larger and larger multi-month collections until eventually I release one giant list for all time and shutdown the blog.Just kidding. I will get back to actual, non-link blogging before too long…a
… [continue reading]
I was privileged to receive a reply from Robin Hanson on my critique of his largely excellent book The Elephant in the Brain with co-author Kevin Simler. I think in several cases he rebutted something other than what I argued, but I encourage you to read it and judge for yourself.
Given the high-profile book reviews that are probably forthcoming from places like the Wall Street Journal, I thank Robin for taking the time to engage with the little guys!
I’ll follow Robin’s lead and switch to first names.
Some say we should have been more academic and detailed, while other say we should have been more accessible and less detailed….Count Jess as someone who wanted a longer book.
It’s true that I’d have preferred a longer book with more details, but I think I gestured at ways Kevin and Robin could hold length constant while increasing convincingness. And there are ways of keeping the book accessible while augmenting the rigor (e.g., endnotes), although of course they are more work.
Yes for each motive one can distinguish both a degree of consciousness and also a degree of current vs past adaptation. But these topics were not essential for our main thesis, making credible claims on them takes a lot more evidence and argument, and we already had trouble with trying to cover too much material for one book.
I was mostly happy with how the authors handled the degree of consciousness. However, I think the current- vs past-adaptation distinction is very important for designing institutions, which Kevin and Robin correctly list as one of the main applications of the book’s material. For instance, should the arXiv host comments on papers, and how should they be implemented to avoid pissing contests?… [continue reading]
Drawing on a large academic literature in topics like sociology, behavioral economics, anthropology, and psychology, and especially the (generalized) theory of signalling, Robin Hanson has assembled a large toolbox for systemically understanding hypocrisy, i.e., the ways in which people’s actions systematically and selfishly deviate from their verbalized explanations. Although he would be the first to admit that many of these ideas have been discovered and rediscovered repeatedly over centuries (or millennia) with varying degrees of clarity, and although there is much I am not convinced by, I find the general framework deeply insightful, and his presentation to be more clear, analytical, and descriptive (rather than disruptively normative) than other accounts. Most of this I have gathered from his renowned blogging at Overcoming Bias, but I have always wished for a more concise (and high status!) form factor that I could point others to. At long last, Hanson and his co-author Kevin Simler have written a nice book that largely satisfies me: The Elephant in the Brain (Amazon). I highly recommend it.
The reason I title these sorts of blog posts “Comments on…” is so I can present some disorganized responses to a work without feeling like I need to build a coherent thesis or pass overall judgment. I do not summarize the book below, so this post will mostly be useful for people who have read it. (You might think of this as a one-sided book club discussion rather than a book review.) But since I will naturally tend to focus on the places where I disagree with the authors, let me emphasize: the basic ideas of this book strike me as profound and probably mostly true.… [continue reading]