DL 6890 Deep Learning
Paper Presentations

---

1. Transformer Models:
   • presented by Taiman (Reformer) and XYZ (?), Apr 30, 1:00pm
   • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al., NAACL 2019.
   • ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, Lan et al., ICLR 2020.
   • Reformer: The Efficient Transformer, Kitaev et al., ICLR 2020.
     • Reformer on Google AI Blog, 2020.
2. Physics Informed Trainining of NNs:
   • presented by Yingnan Zhang, April 30, 1:25pm
   • Physics Informed Deep Learning, Raissi et al., 2019
   • Quantifying the effect of quarantine control in Covid-19 infectious spread using machine learning, Dandekar and Barbastathis, 2020
   • On the Convergence and generalization of Physics Informed Neural Networks,
3. Memory Augmented Networks:
   • presented by Krerkkiat Chusap, Apr 30, 1:50pm
   • Hybrid computing using a neural network with dynamic external memory, Graves et al., Nature 2016.
   • Neural Turing Machines, Graves et al., DeepMind 2014.
4. Deep Learning for Music:
   • presented by Ben Carman, Apr 30, 2:15pm
   • Music Transformer, Huang et al., ICLR 2019
     • Magenta blog on Music Transformer, 2019.
     • Play with Music Transformer,
   • Encoding Musical Style with Transformer Autoencoders, Choi et al., ICLR 2020.
5. Normalization in NNs:
   • presented by Edward Matovu, April 30, 2:40pm
   • Self-Normalizing Neural Networks, Klambauer et al., NIPS 2017.
   • Layer Normalization, Ba et al., NIPS 2016.
   • Weight normalization: A simple reparameterization to accelerate training of deep neural networks, Salimans et al., NIPS 2016.
   • Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Ioffe and Szegey, ICML 2015.
6. Adversarial Examples:
   • presented by Siqin Liu, Apr 30, 3:05pm.
   • Intriguing properties of neural networks, Szegedy et al., ICLR 2014.
   • Explaining and Harnessing Adversarial Examples, Goodfellow et al., ICLR 2015.
   • DeepFool: a simple and accurate method to fool deep neural networks, Moosavi et al., CVPR 2016
   • Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples, Athalye et al., ICML 2018
   • Adversarial Examples Are a Natural Consequence of Test Error in Noise, Ford et al., ICML 2019.
7. Adaptive Gradient Methods:
   • presented by XYZ, Apr 28.
   • Adaptive Gradient Methods with Dynamic Bound of Learning Rate, Luo et al., ICLR 2019
8. Memory Efficient Backpropagation:
   • presented by XYZ, Apr 28.
   • The Reversible Residual Network: Backpropagation Without Storing Activations, Gomez et al., NIPS 2017.
9. Image Segmentation and Generation: Upsampling through Transposed Convolution and Max Unpooling:
   • presented by XYZ, April 28.
   • Fully Convolutional Networks for Semantic Segmentation, Long et al., CVPR 2015.
   • A Guide to Transposed Convolutions, Dumoulin and Visin, CoRR 2018.
10. Bayesian Neural Networks:
    • presented by XYZ, April 28.
    • Weight Uncertainty in Neural Networks, Blundell et al., ICML 2015.
    • Bayesian Methods for Machine Learning, Coursera.
    • Bayesian Methods for Neural Networks, slides from course by Aaron Courville.
    • Chapter 10 in Neural Networks for Pattern Recognition, book by Christopher Bishop, 1995.
11. Learning Curves of NN models:
    • presented by XYZ, April 28.
    • Scaling Laws for Neural Language Models, Kaplan et al., OpenAI 2020.
    • A Constructive Prediction of the Generalization Error Across Scales, Rosenfeld et al., ICLR 2020.
12. Deep Networks and Generalization:
    • presented by XYZ, Apr 28.
    • On Generalization and Regularization in Deep Learning, Lemberger, CoRR 2017.
    • Understanding deep learning requires rethinking generalization, Zhang et al., ICLR 2017.
    • Musings on Deep Learning: Properties of SGD, Zhang et al., CBMM Memo, MIT 2017.
    • Opening the black box of Deep Neural Networks via Information, Schwartz-Ziv and Tishby, ICRI-CI 2017.
    • Scalable Mutual Information Estimation using Dependence Graphs, Noshad et al., ICASSP 2019.
13. Differentiable DSP:
    • presented by XYZ, April 28
    • DDSP: Differentiable Digital Signal Processing, Engel et al., ICLR 2020
    • Magent Blog entry on DDSP, 2020.
14. Evolutionary search of ML algorithms:
    • presented by XYZ, April 28
    • AutoML-Zero: Evolving Machine Learning Algorithms From Scratch, Real et al., Google 2020.
    • Evolving Normalization-Activation Layers, Liu et al., Google 2020.