CEng 783 - Deep Learning - Fall 2019

Image of the “METU Tree of Science” generated using the Deep Dream Generator.

Instructor: Asst. Prof. Dr. Emre Akbas; e-mail: emre at ceng dot metu.edu.tr; Office: B-208; Office hours by apppointment
Teaching Assistant: Nermin Samet and Merve Tapli are kindly volunteering. Thank you, Nermin and Merve!
Lectures: Tuesdays 13:40-16:30 at ~~BMB-2~~ BMB-5
Online communication: (e-mail list, forum, homework submissions) https://odtuclass.metu.edu.tr/
Syllabus: pdf

Announcements

Oct 16, 2019 - Homework assignment #2 is announced.
Oct 11, 2019 - Nermin Samet and Merve Tapli will be voluntary TAs of the course. Thank you, Nermin and Merve!
Sep 24, 2019 - Students with these IDs can add the class during the add/drop period next week.
Sep 11, 2019 - I am receiving many e-mails about taking this course as a special student, as an undergrad, from other departments/universities, etc. I am not able to respond each of them. If you want to take the course, please make sure you come to the first lecture.

Late submission policy

Any work, e.g. an assignment solution, that is submitted late past its deadline will receive -10 points per day delayed. For example, if the deadline is Oct 14, 23:55 and a student submits his/her work anytime on Oct 15, that work will be evaluated over 90 instead 100.

Detailed syllabus:

An important note: Lecture slides provided below are by no means a “complete” resource for studying. I frequently use the board in class to supplement the material in the slides.

NOTE: Below, links to slides and homework material are broken because I removed the files. Recent versions of these files can be found in the Fall 2020 version of the course.

Week 1

Misc info about the course.
Reading assignment for next week: “Chapter 5: Machine Learning Basics” from the book “Deep Learning.”
Assignment 1 is due Oct 6 via ODTUClass.

Week 2

Lecture topics: A high-level introduction to Deep Learning (slides)
Lecture topics: Machine learning background and basics (slides)
Colab notebook for the in-class hands-on demo on binary classification, gradient descent, hinge loss.
Optional reading: “Reconciling modern machine learning practice and the bias-variance trade-off”, PNAS 2019.

Week 3

Lecture topics: Biological neuron, artificial neuron, Perceptron, Multilayer Perceptrons, Artificial Neural Networks, Backpropagation, Activation Functions, Stochastic Gradient Descent, Momentum (slides)
Adding regularization to the hinge loss classifier: Colab notebook
Building, training and evaluating a basic MLP: Colab notebook
Reading assignment for next week: “Chapter 8: Optimization for trainig deep models” from the book “Deep Learning.”

Week 4

Lecture topics: Convolutional neural networks, convolution, connectivity types, pooling, AlexNet, data augmentation, dropout. (slides)
Building, training and evaluating a basic CNN: Colab notebook
Reading assignment for next week: LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.
Assignment 2 is due Oct 29 via ODTUClass. The rest of the files can be found at ODTUClass.

Week 5

Lecture topics: Convolutional neural networks continued, multiclass hinge loss, derivation of cross-entropy loss, notes on initializing neural networks, implementing backpropagation in a modular way, variants of stochastic gradient methods, adaptive learning rate methods. (slides)
Reading assignment for next week:
- He, K., Zhang, X., Ren, S. and Sun, J. Deep residual learning for image recognition. In CVPR 2016.
- Lin, T.Y., Goyal, P., Girshick, R., He, K. and Dollár, P. Focal Loss for Dense Object Detection. In ICCV 2017.

Week 6

No lecture due to national Republic Day. Lecture topics of this week will be distributed to other weeks.

Week 7

Midterm exam held in class.

Week 8

Lecture topics: some applications of ConvNets. (slides)
Building a basic object detector: Colab notebook
Reading assignment for next week: A. Karpathy’s blog post on recurrent neural networks

Week 9

Lecture topics: Recurrent Neural Networks, LSTM. (slides)
Reading assignment for next week: Vinyals, O., Toshev, A., Bengio, S. and Erhan, D. Show and Tell: A Neural Image Caption Generator. In CVPR 2015.

Week 10

Lecture topics: Some applications of RNNs (character-level language modeling, word-embeddings, image captioning, machine translation, attention mechanism, image generation, external memory models) (slides)
Building a basic sequence-to-sequence translation model (for simple arithmetic operations): Jupyter notebook

Week 11

Lecture topics: Deep generative modeling (slides)

Week 12

Project progress demos/presentations in class.

Week 13

Lecture topics: A brief intro to deep reinforcement learning (slides)
Lecture topics: Some notes on deep hierarchies in human/biological vision. (slides)
Misc topics: neural architecture search, importance of sampling during training, self-supervision, transformers.

Content will be added here as we go…