CEng 783 - Deep Learning - Fall 2016

[Image generated by http://deepdreamgenerator.com/]

Instructor: Asst. Prof. Dr. Emre Akbas; e-mail: emre at ceng dot metu.edu.tr; Office: B-202; Office hours by apppointment
Teaching Assistant: Arman Afrasiyabi is kindly volunteering.
Lectures: Mondays 12:40-15:30 at BMB-4
Online communication: (e-mail list, forum, homework submissions) https://odtuclass.metu.edu.tr/
Syllabus: pdf

Announcements

02/11/16: We will have the midterm exam on Monday, Nov 21 during our regular class hours.
25/10/16: Please check ODTUClass for project proposal announcement and submission.
10/10/16: Those students who I did not reject, please send me your METU student ID numbers. I will directly add you to the course.
10/10/16: Arman Afrasiyabi, a member of the CEng Imagelab, is kindly volunteering as the teaching assistant for the class. Thank you, Arman.
04/10/16: Assignment 1 is announced. Due date is Oct 14.
03/10/16: Those who want to register to the class but could not due to the limited capacity should e-mail me about their request by Wednesday, Oct 5, 23:55. In your e-mail, please briefly describe (no more than 3-4 sentences per item below)
- your machine learning background and
- how useful taking the class is going to be for your research or work.

Late submission policy

Any work, e.g. an assignment solution, that is submitted late past its deadline will receive -10 points per day delayed. For example, if the deadline is Oct 14, 23:55 and a student submits his/her work anytime on Oct 15, that work will be evaluated over 90 instead 100.

Detailed syllabus:

An important note: Lecture slides provided below are by no means a “complete” resource for studying. I frequently use the board in class to supplement the material in the slides.

NOTE: Below, links to slides and homework material are broken because I removed the files. Recent versions of these files can be found in the Fall 2017 version of the course.

Week 1

Lecture topics: A high-level introduction to Deep Learning (slides.)
Reading assignment for next week: “Chapter 5: Machine Learning Basics” from the book “Deep Learning.”
Assignment 1 (You also need this file which I forgot to include in hw1.zip.)

Week 2

Lecture topics: Machine learning background and basics (slides.)
Files for the in-class hands-on demo on binary classification, gradient descent, hinge loss: download

Week 3

Lecture topics: Biological neuron, artificial neuron, Perceptron, Multilayer Perceptrons, Artificial Neural Networks, Backpropagation, Activation Functions, Stochastic Gradient Descent, Momentum (slides.)
Adding regularization to the hinge loss classifier, using built-in solvers: download
Resources for in-class hands-on Tensorflow demo:
- Implementing Linear SVM using Tensorflow: https://github.com/eakbas/tf-svm
- Softmax, ANN: https://github.com/eakbas/tf-svm
- Extremely good examples: https://github.com/aymericdamien/TensorFlow-Examples
Reading assignment for next week: “Chapter 8: Optimization for trainig deep models” from the book “Deep Learning.”

Week 4

Lecture topics: Convolutional neural networks, convolution, connectivity types, pooling, AlexNet (slides.)
Tensorflow CNN demo/tutorial postponed to next week.
Reading assignment for next week: LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.

Week 5

Lecture topics: Convolutional neural networks continued, multiclass hinge loss, derivation of cross-entropy loss, notes on implementing backpropagation in a modular way (slides.)
In class Tensorflow ConvNet demo by Arman Afrasiyabi (files.)
Assignment 2 is announced. It is on implementing a modular backpropagation network and setting up, training, testing of ConvNets in Tensorflow. Download it from ODTUclass.
Reading assignment for next week:
- He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385.
- Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems (pp. 91-99).

Week 6

Lecture topics: Variants of stochastic gradient descent, adaptive learning rate methods. Some applications of ConvNets. (slides)
Reading assignment for next week: A. Karpathy’s blog post on recurrent neural networks

Week 7

Lecture topics: Recurrent Neural Networks, LSTM. (slides)

Week 8

No class. Midterm exam.

Week 9

Lecture topics: Midterm exam solutions, RNN recap, LSTM recap, Applications of RNNs. (slides)
Assignment 3 is announced. Please check ODTUClass.

Week 10

Lecture topics: Generative models, Boltzmann Machines, Restricted Boltzmann Machines, Contrastive Divergence, Generative Adversarial Networks, Autoencoders, Variational Autoencoders. (slides)
Nice reading: Salakhutdinov, R. (2015). Learning deep generative models. Annual Review of Statistics and Its Application, 2, 361-385 (link).

Week 11

Project progress demos/presentations in class.

Week 12

No class.

Week 13

Introduction to reinforcement learning. Concepts, Markov Decision Process, Bellman Equation, Q-learning. Deep Q-learning. Deep policy gradients. Applications of deep reinforcement learning. (slides)

Content will be added here as we go…