CEng 501 - Deep Learning - Fall 2023

Images generated by the “Stable Diffusion” method. From left to right: “a bunch of students learning deep neural networks” by Picasso, Van Gogh and Da Vinci.

Instructor: Emre Akbas; e-mail: emre at ceng dot metu.edu.tr; Office: B-208; Office hours: click + by apppointment
Lectures: Monday 9:40-12:30 at BMB-4
Online communication: (forum, homework submissions) https://odtuclass.metu.edu.tr/
Syllabus: pdf

Announcements

Oct 9, 2022 - Homework assignment 1 is announced. See Week 1 below.
Oct 9, 2022 - Student with these IDs can add the class during add-drops this week. Those who are already registered do not need to do anything.
Sep 28, 2023 - Due to conflict with the Opening Ceremony of the university (see “[GENEL-DUYURU: 11661] 2023 - 2024 Eğitim-Öğretim Yılı Açılış Töreni”), our first meeting will be on October 9th, instead of October 2nd. See you all on October 9th in BMB4.
Sep 24, 2023 - I am receiving too many e-mails about taking this course as a special student, as an undergrad, from other departments/universities, etc. I am not able to respond each of them. If you want to take the course, please make sure you attend the first lecture. And, please see my answers to frequently asked questions.

Late submission policy

Any work, e.g. an assignment solution, that is submitted late past its deadline will receive -10 points per day delayed. For example, if the deadline is Oct 14, 23:55 and a student submits his/her work anytime on Oct 15, that work will be evaluated over 90 instead 100.

Detailed syllabus

An important note: Lecture slides provided below are by no means a complete resource for studying. I frequently use the board in class to supplement the material in the slides.

Week 1

No class due to opening ceremony.

Week 2

Introduction and course logistics (slides 1, slides 2).
Reading assignment for next week: “Chapter 4: Numerical Computation” and “Chapter 5: Machine Learning Basics” from the book “Deep Learning.”
Assignment 1 is due Oct 18. Submit your solution over ODTUClass.

Week 3

Lecture topics: Machine learning background and basics (slides).
Reading assignment for next week: sections 8.1 and 8.3 from “Chapter 8: Optimization for training deep models” from the book “Deep Learning.”

Week 4

Lecture topics: Model selection, hyper-parameter tuning, maximum likelihood and maximum aposteriori esitmations; Biological neuron, artificial neuron, Perceptron, Multilayer perceptrons (slides).
Hands-on tutorial: developing a basic classifier using hinge loss: Colab notebook.
Reading assignment for next week: LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.

Week 5

Lecture topics: Multilayer Perceptrons, Activation Functions, Backpropagation, Stochastic Gradient Descent, Momentum (slides)
Lecture topics: convolutional neural networks (part 1) (slides)

Week 6

Lecture topics: Convolutional neural networks continued, multiclass hinge loss, derivation of cross-entropy loss, implementing backpropagation in a modular way, AlexNet, data augmentation, dropout. (slides)
Colab notebook on building, training and evaluating a basic MLP.
Colab notebook on building, training and evaluating a basic CNN.
Reading assignment for next week: He, K., Zhang, X., Ren, S. and Sun, J. Deep residual learning for image recognition. In CVPR 2016.

Week 7

Lecture topics: implementing a convolutional layer, initialization of neural networks, normalization layers, adaptive learning rate methods (slides).

Week 8

Lecture topics: some notable applications of CNNs and major CNN architectures (slides).
Colab notebook on implementing a bare-bones object detector.
Homework assignment 2 announced on ODTUClass.

Week 9

Lecture topics: Recurrent Neural Networks, Backpropagation through time, teacher forcing, the challenge of long-term dependencies, LSTM, GRU (slides).
Reading assignment: “The Unreasonable Effectiveness of Recurrent Neural Networks” by A. Karpathy.

Week 10

Lecture topics: LSTM, GRU, and some applications of RNNs (slides).
A RNN example: solving simple arithmetic operations using a sequence-to-sequence encoder-decoder model: Colab notebook (in Keras)

Week 11

Lecture topics: Attention, self-attention, Transformer, some notable applications of transformers (slides).
Reading assignment: Vaswani et al., “Attention is All you Need”, in NIPS 2017.

Week 12

Lecture topics: Unsupervised/generative modeling, energy-based models, Boltzmann machines, Restricted Boltzmann machines, deep belief networks, generative adversarial networks, autoencoders, variational autoencoders (slides).

Week 13

Lecture topics: A brief intro to deep reinforcement learning (slides).

Week 14

New year’s day (Jan 1, 2024). No class.

Frequently Asked Questions about taking the course

I am receiving too many e-mails about taking the course. Unfortunately, I cannot reply them one by one. Below are answers to common questions.

Q0: Is it going to be an online or face-to-face course?

A0: Face to face.

Q1: Can I take the course?

A1: There is a huge demand for the course from all kinds of backgrounds. Thanks. However, I have the responsibility to evaluate your learning outcomes and grade you. Therefore, I need to limit the number of seats. Based on my previous years’ experience, this limit will be around 35-40.

Since this is a graduate METU CENG course, I need to give priority to the graduate students in our department. Here is the priority order that I will use to accept students to the class. From high to low priority:

Grad students from METU CENG,
A limited number of 4th year undergraduate students from METU CENG,
Grad students from other METU departments,
Special students (see http://oidb.metu.edu.tr/ozel-ogrenci, you need to be a grad student in some other university to be eligible).

I must note that the first two categories (METU CENG students) almost fill up the whole capacity. So, unfortunately, there will not be much room for the remaining two categories.

Also, precedence will be given to students who are actively doing research in machine learning and related areas. This course is not a PyTorch or Keras tutorial, we intend to go beyond the “user” level.

You might want to check out the other two DL courses given at Multimedia Informatics and Electrical Engineering departments.

Machine learning background is required. If you have not taken a machine learning course before, please do not take this course.

Fluency in Python is required.

Q2: How can I register for the course?

A2: Come to the first lecture. In the first lecture, I will collect information from the participants and then, will decide (based on my answer A1 above) on who will be able to register. This enrollment list will be announced in a couple of hours following the first lecture. Students listed in this enrollment list will be able to add the course during the add-drop period.

Q3: I was able to take the course during the regular interactive registration. Should I worry about not being accepted?

A3: No worries. You will stay.

Q4: Can I take this course as a special student?

A4: Possible but unlikely. Please see my answer A1 above.

Q5: Even if I don’t officially register for the class, can I audit it?

A5: Yes, definitely.