COMP9444 Neural Networks and Deep Learning
Term 3, 2020

Exercises 4: PyTorch

1. Adjusting the Learning Rate and Momentum

   The program slope.py solves the simplest possible machine learning task:

   solve   F(x) = A*x   such that   F(1)=1

   1. Run the program by typing

      python3 slope.py --lr 0.1
      

      Try running the code using each of the following values for learning rate:

      0.01, 0.1, 0.5, 1.0, 1.5, 1.9, 2.0, 2.1
      

      Describe what happens in each case, in terms of the success and speed of the algorithm.

   2. Now add momentum, by typing:

      python3 slope.py --mom 0.1
      

      Try running the code with learning rate kept at the default value of 1.9 but with momentum equal to 0.1, 0.2, etc. up to 0.9. For which value of momentum is the task solved in the fewest epochs? What happens when the momentum is 1.0? What happens when it is 1.1?

2. Learning the XOR Task

   The program xor.py trains a 2-layer neural network on the XOR task.

   1. Run the code ten times by typing

      python3 xor.py
      

      For how many runs does it reach the global minimum? For how many runs does it reach a local minimum?

   2. Keeping the learning rate fixed at 0.1, can you find values of momentum (--mom) and initial weight size (--init) for which the code converges relatively quickly to the global minimum on virtually every run?