Deep Learning

Difference between LSTMs and GRUs

Problem

Which of the following is not true with respect to LSTMs and GRUs?

Options

Answer

The LSTM cells use tanh and sigmoid non-linearities in its update equations while the GRU cells use only tanh.

Solution

The update equation of the GRU cells also uses both tanh and sigmoid functions.

---

Discussing dropout

Problem

Which of the following are true regarding dropout?

Options

Answer

Dropout needs to be turned off during testing.

Solution

Dropout needs to be turned off during testing as we want the entire representational capacity of the model during testing. Dropout is similar to a bagging model but there are a few key differences. First in bagging the models are trained independently of each other whereas the dropout models share parameters with each other. Second, in bagging each individual model is trained to convergence while that is not the case in dropout (where each model is trained for 1 step). Further D is incorrect since dropout helps reduce overfitting.

---