– Welcome to class
– Non-linear functions
@ One of the reasons why ReLU is better in deep networks than say Sigmoid is that the gradient in backward pass after each sigmoid nonlinearity gets smaller (multiplied by around 0.25), but in ReLU-like nonlinearities, the gradient does not get smaller after each layer (the gradient is one in positive part)
– Q&A
– Softargmax and softargmin
– Logsoftargmax
– Cost functions
– Architectures: multiplicative interaction
@Alfredo Canziani ~ the topic is "multiplicative modules", thank you
– Mixture of experts
at Yann says that we can do a Non-linear classification with mixture of linear classifiers which are gated, isn't it still linear classifier? Why is it non-linear, what is it that makes it non-linear classification.
– Parameter transformations
What is the meaning of the update rule when the parameter vector is the output a function [at time ] ? As the name implies, w is the output of a function, so how can you update the output?
