– Week 12 – Practicum
– Attention
Thanks a lot for the video Alfredo. Very nice explanations, and e we can see that you are putting lot of efforts in pos-editing (like the add up voice @ :-)) Keep doing the great work 👏
isn't it that A would just be an identity matrix if we do argmax? And H would than be equal to X... What's the purpose of this process then?
I've had the same question the whole first 16 minutes: "what are the x?" Thank for asking, you made the whole lesson more understandable :)
– Key-value store
Question #2: So we have q = Wq * x and k = Wk * x. Why don't we just learn W = Wq * Wk^-1 and compare q = W * x to all of the x? (You mentioned that that would be the cross-attention, but then how the two are different?..)
– Transformer and PyTorch implementation
-convolutional layer at is kinda checking an attention value with a threshold? Kinda if the attention value is less than the threshold, then we set this value to 0 ?Or mb it is just a useful trick to split our huge dxt matrix onto t d-vectors (by using all the kernels equal to 1)? 🤯
when my favorite song starts playing on the radioThank you Alfredo Sir, I now know how transformers and self attention workVery grateful to you
