Lec1) introduction and Word vectors

One hot vector representation , each column represent single word = impossible to represent all words

there is no similarity for one hot vectors. (orthogonal)

 

Distributional semantics = A word's meaning is given by the words that frequently appear close-by

Word2Vec

given some window size , iterate through sentence and update the word vectors

given center word algorithm tries to predict word that is close to center word which is in window size.

first product meaning will go through all the texts and second means will iterate through given window size.

only parameter here which is theta , are word vectors.

In objective function T is for scaling factor. 

Log is known to be good empirically

we use 2 vectors for convenience. u for context word. v for center word.

upper part is calculating inner product with outside word vector and center word vector 

lower part is for normalizing

 

softmax

exp makes large x bigger and there will be no zero.  

every word has 2 vector(u,v) that is why dimension is 2dV and this theta is only parameter we have.

after all the Derivative we have our solution meaning , in order to minmize our loss we need to go this way which is minmizing given acutal outside word vector minus expected outside word vector.