관심있는것들

we can apply filter and let it represent what we want , first filter can be used to filter "polite thing" and so on. we have 2 channels for each kernel size (2,3,4) -> output size is (4,5,6) -> 1 max pooling from each channel and concatenate -> concatenate whole and put it into softmax to tell positve or negative. Regularization Dropout = Create masking vector r of Bernoulli random variable with..

SQUAD evaluation F1 meause is primary , exact match: 1/0 accuracy on whether you match one of the 3 answers F1 = 2*precision*recall / (precision + recall) precision = tp/(tp+fp) , recall = tp/(tp+fn) tp = true positive == true (the model prediction is actually true) , postive ( model said it is true,positive) number of tokens that are shared between the correct answer and the prediction fp = fal..

Statistical Machine Translation (SMT) we are dividing a single conditional probability distribution to Translation Model and Language Model so it can learn more efficiently. searching for all possible y and calculating probability take too long. so we use heuristic search algorithm to search best translation. = decoding Neural Machine Translation (NMT) unlike SMT there are no dividing to smaller..

vanishing gradient happens if intermediate gradients is small? then whole gradient gets smaller if vanishing gradient happen we cannot tell whether there is no dependency between in the data there is dependency between in the data but since our prarmeters are wrong we couldn't capture it. RNN- LM are better at learning from sequential recency than syntatic recency due to vanishing gradient. ther..

Language Modeling is the task of predicting what word comes next N - gram Language Models looking 4 words ahead and predict probability. but this model will predict zero probability, if our training corpus didn't' have particular sentence that we are looking for. => use smoothing technique which makes zero probability to have some probability. same for denominator, if model never seen "students ..

depdency structure shows which words depend on which other words. what word modify other words. up until now we looked at the meaning of words like word vector. we need to understand sentence structure in order to convey complex meaning. but dependency structure is not absolute it can change due to Prepositional phrase attachment ambiguity. Bills on ports and immigration were submitted by Senato..

instead of doing each word error , we calculate windows error and later on we split into (in this case 5) 5 and update our word vector. this things might happen when we fine tune our pretrained word vectors. so what shoudl we do? we only "fine tune" my own word vector when we have only large training dataset. most of the time just using pretrained word vectors are best. our goal is to calculate ..

W are weights (parameter that needs to be adjust) for full dataset we assume that p is 1. controlling theta is same as moving line from right picture we want to minmize loss. Softmax alone are not very powerful since it is linear we need some nonlinearity. we learn word representation and parameter W at the same time. sames as binary logistic regression unit == one neuron we don't want to decide..

most of the word vectors are represented as row Gradient Descent computing naviely takes too much time. Stochastic gradient descent(SGD) = randomly choose small sample(or batch) for each step and do same regular gradient descent, effectively computing faster. Skip - grams = you have 1 center word and predict all the 'outside' words in the context Continous Bag of Words = predict center words fro..

One hot vector representation , each column represent single word = impossible to represent all words there is no similarity for one hot vectors. (orthogonal) Distributional semantics = A word's meaning is given by the words that frequently appear close-by Word2Vec given some window size , iterate through sentence and update the word vectors given center word algorithm tries to predict word that..