[omscs NLP] module 5

Word Embeddings

Term freq - inverse doc freq

image-20260924230458416

inverse doc freq

image-20260924230555102

Multi-hot vector, length is vocab size

image-20260924230704942

Measure The Similarity Documents

image-20260924231500684

or using cosine for the angle between the two vectors

Query vector do matrix multiple with document matrix, get a vector with D length. this is basically a cosine similarity calculation. Argmax or top-k to return most relevant doc.

image-20260924232408558

issues, need to get vocab length vector, not efficient; can’t capture similar words

image-20260924232623171

word embeddings

Vector distance 和cosine similarity的问题是,在onehot上,比如hello 和hi,都会show distance。

image-20260924232933885

解决办法是利用linear layer 把V length vector project到d -dimensions上

image-20260924233304484

由于是onehot,一个1,其它都是0,最后投影到d-dimensions上去的value,其实是有1的那个的weights,最后那个为1的word,对应的是d个weights

image-20260924233453388

onehot会select weights的那一行。

image-20260924233915552

这和之前的RNN的那个onehot –> linear是一样的(左侧)

所以可以用右侧word –> embedding layer来代替 (右侧)

image-20260924234455055

distributional semantics

一个词可以通过从周围的词推断出来

word2vec

相同的词有相似的embedding

train word embedding using surrounding words in a sentence

image-20261004141116571

上面的模型,输入里中间的token n去掉,用surrounding token, predict token n

image-20261004141439288

Skip gram

相反的,keep 一个词,猜一猜周围的词

image-20261004141310180

Skip gram

image-20261004141601625

测量每个word embedding之间的关系

cosine

image-20261004141917604

词pair的关系的delta,也可以推断出另一个词pair

image-20261004163845436

还可以把一个文档里的每个embedding都加在一起,然后normalize

image-20261004164924236