🧠

We use the InfoNCE (참고1) /NT-Xent (참고2) /N-pairs (참고3) loss, which can be interpreted in the following way: 1. We treat each image in the batch as if it had its own class. 2. Then, we have two examples (a pair of augmented views) for each "class". 3. Each view's representation is compared to every possible pair's one (for both augmented versions). 4. We use the temperature-scaled cosine similarity of compared representations as logits. 5. Finally, we use categorical cross-entropy as the "classification" loss

출처

수집시간

2021/10/03 04:39

연결완료

1 more property

참고

positive sample 끼리는 가깝게, negative sample 끼리는 멀게 representation을 정의하기 위해서 보통 InfoNCE NCE(noise contrastive estimation) Loss를 사용합니다. InfoNCE Loss는 위 그림에 L_n 과 같습니다. 초록색은 positive sample과의 관계, 빨간색은 negative sample과의 관계를 의미합니다. InfoNCE Loss를 minimize하면 분자(초록색)는 minimize되고, 분모(빨간색)는 maximize되기 때문에 우리가 원하는 결과를 얻을 수 있습니다. 

Supervised NT-Xent loss (Khosla et al. 2020), let's first look at the self-supervised version of NT-Xent loss. NT-Xent is coined by Chen et al. 2020 in the SimCLR paper and is short for "normalized temperature scaled cross entropy loss‣”. It is a modification of the multi-class N-pair loss with addition of the temperature parameter (𝜏) to scale the cosine similarities. Chen et al. found that an appropriate temperature parameter can help the model learn from hard negatives. 

Multi-class N-pair loss (Sohn 2016) is a generalization of triplet loss* allowing joint comparision‣  among more than one negative samples. When applied on a pair of positive samples 𝐳_𝐢 and 𝐳_𝐣  (𝑦_𝑖=𝑦_𝑗),  where z_i z_j is the inner product, which is equivalent to cosine similarity when both vectors have unit norm. 