PyTorch笔记05----Broadcast自动扩展

Posted on Edited on In Views:

PyTorch的自动扩展

  • Expand，可以维度扩展
  • without copying data，扩展时不用拷贝数据，能节省空间

Key idea

  • Insert 1 dim ahead
  • Expand dims with size 1 to same size

Feature maps: [4, 32, 14, 14] Bias: [32] --> [32, 1, 1] --> [1, 32, 1, 1] --> [4, 32, 14, 14]

Broadcast示意图： Figure1

Why broadcasting

  1. for actual demanding
    • [class, student, scores]
    • Add bias for every students: +5 score
    • [4, 32, 8] + [5.0] (标量)
    • bias.unsqueeze(0).unsqueeze(0).expand_as(A)
  2. memory consumption 节省内存消耗

Is it broadcasting-able?

Match from Last dim 从最后一维匹配（一般最后一维是物理意义上的小维度）

  • If current dim = 1, expand to same
  • If either has no dim, insert one dim and expand to same
  • otherwise, NOT broadcasting-able

例如[8, 32, 8]（8个班，每班32个学生，每个学生修八门课即八个成绩）bias = [5]是 [1]的shape，对八门课都要加，对每个班每个学生都适用 --> [1, 1, 1] ---> [32, 32, 8]

如果仅对某一门课的成绩添加bias，那么bias = [0, 0, 5, ..., 0]是 [8]的shape，每个班每个学生都适用 --> [1, 1, 8] --> [32, 32, 8]

A: [4, 32, 8], bias: [4] NOT broadcasting-able

Situation

Situation 1

A: [4, 32, 14, 14] B: [1, 32, 1, 1] --> [4, 32, 14, 14]

Situation 2

A: [4, 32, 14, 14] B: [14, 14] --> [1, 1, 14, 14] --> [4, 32, 14, 14]

Situation 3

A: [4, 32, 14, 14] B: [2, 32, 14, 14] NOT broadcasting-able

How to understand this behavior?

When it has no dim

  • treat it as all own the same
  • [class, student, scores] + [scores]

When it has dim of size 1

  • treat it shared by all
  • [class, student, scores] + [student, 1]（a学生所有课程 + 0，b学生所有课程 + 1 ...这种情况）

match from LAST dim

+ [1, 1, 1, 1] 等价 + [1]