week3_Module.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. 模型\n",
    "\n",
    "```\n",
    "模型 (nn.Module)\n",
    "|--模型创建\n",
    "|   |--构建网络层\n",
    "|   |   |--卷积层、池化层、激活函数层等\n",
    "|   |--拼接网络层\n",
    "|   |   |--LeNet、AlexNet、ResNet等\n",
    "|--模型初始化\n",
    "|   |--Xavier、Kaiming、均匀分布、正态分布等\n",
    "```\n",
    "\n",
    "## 1.1 模型创建 (以 `LeNet` 为例)\n",
    "\n",
    "### 1). 构建子模块 `__init__()`\n",
    "```python\n",
    "def __init__(self, classes):\n",
    "    super(LeNet, self).__init__()\n",
    "    self.conv1 = nn.Conv2d(3, 6, 5)\n",
    "    self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "    self.fc1 = nn.Linear(16*5*5, 120)\n",
    "    self.fc2 = nn.Linear(120, 84)\n",
    "    self.fc3 = nn.Linear(84, classes)\n",
    "```\n",
    "\n",
    "### 2). 拼接子模块 `forward()`\n",
    "```python\n",
    "def forward(self, x):\n",
    "    out = F.relu(self.conv1(x))\n",
    "    out = F.max_pool2d(out, 2)\n",
    "\n",
    "    out = F.relu(self.conv2(x))\n",
    "    out = F.max_pool2d(out, 2)\n",
    "\n",
    "    out = out.view(out.size(0), -1)\n",
    "    \n",
    "    out = F.relu(self.fc1(out))\n",
    "    out = F.relu(self.fc2(out))\n",
    "    out = self.fc3(out)\n",
    "    \n",
    "    return out\n",
    "```\n",
    "\n",
    "### 综上，构建 `LeNet` 的整个过程：\n",
    "```python\n",
    "class LeNet(nn.Module):\n",
    "    def __init__(self, classes):\n",
    "        super(LeNet, self).__init__() ## super 实现父类函数的调用，这里指的是 LeNet 调用父类 nn.Module 的初始化函数。\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.fc1 = nn.Linear(16*5*5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, classes)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        out = F.relu(self.conv1(x))\n",
    "        out = F.max_pool2d(out, 2)\n",
    "\n",
    "        out = F.relu(self.conv2(x))\n",
    "        out = F.max_pool2d(out, 2)\n",
    "\n",
    "        out = out.view(out.size(0), -1)\n",
    "        \n",
    "        out = F.relu(self.fc1(out))\n",
    "        out = F.relu(self.fc2(out))\n",
    "        out = self.fc3(out)\n",
    "        \n",
    "        return out\n",
    "```\n",
    "\n",
    "## 1.2 模型初始化\n",
    "\n",
    "# 2. `nn.Module` 的详解\n",
    "\n",
    "## 2.1 `torch.nn`\n",
    "\n",
    "### 1). `nn.Module`\n",
    "> 所有网络层的基类，管理网络属性。\n",
    ">\n",
    "> 关于 `nn.Module` 的总结：\n",
    "> &emsp;&emsp; 1. 一个 `module` 可以包含多个子module；\n",
    ">\n",
    "> &emsp;&emsp; 2. 一个 `module` 相当于一个运算，必须实现 `forward()` 函数；\n",
    ">\n",
    "> &emsp;&emsp; 3. 每个 `module` 都有8个有序字典管理它的属性，重点关注 `modules` 和 `parameters`，管理模型和可学习参数；\n",
    "\n",
    "**重点关注：**\n",
    "\n",
    "**`parameters`:** `self._parameters = OrderDict()`，存储管理 `nn.Parameters` 类；\n",
    "\n",
    "**`modules`：**`self._modules = OrderDict()`，存储管理 `nn.Modules` 类；\n",
    "\n",
    "**`buffers`:** `self._buffers = OrderDict()`，存储管理缓冲属性，比如 BN层 中的均值、方差等；\n",
    "\n",
    "\n",
    "### 2). `nn.Parameter`\n",
    "> 张量(Tensor)子类，表示可学习参数，如 weight, bias。\n",
    "\n",
    "### 3). `nn.functional`\n",
    "函数具体实现，比如卷积、池化、激活函数等。\n",
    "\n",
    "### 4). `nn.init`\n",
    "参数初始化方法。\n",
    "\n",
    "# 3. 模型容器 (Containers) 以及 `AlexNet` 的构建\n",
    "```\n",
    "模型容器 (Containers)\n",
    "|--nn.Sequential\n",
    "|   |-- 按顺序 包装多个网络层。\n",
    "|   |-- 顺序性，各网络层之间严格按照顺序执行，常用语 block 构建。(整体上感觉 nn.Sequential 这个模型容器使用更方便)\n",
    "|--nn.MuduleList\n",
    "|   |-- 像 Python 的 list 一样包装多个网络层。\n",
    "|   |-- 迭代性，常用于大量重复网络构建。\n",
    "|--nn.ModuleDict\n",
    "|   |-- 像 Python 的 dict 一样包装多个网络层。\n",
    "|   |-- 索引性，常用语可选择的网络层。\n",
    "```\n",
    "\n",
    "## 3.1 Containers:\n",
    "\n",
    "### 1). `nn.Sequential`\n",
    "\n",
    "> 是 `nn.Module` 的容器，用于按顺序包装一组网络层。\n",
    ">\n",
    "> `nn.Sequential` 的特性：\n",
    "> \n",
    "> &emsp;&emsp; 1. 顺序性：各网络层之间按照严格的顺序进行构建。\n",
    ">\n",
    "> &emsp;&emsp; 2. 自带 `forward()`：在自带的`forward()`中，通过for循环一次执行前向传播。\n",
    "\n",
    "\n",
    "**示例：使用 `nn.Sequential` 构建 `LeNet`**\n",
    "\n",
    "```python\n",
    "## 方法1（网络层的名称使用默认的序号）;\n",
    "class LeNetSequential(nn.Module):\n",
    "    def __init__(self, classes):\n",
    "        super(LeNetSequential, self).__init__()\n",
    "        \n",
    "        self.features = nn.Sequential(\n",
    "            nn.Conv2d(3, 6, 5),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(kernel_size=2, stride=2),\n",
    "\n",
    "            nn.Conv2d(6, 16, 5),\n",
    "            nn.ReLU()\n",
    "            nn.MaxPool2d(kernel_size=2, stride=2),\n",
    "        )\n",
    "\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Linear(16*5*5, 120),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(120, 84),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(84, classes),\n",
    "        )\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = self.features(x)\n",
    "        x = x.view(x.size[0], -1)\n",
    "        x = self.classifier(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## 方法2（网络层的名称进行手动指定）\n",
    "class LeNetSequentialOrderDict(nn.Module):\n",
    "    def __init__(self, classes):\n",
    "        super(LeNetSequentialOrderDict, self).__init__()\n",
    "        \n",
    "        self.features = nn.Sequential(OrderDict({\n",
    "            \"conv1\": nn.Conv2d(3, 6, 5),\n",
    "            \"relu1\": nn.ReLU(inplace=True),\n",
    "            \"pool1\": nn.MaxPool2d(kernel_size=2, stride=2),\n",
    "\n",
    "            \"conv2\": nn.Conv2d(6, 16, 5),\n",
    "            \"relu2\": nn.ReLU(inplace=True),\n",
    "            \"pool2\": nn.MaxPool2d(kernel_size=2, stride=2),\n",
    "        }))\n",
    "\n",
    "        self.classifier = nn.Sequential(OrderDict({\n",
    "            \"fc1\": nn.Linear(16*5*5, 120),\n",
    "            \"relu3\": nn.ReLU(inplace=True),\n",
    "            \"fc2\": nn.Linear(120, 84),\n",
    "            \"relu4\": nn.ReLU(inplace=True),\n",
    "            \"fc3\": nn.Linear(84, classes),\n",
    "        }))\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = self.features(x)\n",
    "        x = x.view(x.size[0], -1)\n",
    "        x = self.classifier(x)\n",
    "        return x\n",
    "```\n",
    "\n",
    "\n",
    "### 2). `nn.ModuleList`\n",
    "> 也是 `nn.Module` 的容器，用于包装一组网络层，并以迭代的方式调用网络层。\n",
    "> \n",
    "> 主要方法：\n",
    "> \n",
    "> `append()`：在 ModuleList 后面添加网络层；\n",
    ">\n",
    "> `extend()`: 拼接两个 ModuleList；\n",
    ">\n",
    "> `insert()`: 指定在 ModuleList 中特定位置插入网络层；\n",
    "\n",
    "\n",
    "**示例：用 `nn.ModuleList` 实现20个全连接层（每层有10个神经元）构成的网络**\n",
    "\n",
    "```python\n",
    "class FCModuleList(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(FCModuleList, self).__init__()\n",
    "        \n",
    "        self.linears = nn.ModuleList([nn.Linear(10,10) for i in range(20)]) ## 用列表生成式进行构造\n",
    "\n",
    "    def forward(self, x):\n",
    "        for i, linear in enumerate(self.linears):\n",
    "            x = linear(x)\n",
    "        return x\n",
    "\n",
    "```\n",
    "\n",
    "### 3. `nn.ModuleDict`\n",
    "> 也是 `nn.Module` 的容器，用于包装一组网络层，以索引的方式调用网络层。\n",
    ">\n",
    "> 主要方法：\n",
    ">\n",
    "> `clear()`: 清空 ModuleDict\n",
    ">\n",
    "> `items()`: 返回可迭代的 key-value\n",
    ">\n",
    "> `keys()`: 返回字典的 key\n",
    ">\n",
    "> `values()`: 返回字典的 value\n",
    ">\n",
    "> `pop()`: 返回一对 key-value，并从字典中删除\n",
    "\n",
    "\n",
    "**示例: Conv + Relu 的组合**\n",
    "```python\n",
    "class ModuleDict(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(ModuleDict, self).__init__()\n",
    "        \n",
    "        self.choices = nn.ModuleDict({\n",
    "            \"conv\": nn.Conv2d(10, 10, 3),\n",
    "            \"pool\": nn.MaxPool2d(3)\n",
    "        })\n",
    "\n",
    "        self.activations = nn.ModuleDict({\n",
    "            \"relu\": nn.ReLU(),\n",
    "            \"prelu\": nn.PReLU()\n",
    "        })\n",
    "\n",
    "    def forward(self, choice, act):\n",
    "        x = self.choices[choice](x)\n",
    "        x = self.activations[act](x)\n",
    "        return x\n",
    "```\n",
    "\n",
    "## 3.2 `AlexNet`的构建\n",
    "> `AlexNet`的特点：\n",
    ">\n",
    "> 1. 采用 ReLU 代替饱和激活函数(比如 Sigmoid)，减轻梯度消失；\n",
    ">\n",
    "> 2. 采用 LRN (Local Response Normalization)，对数据归一化，减轻梯度消失；\n",
    ">\n",
    "> 3. 采用 Dropout 提高全连接层的鲁棒性，增强网络的泛化能力；\n",
    ">\n",
    "> 4. 采用 Data Augmentation；\n",
    "\n",
    "**`AlexNet`的具体实现**\n",
    " \n",
    "```python\n",
    "## 来自于 Pytorch 官方提供的 AlexNet 实现 (torchvision.models.AlexNet())\n",
    "class AlexNet(nn.Module):\n",
    "    def __init__(self, num_classes: int = 1000, dropout: float = 0.5) -> None:\n",
    "        super().__init__()\n",
    "        _log_api_usage_once(self)\n",
    "        self.features = nn.Sequential(\n",
    "            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "            nn.Conv2d(64, 192, kernel_size=5, padding=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "            nn.Conv2d(192, 384, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Conv2d(384, 256, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Conv2d(256, 256, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "        )\n",
    "        self.avgpool = nn.AdaptiveAvgPool2d((6, 6))\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Dropout(p=dropout),\n",
    "            nn.Linear(256 * 6 * 6, 4096),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Dropout(p=dropout),\n",
    "            nn.Linear(4096, 4096),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Linear(4096, num_classes),\n",
    "        )\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = self.features(x)\n",
    "        x = self.avgpool(x)\n",
    "        x = torch.flatten(x, 1)\n",
    "        x = self.classifier(x)\n",
    "        return x\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 神经网络中常用的层\n",
    "\n",
    "## 1. 卷积层\n",
    "\n",
    "> 卷积运算：卷积核在图像上滑动，相应位置上进行乘加（卷积核与图像对应位置进行相乘，然后在把结果相加）；\n",
    "> \n",
    "> 卷积核：也成滤波器、过滤器。\n",
    ">\n",
    "> 卷积维度：一般情况下，卷积核在几个维度上滑动，就是几维卷积。\n",
    "\n",
    "### `nn.Conv2d`\n",
    "功能：对多个二维信号进行二维卷积。\n",
    "\n",
    "```python\n",
    "nn.Conv2d(\n",
    "    in_channels,  ## 输入通道数\n",
    "    out_channels, ## 输出通道数，等价于卷积核个数\n",
    "    kernel_size,  ## 卷积核尺寸\n",
    "    stride=1,     ## 步长 (1 : [1,2,3] -> [2,3,4]; 2: [1,2,3] -> [3,4,5])\n",
    "    padding=0,    ## 填充个数，保持输入输出的图片的尺寸不变\n",
    "    dilation=1,   ## 空洞卷积大小 (\"带孔\"的卷积核，常用于图像分割任务，提升感受野)\n",
    "    groups=1,     ## 设置分组卷积的组数 (常用于模型的轻量化，比如 AlexNet的原始实现)\n",
    "    bias=True,    ## 偏置\n",
    "    padding_mode=\"zeros\"\n",
    ")\n",
    "```\n",
    "图像经过卷积之后的尺寸是如何计算的：\n",
    "\n",
    "（简化版）`Out_size = (In_size - Kernel_size) / stride + 1`\n",
    "（完整版）`H_out = (H_in + 2 x padding - dilation x (kernel_size-1) -1)/stride + 1`\n",
    "\n",
    "### 转置卷积\n",
    "也称为反卷积 (Deconvolution) 和部分跨越卷积 (Fractionally-strided Convolution)，用于对图像进行上采样（图像分割中常用）。\n",
    "\n",
    "简单来说，转置卷积就是将小尺寸图像经过卷积之后得到一个更大尺寸的图像。\n",
    "\n",
    "```python\n",
    "nn.ConvTranspose2d(\n",
    "    in_channels,    ## 输入通道数\n",
    "    out_channels,   ## 输出通道数\n",
    "    kernel_size,    ## 卷积核尺寸\n",
    "    stride=1,       ## 步长\n",
    "    padding=0,      ## 填充个数\n",
    "    output_padding=0,\n",
    "    groups=1,       ## 分组卷积设置\n",
    "    bias=True,      ## 偏置\n",
    "    dilation=1,     ## 空洞卷积大小\n",
    "    padding_mode=\"zeros\"\n",
    ")\n",
    "```\n",
    "\n",
    "## 池化层 `Pooling Layer`\n",
    "作用：对信号进行 \"收集\" 并 \"总结\"。（剔除冗余信息，减少后续计算量；）（大尺寸 --> 小尺寸）\n",
    "\n",
    "\"收集\"：多变少；\n",
    "\n",
    "\"总结\"：最大值/平均值；\n",
    "\n",
    "\n",
    "\n",
    "### `nn.MaxPool2d`\n",
    "对二维信号进行最大池化；\n",
    "\n",
    "```python\n",
    "## MaxPool2d\n",
    "nn.MaxPool2d(\n",
    "    kernel_size, ## 池化核的尺寸\n",
    "    stride=None, ## 步长\n",
    "    padding=0,   ## 填充个数\n",
    "    dilation=1,  ## 池化核之间间隔的大小\n",
    "    return_indices=False, ## 记录池化像素索引（这个在反池化过程中会用到）\n",
    "    ceil_mode=False  ## 尺寸向下取整\n",
    ")\n",
    "```\n",
    "\n",
    "### `nn.AvgPool2d`\n",
    "对二维信号进行平均值池化；\n",
    "\n",
    "\n",
    "```python\n",
    "## AvgPool2d\n",
    "nn.AvgPool2d(\n",
    "    kernel_size,\n",
    "    stride=None,\n",
    "    padding=0,\n",
    "    ceil_mode=False,\n",
    "    count_include_pad=True, ## 填充值用于计算\n",
    "    divisor_override=None  ## 除法因子\n",
    ")\n",
    "```\n",
    "\n",
    "## 反池化\n",
    "对二维像素进行最大值池化上采样（小尺寸 --> 大尺寸）\n",
    "\n",
    "### `nn.MaxUnpool2d`\n",
    "对二维信号进行最大池化上采样。\n",
    "\n",
    "```python\n",
    "## 最大反池化\n",
    "nn.MaxUnpool2d(\n",
    "    kernel_size,\n",
    "    stride=None,\n",
    "    padding=0\n",
    ")\n",
    "\n",
    "## 在前向传播时，需要传入索引indices\n",
    "forward(self, input, indices, output_size=None)\n",
    "```\n",
    "\n",
    "## 线性层 (全连接层)\n",
    "每个神经元与上一层所有神经元相连实现对前一层的线性组合、线性变换。\n",
    "\n",
    "### `nn.Linear`\n",
    "对一维信号（向量）进行线性组合。\n",
    "\n",
    "```python\n",
    "nn.Linear(\n",
    "    in_features,  ## 输入结点数\n",
    "    out_features, ## 输出结点数\n",
    "    bias=True ## 是否需要偏置\n",
    ")\n",
    "```\n",
    "\n",
    "## 激活函数层\n",
    "对特征进行非线性变换，使得多层神经网络有了“深度”的含义。\n",
    "\n",
    "如果没有激活层，那么多层线性层等价于一层线性层！\n",
    "\n",
    "### `nn.Sigmoid`\n",
    "“S形曲线”\n",
    "\n",
    "公式：y = 1/(1+e**(-x))\n",
    "\n",
    "梯度公式：y' = y*(1-y)\n",
    "\n",
    "特性：\n",
    "\n",
    "> 输出值在(0,1)，符合概率分布；\n",
    ">\n",
    "> 导数范围是[0, 0.25]，容易导致梯度消失；\n",
    "> \n",
    "> 输出为非0均值，破坏数据分布。\n",
    "\n",
    "### `nn.tanh`\n",
    "“双曲正切函数”\n",
    "\n",
    "特性：\n",
    "> 输出值在 (-1,1)，数据符合0均值；\n",
    ">\n",
    "> 导数范围是(0,1)，易导致梯度消失；\n",
    "\n",
    "### `nn.ReLU`\n",
    "针对 `Sigmoid`和`tanh`存在的梯度爆炸爆炸问题，提出了`ReLU`的方法缓解梯度爆炸问题。\n",
    "\n",
    "计算公式：y=max(0,x)\n",
    "\n",
    "特性：\n",
    "> 输出值均为正数，负半轴导致死神经元；\n",
    ">\n",
    "> 导数是1，缓解梯度消失，但易引发梯度爆炸。\n",
    "\n",
    "### 针对 `ReLU` 负半轴输出为0的情况，有一些 `ReLU`的变种：\n",
    "**`nn.LeakyReLU`**\n",
    "\n",
    "> 在负半轴增加了一个很小的斜率；(可以通过`negative_slope` 参数来设置负半轴斜率)\n",
    "\n",
    "**`nn.PReLU`**\n",
    "\n",
    "> 斜率为可学习的参数；(可以通过 `init` 参数设置可学习斜率)\n",
    "\n",
    "**`nn.RReLU`**\n",
    "\n",
    "> 斜率是随机的，每次都从一个均匀分布中去随机采样；(可以通过`lower`和`upper`设置均匀分布的下限和上限)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 权重初始化\n",
    "\n",
    "## `Xavier` 初始化\n",
    "针对饱和激活函数，如 Sigmoid, Tanh (保持数据尺度在恰当范围，通常方差为1)\n",
    "\n",
    "`nn.init.xavier_uniform_(m.weight.data)`\n",
    "\n",
    "## `Kaiming` 初始化\n",
    "针对非饱和激活函数，如 ReLU及其变种 (保持数据尺度在恰当范围，通常方差为1)\n",
    "\n",
    "`nn.init.kaiming_normal_(m.weight.data)`\n",
    "\n",
    "## 10种初始化方法：\n",
    "具体选取哪种方法要具体问题具体分析，最终的目的是为了维持网络层的输出值不能太大，也不能太小，尽量保持每一层额度输出值方差为1\n",
    "\n",
    "```\n",
    "1. Xavier 均匀分布\n",
    "2. Xavier 标准正态分布\n",
    "\n",
    "3. Kaiming 均匀分布\n",
    "4. Kaiming 标准正态分布\n",
    "\n",
    "5. 均匀分布\n",
    "6. 正态分布\n",
    "7. 常数分布\n",
    "\n",
    "8. 正交矩阵初始化\n",
    "9. 单位矩阵初始化\n",
    "10. 稀疏矩阵初始化\n",
    "```\n",
    "\n",
    "## 计算激活函数的方差变化尺度：\n",
    "方差变化尺度 = 输入数据的方差 / 激活函数后输出的数据的方差\n",
    "\n",
    "```python\n",
    "nn.init.calculate_gain(nonlinearity,  ## 激活函数名称，比如 Sigmoid, tanh, ReLU\n",
    "                       param=None ## 激活函数的参数，比如 LeakyReLU 的 negative_slpop\n",
    ")\n",
    "```\n",
    "\n",
    "\n",
    "# 损失函数\n",
    "损失函数：衡量模型输出与真实标签的差异。\n",
    "\n",
    "损失函数 (Loss Function): `Loss = f(y^, y)` (计算一个样本的损失)\n",
    "\n",
    "代价函数 (Cost Function): `Cost = mean(f(y^, y))` (计算整个数据集的平均损失)\n",
    "\n",
    "目标函数 (Objective Function): `Obj = Cost + Regularization(正则项)` (训练模型的目的就是得到一个目标函数)\n",
    "\n",
    "## `nn.CrossEntropyLoss`\n",
    "功能：`nn.LogSoftmax()` 与 `nn.NLLLoss()` 结合，进行交叉熵计算。\n",
    "\n",
    "交叉熵：衡量两个分布之间的差异。\n",
    "\n",
    "```python\n",
    "nn.CrossEntropyLoss(\n",
    "    weight=None, ## 各类别的loss设置权值\n",
    "    size_average=None, ## 该参数后续回被去掉\n",
    "    ignore_index=-100, ## 忽略某个类别\n",
    "    reduce=None, ## 该参数后续回被去掉\n",
    "    reduction=\"mean\"  ## 计算模式，可为 none(逐个元素计算)/sum(所有元素求和，返回标量)/mean(加权平均，返回标量)\n",
    ")\n",
    "```\n",
    "\n",
    "**举例：**\n",
    "```python\n",
    "inputs = torch.tensor([1,2],[1,3],[1,3], dtype=torch.float) ## inputs 的数据类型是 float\n",
    "target = torch.tensor([0,1,1], dtype=torch.long) ## target 的数据类型是 long\n",
    "\n",
    "loss_func = nn.CrossEntropyLoss(weight=None, reduction=\"none\")\n",
    "loss_none = loss_func(inputs, target)\n",
    "```\n",
    "\n",
    "```python\n",
    "weights = torch.tensor([1,2], dtype=torch.float) ## 每个类别都要设置 weight，组成一个 list\n",
    "## 如果有10个类别，那这里 weights 中list的长度是10，需要有10个元素。\n",
    "## 如果 reduction = \"mean\" ，那么weights不用太关注它的尺度，只需要关注各类别之间的比例即可；\n",
    "\n",
    "loss_func = nn.CrossEntropyLoss(weights=weights, reduction=\"None\")\n",
    "loss_none = loss_func(inputs, target)\n",
    "```\n",
    "\n",
    "## `nn.NLLLoss`\n",
    "功能：实现负对数似然函数中的**负号功能**。(就是将输入值取负号)\n",
    "\n",
    "```python\n",
    "nn.NLLLoss(\n",
    "    weight=None, ## 各类别的loss的权值\n",
    "    size_average=None, \n",
    "    ignore_index=-100,\n",
    "    reduction=None,\n",
    "    reduction=\"mean\" ## 计算模式\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.BCELoss`\n",
    "功能：二分类交叉熵。\n",
    "\n",
    "注意：输入值的取值在[0,1]之间。\n",
    "\n",
    "```python\n",
    "nn.BCELoss(\n",
    "    weight=None, ## 权重设置\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\" ## 计算模式\n",
    ")\n",
    "```\n",
    "\n",
    "```python\n",
    "inputs = torch.tensor([[1,2],[2,2],[3,4],[4,5]], dtype=torch.float) ## 这里的 inputs 和 target 都是 float类型\n",
    "target = torch.tensor([[1,0],[1,0],[0,1],[0,1]], dtype=torch.float)\n",
    "\n",
    "inputs = torch.sigmoid(inputs) ## 需要用 sigmoid 将 inputs 缩放到 0-1 之间\n",
    "loss_func = nn.BCELoss(weights=None, reduction=\"none\")\n",
    "\n",
    "loss_none = loss_func(inputs, target)\n",
    "```\n",
    "\n",
    "## `nn.BCEWithLogitsLoss`\n",
    "功能：结合 Sigmoid 与二分类交叉熵。\n",
    "\n",
    "注意：网络最后不加 sigmoid 函数。\n",
    "\n",
    "```python\n",
    "nn.BCEWithLogitsLoss(\n",
    "    weights=None,\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\", ## 计算模式\n",
    "    pos_weight=None ## 正样本的权值 (当 pos:neg = 1:3 时，可以将pos_weight设置为3)，需要是一个 tensor\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.L1Loss`\n",
    "功能：计算 inputs 和 target 之差的绝对值。\n",
    "\n",
    "```python\n",
    "nn.L1Loss(\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\"\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.MSELoss`\n",
    "功能：计算 inputs 和 target 之差的平方。\n",
    "\n",
    "```python\n",
    "nn.MSELoss(\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\" ## 计算模式\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.SmoothL1Loss`\n",
    "功能：平滑的 L1Loss (减轻离群点对模型的影响)\n",
    "\n",
    "```python\n",
    "nn.SmoothL1Loss(\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\"\n",
    ")\n",
    "```\n",
    "\n",
    "## `PoissonNLLLoss`\n",
    "功能：泊松分布的负对数似然损失函数。(输出符合泊松分布的时候可以使用)\n",
    "\n",
    "```python\n",
    "nn.PoissonNLLLoss(\n",
    "    log_input=True, ## 输入是否为对数形式，决定计算公式\n",
    "    full=False, ## 计算所有的 loss，默认为False\n",
    "    size_average=None,\n",
    "    eps=1e-8, ## 修正项，避免log(input)为nan\n",
    "    reduce=None,\n",
    "    reduction=\"mean\"\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.KLDivLoss`\n",
    "功能：计算 KLD，KL散度，相对熵。\n",
    "\n",
    "注意：需提前将输入进行取log_probabilities，比如通过 `nn.logsoftmax()`\n",
    "\n",
    "```python\n",
    "nn.KLDivLoss(\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\" ## 计算模式，包括 none/sum/mean/batchmean\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.MarginRankingLoss`\n",
    "功能：计算两个向量之间的相似度，用于排序任务。\n",
    "\n",
    "注意：该方法计算两组数据之间的差异，返回一个 `n*n` 的 loss 矩阵。\n",
    "\n",
    "```python\n",
    "nn.MarginRankingLoss(\n",
    "    margin=0.0, ## 边界值，x1与x2之间的差异值\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\" # 计算模式，包括 none/sum/mean\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.MultiLabelMarginLoss`\n",
    "功能：多标签边界损失函数。\n",
    "\n",
    "```python\n",
    "nn.MultiLabelMarginLoss(\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\" ## 计算模式，包括 none/sum/mean\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.SoftMarginLoss`\n",
    "功能：计算二分类的 logistic 损失。\n",
    "\n",
    "```python\n",
    "nn.SoftMarginLoss(\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\"\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.MultiLabelSoftMarginLoss`\n",
    "功能： `SoftMarginLoss` 多标签版本。\n",
    "\n",
    "```python\n",
    "nn.MultiLabelSoftMarginLoss(\n",
    "    weight=None,\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\"\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.MultiMarginLoss`\n",
    "功能：计算多酚类的折页损失。\n",
    "\n",
    "```python\n",
    "nn.MultiMarginLoss(\n",
    "    p=1, ## 可选1或2\n",
    "    margin=1.0, ## 边界值\n",
    "    weight=None,\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\"\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.TripletMarginLoss`\n",
    "功能：计算三元损失，人脸验证中常用。\n",
    "\n",
    "```python\n",
    "nn.TripletMarginLoss(\n",
    "    margin=1.0, ## 边界值\n",
    "    p=2.0, ## 范数的阶，默认为2\n",
    "    eps=1e-6,\n",
    "    swap=False,\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\"\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.HingeEmbeddingLoss`\n",
    "功能：计算两个输入的相似性，常用于 非线性embedding 和半监督学习。\n",
    "\n",
    "注意：输入x应为两个输入之差的绝对值。\n",
    "\n",
    "```python\n",
    "nn.HingeEmbeddingLoss(\n",
    "    margin=1.0, ## 边界值\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\"\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.CosineEmbeddingLoss`\n",
    "功能：采用cos相似度计算两个输入的相似性。\n",
    "\n",
    "```python\n",
    "nn.CosineEmbeddingLoss(\n",
    "    margin=0.0, ## 可取值[-1,1]，推荐为[0,0.5]\n",
    "    size_average=None,\n",
    "    reduce=None,\n",
    "    reduction=\"mean\"\n",
    ")\n",
    "```\n",
    "\n",
    "## `nn.CTCLoss`\n",
    "功能：计算 CTC 损失，解决时序类数据的分类。\n",
    "\n",
    "```python\n",
    "nn.CTCLoss(\n",
    "    blank=0, ## blank label\n",
    "    reduction=\"mean\",\n",
    "    zero_infinity=False ## 无穷大的值或者梯度置0\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 优化器 (`optimizer`)\n",
    "管理并更新模型中可学习参数的值，使模型输出更接近真实标签。\n",
    "\n",
    "**基本属性：**\n",
    "\n",
    "`defaults`：优化器超参数。\n",
    "\n",
    "`state`：参数的缓存，比如 momentum 的缓存。\n",
    "\n",
    "`param_groups`：管理的参数组。\n",
    "\n",
    "`_step_count`：记录更新次数，调整学习率的时候使用。\n",
    "\n",
    "\n",
    "**基本方法：**\n",
    "\n",
    "`zero_grad()`：清除所管理参数的梯度。（Pytorch中的张量梯度不会自动清零）\n",
    "\n",
    "`step()`：执行一步梯度更新。\n",
    "\n",
    "`add_param_group()`：添加参数组。\n",
    "\n",
    "(下面两个主要用于模型的断点恢复训练，每个一段时间保存优化器状态信息)\n",
    "\n",
    "`state_dict()`：获取优化器当前状态信息字典。\n",
    "\n",
    "`load_state_dict()`：将状态信息字典加载到优化器中。\n",
    "\n",
    "## 学习率:\n",
    "\n",
    "梯度下降：`Wi+1 = Wi - LR * g(Wi)`\n",
    "\n",
    "## 优化器：\n",
    "\n",
    "`Momentum (动量，冲量)`：结合当前梯度与上一次更新信息，用于当前更新（通常设置为 0.9）。\n",
    "`vi = m * vi-1 + g(wi)`\n",
    "\n",
    "`wi+1 = wi - lr * vi`\n",
    "\n",
    "## 1. `torch.optim.SGD`\n",
    "\n",
    "```python\n",
    "optim.SGD(\n",
    "    params, ## 管理的参数组\n",
    "    lr,     ## 学习率\n",
    "    momentum=0, ## 动量系数\n",
    "    dampening=0, \n",
    "    weight_decay=0, ## L2 正则化系数\n",
    "    nesterov=False ## 是否采用NAG\n",
    ")\n",
    "```\n",
    "\n",
    "## 2. pytorch 提供的多种优化器\n",
    "|优化器|用途|\n",
    "|--|--|\n",
    "|`optim.SGD`|随机梯度下降法|\n",
    "|`optim.Adagrad`|自适应学习率梯度下降法|\n",
    "|`optim.RMSprop`|Adagrad的改进|\n",
    "|`optim.Adadelta`|Adagrad的改进|\n",
    "|`optim.Adam`|RMSprop结合Momentum|\n",
    "|`optim.Adamax`|Adam增加学习率上限|\n",
    "|`optim.SparseAdam`|稀疏版的Adam|\n",
    "|`optim.ASGD`|随机平均梯度下降|\n",
    "|`optim.Rprop`|弹性反向传播 (full-batch时使用，但是mini-batch中不可用)|\n",
    "|`optim.LBFGS`|BFGS的改进|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 学习率调整策略\n",
    "\n",
    "学习率控制权重更新的步伐（要放到epoch循环中，不是在DataSet循环中）。\n",
    "\n",
    "## 0. `class _LRScheduler`\n",
    "pytorch中所有学习率调整策略都继承该类。\n",
    "\n",
    "主要属性：\n",
    "```\n",
    "optimizer: 关联的优化器\n",
    "last_epoch: 记录epoch数\n",
    "base_lrs: 记录初始学习率\n",
    "```\n",
    "\n",
    "主要方法：\n",
    "```\n",
    "step(): 更新下一个 epoch 的学习率\n",
    "get_lr(): 计算下一个epoch的学习率\n",
    "```\n",
    "\n",
    "## 1. `StepLR`\n",
    "功能：等间隔调整学习率\n",
    "\n",
    "```python\n",
    "lr_scheduler.StepLR(\n",
    "    optimizer,\n",
    "    step_size, ## 调整间隔数【每隔多少个epoch，将学习率乘以gamma值】\n",
    "    gamma=0.1, ## 调整系数【常用的比如 0.1， 0.5等】\n",
    "    last_epoch=-1\n",
    ")\n",
    "```\n",
    "调整方式：lr = lr * gamma\n",
    "\n",
    "\n",
    "## 2. `MultiStepLR`\n",
    "功能：按照给定的间隔调整学习率\n",
    "\n",
    "```python\n",
    "lr_scheduler.MultiStepLR(\n",
    "    optimizer,\n",
    "    milestones, ## 设定调整时刻数【比如，当epoch=[10, 30, 50]时进行学习率调整】\n",
    "    gamma=0.1, ## 调整系数\n",
    "    last_epoch=-1\n",
    ")\n",
    "```\n",
    "调整方式：lr = lr * gamma\n",
    "\n",
    "## 3. `ExponentialLR`\n",
    "功能：按照指数衰减调整学习率\n",
    "\n",
    "```python\n",
    "lr_scheduler.ExponentialRL(\n",
    "    optimizer,\n",
    "    gamma, ## 指数的底\n",
    "    last_epoch=-1\n",
    ")\n",
    "```\n",
    "调整方式：lr = lr * gamma ** epoch\n",
    "\n",
    "## 4. `CosinAnnealingLR`\n",
    "功能：按照cos周期调整学习率\n",
    "\n",
    "```python\n",
    "lr_scheduler.CosineAnnealingLR(\n",
    "    optimizer,\n",
    "    T_max, ## 下降周期\n",
    "    eta_min=0, ## 学习率下限\n",
    "    last_epoch=-1\n",
    ")\n",
    "```\n",
    "\n",
    "## 5. `ReduceLRonPlateau`\n",
    "功能：监控指标，当指标不再变化则调整\n",
    "\n",
    "```python\n",
    "lr_scheduler.ReduceLRonPlateau(\n",
    "    optimizer,\n",
    "    mode=\"min\", ## min/max两种模式\n",
    "    factor=0.1, ## 调整系数\n",
    "    patience=10, ## “耐心”，接受几次不变化（比如，连续10次不变，就调整学习率）\n",
    "    verbose=False, ## 是否打印日志\n",
    "    threshold=0.0001, \n",
    "    threshold_mode=\"rel\",\n",
    "    cooldown=0, ## “冷却时间”，停止监控一段时间\n",
    "    min_lr=0, ## 学习率下限 (学习率达到多少就不会再继续往下降)\n",
    "    eps=1e-08 ## 学习率衰减最小值\n",
    ")\n",
    "```\n",
    "\n",
    "## 6. `LambdaLR`\n",
    "功能：自定义调整策略（自定义方法调整学习率）\n",
    "\n",
    "```python\n",
    "lr_scheduler.LambdaLR(\n",
    "    optimizer,\n",
    "    lr_lambda, ## function or list\n",
    "    last_epoch=-1\n",
    ")\n",
    "```\n",
    "\n",
    "## 学习率调整方法总结：\n",
    "```\n",
    "1. 有序调整：Step, MultiStep, Exponential, CosineAnnealing\n",
    "2. 自适应调整：ReduceLROnPleateau\n",
    "3. 自定义调整：Lambda\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 可视化工具 `TensorBoard`\n",
    "\n",
    "可视化迭代训练过程。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "\n",
    "writer = SummaryWriter(comment=\"test_tensorboard\")\n",
    "for x in range(100):\n",
    "    writer.add_scalar(\"y=2x\", x * 2, x)\n",
    "    writer.add_scalar(\"y=pow(2,x)\", 2**x, x)\n",
    "    writer.add_scalars(\"/data/scalar_group\", {\"xsinx\": x*np.sin(x),\n",
    "                                              \"xcosx\": x*np.cos(x),\n",
    "                                              \"arctanx\": np.arctan(x)}, x)\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再在终端中运行`tensorboard --logdir ./runs`，结果如下：\n",
    "\n",
    "![](./Figures/tensorboard_test.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Tensorboard` 的使用:\n",
    "\n",
    "SummaryWriter\n",
    "\n",
    "功能：提供创建 event file 的高级接口。\n",
    "\n",
    "1. \n",
    "```python\n",
    "    add_scalar(tag, ## 图像标签名，图的唯一标识\n",
    "               scalar_value, ## 要记录的标量\n",
    "               global_step=None, ## x轴\n",
    "               walltime=None)\n",
    "```\n",
    "记录标量。\n",
    "\n",
    "2. \n",
    "```python\n",
    "    add_scalars(main_tag, ## 该图的标签\n",
    "                tag_scalar_dict, ## key是变量的tag，value是变量的值\n",
    "                global_step=None,\n",
    "                walltime=None)\n",
    "```\n",
    "\n",
    "3. \n",
    "```python\n",
    "    add_histogram(tag,\n",
    "                  values, ## 要统计的参数\n",
    "                  global_step, ## y轴\n",
    "                  bins)\n",
    "```\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torch_gpu",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}