Pytorch（二） —— 激活函数、损失函数及其梯度

（二） —— 激活函数、损失函数及其梯度 2.损失函数 3. 求导和反向传播

1.激活函数 1.1 /

δ ( x ) = 1 1 + e − x δ ′ ( x ) = δ ( 1 − δ ) \delta(x)=\frac{1}{1+e^{-x}}\\\delta'(x)=\delta(1-\delta) δ(x)=1+e−x1​δ′(x)=δ(1−δ)

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.sigmoid(x)
plt.plot(x,y)
plt.show()

1.2 Tanh

t a n h ( x ) = e x − e − x e x + e − x ∂ t a n h ( x ) ∂ x = 1 − t a n h 2 ( x ) tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}\\\frac{\ tanh(x)}{\ x}=1-tanh^2(x) tanh(x)=ex+e−xex−e−x​∂x∂tanh(x)​=1−tanh2(x)

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.tanh(x)
plt.plot(x,y)
plt.show()

1.3 ReLU

f ( x ) = m a x ( 0 , x ) f(x)=max(0,x) f(x)=max(0,x)

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.relu(x)
plt.plot(x,y)
plt.show()

1.4

p i = e a i ∑ k = 1 N e a k ∂ p i ∂ a j = { p i ( 1 − p j ) i = j − p i p j i ≠ j p_i=\frac{e^{a_i}}{\sum_{k=1}^N{e^{a_k}}}\\ \frac{\ p_i}{\ a_j}=\left\{ \begin{array}{lc} p_i(1-p_j) & i=j \\ -&i\neq j\\ \end{array} \right. pi​=∑k=1N​eak​eai​​∂aj​∂pi​​={pi​(1−pj​)−pi​pj​​i=ji​=j​

import torch.nn.functional as F
logits = torch.rand(10)
prob = F.softmax(logits,dim=0)
print(prob)

tensor([0.1024, 0.0617, 0.1133, 0.1544, 0.1184, 0.0735, 0.0590, 0.1036, 0.0861,0.1275])

2.损失函数 2.1 MSE

import torch.nn.functional as F
x = torch.rand(100,64)
w = torch.rand(64,1)
y = torch.rand(100,1)
mse = F.mse_loss(y,x@w)
print(mse)

tensor(238.5115)

2.2

import torch.nn.functional as F
x = torch.rand(100,64)
w = torch.rand(64,10)
y = torch.randint(0,9,[100])
entropy = F.cross_entropy(x@w,y)
print(entropy)

tensor(3.6413)

3. 求导和反向传播 3.1 求导

import torch.nn.functional as F
import torch
x = torch.rand(100,64)
w = torch.rand(64,1)
y = torch.rand(100,1)
w.requires_grad_()
mse = F.mse_loss(x@w,y)
grads = torch.autograd.grad(mse,[w])
print(grads[0].shape)

torch.Size([64, 1])

3.2 反向传播

import torch.nn.functional as F
import torch
x = torch.rand(100,64)
w = torch.rand(64,10)
w.requires_grad_()
y = torch.randint(0,9,[100,])
entropy = F.cross_entropy(x@w,y)
entropy.backward()
w.grad.shape

torch.Size([64, 10])

by 2022 06 28

人生 只是 须臾的刹那

人间 只是 天地的夹缝

——————五月天（因为你 所以我）——————