Sparse Network

Requiriment

• PyTorch, Torchvision...

Reference

1. How do neurons operate on sparse distributed representations? A mathematical theory of sparsity, neurons and active dendrites

   • Paper

2. How Can We Be So Dense? The Benefits of Using Highly Sparse Representations

   • Paper
   • Code

3. K-Winner implementation:

   • Discussion

Miscellaneous

1. Sparse Coding
2. Sparse Distributed Representations
3. ISTA Implementation
4. Bayesian Bits
5. Sparse AutoEncoder
6. Sparse AutoEncoder examples
   • Using L1
   • Using KL Divergence
7. Understanding Pytorch hooks

...

$ KL(\rho||\hat{\rho})_{Ber} = \rho \log\left(\dfrac{\rho}{\hat{\rho}}\right) + (1-\rho)\log\left(\dfrac{1-\rho}{1-\hat{\rho}}\right)$

Gradient based on $\hat{\rho }$

${\displaystyle \frac{\partial KL(\rho ||\hat{\rho }{)}_{Ber}}{\partial \hat{\rho }}}={\displaystyle \frac{1-\rho }{1-\hat{\rho }}}-{\displaystyle \frac{\rho }{\hat{\rho }}}=-{\displaystyle \frac{\hat{\rho }-\rho }{(\hat{\rho }-1)\hat{\rho }}}$

TODO

• Blind Spot Convolution
  • Just observe the noisy context of a pixel
  • 'Efficient Blind-Spot Neural Network Architecture for Image Denoising' [Ref]
• Learning Hybrid Sparsity Prior for Image Restoration
  • Paper
• Sparse Linear
  • Instead of activation functions
  • Reference: Extending PyTorch
• Important! Include a configuration where you can set the sparse property with a 'constant prunning' or 'gradual prunning'.
• Notebook with plotting KL Div and gradient estimation