performance loss a little big

[wuzhiyang2016]

I have train alexnet-s model in cifar10 for 100 epochs:

### dense model performance:

accuracy: 8322/10000 (83.220%)
conv layers sparsity:
layer features.0.weight sparsity: 0.002812213039485756
layer features.3.weight sparsity: 0.017063802083333357
layer features.6.weight sparsity: 0.01662303783275465
layer features.9.weight sparsity: 0.011704433111496937
layer features.12.weight sparsity: 0.25820583767361116
layer classifier.0.weight sparsity: 0.004718780517578125
layer classifier.3.weight sparsity: 0.004614830017089844
layer classifier.6.weight sparsity: 0.0021484375000000444

### sparse model performance:
accuracy: 8131/10000 (81.310%)
conv layers sparsity:
layer features.0.weight sparsity: 0.11059458218549123
layer features.3.weight sparsity: 0.9346875
layer features.6.weight sparsity: 0.9564841941550926
layer features.9.weight sparsity: 0.9704597378954475
layer features.12.weight sparsity: 0.2798902723524306
layer classifier.0.weight sparsity: 0.8953094482421875
layer classifier.3.weight sparsity: 0.9820785522460938
layer classifier.6.weight sparsity: 0.046191406249999956

accuracy down nearly 2 percent

so, is there something wrong with me ?

[TimDettmers]

You need to train both networks a bit longer (250 epochs) to get better performance, but the gap between dense and sparse performance will remain if you use the default 5% weights. In the new version of our paper (will be released on Monday), we investigate how much weights do we need to achieve dense performance for each network on CIFAR-10. AlexNet-s needs the most weights among all networks: 50%. So statistically, with 50% weights, you should reach dense performance. You can get good results with less weights and worse results with more weights, but the sparse mean will be in the same 95% confidence interval as the dense mean if you use 50% weights. You should still see a speedup of about 1.3x for dense convolution and 3.0x for sparse convolution with 50% weights.