Inference and lookups

[cvasfi]

Hi, i've noticed a great speedup when i used lookup based convolutions for training my own network, however the speedup on the inference is quite marginal so i have been wondering if i need to change something when i am doing the inference compared to training.

My reasoning comes from the following paragraph from the paper:

After training, we convert P to the indices and the co-efficients tensors I and C for each layer. At test time, we follow equation 5 to efficiently compute the output of each convolutional layer.

Also when i look into inference.py, there are 2 seperate tests for graphs g1 and g2, dense and sparse supposedly. But i can't really understand what's different about them since both build the same alexnet graph with normal convolutions.

[ildoonet]

@cvasfi

I think you're right.

I have implemented the converted version at that paragraph, here : https://github.com/ildoonet/tf-lcnn/blob/master/layers/LookupConvolution2d.py#L196

Here is what you need to do.

1. Compile custom tf operation : https://github.com/ildoonet/tf-lcnn/tree/master/ops

2. You need to refer to inference as inference.py : https://github.com/ildoonet/tf-lcnn/blob/master/inference.py

as you can see, it loads the original model and save some weights, and loads again with custom operation for fast inference.

I will organize the code so that the above process can be performed easily.

PS. If you provide me a github link to your code, I can give you more detailed advice.

[cvasfi]

as you can see, it loads the original model and save some weights, and loads again with custom operation for fast inference.

But both tests inside the inference.py use the same operation: convtype=conf['conv'] (lines 88 and 152).

Does this part have any direct effect on the results currently: extract_dense_weights(sess)? Because i didn't use it in my network.

My model (a modified alexNet) is defined here: https://github.com/cvasfi/light-cnns/blob/master/TinyImagenet/AlexNet_TI.py#L101
It uses lookup_conv2d for all of its convolutions.

and for the inference i build the graph with again lookup_conv2d and run a sess.run to get the accuracy
as can be seen on https://github.com/cvasfi/light-cnns/blob/master/TinyImagenet/main.py#L166

[ildoonet]

@cvasfi

Yes, when you load weights, extract_dense_weightspart will extract Lookup-convolution weights and use it to generate indices and coefficients as the paper.

I will train alexnet and resnet within few weeks to reproduce the performance of the original thesis.

(But I have an assumption that there are implementation techniques to speed up the original article which I don't know yet...)

BTW, what is your sparsity? Sparsity can impact the inference speed.

[stone226]

@ildoonet
I have the similar problem about inference.After trainning the a model with Lcnn,how to inference with the function named extract_dense_weights.I do not understand how the indices and coefficients to be used in inference.py,after they generated from Lookup-convolution weights.

[ildoonet]

@stone226
I will clean some codes and update the documents. It is a little bit tricky, You should check https://github.com/ildoonet/tf-lcnn/blob/master/inference.py

[cvasfi]

Thanks for the tips, i managed to use custom_op after using extract_dense_weights. However it has been slower than regular convolution so i decided to check the sparsities like you did in the inference.py and it seems to be 0. I have trained with lookup conv2d with initial sparsity of 0.01 and dict_size of 30 with lambda=1.0

Any idea what could be the issue?

[ildoonet]

Did you check the 'density' like here? -- https://github.com/ildoonet/tf-lcnn/blob/master/inference.py#L119

if density is zero, it means all elements are 'ZERO' so sparse matrix have nothing to compute. It should be really fast but its output is meaning-less since it didn't compute anything.

If sparsity is zero(=density is 1.0), it means every elements are non-zero so sparse matrix have lots of thing to compute, in this case, generally normal convolutional operation is way faster.

So the bottom line is...

You need to tune your initial sparsity and lambda, to get a proper number of sparsity. In my case, when sparsity is near 0.0x~0.1x after whole training, sparse convolution operation is way faster than normal convolution.

[stone226]

now,as I have trouble in compling custom_op,I set the mode with 'tf_op',when i run the inference.py.But,there is an error which is found in the function 'tf.sparse_tensor_dense_matmul(sparse_kernel, img2col)'.The error information is "ValueError: Shape must be rank 2 but is rank 4 for 'layer1/SparseTensorDenseMatMul/SparseTensorDenseMatMul' (op: 'SparseTensorDenseMatMul') with input shapes: [1233,4], [1233], [4], [363,25]."

[ildoonet]

Well.. 'tf_op' is not recommended. tf-op was implemented just for test purposes and It is not suitable for real usage. (despite this fact, I will look into that...)

You need to compile custom-op as readme, and use it without modification of the mode.

[stone226]

I compile the custom_op,and set the mode with 'custom_op'.However,there is another problem that when I input a test image,the predicted result which generated from graph1(g1)is right,then the other predicted result which generated from graph2(g2)is always wrong.what's more,the wrong predicted result is the same,when change the input test image every time.
I load model config just as the training code,not the file named conf.json generated from training,becasue when I used conf.json file in inference code,there some errors.I doubt that the above problem is caused by these changes in my inference.could you give some advices and upload your 'conf.json' and 'train_conf.json'

[amcinto]

I ran into a similar error I think you just need to modify the inference.py file. If you change parts of the Inference.py to load the correct model that was trained it should work. I think there is an error that is happening but that should be a work around. I ran into a similar error when training but when inference happens then the dense weights are not using the sparsity. How does that work? Do I have to train twice?