Latency Encoding

[Dola47]

Hi @jeshraghian,

Thanks a lot for your response.

Here it is once again because I have not precisely understood how to use the Latency encoding.

I present here a snippet that shows how I use the Latency encoding on my dataset, however, I am highly suspicious that it is incorrect, as the latency encoding is getting way better accuracy than the rate encoding. So, please have a look and If I doing the latency encoding incorrectly, then I will be glad if you can provide me with some code that shows how to use it correctly.

loss_hist = []
val_loss_hist = []
counter = 0

# Outer training loop
for epoch in range(100):
    minibatch_counter = 0
    train_batch = iter(training_generator)

    # Minibatch training loop
    for data_it, targets_it in train_batch:
        data_it = data_it.to(device)
        targets_it = targets_it.to(device)
        batch_size = data_it.shape[0]
        # Spike generator
        spike_data, spike_targets = spikegen.latency(data_it, targets_it, num_outputs=num_outputs, num_steps=num_steps, tau=5, threshold=0.01,
                                                     linear=True, normalize=True, clip=True)
        
        output, mem_rec = net(spike_data.view(num_steps, batch_size, 95, 32, 32))
        log_p_y = log_softmax_fn(mem_rec)
        loss_val = torch.zeros((1), dtype=dtype, device=device)

        # Sum loss over time steps to perform BPTT
        for step in range(num_steps):
            loss_val += loss_fn(log_p_y[step], targets_it)

        # Gradient calculation
        optimizer.zero_grad()
        loss_val.backward(retain_graph=True)
        nn.utils.clip_grad_norm_(net.parameters(), 1)
        
        # Weight Update
        optimizer.step()

        # Store loss history for future plotting
        loss_hist.append(loss_val.item())

        # Val set
        val_data = itertools.cycle(val_generator)
        valdata_it, valtargets_it = next(val_data)
        valdata_it = valdata_it.to(device)
        valtargets_it = valtargets_it.to(device)
        batch_size = valdata_it.shape[0]
        # Val set spike conversion
        val_spike_data, val_spike_targets = spikegen.latency(valdata_it, valtargets_it, num_outputs=num_outputs, num_steps=num_steps, tau=5, threshold=0.01,
                                                             linear=True, normalize=True, clip=True)
        
        # Val set forward pass
        val_output, val_mem_rec = net(val_spike_data.view(num_steps, batch_size, 95, 32, 32))

        # Val set loss
        log_p_yval = log_softmax_fn(val_mem_rec)
        log_p_yval = log_p_yval.sum(dim=0)
        loss_val_val = loss_fn(log_p_yval, valtargets_it)
        val_loss_hist.append(loss_val_val.item())

        # Print val/train loss/accuracy
        if counter % 50 == 0:
              train_printer()
        minibatch_counter += 1
        counter += 1

loss_hist_true_grad = loss_hist
val_loss_hist_true_grad = val_loss_hist

Also just in case if you wonder how my labels look like. I have 8 classes, and I know which parts of my dataset belong to which class. Accordingly, my labels are defined as the following:

y_labels=np.zeros((1438))
y_labels[0:180] = 0
y_labels[180:360] = 1
y_labels[360:540] = 2
y_labels[540:720] = 3 
y_labels[720:900] = 4
y_labels[900:1079] = 5
y_labels[1079:1258] = 6
y_labels[1258:1438] = 7

In consequence, the spike_data is just tensors consisting of ones and zeros, and spike_targets is their corresponding labels.

Looking forward to your response as soon as possible.

Thanks.

[jeshraghian]

Hey @Dola47,

• I see you're applying latency to the input, though you are training your network to create a rate-coded output. There's nothing wrong with this approach, but I just want to make sure you are aware.

• Are you using the values in y_labels as the targets in targets_it?

• What does 95 represent in the third dimension?

[Dola47]

Hi @jeshraghian ,

I see your applying latency to the input, though you are training your network to create a rate-coded output. There's nothing wrong with this approach, but I just want to make sure you are aware.

That is not actually my aim, I need to have everything Latency encoded. I see then that I do not know how to insert or use my labels when I latency encodes my data. It would be helpful then if you can show me how to do that. Can you provide some modification to the above code to have the data, and labels latency encoded and trained upon that?

Moreover, can you provide some clarification on how could it be an acceptable approach to latency encode the input data, and keep the network learning rate encoded output? It does not seem logical to me. Also, I got very good results out of it and the networks have been able to learn all of the training data, but I can not see how the network is really working and how it learned anything :).

Are you using the values in y_labels as the targets in targets_it?

That's correct.

What does 95 represent in the third dimension?

I am considering it as my number of channels, my input dataset is of the shape of 95, 32, 32.

Looking forward to your response.

Thanks.

[jeshraghian]

That is not actually my aim, I need to have everything Latency encoded, then I do not know how to insert or use my labels when I latency encodes my data. It would be helpful then if you can show me how to do that. Can you provide some modification to the above code to have the data, and labels latency encoded and trained upon that?

Unfortunately I don't have the time to write your training loop, though I may include it in future tutorials.

Until then, here are some issues with your code that suggest you're training a rate coded output, and not latency:

loss_fn(log_p_y[step], targets_it) - your target is encouraging your network to fire at every single time step. This is effectively rate coding.

If you want latency coding at your output, your loss must only promote the correct class to fire first. I am assuming loss_fn is implementing cross entropy or negative log likelihood. You should see that your current target is trying make your correct class fire at every single time step. There is no official best practice to this yet. I would recommend using mean square error on the membrane. Then your target needs to be a tensor that has the correct class firing first, and incorrect classes firing later (or not firing at all). These can be achieved natively in numpy and PyTorch.

Your next problem will be train_printer(). If you haven't modified it from the code in the tutorial, you are measuring accuracy by counting the total number of output spikes. This is incorrect - latency coding should only count the class that fires first.

[jeshraghian]

but It threw a memory error

That most likely happened for large datasets/large time-steps and would require either smaller batches, smaller num_steps, or multiple GPUs.

I do not know why the warning raised, but anyways the problem is resolved now.

Glad to hear it. I have almost finished an updated form of target encoding functions.

[Dola47]

Hi @jeshraghian,

Any updates regarding the time to first spike target encoding and MSE Loss and decoding?

Thanks.

[jeshraghian]

The latest changes include time-to-first-spike encoding for inputs using spikegen.rate(input, num_steps=num_steps, first_spike_time=fst) and for targets using spikegen.targets_rate(targets, num_classes=num_classes, num_steps=num_steps, first_spike_time=fst).

Take a look at the documentation - there is a lot of info there on how to use it.

As a sample of how to apply MSE Loss:

loss_fn = nn.MSELoss()

targets_mem = spikegen.targets_rate(targets, num_classes=10, on_target=1.05, off_target=0.25)  # returns one-hot encoding where incorrect classes are 0.25, correct classes are 1.05
# You could of course leave on_target and off_target to default values for a simple one-hot-encoding, but I like to encourage the membrane to overshoot the threshold by a little bit to guarantee firing, and I like to ensure incorrect classes are semi-ready to fire.

# feedforward pass
spk_rec, mem_rec = net(data_it.view(batch_size, -1))

# initialize loss to 0
loss_val = torch.zeros((1), dtype=dtype, device=device)

# accumulate MSE loss for all time steps
for step in range(num_steps):
    loss_val += loss_fn(mem_rec[step], targets_mem)

loss_val.backward()

Note that targets_mem is not time-varying, so it is a 2D tensor.
If you were to modify certain arguments, such as first_spike_time, correct_rate, incorrect_rate, then it would be time-varying. You would then iterate through targets_mem by modifying the loss calculation to:

for step in range(num_steps):
    loss_val += loss_fn(mem_rec[step], targets_mem[step])

[Dola47]

@jeshraghian

Thanks a lot, it is becoming clearer now.

What about how to calculate the accuracy, for the rate encoding we were actually counting the no of spikes. If we now latency encode the targets and the data, how you gonna consider the class that spiked first and classify it as the correct class?

Looking forward to your response.

Thanks.

[jeshraghian]

I don't have a code snippet for that yet, so you'll need to try implementing this yourself.

But conceptually, if your output spk_rec is of size num_steps x batch_size x num_classes then you need to find the index for where the first (if any) '1' occurs in the num_steps dimension.

The predicted class would then be the minimum index across the num_classes dimension.

[Wang-jun-yu]

Do you have any example code that uses Latency Encoding to recognize the MNIST dataset? If so, could you please provide a link to it?

[Dola47]

Hi @Wang-jun-yu,

Sorry for the late reply.

I remember that I faced a problem in calculating the loss, and the network was not trained appropriately. Accordingly, I have not looked further into it.

Unfortunately, I do not have an implementation on MNIST. It is just a quick trial on a private dataset.

Thanks.

[Spice-monkey]

hey, I am also searching for the examples for Latency Encoding. If you find the relevant code, please contact with me.Thanks.

[SevillaFe]

Hello @jeshraghian , good afternoon,
I would like to know if there are any examples of using latency encoding applied to regression tasks or if you know of any material that could help.
Thanks to all