Stablizing training metrics

[sondosaabed]

Body

Hey Githubbers, I'm not sure if this is the right discussion to ask the question if not let me know which space
I' training a deep learning model (Deep CNN architecture) the split rule that I am using is 80:10:10 train:test: validation
14000:2000:2000 instances

The following curves show the learning metrics, however, it doesn't start smoothly but it follows the trend of good fitting and converges. What could be the reason for the fluctuation before the 20th epoch? and how do I make a smoother curve? Note that I'm using dropout regularization and batch normalization techniques I am also using LR schedule

Guidelines

• I have read and understood this category's guidelines before making this post.

[CopperEagle]

If the orange line was your training loss, I'd be more worried. Higher fluctuations in the validation error are fairly common, also for dropout and batchnorm. Admittedly though, this is quite a high variablility. What do you see if you have a lower LR for the first 20 epochs? If you suspect there may be an error in the code, you could share it. But again, it's probably fine.

[sondosaabed]

Hii Eagle Copper
I tried it, it's better and converges more clearly now. The fluctuation that still exists, could it be related to what factors you think? is it the validation dataset variability related?

[CopperEagle]

Using the moving average over the orange curve, it looks better. For your first training run, the validation accuracy is basically around 40% until epoch 20. This time around, it does fallow the training accuracy a lot more when you apply the moving average. The variability is fairly usual, and the hickup at epoch 20 is the exception - in the first run it was the rule.

One thing to consider is the batch size. If it is too small, then you do many steps but with bad aim. Kind of like a drunk person trying to reach a valley. This can cause the variability in the validation as well. In this case, a lower LR can mitigate this symptom, but it is an inefficient fix. Trying a higher batch size (between 512 and 128 are typically good values) may be interesting.

Also I don't know what LR you use at the beginning. In projects I participated it was not unusual to use 10^-4 or even 10^-5 from the start. This may give a smoother curve but also longer coffee breaks 😆

[sondosaabed]

Hi Eagle :)
I don't know I usually use the default LR by the optimizer 😂 I'm new to using the Reduce LR On Plateau. I was told you can figure the right ones by experimenting, do you have a suggestion or a best practice on how to find the right training config? I came across this do you recommend ?
https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/

You are actually right I'm using a small batch size of 32. I'm gonna try now with 128 let's see what happens 🌠

[sondosaabed]

These were the outputs for the batch size of 128 🤔

When you have the time, take a look at my notebook, i'd like some code review if you have the time or enjoy doing so 😂
https://github.com/sondosaabed/Iris-of-eyes-recognition

[CopperEagle]

Code looks good 👍

One thing that could be happening is a bit overregularisation? Apparently some research has shown that dropout on convolutional layers should range from 0.1 to 0.2. You could also remove one or two batchnorm layers and see whether the validation error gets worse from this.

Also given the big size of the images, batch size of 32 is fairly small. I recommend at least 128, maybe try 256.

Grid search is the equivalent of a systematic bazooka - fire around enough until some configuration works. You seem to have quite good compute as each iteration only takes a few miliseconds. However, it may not be necessary for you. I mean... you do have 99.3% accuracy, which is quite impressive. 👍

What I would recommend, however, is to use data augmentation. Even if this makes the results worse - you are making it more reliable. What I recognized when looking at the data is that all eyes seemed to have the same distance to the camera with the same lighting conditions. This may be due to how the data was collected. Obviously, this won't happen for a phone app. Augmented data should also be mixed into the training and test cases of course. Include all possible to cover different lighting conditions, distance to the camera, etc.

If you find your performance dropping too much, you could consider using a pretrained network and continue training it. These models have seen "more of the world" and may be able to deal better with lighting conditions, etc.