Deep-learning-based registration example #2137

sarathknv · 2020-04-21T20:19:00Z

A simple example of deep-learning-based non-rigid image registration with the MNIST dataset, taking ideas from

An Unsupervised Learning Model for Deformable Medical Image Registration
Non-rigid image registration using fully convolutional networks with deep self-supervision

Framework

An encoder-decoder network is trained to output a dense displacement field that is used to warp the moving image to align with the fixed image. I've used MNIST images of one class, same as that of the fixed image, for training and testing.

Sample results

Colab notebook

https://colab.research.google.com/drive/1fTzz1aT2sb8oAXRO1-dr6O_IR6dof36e

Training takes about 5 minutes on GPU.

sarathknv · 2020-05-24T16:16:49Z

I've refactored my code a little. @BramshQamar @skoudoro @ShreyasFadnavis @Garyfallidis your thoughts on this.

Usage:

from dipy.nn.registration import SimpleFCN2d
from dipy.nn.metrics import NormalizedCrossCorrelation, MeanSquaredError


ncc = NormalizedCrossCorrelation()

model = SimpleFCN2d()
model.compile(optimizer='adam', loss=ncc)

# train
model.fit(x_train, y_train, validation_split=0.2, batch_size=8, epochs=10)

# evaluate on test set
model.evaluate(x_test, y_test, batch_size=8)

# save weights
model.save_weights('./weights_fcn2d.h5')

# load_weights
model.load_weights('weights_fcn2d.h5')

# for single/batch input
moved = model.predict_on_batch({'moving': moving, 'static': static})

If this API is okay, I'll go ahead and clean it and add the test file.

codecov · 2020-05-24T16:50:47Z

Codecov Report

Merging #2137 into master will decrease coverage by 2.12%.
The diff coverage is 78.16%.

@@            Coverage Diff             @@
##           master    #2137      +/-   ##
==========================================
- Coverage   91.37%   89.25%   -2.13%     
==========================================
  Files         251      259       +8     
  Lines       32514    32864     +350     
  Branches     3423     3444      +21     
==========================================
- Hits        29711    29333     -378     
- Misses       2057     2792     +735     
+ Partials      746      739       -7

Impacted Files	Coverage Δ
dipy/nn/registration/fcn.py	`24.19% <24.19%> (ø)`
dipy/nn/registration/__init__.py	`66.66% <66.66%> (ø)`
dipy/nn/registration/dataloader.py	`80.76% <80.76%> (ø)`
dipy/nn/metrics.py	`85.00% <85.00%> (ø)`
dipy/nn/tests/test_metrics.py	`86.36% <86.36%> (ø)`
dipy/nn/tests/test_unet.py	`91.80% <91.80%> (ø)`
dipy/nn/registration/_utils.py	`94.11% <94.11%> (ø)`
dipy/nn/registration/unet.py	`94.66% <94.66%> (ø)`
dipy/viz/app.py	`4.46% <0.00%> (-72.96%)`	⬇️
dipy/viz/tests/test_apps.py	`24.56% <0.00%> (-70.18%)`	⬇️
... and 22 more

ShreyasFadnavis · 2020-05-24T19:22:28Z

Hi @sarathknv ... I quite like this API design. Would like some other people's opinions on this!

@skoudoro did you get a chance to see if we can unify the metrics that we already have in DIPY with TF?

@finalelement can you take a look too? Does this abide by your PR? Any other stuff you'd like the API to have/ we missed out on?

skoudoro · 2020-05-25T00:49:12Z

Hi @sarathknv,

Thank you for this update!

Personally, I prefer when the compile method is in the __init__ of your class like MultipleLayersPerceptron class. I do not think the user needs this extra work. However, I like that your loss parameter is a callable. I think we should keep this point.
Concerning load_weights/save_weights, I think it is good, and you should keep these methods.
I am not sure what is the difference between predict and predict_on_batch. Can you tell me more? Can it just be an option of predict?
Concerning the metric. Do you think you will add more methods to these classes? If not, we do not need a class and you can keep them as a function with a decorator. A class with only __init__ and __call__ methods is a sign that it can be simpler.
There a lot of code.... too much code... it will be hard to review. Please, consider 1 PR = 1 model. You should avoid adding 3-4 models on the same PR. Let's focus on 1 model and make sure that we all agree with the API.

Thank you

sarathknv · 2020-05-25T20:51:40Z

Thanks for the feedback @ShreyasFadnavis and @skoudoro ☺️

Personally, I prefer when the compile method is in the init of your class like MultipleLayersPerceptron class. I do not think the user needs this extra work. However, I like that your loss parameter is a callable. I think we should keep this point.

I agree with the idea of keeping the API simple. But I also think that it's good to have the compile method accessible to the user. If we want to change the parameters of loss or optimizer, like the learning rate, or maybe the optimizer itself, we can just call the compile method to update them in the model, without affecting the model weights. I generally try a few learning rates on sample data before running the full-scale experiment, to get a feel of how long it would take to train. So I thought it would be nice to have it as a method. ☺️

We could do something like this

class CNN(object):
    def __init__(self, loss=None, optimizer='adam')
        self.model.compile(loss=loss, optimizer=optimizer)

    def compile(self, optimizer='adam', loss=None):
        self.model.compile(optimizer=optimizer, loss=loss)

Concerning load_weights/save_weights, I think it is good, and you should keep these methods.

👍

I am not sure what is the difference between predict and predict_on_batch. Can you tell me more? Can it just be an option of predict?

predict is for inputs with many samples. It internally processes the data batch by batch. For inputs with few samples predict has overhead and is slower than predict_on_batch. This StackOverflow thread clears up the differences and also has some benchmarks.

From the TensorFlow docs

Computation is done in batches. This method is designed for performance in large scale inputs. For small amount of inputs that fit in one batch, directly using call is recommended for faster execution, e.g., model(x), or model(x, training=False)

If you think predict and predict_on_batch would be confusing to the user, we can have a __call__ method calling the predict_on_batch method. Then inference usage would be simple

model = CNN()
moved = model({'moving': moving, 'static': static})

Concerning the metric. Do you think you will add more methods to these classes? If not, we do not need a class and you can keep them as a function with a decorator. A class with only init and call methods is a sign that it can be simpler.

Changed the metrics from classes to functions with decorators. 👍

There a lot of code.... too much code... it will be hard to review. Please, consider 1 PR = 1 model. You should avoid adding 3-4 models on the same PR. Let's focus on 1 model and make sure that we all agree with the API.

Removed VoxelMorph models. 👍

skoudoro · 2020-05-25T21:23:26Z

We could do something like this

class CNN(object):
 def __init__(self, loss=None, optimizer='adam')
     self.model.compile(loss=loss, optimizer=optimizer)

 def compile(self, optimizer='adam', loss=None):
     self.model.compile(optimizer=optimizer, loss=loss)

ok, a great hybrid solution, I am fine with that 👍.

predict is for inputs with many samples. It internally processes the data batch by batch. For inputs with few samples predict has overhead and is slower than predict_on_batch. This StackOverflow thread clears up the differences and also has some benchmarks.

Ok, thanks for the reference, I will read more about it.

If you think predict and predict_on_batch would be confusing to the user, we can have a call method calling the predict_on_batch method. Then inference usage would be simple

Yes, I think they are confusing. It needs more thinking and we need to be explicit so I would not recommend using __call__ here.

Removed VoxelMorph models. 👍

Thank you for the cleaning!

BramshQamar · 2020-05-27T22:09:36Z

dipy/nn/registration/_utils.py

+        raise ImportError('Please upgrade to TensorFlow 2+')
+
+
+def regular_grid_2d(height, width):


Missing docstring

BramshQamar

Nice work @sarathknv !

This PR needs documentation.

dipy/nn/registration/_utils.py

BramshQamar · 2020-05-27T22:10:29Z

dipy/nn/registration/fcn.py

+__all__ = ["SimpleFCN2d"]
+
+
+class SimpleFCN2d(object):


All functions in this class missing docstring

BramshQamar · 2020-05-27T22:26:34Z

I've refactored my code a little. @BramshQamar @skoudoro @ShreyasFadnavis @Garyfallidis your thoughts on this.

Usage:

from dipy.nn.registration import SimpleFCN2d
from dipy.nn.metrics import NormalizedCrossCorrelation, MeanSquaredError


ncc = NormalizedCrossCorrelation()

model = SimpleFCN2d()
model.compile(optimizer='adam', loss=ncc)

# train
model.fit(x_train, y_train, validation_split=0.2, batch_size=8, epochs=10)

# evaluate on test set
model.evaluate(x_test, y_test, batch_size=8)

# save weights
model.save_weights('./weights_fcn2d.h5')

# load_weights
model.load_weights('weights_fcn2d.h5')

# for single/batch input
moved = model.predict_on_batch({'moving': moving, 'static': static})

If this API is okay, I'll go ahead and clean it and add the test file.

It would be useful to have documentation/tutorial on how to use the API. Something similar to the usage example you have provided here.

Here are the existing tutorials in DIPY.

Tutorial can be added later in a different PR.

finalelement · 2020-05-28T15:21:05Z

@ShreyasFadnavis Thanks for the tag. Will check out the code.

pep8speaks · 2020-06-26T12:45:24Z

Hello @sarathknv, Thank you for updating !

In the file dipy/nn/registration/fcn.py:

Line 138:42: W292 no newline at end of file

In the file doc/examples/nn_deformable_registration_2d.py:

Line 60:76: W291 trailing whitespace
Line 218:76: W291 trailing whitespace
Line 226:79: W291 trailing whitespace

Comment last updated at 2020-08-21 16:32:45 UTC

arokem

Nice work. Here are a few comments/suggestions.

arokem · 2020-06-26T12:53:53Z

dipy/nn/registration/fcn.py

-class SimpleFCN2d(object):
-    def __init__(self, input_shape=(32, 32)):
-        in_channels = 1
+class FCN2d(object):


What does "FCN" stand for? Could we spell this out in the class name?

Really sorry for the late reply. I can be active from now on.

This was some custom architecture that I wrote for the POC. I’ve added a standard model (U-Net) and will be removing fcn.py.

dipy/nn/registration/unet.py

dipy/nn/registration/fcn.py

arokem · 2020-06-26T12:58:17Z

doc/examples/nn_deformable_registration_2d.py

+"""
+
+
+class DataLoader(tf.keras.utils.Sequence):


Is this something that would be worth implementing as library code?

It’s definitely useful, but we need to handle several cases if we are planning to have this in the library.
Current location of the class dipy/nn/registration/dataloader.py

The function of the dataloader object is to feed batches of processed data to the model. The user needs to pass the entire dataset as input either as arrays (shape (num_images, H, W, …, num_channels)) or as a list of file paths, and the dataloader object performs some preprocessing (data transformations, adding dimensions, changing data types) on batches of data and feeds it to the model. Currently, this code only supports NumPy arrays as input, which will work for small datasets that can be loaded into the memory, but a general use case would be file paths as input.

Some cases that we need to handle:

NumPy arrays as input

Single static image and multiple moving images - All the moving images will be registered to this static image (moving shape (N, H, W, ..., C), static shape (H, W, ..., C))

Multiple static and moving images - every moving image has a different static image (moving shape (N, H, W, C), static shape (N, H, W, C))

Lists of file paths as input

Single static image and multiple moving images

Multiple static and moving images

Different file extensions .npy, .h5, .nii, …

arokem · 2020-06-26T13:00:09Z

doc/examples/nn_deformable_registration_2d.py

+the moving images.
+"""
+
+moved = moved.squeeze(axis=-1)  # Remove the channel dim.


Would this little block of code be useful to implement as module code?

This is only needed for single-channel images. Since most images are single-channel I also think it is useful to implement as library code. Do you mean something like this? Add the squeeze option in the predict method

moved = model.predict(data, squeeze=True)

arokem · 2020-06-26T14:57:41Z

doc/examples/nn_deformable_registration_2d.py

+        ax.imshow(images_list[j][i], cmap='gray')
+plt.tight_layout()
+plt.show()
+plt.savefig('sample_results.png')


This figure is empty for me when I run this.

Maybe use fig.savefig instead?

arokem · 2020-06-26T14:59:42Z

doc/examples/nn_deformable_registration_2d.py

+"""
+
+batch_size = 32
+epochs = 100


It looks like performance converges after much fewer epochs. Maybe set this to 25 for the purpose of the example and then add a comment that this could be increased to a larger number for more accurate results?

Garyfallidis · 2020-07-02T21:04:20Z

@sarathknv please enable automatic trailing whitespace removal in your IDE. It will help you resolve pep8 issues faster.

Garyfallidis · 2020-11-10T19:29:22Z

@sarathknv, @BramshQamar, @ShreyasFadnavis, @skoudoro what is the progress here? Does this need a rebase?

Paluck1Arora2 · 2023-01-08T06:31:46Z

Can anyone please provide the dataset of this code "https://github.com/sarathknv/voxelmorph"
data = np.load('drive/My Drive/DIPY/brain_affine_128.npy')
static = np.load('drive/My Drive/DIPY/static.npy')
These lines are mention in voxelmorph.ipynb file

data = np.load('data/t1_moving_128.npy')
static = np.load('data/t1_static_128.npy')
These lines are mention in main.py

jhlegarreta · 2023-01-09T21:54:40Z

#2137 (comment) @Paluck1Arora2 Please post questions in DIPY's gitter room, mailing list or the DIPY wiki:
https://dipy.org/support/

Also, please avoid asking the same question as a comment and opening an issue at the same time:
#2708

Note that issues should primarily be opened to either raise attention about bugs or to propose/ask about new features.

Thanks.

sarathknv added 5 commits April 21, 2020 23:10

added registration example

96fbaf4

minor correction in args

7587acd

fixed typos

7cc1f23

added optional_package for tf and matplotlib

68722d1

refactored a bit

328983d

skoudoro added gsod2020 state: work-in-progress labels May 24, 2020

more refactoring and removed voxelmorph

24cf40c

BramshQamar reviewed May 27, 2020

View reviewed changes

sarathknv added 3 commits May 29, 2020 17:02

added docstrings

568038d

bug fix in grid_sample

0fea652

added test file and tutorial

b42664a

arokem reviewed Jun 26, 2020

View reviewed changes

sarathknv added 6 commits August 18, 2020 22:01

Merge branch 'nipy-dipy-master' into feat-nn

fd404f6

fixed tests and metrics, added dataloader to lib

bf0d3fe

removed trailing white spaces

b4b66a6

updated setup.py

2429eab

updated setup.py

4bfbe53

added optional_package for tf

5483c74

sarathknv added 2 commits August 21, 2020 21:50

updated __init__.py

6c8550c

updated tests

a57ce97

skoudoro force-pushed the master branch 7 times, most recently from 1419292 to ca6268a Compare December 8, 2023 22:25

skoudoro force-pushed the master branch 3 times, most recently from 5935e1e to 765963e Compare January 24, 2024 19:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deep-learning-based registration example #2137

Deep-learning-based registration example #2137

sarathknv commented Apr 21, 2020

sarathknv commented May 24, 2020

codecov bot commented May 24, 2020 •

edited

ShreyasFadnavis commented May 24, 2020

skoudoro commented May 25, 2020

sarathknv commented May 25, 2020

skoudoro commented May 25, 2020

BramshQamar May 27, 2020

BramshQamar left a comment

BramshQamar May 27, 2020

BramshQamar commented May 27, 2020

finalelement commented May 28, 2020

pep8speaks commented Jun 26, 2020 •

edited

arokem left a comment

arokem Jun 26, 2020

sarathknv Jan 17, 2021

arokem Jun 26, 2020

sarathknv Jan 17, 2021

arokem Jun 26, 2020

sarathknv Jan 17, 2021

arokem Jun 26, 2020

arokem Jun 26, 2020

arokem Jun 26, 2020

Garyfallidis commented Jul 2, 2020

Garyfallidis commented Nov 10, 2020

Paluck1Arora2 commented Jan 8, 2023

jhlegarreta commented Jan 9, 2023 •

edited

		raise ImportError('Please upgrade to TensorFlow 2+')


		def regular_grid_2d(height, width):

Deep-learning-based registration example #2137

Are you sure you want to change the base?

Deep-learning-based registration example #2137

Conversation

sarathknv commented Apr 21, 2020

Framework

Sample results

Colab notebook

sarathknv commented May 24, 2020

codecov bot commented May 24, 2020 • edited

Codecov Report

ShreyasFadnavis commented May 24, 2020

skoudoro commented May 25, 2020

sarathknv commented May 25, 2020

skoudoro commented May 25, 2020

Choose a reason for hiding this comment

BramshQamar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BramshQamar commented May 27, 2020

finalelement commented May 28, 2020

pep8speaks commented Jun 26, 2020 • edited

Comment last updated at 2020-08-21 16:32:45 UTC

arokem left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Garyfallidis commented Jul 2, 2020

Garyfallidis commented Nov 10, 2020

Paluck1Arora2 commented Jan 8, 2023

jhlegarreta commented Jan 9, 2023 • edited

codecov bot commented May 24, 2020 •

edited

pep8speaks commented Jun 26, 2020 •

edited

jhlegarreta commented Jan 9, 2023 •

edited