Spatial Transformer Network with Affine, Projective and Elastic Transformations

The Spatial Transformer Network [1] allows the spatial manipulation of data within the network.

A Spatial Transformer Network implemented in Tensorflow 0.9 by Daniyar Turmukhambetov (@dkantz) [2] and based on [3] (which is also in [4]), [5] and [6].

This tensorflow implementation supports Affine, Projective and Elastic (Thin Plate Spline [7]) Transformations.

The original code has been updated in a number of ways by Steven Bamford (@bamford):

• now compatibile with TensorFlow's eager execution model
• transformation classes now inherit from Keras Layer
• include a Restricted Transformation (see below for explanation)
• optional edge masking with constant value, rather than nearest neighbour
• option to preserve flux (in Affine and Restricted Transformation).

How to use

from spatial_transformer import AffineTransformer, ProjectiveTransformer, ElasticTransformer, RestrictedTransformer

# Initialize
outsize = [300, 300]
stl1 = AffineTransformer(outsize)
stl2 = ProjectiveTransformer(outsize)
stl3 = ElasticTransformer(outsize)
stl4 = RestrictedTransformer(outsize)

# Transform 
y1 = stl1.transform(U, theta1)
y2 = stl2.transform(U, theta2)
y3 = stl3.transform(U, theta3)
y4 = stl4.transform(U, theta4)

Examples

Input

AffineTransformer

example_affine.py shows how to use AffineTransformer. Note, affine transformations preserve parallel lines.

ProjectiveTransformer

example_project.py shows how to use ProjectiveTransformer. Note, parallel lines are not parallel after transformation.

ElasticTransformer

example_elastic.py shows how to use ElasticTransformer. Here, deformations are defined with Thin Plate Splines on a 4x4 grid of control points.

RestrictedTransformer

example_affine.py shows how to use RestrictedTransformer. This behaves similarly to AffineTransformer, but takes more directly comprehensible parameters, designed in such a way as to restrict possible transformations. The transformation parameters, theta, are (x_scale, y_scale, rotation, x_translation, y_translation), where the scales are logarithms of the actual scale factor and the rotation is given as tan(angle/2). This prevents reflections and limits rotations to ±180 degrees. The parameterisation also makes it easier to apply further external restrictions on the transformations (e.g. not allowing rotations, only allowing isotropic scaling, etc.).

By default the transformations are applied in the order: scale, then rotate, then translate. This is appropriate (and vital) when reintroduce transformations to a 'standardised' image. However, for transforming an input image to a 'standardised' (centred, derotated, circularised) format, the reverse order is required: translate, then rotate, then scale. This is available by specifying reverse=True. Note that previously only the 'reverse' order was implemented; the default behaviour was changed in commit 7b5f28 on 2020-11-22.

Bilinear and Bicubic Interpolation

example_interp.py shows how to use Bilinear and Bicubic interpolation methods.

Bilinear:

Spatial Transformer Output:

Tensorflow Output:

Normalized absolute difference:

Bicubic:

Spatial Transformer Output:

Tensorflow Output:

Normalized absolute difference:

Also, the interpolation doesn't have the bug at the edges, as in [2] and [3]. See tensorflow/models#193 for details.

References

[1] Jaderberg, Max, et al. "Spatial Transformer Networks." arXiv preprint arXiv:1506.02025 (2015)

[2] https://github.com/dantkz/spatial-transformer-tensorflow

[3] https://github.com/tensorflow/models/tree/master/transformer/transformerlayer.py

[4] https://github.com/daviddao/spatial-transformer-tensorflow

[5] https://github.com/skaae/transformer_network/blob/master/transformerlayer.py

[6] https://github.com/Lasagne/Lasagne/blob/master/lasagne/layers/special.py

[7] Fred L. Bookstein. "Principal warps: thin-plate splines and the decomposition of deformations." IEEE Transactions on Pattern Analysis and Machine Intelligence. (1989) http://doi.org/10.1109/34.24792