Skip to content

cgnorthcutt/benchmarking-keras-pytorch

Repository files navigation

Benchmarking Keras and PyTorch Pre-Trained Models

Benchmarks for every pre-trained model in PyTorch and Keras-Tensorflow. Benchmarks are reproducible.

Why this is helpful

Combining Keras and PyTorch benchmarks into a single framework lets researchers decide which platform is best for a given model. For example resnet architectures perform better in PyTorch and inception architectures perform better in Keras (see below). These benchmarks serve as a standard from which to start new projects or debug current implementations.

For researchers exploring Keras and PyTorch models, these benchmarks serve as a standard from which to start new projects or debug current implementations.

Many researchers struggle with reproducible accuracy benchmarks of pre-trained Keras (Tensorflow) models on ImageNet. Examples of issues are here1, here2, here3, here4, and here5.

In Keras, the published benchmarks on Keras Applications cannot be reproduced by exactly copying the associated code. In fact, the reported accuracies are usually higher than the actual accuries.

I dive slightly deeper into the reproducibility issues of Keras in the associated blog post.

Benchmark Results on ImageNet

The actual validation set accuracies on ImageNet for all Keras and PyTorch models (verified on macOS 10.11.6, Linux Debian 9, and Ubuntu 18.04).

Platform Model Acc@1 Acc@5 Rank@1 Rank@5
Keras 2.2.4 nasnetlarge 80.83 95.27 1 1
Keras 2.2.4 inceptionresnetv2 78.93 94.45 2 2
PyTorch 1.0 resnet152 77.62 93.81 3 3
Keras 2.2.4 xception 77.18 93.49 4 4
PyTorch 1.0 densenet161 76.92 93.49 5 5
PyTorch 1.0 resnet101 76.64 93.30 6 6
PyTorch 1.0 densenet201 76.41 93.18 7 7
Keras 2.2.4 inceptionv3 76.02 92.90 8 8
PyTorch 1.0 densenet169 75.59 92.69 9 9
PyTorch 1.0 resnet50 75.06 92.48 10 10
Keras 2.2.4 densenet201 74.77 92.32 11 11
PyTorch 1.0 densenet121 74.07 91.94 12 12
Keras 2.2.4 densenet169 73.92 91.76 13 13
PyTorch 1.0 vgg19_bn 72.90 91.35 14 14
PyTorch 1.0 resnet34 72.34 90.84 15 16
PyTorch 1.0 vgg16_bn 72.29 91.01 16 15
Keras 2.2.4 densenet121 72.09 90.70 17 17
Keras 2.2.4 nasnetmobile 71.59 90.19 18 19
PyTorch 1.0 vgg19 71.19 90.40 19 18
PyTorch 1.0 vgg16 70.66 89.93 20 20
Keras 2.2.4 resnet50 70.35 89.55 21 22
PyTorch 1.0 vgg13_bn 70.12 89.56 22 21
Keras 2.2.4 mobilenetV2 69.98 89.49 23 23
PyTorch 1.0 vgg11_bn 69.36 89.06 24 24
PyTorch 1.0 inception_v3 69.25 88.69 25 25
Keras 2.2.4 mobilenet 69.02 88.48 26 27
PyTorch 1.0 vgg13 68.69 88.65 27 28
PyTorch 1.0 resnet18 68.37 88.56 28 26
PyTorch 1.0 vgg11 67.85 88.11 29 29
Keras 2.2.4 vgg19 65.58 86.54 30 30
Keras 2.2.4 vgg16 65.24 86.20 31 31
PyTorch 1.0 squeezenet1_0 56.49 79.06 32 33
PyTorch 1.0 squeezenet1_1 56.42 79.21 33 32
PyTorch 1.0 alexnet 54.50 77.64 34 34

To Reproduce

Get the ImageNet validation dataset

  • Download

  • Preprocess/Extract validation data

    • Once ILSVRC2012_img_val.tar is downloaded, run:
    # Credit to Soumith: https://github.com/soumith/imagenet-multiGPU.torch
    $ cd ../ && mkdir val && mv ILSVRC2012_img_val.tar val/ && cd val && tar -xvf ILSVRC2012_img_val.tar
    $ wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash

Reproduce in 10 seconds

The top 5 predictions for every example in the ImageNet validation set have been pre-computed for you here for Keras models and here for PyTorch models. These are automatically used by the following code which takes a few seconds to run:

$ git clone https://github.com:cgnorthcutt/imagenet-benchmarking.git
$ cd benchmarking-keras-pytorch
$ python imagenet_benchmarking.py /path/to/imagenet_val_data

Reproduce model outputs (hours)

You can also reproduce the inference-time output of each Keras and PyTorch model without using the pre-computed data. Inference for Keras takes a long time (5-10 hours) because I compute the forward pass on each example one at a time and avoid vectorized operations: this was the only approach I found would reliably reproduce the same accuracies. PyTorch is fairly quick (less than one hour). To reproduce:

$ git clone https://github.com:cgnorthcutt/imagenet-benchmarking.git
$ cd benchmarking-keras-pytorch
$ # Compute outputs of PyTorch models (1 hour)
$ ./imagenet_pytorch_get_predictions.py /path/to/imagenet_val_data
$ # Compute outputs of Keras models (5-10 hours)
$ ./imagenet_keras_get_predictions.py /path/to/imagenet_val_data
$ # View benchmark results
$ ./imagenet_benchmarking.py /path/to/imagenet_val_data

You can control GPU usage, batch size, output storage directories, and more. Run the files with the -h flag to see command line argument options.

Tips for Keras

One of the goals of this project is to help reconcile issues with reproducibility in Keras pre-trained models. The way I deal with these issues is three-fold. In Keras I

  1. avoid batches during inference.
  2. run each example one at a time. This is silly slow, but yields a reproducible output for every model.
  3. only run models in local functions or use with clauses to ensure no aspects of a previous model persist in memory when the next model is loaded.

Example use cases from Pervasive Label Errors Paper

# Compare standard benchmarking versus cleaned and corrected benchmarking
# Requires providing index of all test examples EXCEPT non-consensus errors (ambiguous or multi-class)
# Requires providing the labels of the remaining test examples, including the corrected labels of the honeypot.
for consensus in {3..5}; do python imagenet_benchmarking.py -p pytorch_imagenet -o imagenet_benchmarks/cleaned_and_corrected_$consensus.csv -j /home/cgn/cgn/label-errors/experiments/mturk_processing/imagenet_masks/cleaned_and_corrected_labels_idx_$consensus.npy -l /home/cgn/cgn/label-errors/experiments/mturk_processing/imagenet_masks/cleaned_and_corrected_labels_$consensus.npy /datasets/datasets/imagenet/val; done

# Honeypot benchmarking -- consensus corected labels versus original labels
for consensus in {3..5}; do python imagenet_benchmarking.py -p pytorch_imagenet -o imagenet_benchmarks/consensus$consensus.csv -j /home/cgn/cgn/label-errors/experiments/mturk_processing/imagenet_masks/mturk_actual_labels_idx_$consensus.npy -l /home/cgn/cgn/label-errors/experiments/mturk_processing/imagenet_masks/mturk_actual_labels_$consensus.npy /datasets/datasets/imagenet/val; done

# ANOTHER AN EXAMPLE -- THIS RESULT IS NOT IN THE PAPER
# Compare standard benchmarking versus cleaned benchmarking
for consensus in {3..5}; do python imagenet_benchmarking.py -p pytorch_imagenet -o imagenet_benchmarks/test_noise_removed_$consensus.csv -i /home/cgn/cgn/label-errors/experiments/mturk_processing/imagenet_masks/mturk_noise_indices_$consensus.npy /datasets/datasets/imagenet/val; done

License

Copyright (c) 2019 Curtis Northcutt. Released under the MIT License. See LICENSE for details.