Skip to content

Tabularised feature comparison between Catch, doctest and lest C++ test frameworks

License

Notifications You must be signed in to change notification settings

martinmoene/catch-lest-other-comparison

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

85 Commits
 
 
 
 

Repository files navigation

Catch–doctest–lest feature comparison

Tabularised feature comparison between Catch, doctest and lest C++ test frameworks.

Skip to the feature comparison.

Note: This is an initial draft, it is incomplete and likely contains errors.

Ideas, additions, corrections, signaling omissions, etc. welcome!


The frameworks in this comparison share the following properties:

  • The frameworks are available as single-file header-only library.
  • Test cases are described by a string, they are not identified by a function name.
  • There is a single assertion macro for expressions.
    Expressions are decomposed and original expression code can be shown alongside the values they represent.
  • Fixtures can be created inside a test case. They are called sections or sub cases.

Advent of Catch Expression-decomposing assertions and function-level fixtures (sections) were introduced by Kevlin Henney in his talk Rethinking Unit Testing in C++ at ACCU London in May 2010 [1]. At that time, Phil Nash was working on a testing framework called YACUTS (Yet Another C++ Unit Test System) that would bring similar capabilities, be it a little more intrusively [2]. Kevlin's talk inspired Phil to develop Catch [3].

Kevlin's emphasis on simplicity in testing and the desire to understand how expression decomposition works inspired Martin Moene to write lest [4]. The code of lest is intentionally kept small, so that it should be fairly easy to gain insight in how things work.

Accompanying test code with the code to test was one of the drivers for Viktor Kirilov to write doctest [5]. Others are to enable using doctest with many different compilers and to shine in compile-time and run-time performance. At CppCon 2017 Viktor gave a presentation that explains many techniques used in doctest and Catch [23].

 

Feature doctest Catch-1 Catch-2 lest Notes
License MIT BSL-1.0 BSL-1.0 BSL-1.0  
           
C++ standard related          
C++ standard C++98 C++98 C++11 C++11  
Can work without exceptions - - - reduced functionality
Requires RTTI - - - -  
           
Fields of use          
Desktop  
Embedded (tiny) ? ? ? ?  
Embedded (largish) ? ? ?  
Application, tool ? ?  
Library, framework ?  
Operating System ? ? ? lest: IncludeOS
           
Documentation of framework         see [22]
First contact ? ? ? ? new users
Education ? ? ? ? new & existing users
Support ? ? ? ? experienced users
Troubleshooting ? ? ? ? annoyed users
Internals ? ? ? ? fellow developers
Reference ? ? ? ? everyone
Try it online on wandbox
Forum - - Catch: Google group
Chat - - - doctest: Gitter  
StackOverflow tag - - see [19]
           
Test organisation          
Suites of tests - - -  
Fixtures (sections) doctest: sub case
Fixtures (class-based) -  
Type-parameterised tests -  
Value-parameterised tests - - - -  
Auto-registration of tests  
Tabularised tests - - - array of lambdas
Allow tests accompany code - Catch: not recommended...
Allow tests in header files - ...to avoid long compilation times
           
Assertions          
BDD style scenarios  
Assert expressions  
Assert exceptions  
Assert abortion (death) - - - -  
Assert assertions (death) - - - -  
Expression decomposition see [2]
Floating point comparison, approx  
Floating point comparison, ULP - - - see stf [10]
Hamcrest matchers (built-in) - some some see [12][13][14][15]
Hamcrest matchers (user-definable) - -  
Macros with and without prefix CATCH_CHECK(), CHECK()
           
Other test facilities          
Mocking support via 3rd party
Logging facility -  
Logging levels - - - -  
Checkpoints - - - - see Boost.Test [11]
Test decorators - - - see [16]
Test data generators - - - Cach-1: undocumented
Property-based testing - - - -  
Obtain name of current test ? ? - Catch-1: undocumented
           
API / seams          
Reporting user-defined types  
Reporter API - -  
Event listeners - -  
Run-time context -  
           
Reporting formats          
Console, multi-line -  
Console, single-line (compact) - similar to compiler error
JUnit - - result at end
XML - - streaming
TeamCity - -  
TAP - -  
Automake - -  
           
Reporting options          
Colourised output (run-time)    
Colourised output (compile-time) - -  
Literal suffix u, l, f - - - compile-time
           
Test execution          
Concurrent execution of tests - - - -  
Isolated execution of tests - - - -  
Limit test execution time - - - doctest: via test decorator
Signal tests without assertions - -  
Supports floating point exceptions - - - -  
Supports Posix signals -  
Supports Windows SEH -  
           
Compile-time control          
Improve compilation speed - reduced functionality
Omit macros without prefix  
Omit std::wstring - - -  
Omit std::cout, cerr, clog - -  
Specify terminal color system - none, ansi, win32
Specify terminal width - -  
Control handling of POSIX signals -  
Control handling of SE on Windows -  
Enable leak checking on Windows - - uses CRT Debug Heap
Enable use of __COUNTER__ - - for internal unique names
           
Run-time control, commandline          
Help screen  
List selected suites - - -  
List selected tests  
List tags of selected tests -  
Select tests by suite - - -  
Select tests by name  
Select tests by tag  
Select tests by file - Catch: via filename as tags
Select section by name -  
Select supports regexp - -  
Omit assertions expected to throw - -  
Count selected tests - -  
Abort at N-th failure lest: N is 1
Break into debugger -  
Report passing tests  
Time duration of tests  
Control order of tests  
Repeat tests -  
           
IDE integration         see [17]
Output-window-compatible reports click through to source
C++ Builder ? ? ? ?  
CLion -  
Code::Blocks ? ? ? ?  
CodeLite ? ? ? ?  
Eclipse CDT ? ? ? ?  
KDevelop ? ? ? ?  
Qt Creator ? ? ?  
Visual-MinGW ? ? ? ?  
Visual Studio - Catch and doctest: via ReSharper C++
Visual Studio Code - - - - see issue
           
Platforms          
AIX / XL ? 11.1.0 ? ? Catch-1: 2013
macOS / clang ? ? ? ?  
macOS / gcc 4.4-6 ? ? ? source: Travis
macOS / XCode / clang 6.4-8.3 7.3-8 7.3-8 ? source: Travis
macOS / XCode / gcc 6.4-7.3 7.3-8 7.3-8 ? source: Travis
Linux / clang 3.4-4 3.4-3.8 3.5-3.8 3.6-3.7 source: Travis
Linux / gcc 4.4-6 4.4-6 4.8-6 4.9-5 source: Travis
Solaris / SunPro ? 5.12.0 ? ? Catch-1: 2013
Windows / clang ? ? ? 3.2+  
Windows / gcc 7 ? ? 4.8.1+  
Windows / Visual C++ 9-15 10-15 14-15 12-15 Catch-1: possibly from 8
Windows / Cygwin / clang ? ? ? ?  
Windows / Cygwin / gcc ? ? ? ?  
Windows / msys2 / clang ? ? ? ?  
Windows / msys2 / gcc ? ? ? ?  
           
Quality of project         note: challenge 42
Code of conduct - -  
Follows CII best practices partly partly partly see [21]
           
Quality of code          
Unit testing details below
CI testing details below
CI code analysis - - - details below
           
Testing of framework          
Unit tests  
Tests itself  
Baseline tests -  
           
CI testing          
Travis  
Appveyor  
Coverage scan - -  
Coverity scan - - -  
           
CI static code analysis          
clang-tidy - - -  
CppCheck - - -  
CppCoreCheck - - - -  
PVS-Studio - - - -  
OCLint - - -  
ReSharper C++ - - - -  
Visual Studio - - -  
           
Code-related          
Compile-time performance, relative ☆☆☆ ☆☆ ? ? see [20]
Run-time performance, relative ☆☆☆ ☆☆ ? ? see [20]
Compiler-warning friendly ☆☆☆ ☆☆ ☆☆ ☆☆  
Compiles with -Werror -Wall -Wextra  
Can provide canned main() -  
Only depends on C++ std library  
Single-file header-only  
Size, LOC 4,500 9,100 8,800 1,000 see [18]
           
Distribution of framework          
GitHub single-file download from the landing page
conan -  
hunter ? - -  
vcpkg -  
Linux and other (search) search search search on Repology
           
Feature doctest Catch-1 Catch-2 lest Notes

References

[1] Kevlin Henney. Rethinking Unit Testing in C++ (Video). Skills Matter, May 2010.
[2] Phil Nash. The Ultimate C++ Unit Test Framework. Blog, May 2010.
[3] Phil Nash. Unit Testing in C++ and Objective-C just got easier. Blog announcement, December 2010.
[4] Martin Moene. lest errors escape testing. Blog announcement, June 2013.
[5] Viktor Kirilov. doctest – the Lightest C++ Unit Testing Framework. Announcement in Overload 137, February 2017.
[6] Phil Nash. Catch-1 on GitHub. 2010.
[7] Phil Nash. Catch-2 on GitHub. 2017.
[8] Viktor Kirilov. doctest on GitHub. 2014.
[9] Martin Moene. lest on GitHub. 2013.
[10] Joel Falcou. Design rationale for using ULP with stf.
[11] Gennadiy Rozental and Raffi Enficiaud. Checkpoints in Boost.Test.
[12] Matchers in Catch-1.
[13] Matchers in Catch-2.
[14] Matchers in lest: via hamlest.
[15] Hamcrest 1.3 Quick Reference (PDF).
[16] Decorators in doctest.
[17] Wikipedia. List of C++ IDEs.
[18] CodeBlocks IDE. Non-empty, non-comment lines, via File > Properties.
[19] StackOverflow. Questions tagged catch-unit-test.
[20] Viktor Kirilov. doctest Benchmarks. May 2017.
[21] Linux Foundation (LF). CII Best Practices.
[22] Jacob Kaplan-Moss. Writing great documentation.
[23] Viktor Kirilov. Mix Tests and Production Code With Doctest - Implementing and Using the Fastest Modern C++ Testing Framework (slides). CppCon 2017.

About

Tabularised feature comparison between Catch, doctest and lest C++ test frameworks

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published