BUG: fixed race condition setting m_ANTSAssociate #4625

seanm · 2024-04-29T21:14:24Z

Found with thread sanitizer (TSan) running the itkANTSNeighborhoodCorrelationImageToImageRegistrationTest unit test.

There was already a m_ANTSAssociateOnceFlag flag for use with std::call_once, so I just used that in this place too (it's already used elsewhere).

Partial backtrace from TSan:

WARNING: ThreadSanitizer: data race (pid=79176)
  Write of size 8 at 0x000108f02240 by thread T2:
    #0 void itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution_impl<itk::ThreadedIndexedContainerPartitioner>(itk::IdentityHelper<itk::ThreadedIndexedContainerPartitioner>, itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.hxx:118 (ITKMetricsv4TestDriver:arm64+0x1004bfe50)
    #1 itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution(itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.h:235 (ITKMetricsv4TestDriver:arm64+0x1004bf9c0)
    #2 itk::DomainThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreaderCallback(void*) itkDomainThreader.hxx:123 (ITKMetricsv4TestDriver:arm64+0x100259b6c)
    #3 std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()::operator()() const itkThreadPool.h:92 (ITKMetricsv4TestDriver:arm64+0x1007d3228)
    #4 decltype(std::declval<void* (*&)(void*)>()(std::declval<itk::PoolMultiThreader::ThreadPoolInfoStruct*>())) std::__1::__invoke[abi:v160006]<std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()&>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&) invoke.h:394 (ITKMetricsv4TestDriver:arm64+0x1007d31a4)

  Previous write of size 8 at 0x000108f02240 by thread T14:
    #0 void itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution_impl<itk::ThreadedIndexedContainerPartitioner>(itk::IdentityHelper<itk::ThreadedIndexedContainerPartitioner>, itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.hxx:118 (ITKMetricsv4TestDriver:arm64+0x1004bfe50)
    #1 itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution(itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.h:235 (ITKMetricsv4TestDriver:arm64+0x1004bf9c0)
    #2 itk::DomainThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreaderCallback(void*) itkDomainThreader.hxx:123 (ITKMetricsv4TestDriver:arm64+0x100259b6c)
    #3 std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()::operator()() const itkThreadPool.h:92 (ITKMetricsv4TestDriver:arm64+0x1007d3228)
    #4 decltype(std::declval<void* (*&)(void*)>()(std::declval<itk::PoolMultiThreader::ThreadPoolInfoStruct*>())) std::__1::__invoke[abi:v160006]<std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()&>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&) invoke.h:394 (ITKMetricsv4TestDriver:arm64+0x1007d31a4)

PR Checklist

No API changes were made (or the changes have been approved)
No major design changes were made (or the changes have been approved)
Added test (or behavior not changed)
Updated API documentation (or API not changed)
Added license to new files (if any)
Added Python wrapping to new files (if any) as described in ITK Software Guide Section 9.5
Added ITK examples for all new major features (if any)

Refer to the ITK Software Guide for
further development details if necessary.

seanm · 2024-04-29T22:34:50Z

@dzenanz @hjmjohnson @N-Dekker I don't know this code at all, but I noticed the existence of m_ANTSAssociateOnceFlag already, so this fix seems good to me.

BTW I've created a nightly cdash submission than uses Thread Sanitizer: https://open.cdash.org/viewTest.php?onlyfailed&buildid=9578838

I will triage the other test failures...

N-Dekker · 2024-04-30T09:32:17Z

.../include/itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.hxx

-  /* call base method */
-  /* Store the casted pointer to avoid dynamic casting in tight loops. */
-  this->m_ANTSAssociate = dynamic_cast<TNeighborhoodCorrelationMetric *>(this->m_Associate);
-  if (this->m_ANTSAssociate == nullptr)
-  {
-    itkExceptionMacro("Dynamic casting of associate pointer failed.");
-  }
+  std::call_once(this->m_ANTSAssociateOnceFlag, [this]() {
+    /* Store the casted pointer to avoid dynamic casting in tight loops. */
+    this->m_ANTSAssociate = dynamic_cast<TNeighborhoodCorrelationMetric *>(this->m_Associate);
+    if (this->m_ANTSAssociate == nullptr)
+    {
+      itkExceptionMacro("Dynamic casting of associate pointer failed.");
+    }
+  });


Nice that you found this potential race condition. 👍 I'm sorry I don't know the code either, and I find it hard to understand. 🤷 At

ITK/Modules/Registration/Metricsv4/include/itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.hxx

Line 42 in 13ccff6

std::call_once(this->m_ANTSAssociateOnceFlag, [this, &associate]() { this->m_ANTSAssociate = associate; });

The m_ANTSAssociateOnceFlag apparently ensures that m_ANTSAssociate = associate is only executed once during the lifetime of this member variable. Which means that it cannot be reset to a different "associate" anymore, during its lifetime. Is that OK?

It appears that m_Associate is a protected member of DomainThreader. It is assigned in DomainThreader::Execute:

ITK/Modules/Core/Common/include/itkDomainThreader.hxx

Lines 54 to 57 in f408b77

void

DomainThreader<TDomainPartitioner, TAssociate>::Execute(TAssociate * enclosingClass, const DomainType & completeDomain)

{

this->m_Associate = enclosingClass;

So then I guess the question is: could this "threader" be executed multiple times for different "associates"?

It seems to me that a mutex (member variable)+ lock_guard variables (locally) would be more appropriate than call_once + once_flag, in this case. But again, I don't know the code either.

But then again, it seems a bit "overdone" to have a separate data member (m_ANTSAssociate), just to store the result of dynamic_cast<TNeighborhoodCorrelationMetric *>(Superclass::m_Associate), and then try to keep this data member up-to-date and thread-safe.

I mean: apparently the (original) code tries to avoid expensive dynamic_cast's. But extra member data and mutex locking isn't for free either.

The this->m_ANTSAssociate should be change to just a function level variable with the results of the dynamic cast? This seems like it may have been an "optimization" which is not worth the complexity, with some unexpected cost?

Looks like this was done by @songgang back in 2012 in 651dcd0. Are you still around @songgang ?

@seanm Looks like m_ANTSAssociate was actually introduced one month before songgang's commit already! Commit 9614409 authored by Michael Stauffer and committed by Hans (@hjmjohnson) has:

ITK/Modules/Registration/Metricsv4/include/itkANTSNeighborhoodCorrelationImageToImageMetricv4DenseGetValueAndDerivativeThreader.h

Lines 135 to 137 in 9614409

/** Internal pointer to the metric object in use by this threader.

* This will avoid costly dynamic casting in tight loops. */

TNeighborhoodCorrelationMetric * m_ANTSAssociate;

And its commit text says:

Remove calls to dynamic_cast in image metric threaders. Replace with cached pointer to metric object. Speed gain of several %.

Now I wonder if that speed gain will still be there when thread safety is taken into account 🤷 . Moreover, new compilers may yield different performance results anyway!

I've looked a little more...

The string Store the casted pointer to avoid dynamic casting in tight loops appears 7 times, so seems, at least at one time, people thought it was a useful optimization (or got copy-pasted :))

So then I guess the question is: could this "threader" be executed multiple times for different "associates"?

I think the answer is no. The string m_Associate "only" appears 108 times, and almost always it's read, not written. I can't find a setter for the ivar in fact. Only set is in DomainThreader<TDomainPartitioner, TAssociate>::Execute.

seanm · 2024-05-23T17:14:15Z

Anyone have further thoughts on this one?

Found with thread sanitizer (TSan) running the `itkANTSNeighborhoodCorrelationImageToImageRegistrationTest` unit test. There was already a `m_ANTSAssociateOnceFlag` flag for use with `std::call_once`, so I just used that in this place too (it's already used elsewhere). Partial backtrace from TSan: ``` WARNING: ThreadSanitizer: data race (pid=79176) Write of size 8 at 0x000108f02240 by thread T2: #0 void itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution_impl<itk::ThreadedIndexedContainerPartitioner>(itk::IdentityHelper<itk::ThreadedIndexedContainerPartitioner>, itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.hxx:118 (ITKMetricsv4TestDriver:arm64+0x1004bfe50) #1 itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution(itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.h:235 (ITKMetricsv4TestDriver:arm64+0x1004bf9c0) #2 itk::DomainThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreaderCallback(void*) itkDomainThreader.hxx:123 (ITKMetricsv4TestDriver:arm64+0x100259b6c) #3 std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()::operator()() const itkThreadPool.h:92 (ITKMetricsv4TestDriver:arm64+0x1007d3228) #4 decltype(std::declval<void* (*&)(void*)>()(std::declval<itk::PoolMultiThreader::ThreadPoolInfoStruct*>())) std::__1::__invoke[abi:v160006]<std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()&>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&) invoke.h:394 (ITKMetricsv4TestDriver:arm64+0x1007d31a4) Previous write of size 8 at 0x000108f02240 by thread T14: #0 void itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution_impl<itk::ThreadedIndexedContainerPartitioner>(itk::IdentityHelper<itk::ThreadedIndexedContainerPartitioner>, itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.hxx:118 (ITKMetricsv4TestDriver:arm64+0x1004bfe50) #1 itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution(itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.h:235 (ITKMetricsv4TestDriver:arm64+0x1004bf9c0) #2 itk::DomainThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreaderCallback(void*) itkDomainThreader.hxx:123 (ITKMetricsv4TestDriver:arm64+0x100259b6c) #3 std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()::operator()() const itkThreadPool.h:92 (ITKMetricsv4TestDriver:arm64+0x1007d3228) #4 decltype(std::declval<void* (*&)(void*)>()(std::declval<itk::PoolMultiThreader::ThreadPoolInfoStruct*>())) std::__1::__invoke[abi:v160006]<std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()&>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&) invoke.h:394 (ITKMetricsv4TestDriver:arm64+0x1007d31a4) ```

github-actions bot added type:Bug Inconsistencies or issues which will cause an incorrect result under some or all circumstances area:Registration Issues affecting the Registration module labels Apr 29, 2024

dzenanz approved these changes Apr 29, 2024

View reviewed changes

N-Dekker reviewed Apr 30, 2024

View reviewed changes

seanm force-pushed the race-call-once branch from 46eed47 to c48500b Compare May 30, 2024 13:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: fixed race condition setting m_ANTSAssociate #4625

BUG: fixed race condition setting m_ANTSAssociate #4625

seanm commented Apr 29, 2024

seanm commented Apr 29, 2024

N-Dekker Apr 30, 2024 •

edited

N-Dekker Apr 30, 2024

N-Dekker Apr 30, 2024 •

edited

blowekamp Apr 30, 2024

seanm Apr 30, 2024

N-Dekker Apr 30, 2024

seanm May 6, 2024

seanm commented May 23, 2024

	void
	DomainThreader<TDomainPartitioner, TAssociate>::Execute(TAssociate * enclosingClass, const DomainType & completeDomain)
	{
	this->m_Associate = enclosingClass;

	/** Internal pointer to the metric object in use by this threader.
	* This will avoid costly dynamic casting in tight loops. */
	TNeighborhoodCorrelationMetric * m_ANTSAssociate;

BUG: fixed race condition setting m_ANTSAssociate #4625

Are you sure you want to change the base?

BUG: fixed race condition setting m_ANTSAssociate #4625

Conversation

seanm commented Apr 29, 2024

PR Checklist

seanm commented Apr 29, 2024

N-Dekker Apr 30, 2024 • edited

Choose a reason for hiding this comment

N-Dekker Apr 30, 2024

Choose a reason for hiding this comment

N-Dekker Apr 30, 2024 • edited

Choose a reason for hiding this comment

blowekamp Apr 30, 2024

Choose a reason for hiding this comment

seanm Apr 30, 2024

Choose a reason for hiding this comment

N-Dekker Apr 30, 2024

Choose a reason for hiding this comment

seanm May 6, 2024

Choose a reason for hiding this comment

seanm commented May 23, 2024

N-Dekker Apr 30, 2024 •

edited

N-Dekker Apr 30, 2024 •

edited