Do you think dask would be a suitable tool for this?

@MKastek also mentioned concurrent.futures.

@namurphy I have little experience with either, but this is a simple parallelization problem so I'm sure many libraries could work. I've used multiprocessing recently, so I know that could work.

I'll give this a try

I think dask would be the superior option since it allows both local and distributed computation with a high level API

@JaydenR2305 great, thanks for taking a look at it! As you do, keep in mind also issue #1725: whatever parallel solution we will also eventually want to extend to working with multiple grids. Something to think about when choosing an implementation.

Hello, I've been looking into this issue, and I would like to give it a try as my first contribution.

@Sobeskes — awesome!

@JaydenR2305 — were you still planning to work on this?

Many thanks!

@Sobeskes awesome! Let me know if I can be of assistance.

I believe @JaydenR2305 is working on some other projects right now, so you're welcome to take a crack at this if you'd like.

Note that there are currently a few other PRs working on this code:

#1274 Is almost complete and won't change the Tracker class, although we are moving some code around the package so the location of that function will change

#1799 builds on Main and is almost complete

#1802 builds on #1799 and will requires more work + a full review before it will be done. This PR does make some improvements to some of the internal workings of Tracker, as well as adding new functionality.

I would recommend playing with the parallelization on a fork of #1802: that way you're building on the upgrades to Tracker that I've put in there.

Hm, this issue has been harder to tackle than I thought! I made a few implementations using different parallelization libraries (threading, concurrent.futures, etc) and was unable to manage to get anything that performed better than the serial implementation. The serial implementation ran the test suite ran in around 30 seconds, but the best performing parallelized implementation I created using the concurrent.futures library took around 70 seconds to run. It might be that the overhead associated with creating different threads is too high, or that I'm not understanding the problem correctly. It seems like the most potential for parallelization occurs with the interpolation of Ex, Bx, etc.

@Sobeskes, thank you for looking into this! That's really helpful information. Did dask happen to be one of the packages that you tried?

@Sobeskes interesting - at what level are you parallelizing? I would suggest cutting the list of particles into sub-lists (one per core) at the very beginning and then running them entirely separately on different cores from then on, only recombining in the detector plane.

I intentionally chose the test problems to be fast, so they many not accurately represent the gains from parallelization on a realistic run. It only takes 30 seconds because it's running several dozen of them: each individual run is probably close to a second. Try something with a 200x200x200 grid and 1E6 particles and see if you see a difference then: that should take like 20-30 minutes on the serial version.

I would definitely recommend trying this with dask.array as well!