Re: Summary checkpoint in Joss review

[kazewong]

The authors claim

1. pocoMC can speed up the sampling procedure by orders of magnitude when compared to alternatives such as nested sampling.
2. Parallelisation to computing clusters manifests linear scaling.

Regarding 1., although I do expect a speed-up coming from the reparametrization using normalizing flow, it would be better to see a demonstration of the speed-up to some degree. I know wall time demonstration is a bit annoying due to many other factors in optimizing a program, so a reduction in the total number of calls of the likelihood + benchmarking of normalizing flow would be apperciated.

Regarding 2., I don't see this parallelisation benchmark anywhere in the doc. Can the author provide a small example to show the scaling relation to some moderate number of cores (e.g. 32-128)?

#openjournals/joss-reviews#4634

[minaskar]

Thank you for your comments.

Regarding the first comment, performance claims were the subject of the accompanying paper. In that paper, we quote the number of likelihood calls for a number of artificial and realistic posterior targets. For the artificial examples, the number of calls is a good metric of the speed-up as the flow training time is negligible. However, we can provide the walltime or each example and method (PMC/SMC/NS) if you deem it necessary.

As for the second comment, we can provide a toy example in which the likelihood calculation is artificially delayed (e.g. time.sleep(...)) and show the speed-up for a varying number of cores.

[kazewong]

@minaskar I wasn't aware of the companion paper since it is "In prep" in the draft, but I did find it on arXiv now, so I am happy with the number of call reductions stated in that paper.

For the second comment, I am not exactly sure why you would need a delayed likelihood to show this. I think it is sufficient to take either the Gaussian mixture or the Rosenbrock example, run them on a varying number of cores, record the wall time and plot the wall time against the number of cores.

By the way, this is a comment related to the accompanying paper, so this shouldn't affect the JOSS review process:
In the gravitational wave example, it would be nice to show the configuration of your run. The reason why I am asking is the RA and DEC distribution seems pretty unimodal, which is usually the case only the signal is generated with at least 3 detectors operating. There are other features in the posterior that may raise suspicion, adding the configuration you used to generate that will at least help the readers understanding whether certain features are expected or not.

[minaskar]

Hi @kazewong and apologies for the delayed response.

The main reason that I think I need to use a "delayed" likelihood is that the Gaussian mixture or the Rosenbrock example likelihoods are very fast. Both of them are vectorised, but even if they weren't the computational communication overhead (order of 10ms) introduced by multiprocessing or MPI would dominate the total computational cost. This means that in cases in which the likelihood is cheap the parallel version will be slower than the serial. However, as the PMC algorithm (or even the vanilla SMC) is "embarrassingly parallel" scaling to many CPUs is linear assuming that the cost of a single likelihood evaluation is greater than ~10ms. We included a short paragraph on this in the Discussion section of the accompanying paper.

Many thanks for the suggestion regarding the accompanying paper, we will certainly look into this.