Linux vs Windows: different results

[marcingretl]

Describe the bug
While running the same experiment on Linux and Windows I'm getting different results.

To Reproduce
Steps to reproduce the behavior:

library("BAS")

prefix = .Platform$OS.type
sink(paste(prefix, "_logit_bas.txt", sep=""), append=FALSE, split=FALSE)

data("birthwt", package = "MASS")
y <- birthwt$lo
x <- data.frame(birthwt[,-1])
x <- x[,-9]
x$race <- as.factor(x$race)

Nrep = 10000
set.seed(1000000)
bas.time <- system.time(birthwt.bas.glm <- bas.glm(y ~ ., data=x,
family=binomial(link = "logit"),
method="MCMC",
MCMC.iterations=Nrep,
laplace=TRUE))
print("bas.glm results for birthwt:")
print(bas.time)
print(birthwt.bas.glm$modelprior)
summary(birthwt.bas.glm)
coef(birthwt.bas.glm)

Expected behavior
Posterior results should be identical.

Additional context
[Linux]
[1] "bas.glm results for birthwt:"
użytkownik system upłynęło
0.160 0.017 0.197
$family
[1] "Beta-Binomial"

$hyper.parameters
[1] 1 1

attr(,"class")
[1] "prior"
P(B != 0 | Y) model 1 model 2 model 3 model 4
Intercept 1.0000 1.000000 1.0000000 1.0000000 1.0000
age 0.1422 0.000000 0.0000000 0.0000000 0.0000
lwt 0.4241 0.000000 0.0000000 1.0000000 1.0000
race2 0.2564 0.000000 0.0000000 0.0000000 0.0000
race3 0.1971 0.000000 0.0000000 0.0000000 0.0000
smoke 0.2999 0.000000 0.0000000 0.0000000 0.0000
ptl 0.3185 0.000000 1.0000000 0.0000000 0.0000
ht 0.4026 0.000000 0.0000000 0.0000000 1.0000
ui 0.2511 0.000000 0.0000000 0.0000000 0.0000
ftv 0.0841 0.000000 0.0000000 0.0000000 0.0000
BF NA 0.482511 0.4451593 0.2986569 1.0000
PostProbs NA 0.342100 0.0417000 0.0334000 0.0322
R2 NA 0.000000 0.0289000 0.0255000 0.0577
dim NA 1.000000 2.0000000 2.0000000 3.0000
logmarg NA -118.268722 -118.3492936 -118.7484304 -117.5400
model 5
Intercept 1.0000000
age 0.0000000
lwt 0.0000000
race2 0.0000000
race3 0.0000000
smoke 0.0000000
ptl 0.0000000
ht 0.0000000
ui 1.0000000
ftv 0.0000000
BF 0.1898685
PostProbs 0.0235000
R2 0.0216000
dim 2.0000000
logmarg -119.2013938

Marginal Posterior Summaries of Coefficients:

Using BMA

Based on the top 346 models
post mean post SD post p(B != 0)
Intercept -0.1204457 1.1600878 1.0000000
age -0.0052729 0.0187699 0.1422000
lwt -0.0067728 0.0091024 0.4241000
race2 0.2642035 0.5250595 0.2564000
race3 0.1501464 0.3643195 0.1971000
smoke 0.2373405 0.4224629 0.2999000
ptl 0.2135715 0.3678763 0.3185000
ht 0.6881534 0.9508727 0.4026000
ui 0.2097175 0.4255995 0.2511000
ftv -0.0008163 0.0509175 0.0841000

[Windows]
[1] "bas.glm results for birthwt:"
użytkownik system upłynęło
0.42 0.00 0.42
$family
[1] "Beta-Binomial"

$hyper.parameters
[1] 1 1

attr(,"class")
[1] "prior"
P(B != 0 | Y) model 1 model 2 model 3 model 4
Intercept 1.0000 1.000000 1.0000000 1.0000 1.0000000
age 0.1164 0.000000 0.0000000 0.0000 0.0000000
lwt 0.3539 0.000000 0.0000000 1.0000 1.0000000
race2 0.2104 0.000000 0.0000000 0.0000 0.0000000
race3 0.1875 0.000000 0.0000000 0.0000 0.0000000
smoke 0.2750 0.000000 0.0000000 0.0000 0.0000000
ptl 0.2575 0.000000 1.0000000 0.0000 0.0000000
ht 0.3326 0.000000 0.0000000 1.0000 0.0000000
ui 0.2133 0.000000 0.0000000 0.0000 0.0000000
ftv 0.0741 0.000000 0.0000000 0.0000 0.0000000
BF NA 0.482511 0.4451593 1.0000 0.2986569
PostProbs NA 0.444100 0.0376000 0.0321 0.0260000
R2 NA 0.000000 0.0289000 0.0577 0.0255000
dim NA 1.000000 2.0000000 3.0000 2.0000000
logmarg NA -118.268722 -118.3492936 -117.5400 -118.7484304
model 5
Intercept 1.0000000
age 0.0000000
lwt 0.0000000
race2 0.0000000
race3 0.0000000
smoke 0.0000000
ptl 0.0000000
ht 0.0000000
ui 1.0000000
ftv 0.0000000
BF 0.1898685
PostProbs 0.0173000
R2 0.0216000
dim 2.0000000
logmarg -119.2013938

Marginal Posterior Summaries of Coefficients:

Using BMA

Based on the top 361 models
post mean post SD post p(B != 0)
Intercept -0.2581245 1.0803783 1.0000000
age -0.0042545 0.0169919 0.1164000
lwt -0.0056017 0.0086101 0.3539000
race2 0.2212985 0.4945867 0.2104000
race3 0.1454458 0.3610647 0.1875000
smoke 0.2216885 0.4157675 0.2750000
ptl 0.1707630 0.3389293 0.2575000
ht 0.5669318 0.9006299 0.3326000
ui 0.1768643 0.3972297 0.2133000
ftv -0.0002828 0.0477916 0.0741000

[merliseclyde]

Sounds like this could be different random number generators leading to different results as the number of models sampled differs in the two runs despite the same seed. See https://stackoverflow.com/questions/48626086/same-seed-different-os-different-random-numbers-in-r#:~:text=Linux%20gives%20a%20different%20random,results%20will%20agree%20or%20not) and one solution that suggests

set.seed(10, kind = "Mersenne-Twister", normal.kind = "Inversion"); rnorm(1)

to see if you get the same random numbers. Note the top model is the same in both (Null) and has the same marginal likelihood, so it may be a matter of which models are being sampled.

[merliseclyde]

Interestingly I get 312 unique models when running on MAC OSX R 4.3.2.

Do you get different results if you enumerate all models (use method="deterministic") for sampling.?

[marcingretl]

Using method="deterministic" produces exactly the same results, so it must be an issue related with pseudo-random sequence.

[marcingretl]

Well, this is what I got typing '> set.seed(10, kind = "Mersenne-Twister", normal.kind = "Inversion"); rnorm(10)' on Linux machine (Debian, R-4.3.2 from distro repos) and Windows machine (10 Pro, R-4.3.3):
[Linux]
[1] 0.01874617 -0.18425254 -1.37133055 -0.59916772 0.29454513 0.38979430
[7] -1.20807618 -0.36367602 -1.62667268 -0.25647839

[Windows]
[1] 0.01874617 -0.18425254 -1.37133055 -0.59916772 0.29454513 0.38979430
[7] -1.20807618 -0.36367602 -1.62667268 -0.25647839

As you can see sequences are identical, so maybe an issue is BAS-related?