Open3 doesn't write on the right STDOUT

[n-rodriguez]

Hi there!

It seems that there is an issue with Open3 and JRuby :

#!/usr/bin/env ruby

require "open3"

def capture3_args_for(cmd, dir)
  ["git", *cmd]
end

cmd = 'version'
dir = nil

puts *capture3_args_for(cmd, dir).inspect
out, err, status = Open3.capture3(*capture3_args_for(cmd, dir))

puts "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@"
puts "out.inspect: #{out.inspect}"
puts "err.inspect: #{err.inspect}"
puts "status.inspect: #{status.inspect}"
puts "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@"

With Ruby 3.3.1 :

nicolas@MacBook-Pro-de-Nicolas:~/PROJECTS/CONCERTO/concerto$ ./toto.rb
["git", "version"]
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
out.inspect: "git version 2.45.0\n" # GOOD
err.inspect: ""
status.inspect: #<Process::Status: pid 13555 exit 0>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

With JRuby 9.4.7.0 :

nicolas@MacBook-Pro-de-Nicolas:~/PROJECTS/CONCERTO/concerto$ ./toto.rb
["git", "version"]
git version 2.45.0 # BAD
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
out.inspect: "" # BAD
err.inspect: ""
status.inspect: #<Process::Status: pid 13749 exit 0>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Extracted from : rubygems/rubygems#7651

Thank you!

[ntkme]

I saw that you're on JDK 21 from the rubygems thread. Can you please try JDK 17 instead?

There is a known bug in JDK 21 that's affecting subprocess stdio for JRuby in many different ways: #8069

[n-rodriguez]

I saw that you're on JDK 21 from the rubygems thread. Can you please try JDK 17 instead?

Hi there! Thanks for your answer!

It doesn't work neither with JDK17 :

#!/usr/bin/env ruby

require "open3"
require 'java'

java_import java.lang.System

def capture3_args_for(cmd, dir)
  ["git", *cmd]
end

cmd = 'version'
dir = nil

puts %x(java --version)
puts System.getProperties["java.runtime.version"]
puts JRUBY_VERSION
puts *capture3_args_for(cmd, dir).inspect
out, err, status = Open3.capture3(*capture3_args_for(cmd, dir))

puts "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@"
puts "out.inspect: #{out.inspect}"
puts "err.inspect: #{err.inspect}"
puts "status.inspect: #{status.inspect}"
puts "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@"

nicolas@MacBook-Pro-de-Nicolas:~/PROJECTS/CONCERTO/concerto$ ./foo.rb
openjdk 17.0.11 2024-04-16
OpenJDK Runtime Environment Homebrew (build 17.0.11+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.11+0, mixed mode, sharing)
17.0.11+0
9.4.7.0
["git", "version"]
git version 2.45.1
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
out.inspect: ""
err.inspect: ""
status.inspect: #<Process::Status: pid 39796 exit 0>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

[n-rodriguez]

Strangely it works with TruffleRuby :

nicolas@MacBook-Pro-de-Nicolas:~/PROJECTS/CONCERTO/concerto$ ./foo.rb
setrlimit to increase file descriptor limit failed, errno 22
openjdk 17.0.11 2024-04-16
OpenJDK Runtime Environment Homebrew (build 17.0.11+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.11+0, mixed mode, sharing)
3.2.2
["git", "version"]
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
out.inspect: "git version 2.45.1\n"
err.inspect: ""
status.inspect: #<Process::Status: pid 40111 exit 0>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

[headius]

Okay, it's probably a bug in our version of open3.

[ntkme]

I tested the following:

• OpenJDK Runtime Environment Homebrew (build 11.0.23+0) works
• OpenJDK Runtime Environment Homebrew (build 17.0.11+0) broken
• OpenJDK Runtime Environment Homebrew (build 21.0.3) broken
• OpenJDK Runtime Environment Temurin-17.0.11+9 (build 17.0.11+9) works
• OpenJDK Runtime Environment Temurin-21.0.3+9 (build 21.0.3+9-LTS) works

[ntkme]

Looks like this is an issue with Homebrew JDK...

[headius]

I tested the following

Ok I really did not expect that! Of the linked libraries, libSystem seems like the most likely candidate but I have no idea what could cause such a visible change.

I think a next step would be to build the same version manually on MacOS and test it.

It seems less likely to be something JRuby is doing wrong but we will keep this open until we have an explanation.

[headius]

If a hand-build version of the homebrew sources works ok, I think we would want to file a bug with them. There may have been a problem introduced when their build was created.

[ntkme]

This has to do with how $JAVA_HOME is set:

$ export JAVA_HOME=/opt/homebrew/opt/java
$ jruby test.rb                          
openjdk 21.0.3 2024-04-16
OpenJDK Runtime Environment Homebrew (build 21.0.3)
OpenJDK 64-Bit Server VM Homebrew (build 21.0.3, mixed mode, sharing)
21.0.3
9.4.7.0
["git", "version"]
git version 2.39.3 (Apple Git-145)
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
out.inspect: ""
err.inspect: ""
status.inspect: #<Process::Status: pid 30098 exit 0>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

$ export JAVA_HOME=/opt/homebrew/opt/java/libexec/openjdk.jdk/Contents/Home 
$ jruby test.rb                                                            
openjdk 21.0.3 2024-04-16
OpenJDK Runtime Environment Homebrew (build 21.0.3)
OpenJDK 64-Bit Server VM Homebrew (build 21.0.3, mixed mode, sharing)
21.0.3
9.4.7.0
["git", "version"]
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
out.inspect: "git version 2.39.3 (Apple Git-145)\n"
err.inspect: ""
status.inspect: #<Process::Status: pid 29884 exit 0>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

[n-rodriguez]

I confirm :

nicolas@MacBook-Pro-de-Nicolas:~/PROJECTS/CONCERTO/concerto$ echo $JAVA_HOME
/opt/homebrew/opt/openjdk@17
nicolas@MacBook-Pro-de-Nicolas:~/PROJECTS/CONCERTO/concerto$ export  JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
nicolas@MacBook-Pro-de-Nicolas:~/PROJECTS/CONCERTO/concerto$ jruby foo.rb
openjdk 17.0.11 2024-04-16
OpenJDK Runtime Environment Homebrew (build 17.0.11+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.11+0, mixed mode, sharing)
17.0.11+0
9.4.7.0
["git", "version"]
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
out.inspect: "git version 2.45.1\n"
err.inspect: ""
status.inspect: #<Process::Status: pid 14828 exit 0>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

[headius]

Could one of you paste the output of jruby --environment with the bad JAVA_HOME? It could still be something wrong with homebrew for OpenJDK, but it is possible our startup script is calculating the JDK path incorrectly.

[headius]

Oh I see there is a PR to fix this now!

[ntkme]

@headius The issue is that in homebrew they created symlinks for java binaries in order to have executables available at the path/to/Cellar/bin. For example:

/opt/homebrew/opt/openjdk/bin/java is a symlink to /opt/homebrew/opt/openjdk/libexec/openjdk.jdk/Contents/Home/bin/java, however /opt/homebrew/opt/openjdk does not have the same directory structure and content as /opt/homebrew/opt/openjdk/libexec/openjdk.jdk/Contents/Home, thus cannot be used as $JAVA_HOME.

If the user incorrectly set the $JAVA_HOME, the $JAVACMD is set to the symlink that can start Jruby, but it does not work properly.

I’ve sent a PR to handle this kind of case where user passes an incorrect $JAVA_HOME containing $JAVA_HOME/bin/java: #8242

[headius]

@ntkme Thank you!