Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Potential issue when multi-threading testing using virtuoso jdbc driver #1242

Open
suyZhong opened this issue Feb 7, 2024 · 8 comments
Open

Comments

@suyZhong
Copy link

suyZhong commented Feb 7, 2024

I am currently testing Virtuoso by building docker image following this: https://github.com/openlink/vos-reference-docker. Recently I frequently triggered the following crash in Virtuoso (I could reproduce on latest manual build from 604fdc7, openlink/virtuoso-opensource-7:latest, pkleef/virtuoso:latest):

06:22:35 Version 07.20.3238-pthreads for Linux as of Feb  7 2024 (604fdc7)
06:22:35 uses OpenSSL 1.1.1f  31 Mar 2020
06:22:35 uses parts of PCRE, Html Tidy
06:22:35 Database version 3126
06:22:35 SQL Optimizer enabled (max 1000 layouts)
06:22:36 Compiler unit is timed at 0.000109 msec
06:22:37 Roll forward started
06:22:37 Roll forward complete
06:22:37 Checkpoint started
06:22:37 Checkpoint finished, log reused
06:22:37 HTTP/WebDAV server online at 8890
06:22:37 Server online at 1111 (pid 1)
B')) BETWEEN CAST(database0_t0.c1 AS VARCHAR) AND database0_t1.c1) UNION ALL SELECT database0_t0.c0, database0_t0.c1 FROM database0_t0, database0_t1 WHERE (NOT (((06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x5b7dd9) [0x55f4b7e8fdd9]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x5b7e6d) [0x55f4b7e8fe6d]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x118b4e) [0x55f4b79f0b4e]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x11a76a) [0x55f4b79f276a]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x2a8228) [0x55f4b7b80228]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x2a864e) [0x55f4b7b8064e]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x26ebc6) [0x55f4b7b46bc6]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x26f81e) [0x55f4b7b4781e]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x26b9f7) [0x55f4b7b439f7]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x26bd97) [0x55f4b7b43d97]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x27318c) [0x55f4b7b4b18c]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x26b9f7) [0x55f4b7b439f7]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x26bc88) [0x55f4b7b43c88]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x2a701d) [0x55f4b7b7f01d]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x26b9f7) [0x55f4b7b439f7]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x276d1e) [0x55f4b7b4ed1e]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x278637) [0x55f4b7b50637]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0xf7d87) [0x55f4b79cfd87]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0xfea66) [0x55f4b79d6a66]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0xff059) [0x55f4b79d7059]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x270c96) [0x55f4b7b48c96]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x2714ae) [0x55f4b7b494ae]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x26b9f7) [0x55f4b7b439f7]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x277cd3) [0x55f4b7b4fcd3]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x280111) [0x55f4b7b58111]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x2804c7) [0x55f4b7b584c7]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x280614) [0x55f4b7b58614]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x5bc6a2) [0x55f4b7e946a2]
06:22:49 /opt/virtuoso-opensource/bin/virtuoso-t(+0x5c35be) [0x55f4b7e9b5be]
06:22:49 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f22a9391609]
06:22:49 /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f22a915f353]
06:22:49 GPF: gate.c:160 Zero DP in page_fault_map_sem
GPF: gate.c:160 Zero DP in page_fault_map_sem

Scenario:

Start testing with 2 threads (2 JDBC connections) by sending random SQL queries, after several seconds, close my testing program (The JDBC connection might not be closed properly).

Then I start testing with another 2 threads, and the crash happens. There might be some issues when handling the previous queries, as there's one strange string B')) BETWEEN CAST(database0_t0.c1 AS VARCHAR) AND database0_t1.c1) UNION ALL SELECT database0_t0.c0, database0_t0.c1 FROM database0_t0, database0_t1 WHERE (NOT ((( in the server log. Here's the full server log in case it's useful.

I think it might be regardless of what kind of SQL queries I used since I could not easily reproduced by using isql.

Kindly inform me if I should provide more information.

@TallTed
Copy link
Collaborator

TallTed commented Feb 7, 2024

As it sounds like this only happens when connecting via JDBC, please provide full version details for both the Virtuoso JBDC Driver and the JVM/JDK/JRE in use. If you notice that either of these is not current, please retest with a current version of both, and either confirm that results are the same or provide any different results you see.

It may also be necessary for us to know what Java tool you're using to make the JDBC connection and queries.

@TallTed
Copy link
Collaborator

TallTed commented Feb 7, 2024

One more thing — If possible, it would be helpful if you could also provide the complete JDBC connect string(s) in use during these tests.

@suyZhong
Copy link
Author

suyZhong commented Feb 8, 2024

Sure. Here's my environment:

> java --version
openjdk 17.0.8.1 2023-08-24
OpenJDK Runtime Environment (build 17.0.8.1+1-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 17.0.8.1+1-Ubuntu-0ubuntu122.04, mixed mode, sharing)

For other settings:

  • Virtuoso JDBC: virtjdbc4_3 version: 3.123
  • Connecting string: jdbc:virtuoso://localhost:10020/UID=dba/PWD=dba
  • Starting command: docker run --name virtuoso-test -p 10020:1111 -e DBA_PASSWORD=dba openlink/virtuoso-opensource-7:latest
  • Java package: java.sql

Messages from the client side:

virtuoso.jdbc4.VirtuosoException: Virtuoso Communications Link Failure (timeout) : Connection to the server lost
	at virtuoso.jdbc4.VirtuosoFuture.nextResult(VirtuosoFuture.java:63)
	at virtuoso.jdbc4.VirtuosoStatement.close_rs(VirtuosoStatement.java:206)
	at virtuoso.jdbc4.VirtuosoStatement.close(VirtuosoStatement.java:229)
        ...

This happened when I tried to execute the below code:

Connection conn = DriverManager.getConnection(JDBCString);
try (Statement s = conn.createStatement()) {
    s.execute(String.format("DROP TABLE %s_t%d", databaseName, i)); //  Connection to the server lost
} catch (SQLException e1) {

@timhaynesopenlink
Copy link
Collaborator

Hi,
Two things.
First, could you make it run trace_on('exec'); before starting your test and post the resultant log, if not too huge?
Second, is any more of the java code source available so we could reproduce it more readily here?
Thanks

@suyZhong
Copy link
Author

suyZhong commented Feb 9, 2024

@timhaynesopenlink Thanks for your instruction. Here's the full log of the server.

Generally, this testing process is to create some tables and send random queries using two threads. The crash would happen if I manually terminated (Ctrl+C the Java program) the first test run and at the time I started the second one. There might be some concurrency issues.

Sorry but the automated testing tool is still under development. I'll try to make a minimal program that could reproduce this issue. However, I don't think there's complex logic in the Java part (just making normal JDBC calls, but concurrently) and the last queries I could observe were usually simple and could be various.

@timhaynesopenlink
Copy link
Collaborator

Thanks for that - good to see the SQL queries as they happen. We'll hopefully have a look into it in the next few days.

@timhaynesopenlink
Copy link
Collaborator

Development are now looking into it. We'll keep you posted.

@timhaynesopenlink
Copy link
Collaborator

timhaynesopenlink commented Feb 13, 2024

Hi,
We have investigated. JDBC not required. The GPF occurs when looping the SQL queries in two (or more) threads. The problem comes when one thread is performing a DML query while the other is performing DDL hitting the same table - e.g., select * from [something that's just going away].

As a rule, mixing DML and DDL in this way is a Bad Idea. You should probably rethink the client application isolation boundaries - define a test block as a create+query*+drop sequence and then each thread performs a given test one at a time, thereby referencing only the tables pertinent to its current test.

I have raised a bug internally for tracking in case we can think of a satisfactory way to avoid this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants