Linux Swap memory fills

[mitchellscnih]

Hello, not sure if anyone has any thoughts on a problem we have seen since trying to migrate to Linux 8 servers. We have a cron script that runs every night and it runs the following procedure to upload ttl file data to Virtuoso:

"ttl")
      load_func="DB.DBA.TTLP_MT(file_to_string_output('$file_name_full'),'','$graph_uri', 255);

For some reason, it fills the swap memory immediately on Linux 8 servers during the cron job, but if I run the script manually during the day, it does not fill up or fills up minimally. I have tried different tests and the only thing I have seen that helps is basically doubling the resources (CPU/Memory), which is not a feasible fix as we have a lot of VMs that run this job. We are currently running this script on Linux 7 servers and there are no issues with swap.

[HughWilliams]

Can you please provide the following information:

• confirm the exact Linux 8 & 7 versions being used with the command cat /etc/os-release.

• The Virtuoso version being used on the Linux 8 and Linux 7 servers, with the command ./virtuoso-t -? run from the bin directory of your Virtuoso installation.

• The complete content of the cron job being run (minus any sensitive info), so we can see exactly how the DB.DBA.TTLP_MT() procedure is being call, which is not clear from the snippet provided.

• The output of running the status(); command run from the isql command line tool on the Linux 8 and Linux 7 servers, so we can see the status of both servers when.

• The output of running the free -m command on the Linux 8 and Linux 7 servers so we can see the memory and swap setup on both.

[mitchellscnih]

Hi Hugh,
Thanks for the quick response. Here is the information requested:

confirm the exact Linux 8 & 7 versions being used with the command cat /etc/os-release.

RHEL 7
Red Hat Enterprise Linux Server release 7.9 (Maipo)
RHEL 8
Red Hat Enterprise Linux release 8.9 (Ootpa)

The Virtuoso version being used on the Linux 8 and Linux 7 servers, with the command ./virtuoso-t -? run from the bin directory of your Virtuoso installation.

RHEL 7
Version 7.2.5.3229-pthreads as of Aug  2 2021
RHEL 8
Version 7.2.6.3233-pthreads as of Nov 28 2023

The complete content of the cron job being run (minus any sensitive info), so we can see exactly how the DB.DBA.TTLP_MT() procedure is being call, which is not clear from the snippet provided.

Here is the main snippet of code and essentially we have a process that generates a ttl file each night and then this script (what we call the upload script) just loads that rdf data into Virtuoso

code snippet

  file_name_local=${data_file##*/}
  file_name_full=$virtuoso_allowed_dir/$file_name_local

  # Log into Virtuoso isql env
  isql_cmd="/usr/local/virtuoso-opensource/bin/isql 1111 dba"
  isql_pwd=" "

  # Call Virtuoso loading functions for different input formats
  load_func=""
  case "$input_format" in
    "rdf")
      load_func="DB.DBA.RDF_LOAD_RDFXML_MT(file_to_string_output('$file_name_full'), '', '$graph_uri');"
    ;;
    "ttl")
      load_func="DB.DBA.TTLP_MT(file_to_string_output('$file_name_full'),'','$graph_uri', 255);"
    ;;
    "nt")
      load_func="DB.DBA.TTLP_MT(file_to_string_output('$file_name_full'),'','$graph_uri', 255);"
    ;;
    "nq")
      load_func="DB.DBA.TTLP_MT(file_to_string_output('$file_name_full'),'','$graph_uri', 512);"
    ;;
    *)
      echo "Input format unacceptable"
      echo
      echo "          Acceptable input formats"
      echo "          1) rdf -- RDF/XML"
      echo "          2) ttl -- Turtle/N3"
      echo "          3) nt -- n-triple"
      echo "          4) nq -- n-quad"
      exit 0
    ;;
  esac
  ${isql_cmd} ${isql_pwd} << EOF &> /opt/apache-tomcat-scheduler/rdf-exports/vload.log
    log_enable(3,1);
    SPARQL CLEAR GRAPH '$graph_uri';
    $load_func
    checkpoint;
    exit;

The output of running the status(); command run from the isql command line tool on the Linux 8 and Linux 7 servers, so we can see the status of both servers when.

Output from RHEL 8 where the swap was filled and the script still runs successfully every day

SQL> status();
REPORT
VARCHAR
_______________________________________________________________________________

OpenLink Virtuoso  Server
Version 07.20.3237-pthreads for Linux as of Nov 21 2023 (f3d88f16b)
Started on: 2023-11-25 10:30 GMT-5

Database Status:
  File size 249561088, 2127616 pages, 1429720 free.
  4000000 buffers, 536354 used, 3 dirty 0 wired down, repl age 65140098 0 w. io 0 w/crsr.
  Disk Usage: 515924 reads avg 0 msec, 0% r 0% w last  0 s, 5412161 writes flush      248.6 MB/s,
    4056 read ahead, batch = 109.  Autocompact 245974 in 139917 out, 43% saved col ac: 3764177 in 32% saved.
Gate:  8152 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap.
Log = /usr/local/virtuoso-opensource-7.2.10/var/lib/virtuoso/db/virtuoso.trx, 50219 bytes
677808 pages have been changed since last backup (in checkpoint state)
Current backup timestamp: 0x0000-0x00-0x00
Last backup date: unknown
Clients: 5403 connects, max 21 concurrent
RPC: 147915 calls, 1 pending, 2 max until now, 0 queued, 51 burst reads (0%), 0 second 0M large, 769M max
Checkpoint Remap 20000 pages, 0 mapped back. 46 s atomic time.
    DB master 2127616 total 1429720 free 20000 remap 1 mapped back
   temp  18688 total 18683 free

Lock Status: 0 deadlocks of which 0 2r1w, 0 waits,
   Currently 1 threads running 0 threads waiting 0 threads in vdb.
Pending:

Client 1111:5383:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5393:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5388:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5390:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5398:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5392:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5391:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5397:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5385:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5394:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5402:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5396:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5403:-5404:  Account: dba, 230 bytes in, 295 bytes out, 1 stmts.
PID: 1771006, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: PENDING, 1 threads.
Locks:

Client 1111:5400:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5386:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5389:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5387:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5401:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5395:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5384:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:5399:  Account: dba, 87 bytes in, 289 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:


Running Statements:
 Time (msec) Text
         124 status()


Hash indexes


117 Rows. -- 125 msec.

Output from RHEL 7 server where swap never fills

SQL> status();
REPORT
VARCHAR
_______________________________________________________________________________

OpenLink Virtuoso  Server
Version 07.20.3229-pthreads for Linux as of Aug  2 2021
Started on: 2023-12-06 06:59 GMT-5

Database Status:
  File size 3160408064, 385792 pages, 204870 free.
  4000000 buffers, 115858 used, 3 dirty 0 wired down, repl age 0 0 w. io 0 w/crsr.
  Disk Usage: 115957 reads avg 0 msec, 0% r 0% w last  0 s, 77349 writes flush      226.9 MB,
    1228 read ahead, batch = 72.  Autocompact 524 in 244 out, 53% saved col ac: 67542 in 23% saved.
Gate:  2580 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap.
Log = /usr/local/virtuoso-opensource-7.2.5.1/var/lib/virtuoso/db/virtuoso.trx, 12116 bytes
161766 pages have been changed since last backup (in checkpoint state)
Current backup timestamp: 0x0000-0x00-0x00
Last backup date: unknown
Clients: 62 connects, max 21 concurrent
RPC: 5452 calls, -40 pending, 2 max until now, 0 queued, 0 burst reads (0%), 0 second 0M large, 468M max
Checkpoint Remap 19095 pages, 0 mapped back. 0 s atomic time.
    DB master 385792 total 204870 free 19095 remap 0 mapped back
   temp  52736 total 52731 free

Lock Status: 0 deadlocks of which 0 2r1w, 0 waits,
   Currently 1 threads running 0 threads waiting 0 threads in vdb.
Pending:

Client 1111:55:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:54:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:58:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:42:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:57:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:49:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:56:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:44:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:45:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:59:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:48:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:62:-63:  Account: dba, 220 bytes in, 291 bytes out, 1 stmts.
PID: 93623, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: PENDING, 1 threads.
Locks:

Client 1111:61:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:53:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:43:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:60:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:51:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:46:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:50:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:52:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:

Client 1111:47:  Account: dba, 87 bytes in, 287 bytes out, 0 stmts.
Transaction status: PENDING, 0 threads.
Locks:


Running Statements:
 Time (msec) Text
          91 status()


Hash indexes


117 Rows. -- 92 msec.

The output of running the free -m command on the Linux 8 and Linux 7 servers so we can see the memory and swap setup on both.

RHEL 7
              total        used        free      shared  buff/cache   available
Mem:          64247       36604       14373           3       13269       27119
Swap:          4095          69        4026

RHEL 8
              total        used        free      shared  buff/cache   available
Mem:          63845       50028        7182           0        6633       13093
Swap:          4095        3852         243

[imitko]

@mitchellscnih

In order to drop redundant buffering of the file in a string output, you can directly use file handler session for RDF_LOAD_RDF_XML*() TTLP*(), so instead of file_to_string_output(), you can call file_open().

HTH

[mitchellscnih]

Thanks @imitko , i did change the call to file_open(). Did not help with the swap memory issue

[HughWilliams]

What is the kernel swappiness setting on both servers?

[mitchellscnih]

I have tried several swappiness settings (0, 10, and the default 60) and same swap problem. I did find this RedHat article on the cgroups fix, and the swap memory didn't fill all the way (4G), but filled more than usual (760M),

[pkleef]

In the article you mentioned, there is a reference to another article Change in swap behavior between RHEL 7 and RHEL 8 kernels - SOLUTION IN PROGRESS

Issue
Seeing swap usage even though swappiness is low and there is plenty of available memory in pagecache.
vm.swappiness is properly configured and the cgroup swappiness issue is addressed already

Resolution
Changes in virtual memory (VM) management between RHEL 7 and 8 may cause different patterns of swap usage even under similar workloads and tunings
Swap usage or a trend towards 100% swap used is not necessarily indicative of memory pressure.
Recommend monitoring other statistics in order to determine if the system is under memory pressure (see Diagnostic Steps)

which indicates that the problem you are seeing with swap is caused by differences in the way the RHEL7 and RHEL8 kernels function.

As you already tried set the cgroup fixes your article mentions, I can only conclude that RedHat is still working on a solution.

I suggest you keep an eye on these two articles and see if RedHat comes up with a kernel fix, or at least some more settings to try. If you have a RedHat support contract, I would definitely contact them to see if you can get more info on when a complete solution is expected.

At this point, I see no reason to assume that the Virtuoso binary is responsible for the swap usage behaviour.

[mitchellscnih]

Hey @pkleef yeah, i did see the solution in progress article which led me to the cgroup. As I read, the cgroup change should have potentially fixed the issue. I can say after doing this and the job ran last night, the swap did not fill but did go up higher than what would be expected limit. I am trying more tests but have to wait till the overnight process runs. Thanks for the response.