user containers hang without any output #6340

martinpitt · 2024-05-06T12:35:23Z

Downstream report: https://issues.redhat.com/browse/RHEL-34650

Downstream report: https://issues.redhat.com/browse/RHEL-34650 Known issue cockpit-project#6340

cockpituous · 2024-05-06T13:29:35Z

centos-10
Ooops, it happened again

# ----------------------------------------------------------------------
# testMemory (__main__.TestCurrentMetrics.testMemory)
Created symlink /etc/systemd/system/packagekit.service → /dev/null.
/usr/libexec/packagekitd: no process found
[0506/132703.697363:WARNING:sandbox_linux.cc(420)] InitializeSandbox() called with multiple threads in process gpu-process.
[0506/132703.716162:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.

DevTools listening on ws://127.0.0.1:9470/devtools/browser/166a051d-2618-4024-8391-5ca388e4c0f9
[0506/132704.405901:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1715002025555.619,"url":"http://127.0.0.2:9391/cockpit/@localhost/*/po.js","networkRequestId":"9029.20"}
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1715002025559.982,"url":"http://127.0.0.2:9391/cockpit/@localhost/*/po.manifest.js","networkRequestId":"9029.19"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9391/cockpit/@localhost/*/po.manifest.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1715002025616.203,"url":"http://127.0.0.2:9391/metrics"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9391/cockpit/@localhost/*/po.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1715002025616.325,"url":"http://127.0.0.2:9391/metrics"}
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:4!5"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
> warning: cockpit.format_{bytes,bits}[_per_sec](..., 1000, undefined) is deprecated.
> log: PackageKit went away from D-Bus
> warning: Resolving coreutils failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: mem-hog.service; invocation ID: 4f5a2528b0c849d6afe4dfdfbf93268a
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:4!20"}
Getting image source signatures
Copying blob sha256:6b83872188a9e8912bee1d43add5e9bc518601b02a14a364c0da43b0d59acf33
Copying blob sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed
Copying config sha256:56a9b5a744a674481a5c034fec45ebf470a58ab844cd0da4d26b89bd27df5e36
Writing manifest to image destination
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 1064, in testMemory
    admin_s.execute(f"""
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on '
                podman run --rm -d --name user-mem-hog localhost/test-busybox /bin/sh -c '
                head -c 300m /dev/zero | tail | sleep infinity'
            '

Wrote screenshot to TestCurrentMetrics-testMemory-centos-10-127.0.0.2-2501-FAIL.png
Wrote HTML dump to TestCurrentMetrics-testMemory-centos-10-127.0.0.2-2501-FAIL.html
Wrote JS log to TestCurrentMetrics-testMemory-centos-10-127.0.0.2-2501-FAIL.js.log
Journal extracted to TestCurrentMetrics-testMemory-centos-10-127.0.0.2-2501-FAIL.log.gz
Removed "/etc/systemd/system/packagekit.service".
> warning: transport closed: disconnected
umount: /run/user/1000: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 1064, in testMemory
    admin_s.execute(f"""
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on '
                podman run --rm -d --name user-mem-hog localhost/test-busybox /bin/sh -c '
                head -c 300m /dev/zero | tail | sleep infinity'
            '

# Result testMemory (__main__.TestCurrentMetrics.testMemory) failed
# 1 TEST FAILED [154s on 255c45970bf9]
not ok 26 test/verify/check-metrics TestCurrentMetrics.testMemory [ND@3]

First occurrence: 2024-05-06T13:29:35.508775+00:00 | revision eff0ba2
Times recorded: 1
Latest occurrences:

2024-05-06T13:29:35.508775+00:00 | revision eff0ba2

cockpituous · 2024-05-06T13:29:55Z

centos-10
Ooops, it happened again

# ----------------------------------------------------------------------
# testCPU (__main__.TestCurrentMetrics.testCPU)
Created symlink /etc/systemd/system/packagekit.service → /dev/null.
dnf: no process found
[0522/091220.723938:WARNING:sandbox_linux.cc(420)] InitializeSandbox() called with multiple threads in process gpu-process.

DevTools listening on ws://127.0.0.1:10149/devtools/browser/28011e1b-2af4-4224-89e0-8ed2a7d1a399
[0522/091220.748849:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
[0522/091221.401259:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716369143002.632,"url":"http://127.0.0.2:9891/cockpit/@localhost/*/po.js","networkRequestId":"8504.20"}
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716369143005.741,"url":"http://127.0.0.2:9891/cockpit/@localhost/*/po.manifest.js","networkRequestId":"8504.19"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9891/cockpit/@localhost/*/po.manifest.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716369143052.897,"url":"http://127.0.0.2:9891/metrics"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9891/cockpit/@localhost/*/po.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716369143056.597,"url":"http://127.0.0.2:9891/metrics"}
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:3!5"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
> warning: cockpit.format_{bytes,bits}[_per_sec](..., 1000, undefined) is deprecated.
> log: PackageKit went away from D-Bus
> log: PackageKit went away from D-Bus
> warning: Resolving coreutils failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: cpu-hog.service; invocation ID: 1e5e0e4cd9b4483f8f3b8efac16ebd60
Running as unit: cpu-piglet.service; invocation ID: 9a4e390272d94b98b663baf7f3e0eb4b
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
Running as unit: load-hog.service; invocation ID: 0ea8745df1eb4a3b8bc0306fe6fb8588
Getting image source signatures
Copying blob sha256:6b83872188a9e8912bee1d43add5e9bc518601b02a14a364c0da43b0d59acf33
Copying blob sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed
Copying config sha256:56a9b5a744a674481a5c034fec45ebf470a58ab844cd0da4d26b89bd27df5e36
Writing manifest to image destination
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

Wrote screenshot to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.png
Wrote HTML dump to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.html
Wrote JS log to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.js.log
Journal extracted to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.log.gz
> warning: transport closed: disconnected
Removed "/etc/systemd/system/packagekit.service".
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

# Result testCPU (__main__.TestCurrentMetrics.testCPU) failed
# 1 TEST FAILED [184s on 99bfdffab578]
not ok 24 test/verify/check-metrics TestCurrentMetrics.testCPU [ND@7]

First occurrence: 2024-05-22T09:15:22.239261+00:00 | revision e9b5147
Times recorded: 1
Latest occurrences:

2024-05-22T09:15:22.239261+00:00 | revision e9b5147

# ----------------------------------------------------------------------
# testCPU (__main__.TestCurrentMetrics.testCPU)
Created symlink /etc/systemd/system/packagekit.service → /dev/null.
dnf: no process found
[0522/105241.608771:WARNING:sandbox_linux.cc(420)] InitializeSandbox() called with multiple threads in process gpu-process.

DevTools listening on ws://127.0.0.1:9826/devtools/browser/b4385462-9197-44ff-9c5a-a0b36f826a67
[0522/105241.640716:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
[0522/105242.296689:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716375163838.986,"url":"http://127.0.0.2:9891/cockpit/@localhost/*/po.manifest.js","networkRequestId":"8674.19"}
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716375163843.293,"url":"http://127.0.0.2:9891/cockpit/@localhost/*/po.js","networkRequestId":"8674.20"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9891/cockpit/@localhost/*/po.manifest.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716375163890.668,"url":"http://127.0.0.2:9891/metrics"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9891/cockpit/@localhost/*/po.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716375163890.788,"url":"http://127.0.0.2:9891/metrics"}
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:3!5"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
> warning: cockpit.format_{bytes,bits}[_per_sec](..., 1000, undefined) is deprecated.
> log: PackageKit went away from D-Bus
> log: PackageKit went away from D-Bus
> warning: Resolving coreutils failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: cpu-hog.service; invocation ID: 0467110ddbcb4d7b958b5e16d0851f2c
Running as unit: cpu-piglet.service; invocation ID: b7435f461b8049549f4c9cb8fc4f31b7
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
Running as unit: load-hog.service; invocation ID: a92397e3873c43098e92930cbe03e8f0
Getting image source signatures
Copying blob sha256:6b83872188a9e8912bee1d43add5e9bc518601b02a14a364c0da43b0d59acf33
Copying blob sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed
Copying config sha256:56a9b5a744a674481a5c034fec45ebf470a58ab844cd0da4d26b89bd27df5e36
Writing manifest to image destination
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

Wrote screenshot to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.png
Wrote HTML dump to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.html
Wrote JS log to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.js.log
Journal extracted to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.log.gz
> warning: transport closed: disconnected
Removed "/etc/systemd/system/packagekit.service".
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

# Result testCPU (__main__.TestCurrentMetrics.testCPU) failed
# 1 TEST FAILED [168s on 59c116b1fc40]
not ok 24 test/verify/check-metrics TestCurrentMetrics.testCPU [ND@7]

First occurrence: 2024-05-22T10:55:27.046381+00:00 | revision 5c699e95a49cb357360b0e80f74cb581410863aa
Times recorded: 1
Latest occurrences:

2024-05-22T10:55:27.046381+00:00 | revision 5c699e95a49cb357360b0e80f74cb581410863aa

# ----------------------------------------------------------------------
# testCPU (__main__.TestCurrentMetrics.testCPU)
Created symlink /etc/systemd/system/packagekit.service → /dev/null.
/usr/libexec/packagekitd: no process found
[0523/103000.703991:WARNING:sandbox_linux.cc(420)] InitializeSandbox() called with multiple threads in process gpu-process.

DevTools listening on ws://127.0.0.1:9858/devtools/browser/8f294278-30b2-4651-983a-57865ba74900
[0523/103000.721023:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
[0523/103001.405978:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716460203076.639,"url":"http://127.0.0.2:9291/cockpit/@localhost/*/po.js","networkRequestId":"13929.20"}
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716460203080.562,"url":"http://127.0.0.2:9291/cockpit/@localhost/*/po.manifest.js","networkRequestId":"13929.19"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9291/cockpit/@localhost/*/po.manifest.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716460203123.562,"url":"http://127.0.0.2:9291/metrics"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9291/cockpit/@localhost/*/po.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716460203123.716,"url":"http://127.0.0.2:9291/metrics"}
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:3!5"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
> warning: cockpit.format_{bytes,bits}[_per_sec](..., 1000, undefined) is deprecated.
> log: PackageKit went away from D-Bus
> log: PackageKit went away from D-Bus
> warning: Resolving coreutils failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: cpu-hog.service; invocation ID: a720010e0fa54f14a3d4a3ea3b09ba24
Running as unit: cpu-piglet.service; invocation ID: 5fb749b223294f8680ad1f488d89d1aa
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
Running as unit: load-hog.service; invocation ID: 34eae145ac75428c864a6dc29433c95a
Getting image source signatures
Copying blob sha256:6b83872188a9e8912bee1d43add5e9bc518601b02a14a364c0da43b0d59acf33
Copying blob sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed
Copying config sha256:56a9b5a744a674481a5c034fec45ebf470a58ab844cd0da4d26b89bd27df5e36
Writing manifest to image destination
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

Wrote screenshot to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.png
Wrote HTML dump to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.html
Wrote JS log to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.js.log
Journal extracted to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.log.gz
> warning: transport closed: disconnected
Removed "/etc/systemd/system/packagekit.service".
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

# Result testCPU (__main__.TestCurrentMetrics.testCPU) failed
# 1 TEST FAILED [169s on ac0f094babad]
not ok 24 test/verify/check-metrics TestCurrentMetrics.testCPU [ND@2]

First occurrence: 2024-05-23T10:32:47.000070+00:00 | revision 874585ab3067deb08c1a957ecda6609453012370
Times recorded: 1
Latest occurrences:

2024-05-23T10:32:47.000070+00:00 | revision 874585ab3067deb08c1a957ecda6609453012370

# ----------------------------------------------------------------------
# testCPU (__main__.TestCurrentMetrics.testCPU)
Created symlink /etc/systemd/system/packagekit.service → /dev/null.
dnf: no process found
[0523/110815.985540:WARNING:sandbox_linux.cc(420)] InitializeSandbox() called with multiple threads in process gpu-process.

DevTools listening on ws://127.0.0.1:10015/devtools/browser/2b0c6f35-c47d-48e9-8621-f3136a0290f8
[0523/110816.012777:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
[0523/110816.675539:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716462498367.378,"url":"http://127.0.0.2:9891/cockpit/@localhost/*/po.manifest.js","networkRequestId":"8359.19"}
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716462498368.54,"url":"http://127.0.0.2:9891/cockpit/@localhost/*/po.js","networkRequestId":"8359.20"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9891/cockpit/@localhost/*/po.manifest.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716462498411.757,"url":"http://127.0.0.2:9891/metrics"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9891/cockpit/@localhost/*/po.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716462498411.876,"url":"http://127.0.0.2:9891/metrics"}
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:3!5"}
> warning: cockpit.format_{bytes,bits}[_per_sec](..., 1000, undefined) is deprecated.
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
> log: PackageKit went away from D-Bus
> warning: Resolving coreutils failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: cpu-hog.service; invocation ID: 12a1e964c799438f9ed0674680faa8f5
Running as unit: cpu-piglet.service; invocation ID: 76e2c8b579c5485fbed407a19f8714eb
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: load-hog.service; invocation ID: 8d8c9518f3e44e46b0d469071ca5e62b
Getting image source signatures
Copying blob sha256:6b83872188a9e8912bee1d43add5e9bc518601b02a14a364c0da43b0d59acf33
Copying blob sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed
Copying config sha256:56a9b5a744a674481a5c034fec45ebf470a58ab844cd0da4d26b89bd27df5e36
Writing manifest to image destination
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

Wrote screenshot to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.png
Wrote HTML dump to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.html
Wrote JS log to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.js.log
Journal extracted to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-3001-FAIL.log.gz
> warning: transport closed: disconnected
Removed "/etc/systemd/system/packagekit.service".
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

# Result testCPU (__main__.TestCurrentMetrics.testCPU) failed
# 1 TEST FAILED [180s on 2594938fa168]
not ok 24 test/verify/check-metrics TestCurrentMetrics.testCPU [ND@7]

First occurrence: 2024-05-23T11:11:12.772906+00:00 | revision 1ce58138eeefc1f6721632be650f60c82e7e00d4
Times recorded: 1
Latest occurrences:

2024-05-23T11:11:12.772906+00:00 | revision 1ce58138eeefc1f6721632be650f60c82e7e00d4

# ----------------------------------------------------------------------
# testCPU (__main__.TestCurrentMetrics.testCPU)
Created symlink /etc/systemd/system/packagekit.service → /dev/null.
/usr/libexec/packagekitd: no process found
[0523/122934.839368:WARNING:sandbox_linux.cc(420)] InitializeSandbox() called with multiple threads in process gpu-process.

DevTools listening on ws://127.0.0.1:10109/devtools/browser/ec4d5610-a2c7-4d22-ac1b-207717795535
[0523/122934.862102:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
[0523/122935.506219:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716467377079.872,"url":"http://127.0.0.2:9291/cockpit/@localhost/*/po.js","networkRequestId":"13794.20"}
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716467377083.058,"url":"http://127.0.0.2:9291/cockpit/@localhost/*/po.manifest.js","networkRequestId":"13794.19"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9291/cockpit/@localhost/*/po.manifest.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716467377116.423,"url":"http://127.0.0.2:9291/metrics"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9291/cockpit/@localhost/*/po.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716467377116.528,"url":"http://127.0.0.2:9291/metrics"}
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:4!5"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
> warning: cockpit.format_{bytes,bits}[_per_sec](..., 1000, undefined) is deprecated.
> log: PackageKit went away from D-Bus
> warning: Resolving coreutils failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: cpu-hog.service; invocation ID: c2f9a2974c9b4788b22c05af6f851c4d
Running as unit: cpu-piglet.service; invocation ID: 0f0927f624614ab99c818e05a8437dd7
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
Running as unit: load-hog.service; invocation ID: 6de88ebe31674b1c82cbfc33d2ab58a4
Getting image source signatures
Copying blob sha256:6b83872188a9e8912bee1d43add5e9bc518601b02a14a364c0da43b0d59acf33
Copying blob sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed
Copying config sha256:56a9b5a744a674481a5c034fec45ebf470a58ab844cd0da4d26b89bd27df5e36
Writing manifest to image destination
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

Wrote screenshot to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.png
Wrote HTML dump to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.html
Wrote JS log to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.js.log
Journal extracted to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.log.gz
> warning: transport closed: disconnected
Removed "/etc/systemd/system/packagekit.service".
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

# Result testCPU (__main__.TestCurrentMetrics.testCPU) failed
# 1 TEST FAILED [173s on 7db54124bfe9]
not ok 24 test/verify/check-metrics TestCurrentMetrics.testCPU [ND@2]

First occurrence: 2024-05-23T12:32:25.692237+00:00 | revision b2384a35c8c2ae25b0ea01a1670cd76241ecf455
Times recorded: 1
Latest occurrences:

2024-05-23T12:32:25.692237+00:00 | revision b2384a35c8c2ae25b0ea01a1670cd76241ecf455

# ----------------------------------------------------------------------
# testCPU (__main__.TestCurrentMetrics.testCPU)
Created symlink /etc/systemd/system/packagekit.service → /dev/null.
/usr/libexec/packagekitd: no process found
[0523/131924.594230:WARNING:sandbox_linux.cc(420)] InitializeSandbox() called with multiple threads in process gpu-process.
[0523/131924.617522:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.

DevTools listening on ws://127.0.0.1:9930/devtools/browser/02067552-3d5c-4ca6-8db1-a8aacc540ca2
[0523/131925.313653:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716470366757.079,"url":"http://127.0.0.2:9291/cockpit/@localhost/*/po.manifest.js","networkRequestId":"13873.19"}
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716470366758.238,"url":"http://127.0.0.2:9291/cockpit/@localhost/*/po.js","networkRequestId":"13873.20"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9291/cockpit/@localhost/*/po.manifest.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716470366795.256,"url":"http://127.0.0.2:9291/metrics"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9291/cockpit/@localhost/*/po.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716470366795.375,"url":"http://127.0.0.2:9291/metrics"}
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:4!5"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
Running as unit: cpu-hog.service; invocation ID: cd56a99fe691456180f4b7088b2c9d8c
Running as unit: cpu-piglet.service; invocation ID: 1b72828a213e45b081bbe32870cb9c9d
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
Running as unit: load-hog.service; invocation ID: 763374131f46471798e2352ecc9c2b9c
Getting image source signatures
Copying blob sha256:6b83872188a9e8912bee1d43add5e9bc518601b02a14a364c0da43b0d59acf33
Copying blob sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed
Copying config sha256:56a9b5a744a674481a5c034fec45ebf470a58ab844cd0da4d26b89bd27df5e36
Writing manifest to image destination
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

Wrote screenshot to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.png
Wrote HTML dump to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.html
Wrote JS log to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.js.log
Journal extracted to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.log.gz
> warning: transport closed: disconnected
Removed "/etc/systemd/system/packagekit.service".
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

# Result testCPU (__main__.TestCurrentMetrics.testCPU) failed
# 1 TEST FAILED [168s on ef1932c135de]
not ok 24 test/verify/check-metrics TestCurrentMetrics.testCPU [ND@2]

First occurrence: 2024-05-23T13:22:09.833915+00:00 | revision 2654a55b1df2edff90f1feed3e44e2b7f8bfb94c
Times recorded: 1
Latest occurrences:

2024-05-23T13:22:09.833915+00:00 | revision 2654a55b1df2edff90f1feed3e44e2b7f8bfb94c

# ----------------------------------------------------------------------
# testCPU (__main__.TestCurrentMetrics.testCPU)
Created symlink /etc/systemd/system/packagekit.service → /dev/null.
dnf: no process found
[0523/135407.401125:WARNING:sandbox_linux.cc(420)] InitializeSandbox() called with multiple threads in process gpu-process.

DevTools listening on ws://127.0.0.1:9334/devtools/browser/e3fcdffa-4e91-4320-b1d4-e7e129013ed5
[0523/135407.428403:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
[0523/135408.088417:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716472450113.24,"url":"http://127.0.0.2:9791/cockpit/@localhost/*/po.manifest.js","networkRequestId":"8557.19"}
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716472450116.395,"url":"http://127.0.0.2:9791/cockpit/@localhost/*/po.js","networkRequestId":"8557.20"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9791/cockpit/@localhost/*/po.manifest.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716472450176.373,"url":"http://127.0.0.2:9791/metrics"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9791/cockpit/@localhost/*/po.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716472450176.502,"url":"http://127.0.0.2:9791/metrics"}
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:4!5"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
> warning: cockpit.format_{bytes,bits}[_per_sec](..., 1000, undefined) is deprecated.
> log: PackageKit went away from D-Bus
> log: PackageKit went away from D-Bus
> warning: Resolving coreutils failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: cpu-hog.service; invocation ID: c17b8309f8f24286b5f7b545711fe4d9
Running as unit: cpu-piglet.service; invocation ID: 848ed4ef1c6d470fa867408833e542a6
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: load-hog.service; invocation ID: b1d5a97996454b529825670444610ade
Getting image source signatures
Copying blob sha256:6b83872188a9e8912bee1d43add5e9bc518601b02a14a364c0da43b0d59acf33
Copying blob sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed
Copying config sha256:56a9b5a744a674481a5c034fec45ebf470a58ab844cd0da4d26b89bd27df5e36
Writing manifest to image destination
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

Wrote screenshot to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2901-FAIL.png
Wrote HTML dump to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2901-FAIL.html
Wrote JS log to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2901-FAIL.js.log
Journal extracted to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2901-FAIL.log.gz
> warning: transport closed: disconnected
Removed "/etc/systemd/system/packagekit.service".
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

# Result testCPU (__main__.TestCurrentMetrics.testCPU) failed
# 1 TEST FAILED [176s on 1e538764cd2e]
not ok 24 test/verify/check-metrics TestCurrentMetrics.testCPU [ND@6]

First occurrence: 2024-05-23T13:57:00.083487+00:00 | revision 3d900192a3b8589c411cee21c21c7996397637f4
Times recorded: 1
Latest occurrences:

2024-05-23T13:57:00.083487+00:00 | revision 3d900192a3b8589c411cee21c21c7996397637f4

# ----------------------------------------------------------------------
# testCPU (__main__.TestCurrentMetrics.testCPU)
Created symlink /etc/systemd/system/packagekit.service → /dev/null.
/usr/libexec/packagekitd: no process found
[0523/135521.816916:WARNING:sandbox_linux.cc(420)] InitializeSandbox() called with multiple threads in process gpu-process.

DevTools listening on ws://127.0.0.1:10004/devtools/browser/cdd24316-9aa8-45d5-92d4-fcda65a0a3f9
[0523/135521.836002:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
[0523/135522.502518:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716472524231.576,"url":"http://127.0.0.2:9291/cockpit/@localhost/*/po.manifest.js","networkRequestId":"13929.19"}
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716472524235.021,"url":"http://127.0.0.2:9291/cockpit/@localhost/*/po.js","networkRequestId":"13929.20"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9291/cockpit/@localhost/*/po.manifest.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716472524286.441,"url":"http://127.0.0.2:9291/metrics"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9291/cockpit/@localhost/*/po.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716472524286.544,"url":"http://127.0.0.2:9291/metrics"}
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:3!5"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
> log: PackageKit went away from D-Bus
> warning: Resolving coreutils failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: cpu-hog.service; invocation ID: 1366d6b05778411595b7f4c879004748
Running as unit: cpu-piglet.service; invocation ID: ab3a66580d294d70b24027fc740148c1
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
Running as unit: load-hog.service; invocation ID: 1011f836114344f4a663b87e234d674e
Getting image source signatures
Copying blob sha256:6b83872188a9e8912bee1d43add5e9bc518601b02a14a364c0da43b0d59acf33
Copying blob sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed
Copying config sha256:56a9b5a744a674481a5c034fec45ebf470a58ab844cd0da4d26b89bd27df5e36
Writing manifest to image destination
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

Wrote screenshot to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.png
Wrote HTML dump to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.html
Wrote JS log to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.js.log
Journal extracted to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.log.gz
> warning: transport closed: disconnected
umount: /run/user/1000: target is busy.
Removed "/etc/systemd/system/packagekit.service".
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

# Result testCPU (__main__.TestCurrentMetrics.testCPU) failed
# 1 TEST FAILED [169s on eb8c24f1c01f]
not ok 24 test/verify/check-metrics TestCurrentMetrics.testCPU [ND@2]

First occurrence: 2024-05-23T13:58:07.729988+00:00 | revision 2654a55b1df2edff90f1feed3e44e2b7f8bfb94c
Times recorded: 1
Latest occurrences:

2024-05-23T13:58:07.729988+00:00 | revision 2654a55b1df2edff90f1feed3e44e2b7f8bfb94c

# ----------------------------------------------------------------------
# testCPU (__main__.TestCurrentMetrics.testCPU)
Created symlink /etc/systemd/system/packagekit.service → /dev/null.
/usr/libexec/packagekitd: no process found
[0523/141748.187260:WARNING:sandbox_linux.cc(420)] InitializeSandbox() called with multiple threads in process gpu-process.

DevTools listening on ws://127.0.0.1:9955/devtools/browser/7e50b517-f1c0-497d-936d-7c08dce31bba
[0523/141748.229987:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
[0523/141748.864085:WARNING:runtime_features.cc(730)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716473870301.526,"url":"http://127.0.0.2:9291/cockpit/@localhost/*/po.manifest.js","networkRequestId":"13852.19"}
CDP: {"source":"network","level":"error","text":"Failed to load resource: the server responded with a status of 404 (ERROR)","timestamp":1716473870304.383,"url":"http://127.0.0.2:9291/cockpit/@localhost/*/po.js","networkRequestId":"13852.20"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9291/cockpit/@localhost/*/po.manifest.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716473870350.396,"url":"http://127.0.0.2:9291/metrics"}
CDP: {"source":"security","level":"error","text":"Refused to execute script from 'http://127.0.0.2:9291/cockpit/@localhost/*/po.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.","timestamp":1716473870350.503,"url":"http://127.0.0.2:9291/metrics"}
> warning: CPU temperature metric closed: {"problem":"No samples received","command":"close","channel":"1:3!5"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
> log: PackageKit went away from D-Bus
> warning: Resolving coreutils failed: {"problem":"not-found","name":null,"message":"not-found"}
> log: PackageKit went away from D-Bus
> warning: loading available updates failed: {"problem":"not-found","name":null,"message":"not-found"}
Running as unit: cpu-hog.service; invocation ID: b6332141635e4c339ade4931cf497e7b
Running as unit: cpu-piglet.service; invocation ID: a83a514221fc460dab5ed7151cf59b16
> warning: failed to poll tuned {"problem":"disconnected","name":"null","message":"Server has closed the connection.","toString":""}
Running as unit: load-hog.service; invocation ID: e9adc262f97949b0bfd1a9d4fd5493fe
Getting image source signatures
Copying blob sha256:6b83872188a9e8912bee1d43add5e9bc518601b02a14a364c0da43b0d59acf33
Copying blob sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed
Copying config sha256:56a9b5a744a674481a5c034fec45ebf470a58ab844cd0da4d26b89bd27df5e36
Writing manifest to image destination
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

Wrote screenshot to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.png
Wrote HTML dump to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.html
Wrote JS log to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.js.log
Journal extracted to TestCurrentMetrics-testCPU-centos-10-127.0.0.2-2401-FAIL.log.gz
> warning: transport closed: disconnected
Removed "/etc/systemd/system/packagekit.service".
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-metrics", line 825, in testCPU
    admin_s.execute(f"podman run --rm -d --name {user_container_name} {self.busybox_image} /bin/dd if=/dev/urandom of=/dev/null")
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 327, in execute
    res = subprocess.run(command_line,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 2113, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/bots/machine/machine_core/timeout.py", line 45, in handle_timeout
    raise RuntimeError(self.error_message)
RuntimeError: Timed out on 'podman run --rm -d --name user-cpu-hog localhost/test-busybox /bin/dd if=/dev/urandom of=/dev/null'

# Result testCPU (__main__.TestCurrentMetrics.testCPU) failed
# 1 TEST FAILED [173s on 2f31acf73140]
not ok 24 test/verify/check-metrics TestCurrentMetrics.testCPU [ND@2]

First occurrence: 2024-05-23T14:20:38.186548+00:00 | revision 2654a55b1df2edff90f1feed3e44e2b7f8bfb94c
Times recorded: 1
Latest occurrences:

2024-05-23T14:20:38.186548+00:00 | revision 2654a55b1df2edff90f1feed3e44e2b7f8bfb94c

Downstream report: https://issues.redhat.com/browse/RHEL-34650 Known issue #6340

martinpitt added the knownissue label May 6, 2024

martinpitt added a commit to jelly/bots that referenced this issue May 6, 2024

naughty: Add patterns for CentOS 10 user podman hang

eff0ba2

Downstream report: https://issues.redhat.com/browse/RHEL-34650 Known issue cockpit-project#6340

martinpitt added a commit that referenced this issue May 6, 2024

naughty: Add patterns for CentOS 10 user podman hang

87bf7a8

Downstream report: https://issues.redhat.com/browse/RHEL-34650 Known issue #6340

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

user containers hang without any output #6340

user containers hang without any output #6340

martinpitt commented May 6, 2024

cockpituous commented May 6, 2024

cockpituous commented May 6, 2024 •

edited

user containers hang without any output #6340

user containers hang without any output #6340

Comments

martinpitt commented May 6, 2024

cockpituous commented May 6, 2024

cockpituous commented May 6, 2024 • edited

cockpituous commented May 6, 2024 •

edited