Syncs don't respect the resource_requirements set in the actor_definition table in the database in latest Airbyte

[ro8inmorgan]

Helm Chart Version

0.67.17

What step the error happened?

During the Sync

Relevant information

I've set the resource_requirements for my connectors in the postgress database in table actor_definition. For example

{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "1", "cpu_request": "0.5", "memory_limit": "1Gi", "memory_request": "1Gi"}}]}

But all syncs revert to the default settings set in the airbyte-env configmap

No matter what I do the actor_defintion resource_requirements is never used. I would like to set a per connector resource base as these widely differ per connector.

For example logs below come from a connector set with the resource_requirements above as you can see it doesn't follow the resource_requirements but uses the settings from airbyte-env configmap which are:

JOB_MAIN_CONTAINER_CPU_LIMIT: "2"
JOB_MAIN_CONTAINER_CPU_REQUEST: "1"
JOB_MAIN_CONTAINER_MEMORY_LIMIT: 2Gi
JOB_MAIN_CONTAINER_MEMORY_REQUEST: 2Gi

Relevant log output

2024-05-14 07:09:42 replication-orchestrator > Attempting to start pod = source-mssql-read-102432-1-sywjb for airbyte/source-mssql:2.0.1 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@6e986dbf[cpuRequest=1,cpuLimit=2,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@440bf54d[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@5c7a9a3e[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=null, stdOut=io.airbyte.config.ResourceRequirements@3dc75835[cpuRequest=0.5,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@752ffce3[hosts=[192.168.240.5, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-05-14 07:09:42 replication-orchestrator > Attempting to start pod = destination-bigquery-write-102432-1-busrh for airbyte/destination-bigquery:2.4.19 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@7761bb99[cpuRequest=1,cpuLimit=2,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@440bf54d[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@4cb85b24[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@389b79d5[cpuRequest=0.5,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@42dcb02c[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts nul

[intelc]

We have the same probelm in Helm 0.64.205 and version 0.57.3. The connector level setting doesn't work. However, connection level works