Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting Lost connection to MySQL server during query (Mysql2::Error::ConnectionError) #117

Open
nimir opened this issue Jan 7, 2024 · 24 comments

Comments

@nimir
Copy link

nimir commented Jan 7, 2024

Env and Versions:

  • Ruby 3.3.0
  • MySQL: 8.2.0 for macos13.5 on arm64 (Homebrew)
  • MySQL Adapter: mysql2 V0.5.5
  • Rails 7.1.2
  • Solid-Queue 0.1.2

Solid_Queue Configurations:

config/solid_queue.yml:

development:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "dc_development_*"
      threads: 2
      processes: 1
      polling_interval: 0.1

Steps to reproduce:

  1. Started a supervisor on shell tab:
bundle exec rake solid_queue:start
  1. Started Rails Console on another tab to queue a job (simple mailer job):
Notifications::MailerJob.perform_later(Payment.last, :successfull)

Solid_Queue Log Trace:

[SolidQueue] Starting Dispatcher(pid=69907, hostname=Mohameds-MacBook-Pro-2.local, metadata={:polling_interval=>1, :batch_size=>500})
[SolidQueue] Starting Worker(pid=69908, hostname=Mohameds-MacBook-Pro-2.local, metadata={:polling_interval=>0.1, :queues=>"dc_development_*", :thread_pool_size=>2})
[SolidQueue] Claimed 1 jobs
/Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:151:in `_query': Lost connection to MySQL server during query (Mysql2::Error::ConnectionError)
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:151:in `block in query'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:150:in `handle_interrupt'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:150:in `query'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:100:in `block (2 levels) in raw_execute'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:1028:in `block in with_raw_connection'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:1000:in `with_raw_connection'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:98:in `block in raw_execute'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.2/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:1143:in `log'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:97:in `raw_execute'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:233:in `execute_and_free'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:23:in `internal_exec_query'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/database_statements.rb:630:in `select'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/database_statements.rb:71:in `select_all'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/query_cache.rb:114:in `select_all'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:14:in `block in select_all'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:1028:in `block in with_raw_connection'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:1000:in `with_raw_connection'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:10:in `select_all'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/relation/calculations.rb:286:in `block in pluck'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/relation.rb:1003:in `skip_query_cache_if_necessary'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/relation/calculations.rb:282:in `pluck'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/queue_selector.rb:52:in `prefixed_names'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/queue_selector.rb:37:in `eligible_queues'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/queue_selector.rb:31:in `queue_names'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/queue_selector.rb:27:in `none?'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/queue_selector.rb:15:in `scoped_relations'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/ready_execution.rb:11:in `claim'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/worker.rb:35:in `block in poll'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/poller.rb:16:in `with_polling_volume'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/worker.rb:34:in `poll'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/worker.rb:19:in `run'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/runnable.rb:47:in `block in do_start_loop'
	from <internal:kernel>:187:in `loop'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/runnable.rb:44:in `do_start_loop'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/runnable.rb:39:in `start_loop'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/runnable.rb:13:in `start'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:123:in `block in start_fork'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:122:in `fork'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:122:in `start_fork'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:152:in `replace_fork'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:134:in `block in reap_and_replace_terminated_forks'
	from <internal:kernel>:187:in `loop'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:130:in `reap_and_replace_terminated_forks'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:51:in `block in supervise'
	from <internal:kernel>:187:in `loop'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:47:in `supervise'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:26:in `start'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:14:in `start'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/tasks.rb:4:in `block (2 levels) in <main>'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `block in execute'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `each'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `execute'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `invoke_with_call_chain'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:188:in `invoke'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:182:in `invoke_task'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block (2 levels) in top_level'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `each'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block in top_level'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:147:in `run_with_threads'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:132:in `top_level'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:83:in `block in run'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:208:in `standard_exception_handling'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:80:in `run'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/exe/rake:27:in `<top (required)>'
	from /Users/nimir/.rbenv/versions/3.3.0/bin/rake:25:in `load'
	from /Users/nimir/.rbenv/versions/3.3.0/bin/rake:25:in `<top (required)>'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:58:in `load'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:58:in `kernel_load'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:23:in `run'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli.rb:451:in `exec'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli.rb:34:in `dispatch'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli.rb:28:in `start'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/exe/bundle:28:in `block in <top (required)>'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/exe/bundle:20:in `<top (required)>'
	from /Users/nimir/.rbenv/versions/3.3.0/bin/bundle:25:in `load'
	from /Users/nimir/.rbenv/versions/3.3.0/bin/bundle:25:in `<main>'
[SolidQueue] Restarting fork[67633] (status: 1)
[SolidQueue] Starting Worker(pid=67637, hostname=Mohameds-MacBook-Pro-2.local, metadata={:polling_interval=>0.1, :queues=>"dc_development_*", :thread_pool_size=>2})
/Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:151:in `_query': Mysql2::Error::ConnectionError: Lost connection to MySQL server during query (ActiveRecord::ConnectionFailed)
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:151:in `block in query'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:150:in `handle_interrupt'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:150:in `query'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:100:in `block (2 levels) in raw_execute'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:1028:in `block in with_raw_connection'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:1000:in `with_raw_connection'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:98:in `block in raw_execute'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.2/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:1143:in `log'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:97:in `raw_execute'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:233:in `execute_and_free'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:23:in `internal_exec_query'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/database_statements.rb:630:in `select'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/database_statements.rb:71:in `select_all'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/query_cache.rb:114:in `select_all'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:14:in `block in select_all'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:1028:in `block in with_raw_connection'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:1000:in `with_raw_connection'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/mysql2/database_statements.rb:10:in `select_all'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/relation/calculations.rb:286:in `block in pluck'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/relation.rb:1003:in `skip_query_cache_if_necessary'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.2/lib/active_record/relation/calculations.rb:282:in `pluck'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/queue_selector.rb:65:in `paused_queues'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/queue_selector.rb:31:in `queue_names'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/queue_selector.rb:27:in `none?'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/queue_selector.rb:15:in `scoped_relations'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/app/models/solid_queue/ready_execution.rb:11:in `claim'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/worker.rb:35:in `block in poll'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/poller.rb:16:in `with_polling_volume'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/worker.rb:34:in `poll'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/worker.rb:19:in `run'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/runnable.rb:47:in `block in do_start_loop'
	from <internal:kernel>:187:in `loop'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/runnable.rb:44:in `do_start_loop'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/runnable.rb:39:in `start_loop'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/runnable.rb:13:in `start'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:123:in `block in start_fork'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:122:in `fork'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:122:in `start_fork'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:152:in `replace_fork'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:134:in `block in reap_and_replace_terminated_forks'
	from <internal:kernel>:187:in `loop'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:130:in `reap_and_replace_terminated_forks'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:51:in `block in supervise'
	from <internal:kernel>:187:in `loop'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:47:in `supervise'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:26:in `start'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/supervisor.rb:14:in `start'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/tasks.rb:4:in `block (2 levels) in <main>'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `block in execute'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `each'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `execute'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `invoke_with_call_chain'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:188:in `invoke'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:182:in `invoke_task'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block (2 levels) in top_level'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `each'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block in top_level'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:147:in `run_with_threads'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:132:in `top_level'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:83:in `block in run'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:208:in `standard_exception_handling'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:80:in `run'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/exe/rake:27:in `<top (required)>'
	from /Users/nimir/.rbenv/versions/3.3.0/bin/rake:25:in `load'
	from /Users/nimir/.rbenv/versions/3.3.0/bin/rake:25:in `<top (required)>'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:58:in `load'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:58:in `kernel_load'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:23:in `run'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli.rb:451:in `exec'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli.rb:34:in `dispatch'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/cli.rb:28:in `start'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/exe/bundle:28:in `block in <top (required)>'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
	from /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.4/exe/bundle:20:in `<top (required)>'
	from /Users/nimir/.rbenv/versions/3.3.0/bin/bundle:25:in `load'
	from /Users/nimir/.rbenv/versions/3.3.0/bin/bundle:25:in `<main>'
@rosa
Copy link
Member

rosa commented Jan 8, 2024

Hey @nimir, sorry for the trouble! Could you let me know the output from this in your MySQL console?

show global variables like 'max_connections';

And could you also copy your database.yml configuration? (without passwords and that stuff, just the other parameters)

@nimir
Copy link
Author

nimir commented Jan 8, 2024

Not at all @rosa , and thank you and the team for the great work

mysql> show global variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 151   |
+-----------------+-------+
1 row in set (0.03 sec)

My database.yml file:

default: &default
  adapter: mysql2
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  username: REDACTED
  password: REDACTED
  collation: utf8mb4_unicode_ci

development:
  <<: *default
  database: dmydb_development

test:
  <<: *default
  database: mydb_test

production:
  <<: *default
  host: <%= ENV["PRODUCTION_DB_HOST"] %>
  database: mydb_production
  username: <%= ENV["PRODUCTION_DB_USER"] %>
  password: <%= ENV["PRODUCTION_DB_PASSWORD"] %>

Interacting with the database works fine via the rails console, also the same job is currently working using sidekiq.

@rosa
Copy link
Member

rosa commented Jan 8, 2024

So strange. I haven't been able to reproduce this with your same configuration (except I'm running MySQL 8 in Docker via the 8.0.31 image). I imagine it matches this, but what do you get for the following MySQL settings?

mysql> show global variables like 'net_read_timeout';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| net_read_timeout | 30    |
+------------------+-------+
1 row in set (0.01 sec)

mysql> show global variables like 'net_write_timeout';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| net_write_timeout | 60    |
+-------------------+-------+
1 row in set (0.01 sec)

mysql> show global variables like 'connect_timeout';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| connect_timeout | 10    |
+-----------------+-------+
1 row in set (0.01 sec)

@nimir
Copy link
Author

nimir commented Jan 8, 2024

Weird that I got same output:

mysql> show global variables like 'net_read_timeout';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| net_read_timeout | 30    |
+------------------+-------+
1 row in set (0.07 sec)

mysql> show global variables like 'net_write_timeout';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| net_write_timeout | 60    |
+-------------------+-------+
1 row in set (0.01 sec)

mysql> show global variables like 'connect_timeout';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| connect_timeout | 10    |
+-----------------+-------+
1 row in set (0.00 sec)

@nimir
Copy link
Author

nimir commented Jan 8, 2024

I can add more debug code to solid_queue if that can help but will need your input on where to put it in the gem files?

@rosa
Copy link
Member

rosa commented Jan 8, 2024

Thank you 🙏 Let's try to see if we get anything useful in the development logs. Could you clear your log/development.log file (just deleting it would do) and then run solid queue and enqueue the job, so that you get the failure and then check that log to see if there's anything useful there?

@nimir
Copy link
Author

nimir commented Jan 8, 2024

Ok, here is the fresh log file:

[ActiveJob] Enqueued Notifications::MailerJob (Job ID: 6d355475-26f4-4ed3-af32-9127e6632c81) to SolidQueue(dc_development_mailer) with arguments: #<GlobalID:0x000000012ee50798 @uri=#<URI::GID gid://damancash/Payment/9F8K24229SMW4C69>>, :successfull
[ActiveJob] ↳ (irb):1:in `<main>'
[ActiveJob] [Notifications::MailerJob] [6d355475-26f4-4ed3-af32-9127e6632c81] Performing Notifications::MailerJob (Job ID: 6d355475-26f4-4ed3-af32-9127e6632c81) from SolidQueue(dc_development_mailer) enqueued at 2024-01-08T09:56:55.285563000Z with arguments: #<GlobalID:0x00000001350f8328 @uri=#<URI::GID gid://damancash/Payment/9F8K24229SMW4C69>>, :successfull
[ActiveJob] [Notifications::MailerJob] [6d355475-26f4-4ed3-af32-9127e6632c81] Performed Notifications::MailerJob (Job ID: 6d355475-26f4-4ed3-af32-9127e6632c81) from SolidQueue(dc_development_mailer) in 40.22ms

@rosa
Copy link
Member

rosa commented Jan 8, 2024

Ahh, nothing useful there 😞 Thanks for trying that.

I imagine you got this after starting fresh, but just in case: do you get any errors when running

SolidQueue::Pause.all.pluck(:queue_name)
SolidQueue::ReadyExecution.where("queue_name LIKE ?", "dc_development_%").distinct(:queue_name).pluck(:queue_name)

from your Rails console?

@nimir
Copy link
Author

nimir commented Jan 8, 2024

No errors

❯ rails c
Loading development environment (Rails 7.1.2)
irb(main):001> SolidQueue::Pause.all.pluck(:queue_name)
=> []
irb(main):002> SolidQueue::ReadyExecution.where("queue_name LIKE ?", "dc_development_%").distinct(:queue_name).pluck(:queue_name)
=> []

@rosa
Copy link
Member

rosa commented Jan 8, 2024

Thanks for checking that 🙏 Ok, let's rule out forking here causing issues. Instead of starting Solid Queue via

bundle exec rake solid_queue:start

could you start just a single worker like this? In a Rails console:

worker = SolidQueue::Worker.new(polling_interval: 0.1, queues: "dc_development_*", threads: 2)
worker.start

And then try enqueuing the job and see if you get the error.

To stop it, just do:

worker.stop

@nimir
Copy link
Author

nimir commented Jan 8, 2024

Yes, starting the worker without forking worked like a charm and I didn't get any errors.

❯ rails c
Loading development environment (Rails 7.1.2)
irb(main):001> worker = SolidQueue::Worker.new(polling_interval: 0.1, queues: "dc_development_*", threads: 2)
irb(main):002> worker.start
[SolidQueue] Starting Worker(pid=66198, hostname=Mohameds-MacBook-Pro-2.local, metadata={:polling_interval=>0.1, :queues=>"dc_development_*", :thread_pool_size=>2})
=> #<Thread:0x000000012f69fe58 /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/runnable.rb:37 run>
irb(main):003> Notifications::MailerJob.perform_later(Payment.last, :successfull)
[SolidQueue] Enqueued job {:queue_name=>"dc_development_mailer", :active_job_id=>"5673fe6f-c9fc-4d05-858c-475e03e60daa", :priority=>nil, :scheduled_at=>Mon, 08 Jan 2024 20:29:55.826036000 EET +02:00, :class_name=>"Notifications::MailerJob", :arguments=>{"job_class"=>"Notifications::MailerJob", "job_id"=>"5673fe6f-c9fc-4d05-858c-475e03e60daa", "provider_job_id"=>nil, "queue_name"=>"dc_development_mailer", "priority"=>nil, "arguments"=>[{"_aj_globalid"=>"gid://dc/Payment/9F8K24229SMW4C69"}, {"_aj_serialized"=>"ActiveJob::Serializers::SymbolSerializer", "value"=>"successfull"}], "executions"=>0, "exception_executions"=>{}, "locale"=>"en", "timezone"=>"Cairo", "enqueued_at"=>"2024-01-08T18:29:55.826241000Z", "scheduled_at"=>nil}, :concurrency_key=>nil}
Enqueued Notifications::MailerJob (Job ID: 5673fe6f-c9fc-4d05-858c-475e03e60daa) to SolidQueue(dc_development_mailer) with arguments: #<GlobalID:0x000000012f453500 @uri=#<URI::GID gid://dc/Payment/9F8K24229SMW4C69>>, :successfull
↳ (irb):3:in `<main>'
=>
<<REDACTED_JOB_DETAILS_INCLDUNG_SENSETIVE_INFO>>
:[SolidQueue] Claimed 1 jobs
Performing Notifications::MailerJob (Job ID: 5673fe6f-c9fc-4d05-858c-475e03e60daa) from SolidQueue(dc_development_mailer) enqueued at 2024-01-08T18:29:55.826241000Z with arguments: #<GlobalID:0x000000013c5d1410 @uri=#<URI::GID gid://dc/Payment/9F8K24229SMW4C69>>, :successfull
Performed Notifications::MailerJob (Job ID: 5673fe6f-c9fc-4d05-858c-475e03e60daa) from SolidQueue(dc_development_mailer) in 3.7ms
irb(main):004> worker.stop
=> #<Thread:0x000000012f69fe58 /Users/nimir/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.1.2/lib/solid_queue/processes/runnable.rb:37 dead>

So, how can we debug the forking part to get more useful insight on the cause?

@rosa
Copy link
Member

rosa commented Jan 16, 2024

Hey @nimir! So sorry for the silence and delay in replying after your last comment. I tried to come up with a good way to debug this but in the end, I had to work on other unrelated stuff and lost track of it. I'm quite puzzled about why this might be happening. Did you figure anything else on your own or are you still having the exact same problem?

@nimir
Copy link
Author

nimir commented Jan 19, 2024

Hi @rosa , no worries and I appreciate your support

I thought the issue might be with my version of MySQL so tried downgrading to v8.0 and also testing on fresh apps just to eliminate the possibility I am making something wrong in the configurations but I had no luck. Unfortunately I have limited time to work on this at the time!

@hahwul
Copy link

hahwul commented Jan 22, 2024

Hi @rosa
I'm using AWS RDS and I'm experiencing the same issue. The version is also similar 😭
The connection error appears after restarting fork occurs several times.

  • MySQL: 8.0.35 (aws rds)
  • Ruby: 3.3.0
  • Rails 7.1.3
  • Solid-Queue 0.1.2
[SolidQueue] Restarting fork[71885] (status: )
[SolidQueue] Starting Worker(pid=71887, hostname=******, metadata={:polling_interval=>0.1, :queues=>"*", :thread_pool_size=>5})
[SolidQueue] Restarting fork[71887] (status: )
[SolidQueue] Starting Worker(pid=71993, hostname=******, metadata={:polling_interval=>0.1, :queues=>"*", :thread_pool_size=>5})
[SolidQueue] Restarting fork[71993] (status: )
[SolidQueue] Starting Worker(pid=71994, hostname=******, metadata={:polling_interval=>0.1, :queues=>"*", :thread_pool_size=>5})

/Users/user/.rvm/gems/ruby-3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:151:in `_query': Mysql2::Error::ConnectionError: Lost connection to MySQL server during query (ActiveRecord::ConnectionFailed)
	from /Users/user/.rvm/gems/ruby-3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:151:in `block in query'
	from /Users/user/.rvm/gems/ruby-3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:150:in `handle_interrupt'
	from /Users/user/.rvm/gems/ruby-3.3.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:150:in `query'
``

@nimir
Copy link
Author

nimir commented Jan 22, 2024

I found a work around this morning but I am not sure how sustainable it is since it gives a depreciation warning

Setting reconnect: true in database.yaml
I don't get the connection lost error but i get this warning since this flag seems to be deprecated

WARNING: MYSQL_OPT_RECONNECT is deprecated and will be removed in a future version.

@nimir
Copy link
Author

nimir commented Jan 22, 2024

The reconnect: true work around fails when I am enqueuing a job from another job and I get the lost mysql connection error again! 😢

Update:
I am starting to think that work around didn't work at all and it was just some inconsistency !

@rounders
Copy link

rounders commented Jan 22, 2024

Decided to give solid_queue a try today and I'm running into the same issue that @nimir described.

Env and Versions:

Ruby 3.2.2
MySQL: 8.2.0 (Homebrew)
MySQL Adapter: mysql2 V0.5.5
Rails 7.1.2
Solid-Queue 0.1.2

[SolidQueue] Starting Dispatcher(pid=61704, hostname=fh-imac-2.local, metadata={:polling_interval=>1, :batch_size=>500})
[SolidQueue] Starting Worker(pid=61705, hostname=fh-imac-2.local, metadata={:polling_interval=>0.1, :queues=>"*", :thread_pool_size=>5})
/Users/rounders/.rvm/gems/ruby-3.2.2/gems/mysql2-0.5.5/lib/mysql2/client.rb:151:in `_query': Mysql2::Error::ConnectionError: Lost connection to MySQL server during query (ActiveRecord::ConnectionFailed)
	from /Users/rounders/.rvm/gems/ruby-3.2.2/gems/mysql2-0.5.5/lib/mysql2/client.rb:151:in `block in query'
	from /Users/rounders/.rvm/gems/ruby-3.2.2/gems/mysql2-0.5.5/lib/mysql2/client.rb:150:in `handle_interrupt'
	from /Users/rounders/.rvm/gems/ruby-3.2.2/gems/mysql2-0.5.5/lib/mysql2/client.rb:150:in `query'

Looking at my development.log file I can see that I get about 200 SolidQueue::ReadyExecution Pluck commands after starting solid_queue (using 'bundle exec rake solid_queue:start') before I see the first Mysql2::Error::ConnectionError error.

@hahwul
Copy link

hahwul commented Jan 26, 2024

Yesterday, I also conducted some tests. In my case, when I extended the worker's polling interval period to seconds, the issue did not occur. I'm not sure of the exact cause, but I hope this helps someone :D

development:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "dc_development_*"
      threads: 2
      processes: 1
      polling_interval: 1 # ⬅️ 0.1 to 1

I think, It seems like there might be a key of solution in the MySQL connection settings and the access method of solid_queue.

Changed

  • Although the frequency of connection errors has decreased, it has not been completely resolved 😭
  • This problem only occurs in macOS

@nimir
Copy link
Author

nimir commented Jan 26, 2024

@hahwul thanks for sharing the workaround, unfortunately it didn't work for me ... It just delays the error due to the extended polling interval i suppose!

@gkeswani92
Copy link

Experiencing something similar on ruby 3.2.2 and rails 7.1.3 with all of the above pointers tried

11:51:50 solid_queue.1 | /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:151:in `_query': Lost connection to MySQL server during query (Mysql2::Error::ConnectionError)
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:151:in `block in query'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:150:in `handle_interrupt'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/mysql2-0.5.5/lib/mysql2/client.rb:150:in `query'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/mysql2/database_statements.rb:100:in `block (2 levels) in raw_execute'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:1028:in `block in with_raw_connection'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.3/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:1000:in `with_raw_connection'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/mysql2/database_statements.rb:98:in `block in raw_execute'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.3/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:1143:in `log'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/mysql2/database_statements.rb:97:in `raw_execute'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:233:in `execute_and_free'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/mysql2/database_statements.rb:23:in `internal_exec_query'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:630:in `select'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:71:in `select_all'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/abstract/query_cache.rb:115:in `select_all'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/mysql2/database_statements.rb:14:in `block in select_all'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:1028:in `block in with_raw_connection'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.3/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:1000:in `with_raw_connection'
11:51:50 solid_queue.1 |        from /Users/gaurav/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/mysql2/database_statements.rb:10:in `select_all'

@jairovm
Copy link

jairovm commented Feb 6, 2024

Just my two cents. I got this error on a M1 Pro machine, and this is what I did.

There are no more connection issues now. 🚀

@bmb1603
Copy link

bmb1603 commented Feb 29, 2024

Is there any fix yet? @rosa this seems to happen when starting solid queue but there are no Jobs or too few (< 5) Jobs to claim

@rosa
Copy link
Member

rosa commented Feb 29, 2024

I haven't been able to reproduce this in any way, @brincau, so no, I don't have a fix for it, I'm afraid.

@mountaindog
Copy link

I've seen the same thing: starting a solid queue is fine, but when running a rake command, it generates a MySQL connection error. The Trilogy adaptor seems to work better, but it has some issues when running GitHub Actions.

[SolidQueue] Starting Dispatcher(pid=35671, hostname=xxxxx-mac1.localdomain, metadata={:polling_interval=>1, :batch_size=>500})
[SolidQueue] Starting Worker(pid=35672, hostname=xxxxx-mac1.localdomain, metadata={:polling_interval=>0.1, :queues=>"*", :thread_pool_size=>5})
/Users/xxxxxx/.asdf/installs/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/mysql2-0.5.6/lib/mysql2/client.rb:151:in `_query': Mysql2::Error::ConnectionError: Lost connection to MySQL server during query (ActiveRecord::ConnectionFailed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants