Recreating another CKAN instance (after the default is running successfully) doesn't work

[starship9]

Hello!

I managed to get a CKAN instance running successfully on localhost:8443 (the default). However, subsequent attempts to launch an instance fail.

I have tried changing the ports, container names, and service names in the .env and docker-compose.yml file for solr, datapusher, nginx as well as redis, but I keep getting a dependency failed to start: solr_new is unhealthy.

When I do a docker compose exec solr_new env, I still see the default values; my updates are not reflecting for the new container. Similarly for datapusher_new. I have found the config files for the two in their respective containers (datapusher_settings.py and jetty.xml) that do contain the ports that I probably need to change, but they're read-only.

How do I resolve this, and get a container to run successfully when there already is a CKAN instance running?

[kowh-ai]

Sounds like something got messed up along the way. Are you able to stop all containers and remove all images, networks, volumes and containers plus run a docker system prune then start again?

[starship9]

I am facing the same error, even after a docker system prune. I get a dependency failed to start: solr_2 is unhealthy. Here's a screenshot of my docker container ls:

My workflow is this (after successfully running a clone of this repository as is):

• I clone this repository
• I change the following in the .env file:
  • Container names
  • Host ports
  • CKAN_SITE_URL, CKAN_PORT, CKAN_PORT_HOST
  • CKAN_SOLR_URL
  • CKAN_REDIS_URL
  • CKAN_DATAPUSHER_URL
  • DATAPUSHER_REWRITE_URL
  • NGINX_PORT
  • NGINX_SSLPORT
  • CKAN__HARVEST__MQ__PORT
• I also changed the docker-compose.yml file as follows:
  • Volume names
  • Service names
  • Health checks (to reflect changes made in the .env file)
  • Network names

However, after a docker compose build or even a docker compose build --no-cache, followed by a docker compose up, I see that the datapusher and solr containers are in an unhealthy state. Additionally, a docker compose exec solr_2 env shows me that the port of the solr container is still the default, even after a docker system prune.

I did think of perhaps creating new config files (such as datapusher_settings.py), but it seems that the filesystem for the datapusher container is read-only, so I can't make changes to it.

No luck so far unfortunately!

[kowh-ai]

Are you able to post (here) all of the logs during the startup?

[starship9]

I'm posting the logs for the solr container here, since I get a dependency failed to start: solr_2 is unhealthy message. Got these after a docker compose logs solr_2:

Executing /opt/solr/docker/scripts/solr-precreate ckan /opt/solr/server/solr/configsets/ckan Executing /opt/solr/docker/scripts/precreate-core ckan /opt/solr/server/solr/configsets/ckan Core ckan already exists Starting Solr Java 17 detected. Enabled workaround for SOLR-16463 [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true WARNING: A command line option has enabled the Security Manager WARNING: The Security Manager is deprecated and will be removed in a future release 2023-11-20 10:53:47.452 INFO (main) [] o.e.j.s.Server jetty-10.0.13; built: 2022-12-07T20:13:20.134Z; git: 1c2636ea05c0ca8de1ffd6ca7f3a98ac084c766d; jvm 17.0.7+7 2023-11-20 10:53:47.942 WARN (main) [] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2023-11-20 10:53:47.976 INFO (main) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2023-11-20 10:53:47.984 INFO (main) [] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.2.1 2023-11-20 10:53:47.984 INFO (main) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in standalone mode on port 8983 2023-11-20 10:53:47.984 INFO (main) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.2.1 2023-11-20 10:53:47.984 INFO (main) [] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2023-11-20T10:53:47.984876750Z 2023-11-20 10:53:47.988 INFO (main) [] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_8.log 2023-11-20 10:53:47.993 INFO (main) [] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2023-11-20 10:53:47.995 INFO (main) [] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2023-11-20 10:53:47.995 INFO (main) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.2.1/server/solr/solr.xml 2023-11-20 10:53:48.075 INFO (main) [] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.2.1/lib] 2023-11-20 10:53:49.524 WARN (main) [] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2023-11-20 10:53:49.527 WARN (main) [] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2023-11-20 10:53:49.804 INFO (main) [] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /var/solr/data 2023-11-20 10:53:49.805 INFO (main) [] o.a.s.c.CorePropertiesLocator Cores are: [ckan] 2023-11-20 10:53:49.962 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.0 2023-11-20 10:53:50.101 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.s.IndexSchema Schema name=ckan-2.10 2023-11-20 10:53:50.271 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.s.IndexSchema Loaded schema ckan-2.10/1.6 with uniqueid field index_id 2023-11-20 10:53:50.282 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.c.CoreContainer Creating SolrCore 'ckan' using configuration from instancedir /var/solr/data/ckan, trusted=true 2023-11-20 10:53:50.330 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/ckan], dataDir=[/var/solr/data/ckan/data/] 2023-11-20 10:53:50.356 INFO (main) [] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2023-11-20 10:53:50.504 INFO (main) [] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@548e76f1{solr-jetty-context.xml,/solr,file:///opt/solr-9.2.1/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.2.1/server/solr-webapp/webapp} 2023-11-20 10:53:50.517 INFO (main) [] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2023_11_20.request.log 2023-11-20 10:53:50.522 INFO (main) [] o.e.j.s.AbstractConnector Started ServerConnector@d8305c2{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983} 2023-11-20 10:53:50.524 INFO (main) [] o.e.j.s.Server Started Server@40e10ff8{STARTING}[10.0.13,sto=0] @4949ms 2023-11-20 10:53:51.021 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2023-11-20 10:53:51.042 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2023-11-20 10:53:51.043 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2023-11-20 10:53:51.059 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2023-11-20 10:53:51.059 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2023-11-20 10:53:51.179 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /var/solr/data/ckan/conf 2023-11-20 10:53:51.200 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal} 2023-11-20 10:53:51.210 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2023-11-20 10:53:51.216 INFO (coreLoadExecutor-10-thread-1) [ x:ckan] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1783080032629948416 2023-11-20 10:53:51.222 INFO (searcherExecutor-12-thread-1-processing-ckan) [ x:ckan] o.a.s.c.QuerySenderListener QuerySenderListener done. 2023-11-20 10:53:51.222 INFO (searcherExecutor-12-thread-1-processing-ckan) [ x:ckan] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2023-11-20 10:53:51.231 INFO (searcherExecutor-12-thread-1-processing-ckan) [ x:ckan] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms

[kowh-ai]

Is that the end of the Solr log?...the next lines should be:
...
INFO (main) [] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@61a002b1{solr-jetty-context.xml,/solr,file:///opt/solr-9.2.1/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.2.1/server/solr-webapp/webapp}
2023-11-20 10:21:19.193 INFO (main) [] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2023_11_20.request.log
2023-11-20 10:21:19.198 INFO (main) [] o.e.j.s.AbstractConnector Started ServerConnector@53812a9b{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2023-11-20 10:21:19.200 INFO (main) [] o.e.j.s.Server Started Server@269f4bad{STARTING}[10.0.13,sto=0] @6305ms

[starship9]

This is the entire output from the logs for my solr container! I changed the ports for it in the .env file to 8984, but I'm unable to access the admin portal for solr as well

[starship9]

Just as an add-on... if I already have an instance of CKAN running successfully (with the defaults from this repository), and I need to run another instance from the same machine... what steps do I need to take? My assumption was that I needed to change the ports being used in the .env file for dependencies like solr, datapusher and the ckan service itself, along with changing volume and network names.

Is there something I'm missing out on?

[kowh-ai]

Try it and see...it should be self explanatory...different container names definitely...ports can be the same if the container name (ie: hostname) is different

[starship9]

I changed the port numbers to their defaults, they're all the same except for the CKAN_SITE_URL. This time, most of the dependencies were healthy, apart from the ckan one itself. Upon seeing the logs, this stands out:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "db" to address: Try again

I have changed the container name for db to db_2 in the .env file, for context. I did not see this error before since previously, solr and datapusher were in an unhealthy state.