4.0 container upgrade procedure

Follow the procedure below to upgrade contrail cloud running 4.0 code across minor version, eg, 4.0 build 3052 to 3054. It is assumed that previous version DB schema is compatible with new version. This procedure uses the Cassandra config DB folder backed up from older version and mounts to the new version controller container. This procedure is for controller container (config DB).

Update on 4.0.2 to 4.0.3 upgrade:

For 4.0.2 installation, cassandra DB and zookeeper data for config are host mounted. Cassandra DB for analyticsdb container is also host mounted. Use the commands below to display mounts:

   docker inspect -f {{.Mounts}} controller
   docker inspect -f {{.Mounts}} analyticsdb

delete the /var/log/contrail/controller folder from the host

To work around an issue with zookeeper service crashing, remove the zookeeper data mount in controller container.

Edit the file /opt/contrail/server_manager/ansible/playbooks/contrail_networking_docker_upgrade/playbooks/roles/node/defaults/main.yml

   contrail_controller_data_volumes:
     - "{{ configdb_data_dir }}:/var/lib/cassandra"
   #  - "{{ config_zookeeper_data_dir }}:/var/lib/zookeeper"

   #contrail_controller_volumes: "{{ contrail_container_volumes + contrail_controller_log_volumes + 
   contrail_controller_data_volumes }}"
   contrail_controller_volumes: "{{ contrail_container_volumes + contrail_controller_data_volumes }}"

   re-provision the cluster using 4.0.3 image. The containers should now be 4.0.3.

Check the contrail-status on the containers, check cassandra service status on controller and analyticsdb containers, restart if the service is not started.

   service cassandra status
   service cassandra restart

Below Steps from initial contrail container based images, like 4.0.

1. Shutdown all northbound api services (openstack services and rabbitmq on openstack nodes). This is to ensure no events occur that trigger Cassandra DB changes.

2. shutdown controller container.

   docker stop controller
   

3. Copy the Cassandra data folder from the running controller container to host file system. This step can be omitted if the controller being upgraded is using the volume mounted in the container.

   docker cp controller:/var/lib/cassandra /root/cass/.
   

4. Prepare cluster.json and server.json as required for 4.X Server-Manager provisioning (Refer to Provision using 4.X Server-Manager)

   Add the “Cassandra_data_dir” key in cluster.json [all:vars] section, as shown below.

   "[all:vars]": {
          "cassandra_data_dir": "/root/cass"
   }
   

5. Add the following patch to the SM. This makes the host folder mount inside the controller container. eg. If cassandra_data_dir is /root/cass, it is mounted in container as /var/root/cass.

   Add following lines in /opt/contrail/server_manager/ansible/playbooks/contrail-ansible/roles/common/vars/main.yml

    contrail_volumes:
       - "{{ contrailctl_config_directory }}:/etc/contrailctl/"
    cass_volumes:
       - "{% if cassandra_data_dir is defined and cassandra_data_dir %}{{cassandra_data_dir}}:/var{{ cassandra_data_dir       }}{% endif %}"
    docker_volumes: "{% if cass_volumes is defined and cass_volumes[0] != '' %}{{contrail_volumes + cass_volumes}}{% else %}{{contrail_volumes}}{%endif%}"
   

   change volumes key in /opt/contrail/server_manager/ansible/playbooks/contrail-ansible/roles/node/tasks/controller.yml

    volumes: "{{ docker_volumes }}"
   

6. Provision the cluster and wait for provisioning to be completed. Check the status of provisioning using following command

    server-manager status server --cluster_id <cluster-id>
   

   The provisioning removes previous containers and provisions new containers. The /root/cass folder will be mounted in the container. Use below command on host to verify the mount.

   docker inspect -f {{.Mounts}} controller
   

7. To make the configuration persistent upon controller container restart, add cassandra config section in /etc/contrailctl/controller.conf on the host running the controller container.

   [CASSANDRA]
   commitlog_dir = /var/root/cass/commitlog
   saved_caches_dir = /var/root/cass/saved_caches
   data_dirs = ["/var/root/cass/data"]
   

   run following command from host of the controller container, for configuration to take effect

   docker exec -i controller contrailctl config sync -c controller -F -v -t configure
   

8. change the ownership on the Cassandra data folder and restart Cassandra service

   chown -R cassandra:cassandra /var/root/cass
   service Cassandra restart