Scenario Troubleshooting

VM doesn't have link-local (169.254.x.x) address upon boot

1. check on hypervisor 8085/Snh_ItfReq to see if config is missing (i.e. state in ERROR, vn-null etc.).

   if missing, config has not come from controller

   a. check api-server 8082/virtual-machine-interface/ if routing_instance_refs are present

      if missing,
        a. check if contrail-schema is running on atleast one controller
             if running
               a. check if VMI present in ifmap on all controllers via ifmap localhost 8443 visual visual | grep <vmi-uuid>
                    if missing in ifmap,
                      a. check if rabbitmq is clustered fine and all api servers are connected to it.
                      b. check contrail-api introspect port (8084) for rest/db/messagebus/ifmap traces
                    else
                      a. need to check why contrail-schema didnt establish link between VMI to RI maybe check /var/log/contrail/schema.err
                      b. check contrail-logs --object-types config --object-values followed by --object-id
             else
               a. check last log from contrail to stdout + log file
               b. check connection status http://<config-node-where-schema-active>:8084/Snh_SandeshUVECacheReq?x=NodeStatus
   

VM doesn't ping the default gateway (subnet gateway)

1. Initiate continuous ping from VM to subnet gateway
2. Find tap interface of VM (TODO a. from contrail UI b. agent introspect)
3. tcpdump -ni <tap-intf> - see ICMP requests but no replies received
4. tcpdump -ni pkt0 -X - don't see the packet going to Vrouter-Agent with metadata
5. watch -n1 "dropstats | egrep '[1-9]+'" - observed flow unsable count increases
6. find VRF id for net from introspect (TODO)
7. rt --dump 34 | less observed NH for VJX0 ip is 0 which is discard.
8. Source code check to determine bug in state machine sending from Agent to kernel module.