Cherry pick controller commits from R2.20 to R2.22.x

updating version.info from 2.22 to 2.23 in 2.20 branch
Closes-Bug:#1528370

Change-Id: Ic649422979a926cc5f5b8457c01610b848dc206b

Storage stats daemon fix

Partial-Bug: #1528327
Fixed latency monitor code based on the Ceph 0.94.3 version.
Fixed issues in OSD throughput/IOPs calculation.
Updated code based on the latest Sandesh apis.

Change-Id: I12caf951f84c8b213b1b5ec01371bb68b4c48cb3

Fix contrail-collector back pressure mechanism

contrail-collector DB queue back presssure mechanism was not
working since the DB drop level is initialized to INVALID and
even the water marks levels are INVALID and hence the defer/undefer
callbacks are not called.

Change-Id: Ib28141a69aeed3c4ad6f50abbaed2a285e3e7db2
Partial-Bug: #1528380

Fix Agent crash for flow index tree management

Issue:
------
During a flow index change vrouter-agent triggers a delete
on index tree using new flow handle instead of currently
held flow_handle resulting in flow entry getting associated
to two slots in the flow index tree, which further on flow
entry delete due to aging or eviction never releases the
slot for old flow handle, causing failures for further
insertions in the flow index tree

Fix:
----
Avoid taking flow handle as argument to DeleteByIndex and
use the currently associated flow_handle to remove from tree
Adding assert in DeleteByIndex to catch delete failure
Avoid doing delete from index tree in code paths other than
flow entry index update of flow entry delete.

Add logic for KSync Sock User to Mock vrouter behavior
returning index for an entry if it is already allocated
instead of allocating a new one.

Closes-Bug: 1527425
Change-Id: I10e77fb59650acfdd924a5f1d35d6b8dea03a3f0

Fix discovery dependency issue. Originally made in master branch
via https://review.opencontrail.org/#/c/15749

Change-Id: I5d874de3714074c66fa73bfd7c9119772dc681fd
Partial-Bug: #1530186

Avoid calling get_routing_instances on VN object

Calling get_routing_instances could trigger another read of the VN
if the VN has no routing instance. This is not only inefficient, but
could also cause exception if the VN has disappeared. We can avoid
this by calling getattr.

Change-Id: Ie5500585b9e6c578576276c2c04ec03f32c75112
Partial-Bug: 1528950

Fix Centos 65 agent compilation issues.
Closes-Bug: #1532159

Change-Id: Ia8b77619c80737000d5bd949534c9e0a16967359

Closes-Bug: #1524063, contrail-status is showing contrail-web-ui, even it is not configured, in case of SMLite

Change-Id: I55afc19140b1ce52b3b529a644124705de5ce6a8

Fix a corner case with routing instance delete

Sequence of event that causes the crash
   1. Static route config deleted
   2. Static Route maanger triggers resolve_trigger_ to re-evaluate static
      route config
   3. Before the resolve trigger is invoked routing instance is deleted

Resolve trigger calls ProcessStaticRouteConfig to apply any pending static
route config. ProcessStaticRouteConfig accesses the NULL config pointer of
the routing instance

Fix:
1. Check whether the routing instance is deleted in ProcessStaticRouteConfig
2. Reset the resolve_trigger_ in StaticRouteMgr destructor
3. Add API to disable resolve_trigger_ and Add UT to test delayed processing
of resolve_trigger_

Change-Id: Icb1b9bad340ccefc9fbab75188034ade79a6193a
Closes-bug: #1533435

Fix scons failure due to pip ugrade

Closes-Bug: 1536541
Change-Id: I5138f6fb7d073beddafc9b3c686ef968f21e7e31

Fix uninitialized vrouter params

Issue:
------
uninitialized variable results in bad calculation of number
of MPLS labels for multicast

Fix:
----
initialize vrouter params to 0, handle case if
vrouter_max_labels is lesser than fixed unicast range.

Conflicts:
	src/vnsw/agent/cmn/agent.cc

Closes-Bug: 1535735
Change-Id: Id4a7b74f12728e78dcb5b8b24d0848e560ff0138
(cherry picked from commit eb0488b)

Update task policy for contrail-dns.

When XMPP channel with agent clsoes, the records exported from there
are removed in the cleaner task context. Update the task policy to
ensure the config and bind tasks do not run in parallel with it.

Change-Id: Ic92f147060c39e452742aa67ace7e3a6f3ddc5a3
closes-bug: 1533811

Fix corner case in join processing

This problem can happen when a number of vRouter agents restart in
quick succesion and subscribe to a table and advertise routes into
the table.

Consider the following sequence of events on a route in foo.inet.0:

- Route is added with a path with attribute A and nexthop N1
- XMPP peer P1 subscribes to the table
- As a result a RouteUpdate is created in QBULK (the join queue). The
RouteUpdate has an UpdateInfo with a bitset P1 and a RibOutAttr with
attribute A and nexthop N1
- Another path for the same route is added with nexthop N2
- This new path is ecmp eligible i.e. has same local preference as the
best path
- XMPP peer P2 subscribes to the table
- As a result the existing RouteUpdate in QBULK gets a new UpdateInfo.
A new UpdateInfo is created because RibOutAttr for P2 has attribute A
and nexthops (N1, N2). Note that a different UpdateInfo is required
for different RibOutAttr. This UpdateInfo has a bitset P2.

Notice that we now have a RouteUpdate with 2 UpdateInfos that have the
same attribute A, but different RibOutAttrs (by virtue having different
forwarding nexthops).

The UpdateQueue maintains a set (attr_set_ of type UpdatesByAttr) of
UpdateInfos keyed by BgpAttr and timestamp. The label/nexthops in the
RibOutAttr are not included in key to achieve optimal packing of bgp
updates by attribute. As a result, both the UpdateInfos are inserted
(or rather attempted to be inserted) into attr_set_ with the same key.
This causes a crash when we later try to erase both UpdateInfos from
the attr_set_ when doing export processing for the route.

Note that we run into this case only if join processing for P2 happens
before export processing for the route after the 2nd path got added.
If export processing happens before the join for P2, the RouteUpdate
would move from QBULK to QUPDATE and would have only 1 UpdateInfo with
attribute A and nexthop (N1, N2).

The fix consists of 2 parts:

1. Use the UpdateInfo pointer itself as the final tie-breaker in the
key for UpdateQueue::UpdatesByAttr to ensure we have no duplicates.

2. When traversing the UpdatesByAttr set to build update messages, fix
UpdateQueue::AttrNext to not return an UpdateInfo for same RouteUpdate
as the current UpdateInfo. Doing so invalidates the locking design in
RibOutUpdates and results in a deadlock.

Add unit tests to recreate the above scenario and verify the fix.

Change-Id: I45ce1bbd72d8b6a163a5aa61358491cc1d3f6a93
Closes-Bug: 1536729

Create only 1 ksync-socket

We are creating as many ksync sockets as number of TBB threads even
though we only use only the first socket for all operations. Modified
code to create only one ksync socket.

Change-Id: I2f1bf8558c219fc97402f8192c3d9d6cebacaf98
Fixes-Bug: #1533495

Fix ToR agent crash for duplicate VxLAN-ID

Issue:
------
ToR agent doesnot program/expect QFX to have two logical
switch with same VxLAN ID at any point of time, observing
the same in certain negative test scenarios doesnot allow
ToR agent to recover OVSDB database to sane state

Fix:
----
Allow creation of stale entry with duplicate VxLAN ID,
even though it is not expected, allowing creation helps
to recover OVSDB database to sane state the deletion of
the same on stale entry timeout.

Closes-Bug: 1535093
Change-Id: I32a4fbab665f433d6a5dae7eb185be8e50de53d0
(cherry picked from commit c1cfe79)

Handle VMs with whitespace in the name

vrouter-port-control script fail when there is a whitespace in the vm
name (in fact in any of the arguments).

This patch add a regex based split  on vrouter-port-control to fix that,
so that it will pass the arguments with whitespace in it correctly.

Change-Id: Ibf52dc23321d1c4c7f231cb5cd386afa495de0aa
Fixes-Bug: #1519768
Signed-off-by: hkumarmk <hkumar@d4devops.org>

* Add exclusion between flow table and flow stats collector
Reference for flow entry can be release by flow stats collector
resulting in flow being deleted from flow tree and parallel
modification of flow tree from flow table and flow stats collector
context. Fixing the same.
Closes-bug:#1535040

Change-Id: I8e7c18aaacbe1ed16639917dc51480af55b2da86

src/analytics/db_handler.cc

@@ -119,9 +119,10 @@ void DbHandler::SetDropLevel(size_t queue_count, SandeshLevel::type level,
             Sandesh::LevelToString(level) << "], DB QUEUE COUNT: " << 
             queue_count);
         drop_level_ = level;
-        if (!cb.empty()) {
-            cb();
-        }
+    }
+    // Always invoke the callback
+    if (!cb.empty()) {
+        cb();
     }
 }
 

src/api-lib/tools/install_venv_common.py

@@ -122,7 +122,7 @@ def install_dependencies(self, find_links):
 
         # First things first, make sure our venv has the latest pip and
         # setuptools and pbr
-        self.pip_install(find_links, 'pip>=1.4')
+        self.pip_install(find_links, 'pip<=7.1')
         self.pip_install(find_links, 'setuptools')
         self.pip_install(find_links, 'pbr')
 

src/base/version.info

@@ -1 +1 @@
-2.22
+2.23

src/bgp/bgp_update_queue.cc

@@ -115,6 +115,12 @@ void UpdateQueue::AttrDequeue(UpdateInfo *current_uinfo) {
 //
 // Returns NULL if there are no more updates with the same BgpAttr.
 //
+// Also return NULL if the next UpdateInfo is for the same RouteUpdate. This
+// can happen in corner cases where the label (or the set for ecmp nexthops
+// in case of an XMPP ribout) for a route changes between Join operations for
+// 2 different sets of IPeerUpdates. Returning such an UpdateInfo breaks the
+// locking design in RibOutUpdates and results in a deadlock.
+//
 UpdateInfo *UpdateQueue::AttrNext(UpdateInfo *current_uinfo) {
     tbb::mutex::scoped_lock lock(mutex_);
     UpdatesByAttr::iterator iter = attr_set_.iterator_to(*current_uinfo);
@@ -123,6 +129,9 @@ UpdateInfo *UpdateQueue::AttrNext(UpdateInfo *current_uinfo) {
         return NULL;
     }
     UpdateInfo *next_uinfo = iter.operator->();
+    if (next_uinfo->update == current_uinfo->update) {
+        return NULL;
+    }
     if (next_uinfo->roattr.attr() == current_uinfo->roattr.attr()) {
         return next_uinfo;
     }

src/bgp/bgp_update_queue.h

@@ -18,6 +18,12 @@
 // Looks at the BgpAttr, Timestamp and the associated RouteUpdate but not
 // the Label, in order to achieve optimal packing of BGP updates.
 //
+// Compare the UpdateInfo pointers themselves as the final tie-breaker to
+// handle the case where there are 2 UpdateInfos with the same BgpAttr in
+// the same RouteUpdate. This can happen if the label (or the set for ecmp
+// nexthops in case of an XMPP ribout) for a route changes between the Join
+// operations for 2 different sets of IPeerUpdates.
+//
 struct UpdateByAttrCmp {
     bool operator()(const UpdateInfo &lhs, const UpdateInfo &rhs) const {
         if (lhs.roattr.attr() < rhs.roattr.attr()) {
@@ -32,7 +38,13 @@ struct UpdateByAttrCmp {
         if (lhs.update->tstamp() > rhs.update->tstamp()) {
             return false;
         }
-        return (lhs.update < rhs.update);
+        if (lhs.update < rhs.update)  {
+            return true;
+        }
+        if (lhs.update > rhs.update) {
+            return false;
+        }
+        return (&lhs < &rhs);
     }
 };
 

src/bgp/routing-instance/static_route.cc

@@ -676,6 +676,7 @@ void StaticRouteMgr::RemoveStaticRoutePrefix(const Ip4Prefix &static_route) {
 
 void StaticRouteMgr::ProcessStaticRouteConfig() {
     CHECK_CONCURRENCY("bgp::Config");
+    if (routing_instance()->deleted() || !routing_instance()->config()) return;
     const BgpInstanceConfig::StaticRouteList &list =
         routing_instance()->config()->static_routes();
     typedef BgpInstanceConfig::StaticRouteList::const_iterator iterator_t;
@@ -690,6 +691,7 @@ bool StaticRouteMgr::ResolvePendingStaticRouteConfig() {
 }
 
 StaticRouteMgr::~StaticRouteMgr() {
+    resolve_trigger_->Reset();
     if (static_route_queue_)
         delete static_route_queue_;
 }

src/bgp/routing-instance/static_route.h

@@ -75,6 +75,10 @@ class StaticRouteMgr {
     RoutingInstance *instance_;
     BgpConditionListener *listener_;
     StaticRouteMap  static_route_map_;
+
+    void DisableResolveTrigger() { resolve_trigger_->set_disable(); }
+    void EnableResolveTrigger() { resolve_trigger_->set_enable(); }
+
     void DisableQueue() { static_route_queue_->set_disable(true); }
     void EnableQueue() { static_route_queue_->set_disable(false); }
     bool IsQueueEmpty() { return static_route_queue_->IsQueueEmpty(); }

src/bgp/test/bgp_export_rtupdate_test.cc

@@ -15,7 +15,11 @@ using namespace std;
 class BgpExportRouteUpdateCommonTest : public BgpExportTest {
 protected:
 
-    BgpExportRouteUpdateCommonTest() : rt_update_(NULL), count_(0) {
+    BgpExportRouteUpdateCommonTest()
+        : qid_(RibOutUpdates::QUPDATE),
+          rt_update_(NULL),
+          tstamp_(0),
+          count_(0) {
     }
 
     void InitAdvertiseInfo(BgpAttrPtr attrX, int start_idx, int end_idx) {
@@ -34,6 +38,12 @@ class BgpExportRouteUpdateCommonTest : public BgpExportTest {
         InitUpdateInfoCommon(attrX, start_idx, end_idx, uinfo_slist_);
     }
 
+    void InitUpdateInfo(BgpAttrPtr attrX, uint32_t label,
+        int start_idx, int end_idx, int qid = RibOutUpdates::QUPDATE) {
+        qid_ = qid;
+        InitUpdateInfoCommon(attrX, label, start_idx, end_idx, uinfo_slist_);
+    }
+
     void InitUpdateInfo(BgpAttrPtr attr_blk[], int start_idx, int end_idx,
             int qid = RibOutUpdates::QUPDATE) {
         qid_ = qid;
@@ -750,6 +760,159 @@ TEST_F(BgpExportRouteUpdateTest1, JoinMerge4) {
     }
 }
 
+//
+// Description: Join processing for route that already has join pending updates
+//              for some peers. Export policy accepts the route for other join
+//              peers with same attribute, but different label.
+//              Common attribute for initial join peers and new join peers.
+//              Different labels for initial join peers and new join peers.
+//
+// Old DBState: RouteUpdate in QBULK.
+//              No AdvertiseInfo.
+//              UpdateInfo peer x=[vJoinPeerCount,kPeerCount-1],
+//              attr A + label 100.
+// Join Peers:  Peers x=[0,vJoinPeerCount-1].
+// Export Rslt: Accept peer x=[0,vJoinPeerCount-1], attr A + label 200.
+// New DBState: RouteUpdate in QBULK.
+//              No AdvertiseInfo.
+//              UpdateInfo peer x=[vJoinPeerCount,kPeerCount-1],
+//              attr A + label 100.
+//              UpdateInfo peer x=[0,vJoinPeerCount-1], attr A + label 200.
+//
+TEST_F(BgpExportRouteUpdateTest1, JoinMerge5a) {
+    int qid = RibOutUpdates::QBULK;
+    for (int vJoinPeerCount = 1; vJoinPeerCount < kPeerCount;
+            vJoinPeerCount++) {
+        InitUpdateInfo(attrA_, 100, vJoinPeerCount, kPeerCount-1, qid);
+        Initialize();
+
+        RibPeerSet join_peerset;
+        BuildPeerSet(join_peerset, 0, vJoinPeerCount-1);
+        BuildExportResult(attrA_, 0, vJoinPeerCount-1, 200);
+        RunJoin(join_peerset);
+        table_.VerifyExportResult(true);
+
+        RouteUpdate *rt_update = ExpectRouteUpdate(&rt_, qid);
+        VerifyRouteUpdateDequeueEnqueue(rt_update);
+        EXPECT_EQ(rt_update_, rt_update);
+        VerifyUpdates(rt_update, attrA_, 100, vJoinPeerCount, kPeerCount-1, 2);
+        VerifyUpdates(rt_update, attrA_, 200, 0, vJoinPeerCount-1, 2);
+        VerifyHistory(rt_update);
+
+        DrainAndDeleteRouteState(&rt_);
+    }
+}
+
+//
+// Description: Join processing for route that already has join pending updates
+//              for some peers. Export policy accepts the route for other join
+//              peers with same attribute, but different label.
+//              Common attribute for initial join peers and new join peers.
+//              Different labels for initial join peers and new join peers.
+//              After this the route is exported to all peers with the same
+//              attribute and the label. The attribute and label are same as
+//              that for the new join peers.
+//
+// Old DBState: RouteUpdate in QBULK.
+//              No AdvertiseInfo.
+//              UpdateInfo peer x=[vJoinPeerCount,kPeerCount-1],
+//              attr A + label 100.
+// Join Peers:  Peers x=[0,vJoinPeerCount-1].
+// Export Rslt1:Accept peer x=[0,vJoinPeerCount-1], attr A + label 200.
+// Export Rslt2:Accept peer x=[0,kPeerCount-1], attr A + label 200.
+// New DBState: RouteUpdate in QBULK.
+//              No AdvertiseInfo.
+//              UpdateInfo peer x=[0,kPeerCount-1], attr A + label 200.
+//
+TEST_F(BgpExportRouteUpdateTest1, JoinMerge5b) {
+    int qid = RibOutUpdates::QBULK;
+    for (int vJoinPeerCount = 1; vJoinPeerCount < kPeerCount;
+            vJoinPeerCount++) {
+        InitUpdateInfo(attrA_, 100, vJoinPeerCount, kPeerCount-1, qid);
+        Initialize();
+
+        RibPeerSet join_peerset;
+        BuildPeerSet(join_peerset, 0, vJoinPeerCount-1);
+        BuildExportResult(attrA_, 0, vJoinPeerCount-1, 200);
+        RunJoin(join_peerset);
+        table_.VerifyExportResult(true);
+
+        RouteUpdate *rt_update = ExpectRouteUpdate(&rt_, qid);
+        VerifyRouteUpdateDequeueEnqueue(rt_update);
+        EXPECT_EQ(rt_update_, rt_update);
+        VerifyUpdates(rt_update, attrA_, 100, vJoinPeerCount, kPeerCount-1, 2);
+        VerifyUpdates(rt_update, attrA_, 200, 0, vJoinPeerCount-1, 2);
+        VerifyHistory(rt_update);
+
+        BuildExportResult(attrA_, 0, kPeerCount-1, 200);
+        RunExport();
+        table_.VerifyExportResult(true);
+
+        rt_update = ExpectRouteUpdate(&rt_, RibOutUpdates::QUPDATE);
+        VerifyRouteUpdateDequeueEnqueue(rt_update);
+        EXPECT_EQ(rt_update_, rt_update);
+        VerifyUpdates(rt_update, attrA_, 200, 0, kPeerCount-1);
+        VerifyHistory(rt_update);
+
+        DrainAndDeleteRouteState(&rt_);
+    }
+}
+
+//
+// Description: Join processing for route that already has join pending updates
+//              for some peers. Export policy accepts the route for other join
+//              peers with same attribute, but different label.
+//              Common attribute for initial join peers and new join peers.
+//              Different labels for initial join peers and new join peers.
+//              After this the route is exported to all peers with the same
+//              attribute and the label. The attribute and label are different
+//              than those for both the initial and new join peers.
+//
+// Old DBState: RouteUpdate in QBULK.
+//              No AdvertiseInfo.
+//              UpdateInfo peer x=[vJoinPeerCount,kPeerCount-1],
+//              attr A + label 100.
+// Join Peers:  Peers x=[0,vJoinPeerCount-1].
+// Export Rslt1:Accept peer x=[0,vJoinPeerCount-1], attr A + label 200.
+// Export Rslt2:Accept peer x=[0,kPeerCount-1], attr B + label 300.
+// New DBState: RouteUpdate in QBULK.
+//              No AdvertiseInfo.
+//              UpdateInfo peer x=[0,kPeerCount-1], attr B + label 300.
+//
+TEST_F(BgpExportRouteUpdateTest1, JoinMerge5c) {
+    int qid = RibOutUpdates::QBULK;
+    for (int vJoinPeerCount = 1; vJoinPeerCount < kPeerCount;
+            vJoinPeerCount++) {
+        InitUpdateInfo(attrA_, 100, vJoinPeerCount, kPeerCount-1, qid);
+        Initialize();
+
+        RibPeerSet join_peerset;
+        BuildPeerSet(join_peerset, 0, vJoinPeerCount-1);
+        BuildExportResult(attrA_, 0, vJoinPeerCount-1, 200);
+        RunJoin(join_peerset);
+        table_.VerifyExportResult(true);
+
+        RouteUpdate *rt_update = ExpectRouteUpdate(&rt_, qid);
+        VerifyRouteUpdateDequeueEnqueue(rt_update);
+        EXPECT_EQ(rt_update_, rt_update);
+        VerifyUpdates(rt_update, attrA_, 100, vJoinPeerCount, kPeerCount-1, 2);
+        VerifyUpdates(rt_update, attrA_, 200, 0, vJoinPeerCount-1, 2);
+        VerifyHistory(rt_update);
+
+        BuildExportResult(attrB_, 0, kPeerCount-1, 300);
+        RunExport();
+        table_.VerifyExportResult(true);
+
+        rt_update = ExpectRouteUpdate(&rt_, RibOutUpdates::QUPDATE);
+        VerifyRouteUpdateDequeueEnqueue(rt_update);
+        EXPECT_EQ(rt_update_, rt_update);
+        VerifyUpdates(rt_update, attrB_, 300, 0, kPeerCount-1);
+        VerifyHistory(rt_update);
+
+        DrainAndDeleteRouteState(&rt_);
+    }
+}
+
 //
 // Description: Leave processing for route that has scheduled updates or join
 //              pending. The leave peerset does not overlap with the peers that