refactor prep for package rbf #30072

instagibbs · 2024-05-09T13:58:19Z

First few commits of #28984 to set the stage for the package RBF logic.

These refactors are preparation for evaluating an RBF in a multi-proposed-transaction context instead of only a single proposed transaction. Also, carveouts and sibling evictions only should work in single RBF cases so add logic to preclude multi-tx cases in the future.

No behavior changes aside from bailing earlier from failed carve-outs.

DrahtBot · 2024-05-09T13:58:22Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	glozow, sr-gi
Stale ACK	sdaftuar, ismaelsadeeq, theStack

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#29680 (wallet: fix unrelated parent conflict doesn't cause child tx to be marked as conflict by Eunovo)
#29496 (policy: bump TX_MAX_STANDARD_VERSION to 3 by glozow)
#29325 (consensus: Store transaction nVersion as uint32_t by achow101)
#26593 (tracing: Only prepare tracepoint arguments when actually tracing by 0xB10C)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

instagibbs · 2024-05-09T13:58:36Z

suggested by @glozow , can close if this ends up not being helpful

glozow

ACK ff7c71f. Code review + checked that the members get cleared properly in between subpackage evals.

src/validation.cpp

test/functional/mempool_package_onemore.py

ismaelsadeeq

Code Review ACK 6a39183

glozow

reACK 6a39183

src/validation.cpp

instagibbs · 2024-05-15T12:04:50Z

rebased due to merge conflict

glozow · 2024-05-15T12:47:51Z

reACK 186a00e via range-diff

ismaelsadeeq · 2024-05-15T17:18:28Z

re-ACK 186a00e

src/validation.cpp

instagibbs · 2024-05-20T14:15:23Z

src/validation.cpp

-    // because it's unnecessary. Also, CPFP carve out can increase the limit for individual
-    // transactions, but this exemption is not extended to packages in CheckPackageLimits().
-    std::string err_string;
+    // because it's unnecessary.


@sr-gi I reverted the logic change here to reduce review surface, just shortening the comment here instead.

instagibbs · 2024-05-20T15:25:41Z

win64 test failure is minisketch https://github.com/bitcoin/bitcoin/pull/30072/checks?check_run_id=25181774728

glozow

lgtm ACK 6fcd3cc

src/validation.cpp

theStack

Code-review ACK 6fcd3cc

Out of curiosity, I verified that the extended carve-out functional test in 868c3e7 (checking testmempoolaccept and submitpackage results) would also fail on master, but in a slightly different way, as the carve out leads to a failure later (in CheckPackageLimits() vs the earlier PreChecks() for testmempoolaccept; for submitpackage the first tx succeeds, and only the second fails with "too-long-mempool-chain"):

diff

diff --git a/test/functional/mempool_package_onemore.py b/test/functional/mempool_package_onemore.py
index 76e5ad2ff8..a6883f09fe 100755
--- a/test/functional/mempool_package_onemore.py
+++ b/test/functional/mempool_package_onemore.py
@@ -59,10 +59,10 @@ class MempoolPackagesTest(BitcoinTestFramework):
         replaceable_tx = self.wallet.create_self_transfer_multi(utxos_to_spend=[chain[0]])
         txns = [replaceable_tx["tx"], self.wallet.create_self_transfer_multi(utxos_to_spend=replaceable_tx["new_utxos"])["tx"]]
         txns_hex = [tx.serialize().hex() for tx in txns]
-        assert_equal(self.nodes[0].testmempoolaccept(txns_hex)[0]["reject-reason"], "too-long-mempool-chain")
+        assert "package-mempool-limits" in self.nodes[0].testmempoolaccept(txns_hex)[0]["package-error"]
         pkg_result = self.nodes[0].submitpackage(txns_hex)
-        assert "too-long-mempool-chain" in pkg_result["tx-results"][txns[0].getwtxid()]["error"]
-        assert_equal(pkg_result["tx-results"][txns[1].getwtxid()]["error"], "bad-txns-inputs-missingorspent")
+        assert "error" not in pkg_result["tx-results"][txns[0].getwtxid()]
+        assert "too-long-mempool-chain" in pkg_result["tx-results"][txns[1].getwtxid()]["error"]
         # But not if it chains directly off the first transaction
         self.nodes[0].sendrawtransaction(replaceable_tx["hex"])
         # and the second chain should work just fine

src/validation.cpp

theStack

re-ACK a84b574 📦

theStack · 2024-05-22T15:10:08Z

src/validation.cpp

+            // If we are using package feerates, we must be doing package submission.
+            // It also means carveouts and sibling eviction are not permitted.
+            if (m_package_feerates) {
+                Assume(m_package_submission);
+                Assume(!m_allow_carveouts);
+                Assume(!m_allow_sibling_eviction);
+            }
+            if (m_allow_sibling_eviction) Assume(m_allow_replacement);


Kind of unrelated, and more like a "I wish my c++ skills were better" nit: I wonder if there is a way to do the introduced sanity checks already at compile-time, since these flags are all constant for each of the ATMPArgs instances (probably not, or at least not without significant changes in structure...).

sdaftuar · 2024-05-22T18:45:17Z

src/validation.cpp

+    size_t m_conflicting_size{0};
+
+    /** Re-set sub-package state to not leak between evaluations */
+    void ClearSubPackageState() EXCLUSIVE_LOCKS_REQUIRED(cs_main, m_pool.cs)


Do you think it might make sense to explicitly group these variables into a single struct that makes it clear we have one grouping of per-subpackage state? All these variables that are at the same scope level as other things that don't get reset seems maybe a bit messy...

Probably worth doing now to make it more clear over time. Let me know what you think.

Seems more readable to me, thanks.

Looks like the commit message for commit 6b33732 should be updated now, sorry!

touched up the commit message

sdaftuar · 2024-05-22T18:52:47Z

utACK a84b574

Left one nit/question for you, but looks good to me regardless.

sdaftuar · 2024-05-23T13:48:47Z

ACK 63cfa4c

ismaelsadeeq

re-ACK 63cfa4c

theStack

re-ACK 63cfa4c

glozow

ACK 63cfa4c

glozow · 2024-05-23T14:44:56Z

src/validation.cpp

+        size_t m_conflicting_size{0};
+    };
+
+    struct SubPackageState m_subpackage;


Is there a reason for struct (seems like a C thing?)

Suggested change

struct SubPackageState m_subpackage;

SubPackageState m_subpackage;

old habits!

glozow · 2024-05-23T14:50:25Z

src/validation.cpp

@@ -1380,6 +1438,7 @@ PackageMempoolAcceptResult MemPoolAccept::AcceptMultipleTransactions(const std::
        // replacements, we don't need to track the coins spent. Note that this logic will need to be
        // updated if package replace-by-fee is allowed in the future.
        assert(!args.m_allow_replacement);
+        assert(!m_subpackage.m_rbf);


Seems this will need to be deleted immediately in the next PR?

The behavior is not new, but this rule exits earlier than before. Previously, a carve out could have been granted in PreChecks() but then nullified in PackageMempoolChecks() when CheckPackageLimits() is called with the default limits.

No change in behavior. For single transaction acceptance, this is a simple refactor: Workspace::m_all_conflicting Workspace::m_conflicting_fees Workspace::m_conflicting_size Workspace::m_replaced_transactions are now grouped under a new SubPackageState struct that is a member of MemPoolAccept. And local variables m_total_vsize and m_total_modified_fees are now SubpackageState members so they can be accessed from PackageMempoolChecks. We want these to be package-wide variables because - Transactions could conflict with the same tx (just not the same prevout), or their conflicts could share descendants. - We want to compare conflicts with the package fee rather than individual transaction fee. We reset these MemPoolAccept-wide fields for each subpackage evaluation to not cause state leaking, similar to temporary coins.

instagibbs · 2024-05-23T16:11:22Z

sadly, forced to rebased post #29873

Should be pretty trivial to review with something like:
git range-diff master 63cfa4c 2fd34ba504957331f5a08614b6e1f8317260f04d

glozow · 2024-05-23T16:14:54Z

reACK 2fd34ba via range-diff

murchandamus · 2024-05-23T17:15:12Z

src/validation.cpp

+        /** When true, allow sibling eviction. This only occurs in single transaction package settings. */
+        const bool m_allow_sibling_eviction;


I thought sibling eviction was relevant in a TRUC Transaction constellation when there already exist an unconfirmed parent and unconfirmed child, and now a newly submitted transaction is a child of the same parent, but spends a different output than the original child, i.e. not actually in conflict with the original child except for the topological restrictions affecting TRUC Transactions.

If I got all that right, I would posit that "This only occurs in single transaction package settings." is a bit confusing, unless I’m the only person for whom it’s not obvious that "single transaction package settings" is exclusively describing the replacement transaction.

"single transaction subpackage"?

sr-gi

utACK 2fd34ba

Sorry it took me a while to get familiar with the validation logic. I left some non-blocking comments.

sr-gi · 2024-05-16T19:12:36Z

src/validation.cpp

@@ -476,6 +476,9 @@ class MemPoolAccept
         */
        const std::optional<CFeeRate> m_client_maxfeerate;

+        /** Whether CPFP carveout and RBF carveout are granted. */


In 90b60a1

nit (not-blocking):

Suggested change

/** Whether CPFP carveout and RBF carveout are granted. */

/** Whether CPFP and RBF carveouts are granted. */

will touch if I have to retouch the PR

sr-gi · 2024-05-16T19:20:14Z

src/validation.cpp

@@ -948,6 +957,13 @@ bool MemPoolAccept::PreChecks(ATMPArgs& args, Workspace& ws)
        ws.m_ancestors = std::move(*ancestors);
    } else {
        // If CalculateMemPoolAncestors fails second time, we want the original error string.


In 90b60a1

I think this comment is not accurate (it was not all that accurate before that PR either). At this point, CalculateMemPoolAncestors has only been called once. The only case in which this caching applies is in the last else case of this context. I think this should be reworded

My reading of it is if it fails below for a second time, we will want to have cached the first error string.

either way, this PR hopefully doesn't make this confusion worse than status quo?

Yeah, I don't think it makes it worse

sr-gi · 2024-05-16T19:45:13Z

src/validation.cpp

-    std::string err_string;
-    if (txns.size() > 1 && !PackageMempoolChecks(txns, m_total_vsize, package_state)) {
+    // Apply package mempool ancestor/descendant limits.
+    if (!PackageMempoolChecks(txns, m_total_vsize, package_state)) {


In 90b60a1

If we are removing txns.size() > 1 because it is a truism, we should at least assume it to make sure we are not overlooking it

I reverted that change based on your feedback already :)

Oh sorry, this was left on my first pass, which I never submitted 😆

test/functional/mempool_package_onemore.py

sr-gi · 2024-05-23T18:13:21Z

src/validation.cpp

+        // Single transaction contexts only.
+        if (args.m_allow_sibling_eviction && err->second != nullptr) {
+            // We should only be considering where replacement is considered valid as well.
+            Assume(args.m_allow_replacement);


Is there any case in which m_allow_sibling_eviction is not shadowing m_allow_replacement? All the uses I see, they have the same value, or if one is set, the other is assumed.

Is that just the case now, and may it be different in a follow-up? Otherwise, what is the point of having two flags?

In package RBF this changes: f93b602#diff-97c3a52bc5fad452d82670a7fd291800bae20c7bc35bb82686c2c0a4ea7b5b98R528

We don't want sibling eviction attempts to happen during package rbf logic

glozow reviewed May 9, 2024

View reviewed changes

src/validation.cpp Outdated Show resolved Hide resolved

src/validation.cpp Outdated Show resolved Hide resolved

test/functional/mempool_package_onemore.py Show resolved Hide resolved

instagibbs force-pushed the 2024-05-package_rbf_prep branch 2 times, most recently from ed7e6cc to 6a39183 Compare May 9, 2024 16:58

ismaelsadeeq reviewed May 10, 2024

View reviewed changes

DrahtBot requested a review from glozow May 10, 2024 10:58

glozow reviewed May 10, 2024

View reviewed changes

src/validation.cpp Outdated Show resolved Hide resolved

DrahtBot added the Needs rebase label May 15, 2024

glozow mentioned this pull request May 15, 2024

Package Relay Project Tracking #27463

Open

56 tasks

instagibbs force-pushed the 2024-05-package_rbf_prep branch from 6a39183 to 186a00e Compare May 15, 2024 12:01

DrahtBot removed the Needs rebase label May 15, 2024

DrahtBot requested a review from ismaelsadeeq May 15, 2024 12:47

laanwj added the Mempool label May 16, 2024

instagibbs mentioned this pull request May 16, 2024

Cluster size 2 package rbf #28984

Open

instagibbs commented May 16, 2024

View reviewed changes

src/validation.cpp Outdated Show resolved Hide resolved

instagibbs force-pushed the 2024-05-package_rbf_prep branch from 186a00e to 6fcd3cc Compare May 20, 2024 14:10

instagibbs commented May 20, 2024

View reviewed changes

DrahtBot added CI failed and removed CI failed labels May 20, 2024

glozow reviewed May 21, 2024

View reviewed changes

src/validation.cpp Outdated Show resolved Hide resolved

theStack approved these changes May 21, 2024

View reviewed changes

src/validation.cpp Outdated Show resolved Hide resolved

instagibbs force-pushed the 2024-05-package_rbf_prep branch from 6fcd3cc to a84b574 Compare May 22, 2024 14:13

theStack approved these changes May 22, 2024

View reviewed changes

DrahtBot requested a review from glozow May 22, 2024 15:11

sdaftuar reviewed May 22, 2024

View reviewed changes

instagibbs force-pushed the 2024-05-package_rbf_prep branch 2 times, most recently from 7689438 to 63cfa4c Compare May 23, 2024 12:50

DrahtBot requested a review from theStack May 23, 2024 13:48

ismaelsadeeq reviewed May 23, 2024

View reviewed changes

theStack approved these changes May 23, 2024

View reviewed changes

glozow reviewed May 23, 2024

View reviewed changes

instagibbs and others added 5 commits May 23, 2024 12:08

Add description for m_test_accept

57ee302

Add m_allow_sibling_eviction as separate ATMPArgs flag

69f7ab0

cpfp carveout is excluded in packages

cbbfe71

The behavior is not new, but this rule exits earlier than before. Previously, a carve out could have been granted in PreChecks() but then nullified in PackageMempoolChecks() when CheckPackageLimits() is called with the default limits.

Add sanity checks for various ATMPArgs booleans

2fd34ba

instagibbs force-pushed the 2024-05-package_rbf_prep branch from 63cfa4c to 2fd34ba Compare May 23, 2024 16:10

DrahtBot requested review from sdaftuar, theStack and ismaelsadeeq May 23, 2024 16:14

DrahtBot mentioned this pull request May 23, 2024

policy: restrict all TRUC (v3) transactions to 10kvB #29873

Merged

murchandamus reviewed May 23, 2024

View reviewed changes

sr-gi reviewed May 23, 2024

View reviewed changes

	struct SubPackageState m_subpackage;
	SubPackageState m_subpackage;

		/** When true, allow sibling eviction. This only occurs in single transaction package settings. */
		const bool m_allow_sibling_eviction;

	/** Whether CPFP carveout and RBF carveout are granted. */
	/** Whether CPFP and RBF carveouts are granted. */

refactor prep for package rbf #30072

Are you sure you want to change the base?

refactor prep for package rbf #30072

Conversation

instagibbs commented May 9, 2024 • edited

DrahtBot commented May 9, 2024 • edited

Code Coverage

Reviews

Conflicts

instagibbs commented May 9, 2024

glozow left a comment

Choose a reason for hiding this comment

ismaelsadeeq left a comment

Choose a reason for hiding this comment

glozow left a comment

Choose a reason for hiding this comment

instagibbs commented May 15, 2024

glozow commented May 15, 2024

ismaelsadeeq commented May 15, 2024

Choose a reason for hiding this comment

instagibbs commented May 20, 2024

glozow left a comment

Choose a reason for hiding this comment

theStack left a comment

Choose a reason for hiding this comment

theStack left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdaftuar commented May 22, 2024

sdaftuar commented May 23, 2024

ismaelsadeeq left a comment

Choose a reason for hiding this comment

theStack left a comment

Choose a reason for hiding this comment

glozow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

instagibbs commented May 23, 2024

glozow commented May 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sr-gi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

instagibbs commented May 9, 2024 •

edited

DrahtBot commented May 9, 2024 •

edited