feat: implement dynamic pruning strategy #1295

morph-dev · 2024-05-10T15:51:26Z

What was wrong?

Current pruning logic for IdIndexedV1Store is to delete up to 5% of storage, but no more than 100 elements. However, even 100 elements is too slow on big databases (e.g. 35GB that are currently used). It can take up to a second and sometimes even longer. This wouldn't be such a big problem if this wouldn't block other read/writes.

How was it fixed?

The idea is to have dynamic number of items that we delete. The PruningConfig allows us to set parameters, for this dynamic strategy:

what is the optimal pruning duration - default 100-300 milliseconds
by how much to increase/decrease number of items to prune when pruning is faster/slower than optimal - default 20%

Also added logic for handling the zero storage capacity to prune the db (if non-empty) and set radius to zero.

To-Do

Clean up commit history and use conventional commits.

njgheorghita · 2024-05-14T19:40:54Z

It seems like this might be relevant... Taking into account #1228 and that (IMO) we will sooner rather than later want to support this, along with a "max db" size (aka a user has 100 gbs of disk space they want to contribute, we'll actually spin up 10 nodes of 10gbs under the hood). Once that's implemented, we'll have a consistent max db size, which means we really don't have to worry about improving the pruning mechanism to support large dbs and can optimize db performance against a fixed target (eg a 10gb storage). Assuming that my description is the direction we're heading, does this pr still make sense?

pipermerriam · 2024-05-14T19:48:05Z

we'll actually spin up 10 nodes of 10gbs under the hood)

I don't think this is how the architecture will actually work out. I think that under the hood it will be a single 100gb database that is represented in the network across 10 different node-ids.

See: ethereum/portal-network-specs#283 (comment)

morph-dev · 2024-05-15T09:06:50Z

Together with what Piper said, I would add that there is benefit for smaller dbs as well (as we can purge more there).

In general, I would say it's better to monitor and adjust in any scenario. For example:

target of 10GB will perform differently on different devices (e.g. SSD vs HD)
even if you have 1 node of 10GB (i.e. our fixed target), you might also run execution+consensus client and your disk usage is already high

carver

LGTM! Mostly just nit comments that you're welcome to ignore.

I apparently have a strong opinion about the ideal way to calculate the percentage update of the pruning number (more below). But in the end, it's not a blocker because even if the number isn't ideal, I don't foresee any catastrophe. So don't let me stop you from merging it if you disagree about the percentage update 😆

carver · 2024-05-20T20:48:48Z

trin-storage/src/versioned/id_indexed_v1/store.rs

+            self.radius = Distance::ZERO;
+            self.metrics.report_radius(self.radius);


Tiniest of nits: maybe it's worth skipping reporting the radius if it was already 0.

I'm confused with "if it was already 0." part. This functions is called only once during initialization, so it wasn't anything useful at this point (it is MAX by default).

I also believe that it's still useful to report it at least once, so we have at least some data in grafana. Or do you want on purpose not to report it to grafana?

carver · 2024-05-20T20:57:17Z

trin-storage/src/versioned/id_indexed_v1/store.rs

+        if !self.pruning_strategy.should_prune(&self.usage_stats) {
            warn!(Db = %self.config.content_type,
-                "Pruning requested but we are below target capacity. Skipping");
+                "Pruning requested but not needed. Skipping");


Is this ever showing up? It seems strange that the call to prune only happens if should_prune() is true and then prune immediately checks it again inside.

It shouldn't show up, and with current code, it can't (as you said, should_prune is always called right before prune).
I put it in case there is some refactoring or some other changes to the code, in which case we will see it in the logs (therefore warn level, indicating that something got wrong).

carver · 2024-05-20T21:08:20Z

trin-storage/src/versioned/id_indexed_v1/store.rs

            self.metrics.stop_process_timer(delete_timer);
+            self.pruning_strategy
+                .observe_pruning_duration(pruning_start_time.elapsed());


Seems like a bummer to have prune run its own Instant timer, when we're already calculating the time in the metrics. I suppose if stop_process_timer returned the duration it observed (looks like it's an f64 in seconds, but we could convert to a Duration), then observe_duration could reuse it. Do you agree that would be preferable? If you don't feel like doing that, I might do it after this merges, just so it's ready the next time we want to re-use the timing.

I was thinking about the same thing. But I didn't want to complicate this PR any further. I'm fine with doing it myself in the followup PR.

carver · 2024-05-20T21:25:08Z

trin-storage/src/versioned/id_indexed_v1/pruning_strategy.rs

+        max_pruning_count_change_fraction: f64,
+        optimal_pruning_duration_range: Range<Duration>,
+    ) -> Self {
+        if !(0.0..1.0).contains(&target_capacity_fraction) {


Nit: I suppose 1.0 is a valid choice here, which Range seems to exclude. Due to database compression, there was even a time when >1.0 choices might be valid. I haven't observed how SQL performs since the switchover. Anyway it all seems mostly moot since it's hard-coded for now, but just wanted to make a note that I think it would be fine to limit the input up to 2.0 or something, since it would just be about letting people play with configuration values eventually, I don't think it's a massive footgun that we need to panic to avoid letting them play with larger values.

Definitely while coding I had a mental model that target_capacity should we lower than storage_capacity.

Maybe code would work even if that's not the case, but I would say it doesn't makes much sense, because both storage capacity and target capacity represent the raw content size (content_id.len() + content_key.len() + content_value.len()), or at least approximation (there is some extra storage used for indexes and other smaller columns). Irrelevant, but I don't think that there is any compression happening at the moment.

With that being said, I will allow value 1.0. And once we have real use case for using values higher than 1, we can come back to it (in which case we would probably have to rewrite some of the logic).

carver · 2024-05-20T21:27:33Z

trin-storage/src/versioned/id_indexed_v1/pruning_strategy.rs

+                target_capacity_fraction
+            )
+        }
+        if !(0.0..1.0).contains(&max_pruning_count_change_fraction) {


Similarly, it's not totally obvious to me that 1 is a hard upper limit here. If someone wanted to play with 2x the pruning count change, my intuition isn't to panic to prevent them from playing with it.

Edit: I see where the number came from now, but stand by the above comment when paired with the suggestion below.

I actually wanted increasing the max_pruning_count to be harder than decreasing, because downside of being too low is not so bad (we prune less items more frequently) while downside of being too high is that db is blocked for longer.

I added better explanation to the max_pruning_count_change_fraction field.

However, if the behavior that you expected is more common way of doing it, I'm willing to change it in the followup PR.

carver · 2024-05-20T21:35:35Z

trin-storage/src/versioned/id_indexed_v1/pruning_strategy.rs

+                Db = %self.config.content_type,
+                "Pruning was too slow. Decreasing max_pruning_count",
+            );
+            1. - pruning_config.max_pruning_count_change_fraction


Ah, I see now where the upper limit of 1 came from. My intuition before getting to this code is that a change fraction of 1.0 would double on the upside and cut in half on the downside. Those seem like the more symmetric options to me. (ie~ one increase and one decrease takes you back to where you started)

In this approach, choosing a value of .9 nearly doubles on the upside, but is 1/10th on the downside, which is aggressively dropping but not that aggressively climbing (it takes more than 3 upside adjustments to make up for 1 downside adjustment). So if we want one configuration number to govern both directions, my intuition says that this line should be:

Suggested change

1. - pruning_config.max_pruning_count_change_fraction

1. / (1. + pruning_config.max_pruning_count_change_fraction)

ogenev

⛵

trin-storage/src/versioned/id_indexed_v1/pruning_strategy.rs

morph-dev

@carver I do disagree with the proposed changes about pruning number changes, but not strongly :). Willing to discuss it more before merging.

morph-dev · 2024-05-21T11:23:53Z

trin-storage/src/versioned/id_indexed_v1/store.rs

+            self.radius = Distance::ZERO;
+            self.metrics.report_radius(self.radius);


I'm confused with "if it was already 0." part. This functions is called only once during initialization, so it wasn't anything useful at this point (it is MAX by default).

I also believe that it's still useful to report it at least once, so we have at least some data in grafana. Or do you want on purpose not to report it to grafana?

morph-dev · 2024-05-21T11:28:30Z

trin-storage/src/versioned/id_indexed_v1/store.rs

+        if !self.pruning_strategy.should_prune(&self.usage_stats) {
            warn!(Db = %self.config.content_type,
-                "Pruning requested but we are below target capacity. Skipping");
+                "Pruning requested but not needed. Skipping");


It shouldn't show up, and with current code, it can't (as you said, should_prune is always called right before prune).
I put it in case there is some refactoring or some other changes to the code, in which case we will see it in the logs (therefore warn level, indicating that something got wrong).

morph-dev · 2024-05-21T11:32:26Z

trin-storage/src/versioned/id_indexed_v1/store.rs

            self.metrics.stop_process_timer(delete_timer);
+            self.pruning_strategy
+                .observe_pruning_duration(pruning_start_time.elapsed());


I was thinking about the same thing. But I didn't want to complicate this PR any further. I'm fine with doing it myself in the followup PR.

morph-dev · 2024-05-21T15:22:50Z

trin-storage/src/versioned/id_indexed_v1/pruning_strategy.rs

+        max_pruning_count_change_fraction: f64,
+        optimal_pruning_duration_range: Range<Duration>,
+    ) -> Self {
+        if !(0.0..1.0).contains(&target_capacity_fraction) {


Definitely while coding I had a mental model that target_capacity should we lower than storage_capacity.

Maybe code would work even if that's not the case, but I would say it doesn't makes much sense, because both storage capacity and target capacity represent the raw content size (content_id.len() + content_key.len() + content_value.len()), or at least approximation (there is some extra storage used for indexes and other smaller columns). Irrelevant, but I don't think that there is any compression happening at the moment.

With that being said, I will allow value 1.0. And once we have real use case for using values higher than 1, we can come back to it (in which case we would probably have to rewrite some of the logic).

morph-dev · 2024-05-21T15:38:51Z

trin-storage/src/versioned/id_indexed_v1/pruning_strategy.rs

+                target_capacity_fraction
+            )
+        }
+        if !(0.0..1.0).contains(&max_pruning_count_change_fraction) {


I actually wanted increasing the max_pruning_count to be harder than decreasing, because downside of being too low is not so bad (we prune less items more frequently) while downside of being too high is that db is blocked for longer.

I added better explanation to the max_pruning_count_change_fraction field.

However, if the behavior that you expected is more common way of doing it, I'm willing to change it in the followup PR.

morph-dev · 2024-05-22T05:33:33Z

@carver I do disagree with the proposed changes about pruning number changes, but not strongly :). Willing to discuss it more before merging.

I'm going to merge this and do a followup PR if needed.

morph-dev requested review from carver, njgheorghita, KolbyML and ogenev May 10, 2024 15:51

morph-dev self-assigned this May 10, 2024

morph-dev mentioned this pull request May 17, 2024

fix(storage): skip error log if storage config is 0 #1300

Closed

1 task

feat: implement dynamic pruning strategy

e3b1755

morph-dev force-pushed the pruning_strategy branch from 4ac42b6 to e3b1755 Compare May 18, 2024 20:59

carver approved these changes May 20, 2024

View reviewed changes

ogenev approved these changes May 21, 2024

View reviewed changes

njgheorghita approved these changes May 21, 2024

View reviewed changes

trin-storage/src/versioned/id_indexed_v1/pruning_strategy.rs Show resolved Hide resolved

morph-dev commented May 21, 2024

View reviewed changes

fix: pr comments

08d1c69

morph-dev force-pushed the pruning_strategy branch from bff7f5d to 08d1c69 Compare May 21, 2024 16:19

morph-dev merged commit 4d603d4 into ethereum:master May 22, 2024
8 checks passed

morph-dev deleted the pruning_strategy branch May 22, 2024 05:34

morph-dev mentioned this pull request May 22, 2024

refactor: make stop_process_timer return duration #1306

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement dynamic pruning strategy #1295

feat: implement dynamic pruning strategy #1295

morph-dev commented May 10, 2024 •

edited

njgheorghita commented May 14, 2024

pipermerriam commented May 14, 2024

morph-dev commented May 15, 2024

carver left a comment

carver May 20, 2024 •

edited

morph-dev May 21, 2024

carver May 20, 2024

morph-dev May 21, 2024

carver May 20, 2024

morph-dev May 21, 2024

carver May 20, 2024

morph-dev May 21, 2024

carver May 20, 2024

morph-dev May 21, 2024

carver May 20, 2024 •

edited

ogenev left a comment

morph-dev left a comment

morph-dev May 21, 2024

morph-dev May 21, 2024

morph-dev May 21, 2024

morph-dev May 21, 2024

morph-dev May 21, 2024

morph-dev commented May 22, 2024 •

edited

		self.radius = Distance::ZERO;
		self.metrics.report_radius(self.radius);

	1. - pruning_config.max_pruning_count_change_fraction
	1. / (1. + pruning_config.max_pruning_count_change_fraction)

feat: implement dynamic pruning strategy #1295

feat: implement dynamic pruning strategy #1295

Conversation

morph-dev commented May 10, 2024 • edited

What was wrong?

How was it fixed?

To-Do

njgheorghita commented May 14, 2024

pipermerriam commented May 14, 2024

morph-dev commented May 15, 2024

carver left a comment

Choose a reason for hiding this comment

carver May 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carver May 20, 2024 • edited

Choose a reason for hiding this comment

ogenev left a comment

Choose a reason for hiding this comment

morph-dev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

morph-dev commented May 22, 2024 • edited

morph-dev commented May 10, 2024 •

edited

carver May 20, 2024 •

edited

carver May 20, 2024 •

edited

morph-dev commented May 22, 2024 •

edited