[Feat] Add Local Search for Solution Improvement #140

hyeok9855 · 2024-03-15T15:42:39Z

Open for Review; Do Not Merge for Now!

Description

Add local search operators as a post-processing to improve a given solution.
Here, we implement 2-opt for TSP and LocalSearch operator provided by PyVRP for CVRP and CVRPTW.
For other problems (e.g., PDP or scheduling), we couldn't find such a plug-and-play local search operator, and we're looking for contributions to local search for other problems!

Note that I also made some refactorings regarding type hinting.

Motivation and Context

Local search is an essential component for an enhanced CO solver. Many research projects in the NCO field utilize local search operators, such as our recent work GFACS, which trains NN using the solution refined by local search in an off-policy manner.

I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds core functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (update in the documentation)
Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

My change requires a change to the documentation.
I have updated the tests accordingly (required for a bug fix or a new feature).
I have updated the documentation accordingly.

rl4co/envs/common/base.py

rl4co/envs/routing/tsp.py

fedebotu · 2024-03-16T03:05:57Z

rl4co/envs/routing/tsp.py

@@ -161,6 +174,50 @@ def check_solution_validity(td: TensorDict, actions: torch.Tensor):
            == actions.data.sort(1)[0]
        ).all(), "Invalid tour"

+    @staticmethod
+    def improve_solution(td: TensorDict, actions: torch.Tensor, **kwargs) -> torch.Tensor:


(as per above, we can call this local_search)

CC: @leonlan this is what we mentioned during our meeting about having an optional local search operator API in RL4CO with pyvrp ; any thoughts about this?

We are unfortunately dropping support for the TSP-based TwoOpt in the next release (v0.8.0), because this operator is not very effective in VRPs. I spoke to @fedebotu in Slack and these are my suggestions:

If you want a dedicated TwoOpt operator for TSP, you can try to implement this yourself. This should not be too hard (GPT-4 can give the right solution) and you can make this very performant when implemented in say e.g. Cython. Happy to take a look when you have an implementation.

If you want to have local search support for VRPs, the current setup can serve as a good basis. However, some details (e.g., which penalties to set in the CostEvaluator?) need to be addressed. You can always ping me to have a look when you have an implementation.

For now, I assume that you continue to plan using PyVRP v0.7.0 which has support for TwoOpt.

Thank you for your suggestion!
As you suggested, I will replace the pyvrp TwoOpt with another implementation, and accordingly, I'll change the PyVRP version to the latest one.

For other problems like CVRP(TW) or PDP, I think I should use PyVRP, so I will let you know if there's anything to discuss.

About PyVRP (CVRP, CVRPTW, new variants): we can help with that!
About PDP: from what I know, one-on-one pickup and delivery problems are not available right now, is that correct @leonlan ?

One-to-one PDP is not supported indeed.

Do you have any plan for supporting one-to-one pdp by any chance?
If so, when do you think will it be?

There are no plans for supporting PDP in the near future.

fedebotu · 2024-03-16T03:07:24Z

tests/test_envs.py

@@ -48,6 +48,10 @@ def test_routing(env_cls, batch_size=2, size=20):
    env = env_cls(num_loc=size)
    reward, td, actions = rollout(env, env.reset(batch_size=[batch_size]), random_policy)
    env.render(td, actions)
+    try:
+        env.improve_solution(td, actions)


For now this is fine, for the future I think we can move the local_search part in a separate test as it is optional

fedebotu · 2024-03-16T03:19:24Z

Great job! 🚀

Another detail I cannot review above: tt seems that the installation fails for Python=3.8. This should be because pyvrp is only available for Python 3.9 onwards (reference), so maybe we can skip its for that case

leonlan · 2024-03-16T09:10:50Z

rl4co/envs/routing/tsp.py

@@ -161,6 +174,50 @@ def check_solution_validity(td: TensorDict, actions: torch.Tensor):
            == actions.data.sort(1)[0]
        ).all(), "Invalid tour"

+    @staticmethod
+    def improve_solution(td: TensorDict, actions: torch.Tensor, **kwargs) -> torch.Tensor:


We are unfortunately dropping support for the TSP-based TwoOpt in the next release (v0.8.0), because this operator is not very effective in VRPs. I spoke to @fedebotu in Slack and these are my suggestions:

If you want a dedicated TwoOpt operator for TSP, you can try to implement this yourself. This should not be too hard (GPT-4 can give the right solution) and you can make this very performant when implemented in say e.g. Cython. Happy to take a look when you have an implementation.

If you want to have local search support for VRPs, the current setup can serve as a good basis. However, some details (e.g., which penalties to set in the CostEvaluator?) need to be addressed. You can always ping me to have a look when you have an implementation.

For now, I assume that you continue to plan using PyVRP v0.7.0 which has support for TwoOpt.

rl4co/envs/routing/tsp.py

pyproject.toml

rl4co/envs/routing/tsp.py

rl4co/envs/common/base.py

…ption to DeepACO

hyeok9855 · 2024-05-01T19:04:36Z

Changes

Merged the updated main branch
Added the reward augmentation with local search, which is one of the key components of DeepACO.

hyeok9855 · 2024-05-31T22:20:06Z

Changes

Merge the main branch (c.f., [BugFix] Fix the performance issue of DeepACO #170)
Increase the performance further (now it's very close to the performance of the original implementation of DeepACO)
Add CVRP local search based on PyVRP

@fedebotu @leonlan
If you have time, please review the renewed code!

leonlan · 2024-06-01T05:51:44Z

rl4co/envs/routing/cvrp/local_search.py

+    improved_solution, is_feasible = perform_local_search(
+        ls_operator,
+        solution,
+        int(load_penalty * 10**4),  # * 10**4 as we scale the data by 10**4 in `make_data`


Suggested change

int(load_penalty * 10**4), # * 10**4 as we scale the data by 10**4 in `make_data`

int(load_penalty * 10**4), # * 10**4 as we scale the data by 10**4 in `make_data`

I suggest making this value a CONSTANT variable and use it make_data to. Perhaps also add a comment to the CONSTANT explaining why it's needed to scale the data so that users can understand how to modify if needed.

Good point, thanks!

leonlan · 2024-06-01T05:52:06Z

Hi @hyeok9855, I don't have time to go over the code in full detail, but I had a quick glance at the PyVRP part. It looks good to me with a small comment on the constant.

fedebotu · 2024-06-01T06:56:35Z

rl4co/models/zoo/deepaco/antsystem.py

@@ -27,7 +29,10 @@ class AntSystem:
        pheromone: Initial pheromone matrix. Defaults to `torch.ones_like(log_heuristic)`.
        require_logprobs: Whether to require the log probability of actions. Defaults to False.
        use_local_search: Whether to use local_search provided by the env. Default to False.
-        local_search_params: Arguments to be passed to the local_search function.
+        use_nls: Whether to use neural-guided local search provided by the env. Default to False.


What is the difference between use_nls and use_local_search?

nls indicates the local search with neural-guided perturbation, proposed by DeepACO. See here.

fedebotu · 2024-06-01T07:01:07Z

rl4co/envs/routing/tsp/env.py

@@ -15,6 +15,7 @@
 from rl4co.utils.pylogger import get_pylogger

 from .generator import TSPGenerator
+from .local_search import local_search


local_search has numba as a dependency (which is optional), so we could do something like:

try: from .local_search import local_search except: local_search = None

Then in env:

def local_search(...): assert local_search is not None, "Cannot import local_search module. Make sure to have `numba` installed"

Good, thanks!

fedebotu · 2024-06-01T07:04:04Z

rl4co/envs/routing/tsp/env.py

@@ -166,6 +167,19 @@ def check_solution_validity(td: TensorDict, actions: torch.Tensor):
            == actions.data.sort(1)[0]
        ).all(), "Invalid tour"

+    def generate_data(self, batch_size) -> TensorDict:


[Important] We replaced the generate_data with @cbhua this function with the more modular generator function:(e.g. here).

To generate data, we can call: env.generator(...) instead of env.generate_data

fedebotu · 2024-06-01T07:06:46Z

rl4co/models/zoo/deepaco/antsystem.py

+        return heuristic_dist
+
+    @staticmethod
+    def select_start_node_fn(


[Minor, for now]
By default, now we look for this function inside of the environment as done here, which is a bit more modular. But since we have to transfer this functions yet and is not a hard task, no need to do it now

fedebotu · 2024-06-01T07:08:45Z

rl4co/models/zoo/deepaco/antsystem.py

-        return actions, reward  # type: ignore
+        td_cpu = td.detach().cpu()  # Convert to CPU in advance to minimize the overhead from device transfer
+        td_cpu["distances"] = get_distance_matrix(td_cpu["locs"])
+        # TODO: avoid or generalize this, e.g., pre-compute for local search in each env


Yes, we can keep this as todo, but it should be generalized. I think this could be a common classmethod for routing environments

fedebotu · 2024-06-01T07:12:22Z

rl4co/models/zoo/nargnn/encoder.py


-        return heatmaps_logits
+        heatmap += 1e-10 if heatmap.dtype != torch.float16 else 3e-8


Nice, I see a huge trick here!
These values should ideally be constants at the top, e.g:

LOWEST_POSVAL_FP32 = 1e-10 LOWEST_POSVAL_FP16 = 3e-8

The lowest positive value for FP32 is not 1e-10 actually. It is much smaller than that, but 1e-10 is used in DeepACO.

fedebotu · 2024-06-01T07:13:33Z

tests/test_envs.py

@@ -79,6 +79,10 @@ def test_routing(env_cls, batch_size=2, size=20):
 def test_mtvrp(variant, batch_size=2, size=20):
    env = MTVRPEnv(generator_params=dict(num_loc=size, variant_preset=variant))
    reward, td, actions = rollout(env, env.reset(batch_size=[batch_size]), random_policy)
+    try:
+        env.local_search(td, actions)


Did you add the local search to this environment?

We should have this available for all 16 variants with the code @leonlan made

fedebotu · 2024-06-01T07:16:55Z

I think we should skip the local search testing with pyvrp for Python < 3.9, I think that is why testing failed

Like:

@pytest.mark.skipif(sys.version_info < (3, 9))
[...]

Maybe in the near future we could just remove testing Python 3.8 since it's a really old version anyways

fedebotu reviewed Mar 16, 2024

View reviewed changes

leonlan reviewed Mar 16, 2024

View reviewed changes

fedebotu mentioned this pull request Mar 18, 2024

[Feat] Add DeepACO #142

Merged

4 tasks

fedebotu reviewed Mar 18, 2024

View reviewed changes

rl4co/envs/common/base.py Outdated Show resolved Hide resolved

hyeok9855 added 4 commits April 26, 2024 21:28

correct wrong type hintings

b325dbb

Add 2-opt local search for TSPEnv

01f254c

Add tutorial notebook for solution improvement

12b702c

Replace pyvrp-based twoopt with custom one from DeepACO

7ac643d

hyeok9855 force-pushed the local-search branch from 95f24ea to 7ac643d Compare April 26, 2024 15:28

hyeok9855 added 3 commits April 27, 2024 02:28

Add neural-guided neural-guided perturbation and local search (NLS) o…

ccbcd9b

…ption to DeepACO

Merge branch 'main' into local-search

c91ca02

Implement local search based reward augmentation for DeepACO

6e91ecc

separate local search from env.py

77f2183

hyeok9855 force-pushed the local-search branch from c1c5637 to b6520cf Compare May 4, 2024 06:11

add DeepACO-style select_start_node_fn for multistart

d19e20c

hyeok9855 force-pushed the local-search branch from b6520cf to d19e20c Compare May 4, 2024 06:47

hyeok9855 mentioned this pull request May 17, 2024

[BugFix] Fix the performance issue of DeepACO #170

Merged

hyeok9855 added 6 commits May 28, 2024 20:56

Add manual advantage calculation

77defb7

Merge branch 'main' into local-search

a84e5e8

minor fix

6d65f15

resolve the performance drop

1b5f112

add pyvrp-based local search for CVRP

0bf80d5

minor refactoring

5b30322

hyeok9855 requested review from leonlan and fedebotu May 31, 2024 22:20

leonlan reviewed Jun 1, 2024

View reviewed changes

fedebotu reviewed Jun 1, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feat] Add Local Search for Solution Improvement #140

[Feat] Add Local Search for Solution Improvement #140

hyeok9855 commented Mar 15, 2024 •

edited

fedebotu Mar 16, 2024

leonlan Mar 16, 2024 •

edited

hyeok9855 Apr 26, 2024

fedebotu Apr 26, 2024

leonlan Apr 26, 2024

hyeok9855 May 1, 2024

leonlan May 1, 2024

fedebotu Mar 16, 2024

fedebotu commented Mar 16, 2024

leonlan Mar 16, 2024 •

edited

hyeok9855 commented May 1, 2024

hyeok9855 commented May 31, 2024

leonlan Jun 1, 2024

hyeok9855 Jun 1, 2024

leonlan commented Jun 1, 2024

fedebotu Jun 1, 2024

hyeok9855 Jun 1, 2024

fedebotu Jun 1, 2024

hyeok9855 Jun 1, 2024

fedebotu Jun 1, 2024

fedebotu Jun 1, 2024

fedebotu Jun 1, 2024

fedebotu Jun 1, 2024

hyeok9855 Jun 1, 2024

fedebotu Jun 1, 2024

fedebotu commented Jun 1, 2024

	int(load_penalty * 10*4), # 104 as we scale the data by 104 in `make_data`
	int(load_penalty * 10*4), # 104 as we scale the data by 104 in `make_data`


		return heatmaps_logits
		heatmap += 1e-10 if heatmap.dtype != torch.float16 else 3e-8

[Feat] Add Local Search for Solution Improvement #140

Are you sure you want to change the base?

[Feat] Add Local Search for Solution Improvement #140

Conversation

hyeok9855 commented Mar 15, 2024 • edited

Open for Review; Do Not Merge for Now!

Description

Motivation and Context

Types of changes

Checklist

Choose a reason for hiding this comment

leonlan Mar 16, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedebotu commented Mar 16, 2024

leonlan Mar 16, 2024 • edited

Choose a reason for hiding this comment

hyeok9855 commented May 1, 2024

Changes

hyeok9855 commented May 31, 2024

Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leonlan commented Jun 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedebotu commented Jun 1, 2024

hyeok9855 commented Mar 15, 2024 •

edited

leonlan Mar 16, 2024 •

edited

leonlan Mar 16, 2024 •

edited