Having different merge strategies for OmegaConf

[Hammit]

I'm evaluating different config libraries/systems and nothing currently does quite what I want, which is to be able to merge configs in a controlled manner and in different ways. At work we have our own config system that has been developed that caters to this but I'd like to replace if for a pure Python solution.

I see DynaConf can merge, but only top level keys. OmegaConf is better in that respect in that it will merge deeply nested structures, however the config being merging in always overwrites the config to merge into. If a list is being merged, sometimes I would like concat/append and sometimes I would like to overwrite it completely.

I've got a patch to add that feature, just to explore the codebase and possibilities, however I'm not sure if this is something that would be wanted. Something to control the merge strategy being used.

[odelalleau]

Yeah, merging lists is a recurring pain point. I'm pretty sure there have been discussions on this topic (possibly in Hydra) but I can't easily find them. This SO post (https://stackoverflow.com/questions/61315623/omegaconf-can-i-influence-how-lists-are-merged) has a link to #131 as a potential way to control such behavior with type annotations in structured configs.

Another use case of adding more control on the merge behavior would be the ability to completely overwrite a DictConfig node (essentially deleting this node from the config being merged into before doing the merge).

I think overall it's definitely a useful thing to have, but it's a bit unclear what's the best way to implement it. Feel free to suggest ideas though!

[Hammit]

hmmm...Interesting...

The way I've implemented it so far is to have a special token in the list being merged in (currently "OmegaConf.list_merge_append") that triggers the desired behaviour in BaseContainer._list_merge. This token is excluded from the merge.

diff --git a/omegaconf/basecontainer.py b/omegaconf/basecontainer.py
index a83907f..4de9dfb 100644
--- a/omegaconf/basecontainer.py
+++ b/omegaconf/basecontainer.py
@@ -461,7 +461,11 @@ class BaseContainer(Container, ABC):
         elif src._is_interpolation():
             dest._set_value(src._value())
         else:
-            temp_target = ListConfig(content=[], parent=dest._get_parent())
+            if src[0].lower() == 'omegaconf.list_merge_append':
+                content = list(dest)
+            else:
+                content = []
+            temp_target = ListConfig(content=content, parent=dest._get_parent())
             temp_target.__dict__["_metadata"] = copy.deepcopy(
                 dest.__dict__["_metadata"]
             )
@@ -474,7 +478,11 @@ class BaseContainer(Container, ABC):
                     temp_target.append(item)
             else:
                 for item in src._iter_ex(resolve=False):
-                    temp_target.append(item)
+                    if type(item) is str:
+                        if item.lower() != 'omegaconf.list_merge_append':
+                            temp_target.append(item)
+                    else:
+                        temp_target.append(item)

             if list_merge_mode == ListMergeMode.EXTEND:
                 dest.__dict__["_content"].extend(temp_target.__dict__["_content"])

I do like the idea of resolvers though in that there is a hook to add custom functionality. Some sort of callback/hook provided by the user when merging could be useful?

[odelalleau]

The way I've implemented it so far is to have a special token in the list being merged in (currently "OmegaConf.list_merge_append") that triggers the desired behaviour in BaseContainer._list_merge. This token is excluded from the merge.

Makes sense -- feels somewhat hacky but I think it'll be hard to come up with a solution that doesn't.

I do like the idea of resolvers though in that there is a hook to add custom functionality. Some sort of callback/hook provided by the user when merging could be useful?

Not sure how resolvers can help here. Can you clarify?

Overall I feel like what we need is a way to attach metadata to a config node, through the config itself. Ideally this is something that would disappear from the config once created, but would be preserved internally (there is already internal metadata attached to config nodes, that includes flag, but you can't control it through the config).

Alternatively, there could also be an option of the merge() function that allows specifying a different merge behavior, possibly only for specific nodes. This would be simpler, but more limited (in particular it'd be hard to take advantage of it in Hydra).

[Hammit]

When I mentioned resolvers I wasn't be literal but more or less just saying that resolvers provide a way for the user to supply custom behaviour in the form of a callback function. Could that be an option here?

As for discussing the the specific internals of OmegaConf, I'm not sure I can be of assistance. All I did was patch the code to do what I want and apart from what I read to do that I understand very little. Sorry :)

[odelalleau]

So one way to leverage resolvers here would be to merge dicts rather than lists, then use oc.dict.values to convert the result into a list that would represent the “merged list”. I’m on my phone and can’t easily provide an example now, but let me know if you want one later.