Bind tb_logger tag to engine dataset name?

[aksg87]

In this example https://labs.quansight.org/blog/2020/09/pytorch-ignite/

# Attach handler to dump evaluator's metrics every epoch completed
for tag, evaluator in [("training", train_evaluator), ("validation", evaluator)]:
    tb_logger.attach_output_handler(
        evaluator,
        event_name=Events.EPOCH_COMPLETED,
        tag=tag,
        metric_names="all",
        global_step_transform=global_step_from_engine(trainer),
    )

As an alternative, is there a nice way to bind the tag to the engine's dataloader instance/name? This would make it easier to manage the same engine running on different dataloaders?

[sdesrozis]

@aksg87 Thank you for this question.

The tag should be given explicitly at the handler attachment. However, your feedback about our API will be considered with a strong interest! Feel free to provide a few suggestions or improvements 😊

Maybe @vfdev-5 or @trsvchn could have some ideas and thoughts about this topic 😉

[aksg87]

One idea I had was passing a lambda function as a tag that would map the dataloader instance in an engine to a string. Would love to hear your thoughts on this!

Thanks for responding @sdesrozis

[sdesrozis]

Today, tag has to be a str but maybe we could consider callable in addition.

Another idea should to automatically have a tag whether it would be set. In that case, the tag argument would be completely optional (callable or not). The tag could be created regarding to the dataloader class / instance, maybe a dict located wherever. Just a thought, need digging.

[vfdev-5]

As an alternative, is there a nice way to bind the tag to the engine's dataloader instance/name? This would make it easier to manage the same engine running on different dataloaders?

There is a funny way to create a dynamic tag:

import torch
from ignite.engine import Engine, Events

evaluator = Engine(lambda e, b: print(b.item(), end=" "))
train_loader = torch.tensor([1, 2, 3, 4, 5])
val_loader = torch.tensor([11, 21, 31, 41, 51])

def compute_metrics():
    evaluator.run(train_loader)
    evaluator.run(val_loader)

class DataTag:
    def __init__(self, engine):
        self.engine = engine
    def __repr__(self):
        return self.engine.state.dataloader.dname


train_loader.dname = "training"
val_loader.dname = "validation"

@evaluator.on(Events.EPOCH_COMPLETED, tag=DataTag(evaluator))
def show_tag(tag):
    print(f"\n{evaluator.state.dataloader.dname} : tag={tag}")

compute_metrics()
> 
1 2 3 4 5 
training : tag=training
11 21 31 41 51 
validation : tag=validation

Same would work with tensorboard logger:

    class DataTag:        
        def __init__(self, engine):
            self.engine = engine            
        def __repr__(self):
            return self.engine.state.dataloader.dname
    
    
    train_loader.dname = "training"
    val_loader.dname = "validation"
    tb_logger.attach_output_handler(
        train_evaluator,
        event_name=Events.EPOCH_COMPLETED,
        tag=DataTag(train_evaluator),
        metric_names=["loss", "accuracy"],
        global_step_transform=global_step_from_engine(trainer),
    )

[sdesrozis]

Ah ah !! Clever solution using python tricks.

[sdesrozis]

Small variation of the @vfdev-5 crazy solution using an external dict rather than hacking the dataloaders.

class DataTag:
    def __repr__(self):
        return tag_mapping[evaluator.state.dataloader]

tag_mapping = {
    train_loader: "training",
    val_loader: "validation"
}

@evaluator.on(Events.EPOCH_COMPLETED, tag=DataTag())
def show_tag(tag):
    print(f"\n{tag_mapping[evaluator.state.dataloader]} : tag={tag}")

Used with

[aksg87]

Thanks! @vfdev-5

I really appreciate your solution. Nice to know this tip for the future