The reasons behind v8 breaking changes

[mishamyte]

👋 Intro

Hello community!

The v8 release is going to go beta in a short time. 🏎️ This release will has the most breaking changes in Serilog.Sinks.Grafana.Loki lifetime.

A lot of you use this sink in your projects (we appreciate it a lot 💜). During this time authors of the sink have received a lot of questions, feature requests and bug reports. We have collected and analyzed a considerable number of feedbacks, use case scenarios etc. And at the certain time is became clearly that some things need to be done. So the main idea of this post to tell you about the reasons that led to taken decisions.

💡 Changing default text formatter to LokiJsonTextFormatter

LokiJsonTextFormatter has been introduced to sink's users a while ago and quickly became a favorite. In conjunction with Loki's 2.0+ parsing features it is a powerful instrument to work with structured data without losing performance, related to weak label configuration by user (also known as high cardinality problem).

Using this formatter you are receiving all metadata as a part of log entry and it could be easily parsed into labels with json pipeline. Data after parsing could be used for filtering, formatting and graphing.

We started recommending this instrument a while ago and from v8 release we are going to use it is a default formatter instead of MessageTemplateTextFormatter. Also configuration parameter outputTemplate , used by MessageTemplateTextFormatter was removed.

💡 Redesigning behavior of log level metadata and deleting ILabelAwareTextFormatter interface

The discussions about log level exists almost from the very beginning of this sink. Serilog's LogEvent provides LogEventLevel as a separate property.

This sink was designed with an idea that it is a sink for logger, so log level should always be provided as a part of log entry. To my surprise so many people came with "I haven't configured this label, why it is created?". I haven't changed this behavior a long time, cause it was one of core ideas. Also this label was not involved in the process of filtering and have appeared always.

That caused to the creation of ILabelAwareTextFormatter interface and createLevelLabel configuration property in the process of creating LokiJsonTextFormatter .

This was not an elegant decision, but helped us to avoid the duplication of level as labels (one from the sink, second - from parsed log entry) and didn't break a behavior of default formatter. Due to large number of questions, from v7 level label was not created by default. By this was not a salvation we needed.

From v8 we are considering log level as a usual log property. It is added to every log entry and could be mapped to the label, using the standard approach. According to this change interface ILabelAwareTextFormatter and createLevelLabel configuration property have become useless and were removed.

💡 Mapping log event properties to labels

Label is one of most important thinks in Loki. It is a key-value pair, used by Loki to organize events. The main use case of label is querying and filtering the data. By according to internals of Loki, incorrect use of labels led to high cardinality problem. And there are some best practices for labels.

First public version (v3) of this sink has only possibility to exclude some properties from mapping to the labels. This wasn't a good solution, cause you needed to add all exclusion rules explicitly, what becomes a hell with a huge amount of labels (especially in cases with external libs.

We have changed this behavior in v4. It brought us a filtrationMode and filtrationLabels, where you could select one of modes: Include/Exclude and define properties' names to map 'em to the labels. Or you could leave it blank and in this case you will face the high cardinality problem. We always recommended to use Include mode with some always present labels (such a app name). But in last discussion we have found that this behavior is straightforward, but has no sense.

From v8 we propose you a clear way - you define only properties that should be mapped into a labels (or none if you don't need it). For this case use a new propertiesAsLabels configuration parameters. The rest of properties would be passed to a formatted and should be included as a part of log entry.

💡 Properties, mapped to the labels won't be passed to the text formatter

This change is internal and will not be noticed by most of users. Before v8 LokiJsonTextFormatter has received a LogEvent and a list of labels via ILabelAwareTextFormatter. It used labels list to exclude properties, that are already mapped from payload body. This causes no property duplication (similar to level label case).

From now LokiJsonTextFormatter implements ITextFormatter interface and received only list of properties that were not mapped. So pay attention if you provide a custom ITextFormatter to the sink.

💡 Introducing IReservedPropertyRenamingStrategy

Issue #94 describes a interesting bug. In a process of messaging formatting, executed by LokiJsonTextFormatter there could be a problem if a log event property name (provided by your or worse - by external code) equals to some parameter names, used by sink (Message, MessageTemplate etc), two labels with the same name are generated. But only one of them is accepted by Loki.

So v7.1.1 fixes it, checking if keyword is in reserved keyword list. It renames original property name by adding underscore to the start of name.

This could be not acceptable but you organization's naming conventions. So we created a IReservedPropertyRenamingStrategy interface with a simple method string Rename(string originalName);, what gets a original property name and return renamed one. It is called in some cases (but won't limited to in the future):

1. In formatting scenarios (as described above) if property name equals to reserved keyword
2. In case of appending level property. If you have a property named level in your code, it would be renamed, using a provided strategy (this will recursively check for a collision - if output of rename is taken, it would rename it too)
   ⚠️ We strongly recommend to follow a property names guidelines and use PascalCase for properties in log message template.
   🐛 Known issue
   MessageTemplate property is not adjusted in this case and render would be wrong. We are looking the ways to solve this before v8 leaving beta

---

Please, let us know what you are thinking about this changes and give this version a chance to be tested in your business cases 🙏.

Glory to Ukraine! 🇺🇦

[gabrielcerutti]

do you have an estimate of when v8 is gonna be GA? good job and thanks!

[mishamyte]

Thank you!

I almost morally fixed and going to implement #103 in a next days. Than I think we will give a little time to test it. And I hope that release will be in 4-5 of July.

Sorry for delay, but was kicked out mentally by a war. Had no will to write the code at all (

[gabrielcerutti]

Hey! thank you for the quick response! hope the situation in Ukraine gets better soon :(

[mishamyte]

Hello once again, @gabrielcerutti )

I apologize for that release delay, but now v8.0.0 is officially live)

[gabrielcerutti]

Hey @mishamyte! No problem at all and thank you very much! :)

[mishamyte]

Hey folks,

v8.0.0 just was shipped to NuGet.
All docs were updated to the actual version (if you see a mess - please, notice me).

Bug #103 was not solved in that release, according to complexity, but was not classified as show-stopper.

I would love to hear your feedbacks and questions

[sven-n]

Hi, can you help me in upgrading?

I have configured it previously as follows:

// includeLabels: a string array with labels I want to include

.GrafanaLoki(
                uri: "http://loki:3100",
                filtrationLabels: includeLabels,
                filtrationMode: LokiLabelFiltrationMode.Include)

Since the new LokiLabel has a key and a value, I don't know what to put there as value. Can I leave it out? Is this correct?

.GrafanaLoki(
                uri: "http://loki:3100",
                labels: includeLabels.Select(labelString => new LokiLabel() { Key = labelString }))

Thank you :)

[mishamyte]

Hi, can you help me in upgrading?

I have configured it previously as follows:

// includeLabels: a string array with labels I want to include

.GrafanaLoki(
                uri: "http://loki:3100",
                filtrationLabels: includeLabels,
                filtrationMode: LokiLabelFiltrationMode.Include)

Since the new LokiLabel has a key and a value, I don't know what to put there as value. Can I leave it out? Is this correct?

.GrafanaLoki(
                uri: "http://loki:3100",
                labels: includeLabels.Select(labelString => new LokiLabel() { Key = labelString }))

Thank you :)

Hi @sven-n,

I'm sorry, but you are doing a little bit wrong.
Property labels has no changes from previous version. It is a list of static labels, that will be appended to log entries (for example service name).

You should configure propertiesAsLabels and pass there your includeLabels. That means that all labels from includeLabels list would be mapped as a labels to Loki (so you won't need for example |json pipe to extract them

[sven-n]

Ahh, got it, thank you :)

[mishamyte]

You are welcome)