Meilisearch logging to stdout

[hendrikniemann]

Hey meilisearch team and community,

I am running meilisearch (currently version 0.25) in a container in kubernetes on GCP. I have been using and adopting the example manifests. They were very useful and it going quite well.

One thing I noticed is that the logs (namely the health endpoint logs) end up reported as errors in GCP. The reason seems to be, that the logs are written to stderr. I had a similar problem with a different application and this application allowed to change the logging target to stdout.

Here is the log from GCP:

{
  "textPayload": "[2022-06-14T10:05:19Z INFO  actix_web::middleware::logger] 10.84.2.1:45498 \"GET /health HTTP/1.1\" 200 22 \"-\" \"kube-probe/1.21\" 0.000064",
  "insertId": "cmi8d62rknq94vyz",
  "resource": {
    "type": "k8s_container",
    "labels": {
      "cluster_name": "cluster-1",
      "pod_name": "meilisearch-0",
      "namespace_name": "default",
      "container_name": "meilisearch",
      "location": "XXXXXXXXX",
      "project_id": "XXXXXXXXXXXX"
    }
  },
  "timestamp": "2022-06-14T10:05:19.285621218Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/controller-revision-hash": "meilisearch-6994d65955",
    "k8s-pod/app_kubernetes_io/instance": "meilisearch",
    "k8s-pod/app_kubernetes_io/name": "meilisearch",
    "compute.googleapis.com/resource_name": "gke-XXXXXXXXXXXX",
    "k8s-pod/statefulset_kubernetes_io/pod-name": "meilisearch-0"
  },
  "logName": "projects/XXXXXXXXXXXX/logs/stderr",
  "receiveTimestamp": "2022-06-14T10:05:20.065729855Z"
}

I think it would help me to confirm what is going and find a solution if I got some answers to the following questions:

Is meili indeed logging to stdout or is this a problem with my setup?

Did this change in recent versions or is this planned to be changing?

Any ideas how I could change that? I was thinking about redirecting to stdout mabye by changing the CMD of the container.

Any help would be much apprechiated!

[curquiza]

Hello @hendrikniemann
I moved the discussion to the appropriate repository so that the support team (@oluademola) can help you!
Thanks for using Meilisearch 😊

[oluademola]

Hello @hendrikniemann

Indeed meilisearch logs are by default presented to the user in stdout. Also, this hasn't changed, and currently, meilisearch does not support specifying a log path. Although this has been added to the roadmap page I cannot guarantee when or if this feature will be implemented. I think an example of how you can manage logs was mentioned in this discussion but you could also inquire in the Slack community channel to learn how other users have handled this.

Hope this helps. Thank you.

[hendrikniemann]

Okay, it seems like Meili is logging to stdout and the problem is on our side somewhere in the DevOps stack. Thank you for the response. I will mark this post as the answer for others having a similar problem (I cant mark @oluademola's response, as it is a comment).

[hendrikniemann]

Hello @oluademola, it would be cool if you could give this thing another look!

I have been doing another round of digging and I found out that the rust crate documentation for env_logger states that it uses stderr by default. I am not at all familiar with Rust, but the source code does not seem to change the target of the log stream. Furthermore, the middleware::Logger seems to be using the same create for logging.

Redirecting the stderr stream to a file locally confirms the assumption: Meili actually logs everything to stderr.

I am not sure if this is the desirable behaviour. For my setup at least, it creates quite some inconvinience as debug messages are picked up as errors within Google Cloud Logging. I understand that changing this would be a breaking change as others might rely on this in their deployments.

[oluademola]

Hi @hendrikniemann

Thank you for the additional information.

You are right, and sorry for the confusion. We discussed this internally and indeed meilisearch logs to stderr, which is typical and common for software. I am aware of the impact this would have on you, but as I already explained, there are no current plans to change the default logger. However, as you can see on our roadmap page, we are looking into the idea of allowing users to specify a log path in the future.

I hope this helps. Many thanks 🙂

[mercuriete]

@oluademola
I have the exact same problem.
stderr is for error and stdout is for anything else.
Please fix this issue because it is indeed a bug.

• If you want to keep this behavior please keep the error level messages to stderr and the rest messages to stdout like any other Linux application.

• If you can't split your messages please put stdout as default.

I really like your application, thanks for your work.

PS: I think this thread should be moved to issues instead of discussions because this is a bug.

[oluademola]

Hello @mercuriete,

I understand your pain point, and thank you for the feedback. I am not sure where you're drawing your conclusions from, but this isn't a bug but rather a limitation of the product, which can be considered a feature request. @gmourier, if there is no existing discussion on this issue, I can move it to the product repository.

Just to clarify, the reason why everything is going to stderr is because of the default behavior of the rust env_logger, which writes logs to stderr. Although using stdout for non-error logging and stderr for error logging sounds like a good idea and may solve the issue you and OP are experiencing but this is currently the expected behavior of our logger.

Perhaps the @meilisearch/engine-team might have some suggestions or a workaround for you.

[mercuriete]

I understand your point that is in a component standard to rust logger.
But the problem is rust logger is not following linux/unix universal good practices of using stdout and stderr.

kubernetes understand everything goes to stderr is an error (like anybody else) and on some tools for collecting logs... this is translated to an error level.

My advice is you contact with rust logger and ask to follow the linux/unix standard practices.
I will try to do it as well.

thanks for your understanding of the situation.

[mercuriete]

@oluademola
env_logger is no longer mantained
rust-cli/env_logger#159

So I advice you to switch to another solution.

[irevoire]

Hey @mercuriete,
Sorry, I'm late to the party.

stderr is for error and stdout is for anything else.
Please fix this issue because it is indeed a bug.
[...] But the problem is rust logger is not following linux/unix universal good practices of using stdout and stderr.

Actually, you're wrong on this one; as olu said, this is expected and is not a bug.
Also, nothing is really well-defined in the standard, and even less in the industry.
This is what the GNU C library state about both stream

    Variable: FILE * stdout
    The standard output stream, which is used for normal output from the program.

    Variable: FILE * stderr
    The standard error stream, which is used for error messages and diagnostics issued by the program.

So, basically, it comes down to what you consider as a diagnostic or a « normal output » (which doesn't mean anything).
At meilisearch, like most software, we decided to consider every log as a diagnostic.

env_logger is no longer maintained

I'm afraid that's not right, and the issue you linked shows it. epage is the maintainer of the repo, and you can see him working here: https://github.com/rust-cli/env_logger/commits/main

---

Now I don't want to be rude, but please refrain from talking as if you were representing meilisearch on other OSS projects.
Your answer here really shows a lack of respect, and that's not what we want to be affiliated with.
Plus, meilisearch logs are like that because initially, we decided to follow what env_logger was doing.
But if we continue to log everything to stderr after your comment, it's because we decided it was not a priority to fix it, not because we wait for env_logger to do the work for us.
If we were to implement what you want, maybe we'll talk with the maintainer of env_logger or move to another logger entirely, but your issue is 100% with meilisearch, not with env_logger.

[gh2k]

At the risk of being Opinionated, I agree with OP in that I was surprised to see the logs going to stderr.

While there is no published standard, logs-to-stdout, errors-to-stderr is the default for most of the products I have run in production (see: Apache, nginx), as evidenced by the fact that our log ingestor is treating the Meilisearch access log as errors, which is very noisy.