Clean up fluent-bit component construction #1537
Conversation
Use modify module in self log generation.
| @@ -275,6 +275,55 @@ func (p LoggingProcessorNestWildcard) Components(ctx context.Context, tag, uid s | |||
| } | |||
| } | |||
|
|
|||
| type LoggingProcessorLift struct { | |||
There was a problem hiding this comment.
Please don't name these logging processors; we're unlikely to ever expose them as-is, and someone in the future might think that this was already designed and try to use it publicly.
Grep and lift belong as functions (or possibly still structs, for convenience) in the fluentbit package.
There was a problem hiding this comment.
I was actually hoping to eventually expose the actual grep, nest, and lift processors to third-party receivers/processors, and use them to construct auxiliary pipelines. This change gets us part of the way there. I am happy to change them to functions again, and lift them into processors as part of the pipelines change (I'd have to move the other proto-processors that were already present, like NestWildcard).
There was a problem hiding this comment.
I think this suggests that there needs to a new category of processors: internal-only processors that can exist as objects in pipelines but can't be used in user config. Can you just move these into the fluentbit package and change the names slightly (maybe just call it fluentbit.Lift?)? I think they should still be usable as processors since they'll still implement the interface, and it will be clear they're not for user config.
(I think these are fluentbit-specific because nest/lift/etc. are all just done as workarounds specific to fluent-bit, and for an otel version of these third-party processors we wouldn't use them.)
There was a problem hiding this comment.
I suspect we can achieve internal-only processors by defining the Type() method, but not calling RegisterType. Done.
While these translate directly into fluent-bit filters, the actions performed by those are more general (extract values that are nested under a field into a top-level field). That should be translatable to an otel logging filter as well.
There was a problem hiding this comment.
I do not think that avoiding RegisterType is sufficient, because this is a loaded foot-gun. Also, we already have a clear pattern for how these should be implemented - as functions in confgenerator/fluentbit/processors.go that return []fluentbit.Component (and which AFAICT you are not touching in this PR).
While it's theoretically possible to translate these actions into otel filters, we would never do that because they are specific workarounds for fluent-bit bugs. There are better ways to do all of these things in otel without combining these specific action primitives.
| @@ -55,6 +56,40 @@ func healthChecksLogsPath() string { | |||
| return path.Join("${logs_dir}", "health-checks.log") | |||
| } | |||
|
|
|||
| type LoggingProcessorSampleLogs struct { | |||
There was a problem hiding this comment.
This is bespoke enough that it's fine staying as raw fluent-bit config. I don't think it's ever going to be reused in another processor.
There was a problem hiding this comment.
Again, I was hoping to turn this into an internal logging processor that was used in an actual pipeline…
Disallow registering components prefixed with "internal:".
igorpeshansky
force-pushed
the
igorpeshansky-refactor-fluent-bit-components
branch
from
1bdf6dc
to
0a594ed
Compare
Description
Remove explicit construction of fluent-bit components from all third-party application receivers and processors. Factor out all fluent-bit components into helper functions and/or logging processors.
Related issue
How has this been tested?
Checklist: