JS outgoing payload mapper doesn't know if payload is base64-encoded

[dimabarbul]

Summary

Live message value in JS (argument value in mapFromDittoProtocolMsg function) outgoing payload mapper is sometimes base64-encoded, sometimes not. It seems to be dependent on content-type: if payload is TEXT or JSON, then payload is left intact, otherwise it's base64-encoded. There is no way for JS mapper to know it other than check content-type.

Details

My use case is following: I want to allow sending any headers in the message, for example, all headers starting with "x-" (e.g., "x-my-header") should be forwarded as message headers (MQTT 5 in my setup) without "x-" prefix (e.g., "my-header"). The payload should not be changed. Messages might be sent with any content-type. AFAIK, the easiest way to do this is by using JS payload mapper.

I noticed that depending on content-type, that the message is sent with, its payload might be base64-encoded. Looks like, the logic for this is:

    // messages/model/src/main/java/org/eclipse/ditto/messages/model/signals/commands/MessagePayloadSerializer.java
    static <T> void serialize(final Message<T> message, final JsonObjectBuilder messageBuilder,
            final Predicate<JsonField> predicate) {

        final Optional<ByteBuffer> rawPayloadOptional = message.getRawPayload();
        final Optional<T> payloadOptional = message.getPayload();
        final ContentType contentType = message.getContentType().map(ContentType::of).orElse(ContentType.of(""));
        final JsonValue payloadValue;
        if (rawPayloadOptional.isPresent() && !payloadOptional.filter(JsonValue.class::isInstance).isPresent()) {
            final ByteBuffer rawPayload = rawPayloadOptional.get();
            if (MessageDeserializer.shouldBeInterpretedAsTextOrJson(contentType)) {
                payloadValue =
                        interpretAsJsonValue(new String(rawPayload.array(), StandardCharsets.UTF_8), contentType);
            } else {
                final ByteBuffer base64Encoded = BASE64_ENCODER.encode(rawPayload);
                payloadValue = JsonFactory.newValue(new String(base64Encoded.array(), StandardCharsets.UTF_8));
            }
        } else if (payloadOptional.isPresent()) {
            final T payload = payloadOptional.get();
            payloadValue = payload instanceof JsonValue
                    ? (JsonValue) payload
                    : interpretAsJsonValue(payload.toString(), contentType);
        } else {
            payloadValue = null;
        }
        injectMessagePayload(messageBuilder, predicate, payloadValue, message.getHeaders());
    }


    // messages/model/src/main/java/org/eclipse/ditto/messages/model/signals/commands/MessageDeserializer.java
    public static boolean shouldBeInterpretedAsTextOrJson(final ContentType contentTypeHeader) {
        return contentTypeHeader.isText() || contentTypeHeader.isJson();
    }

But in payload mapper there is simply no way to find out if the payload was encoded. No way other than checking content-type by itself. What I did to work this around is:

• created my own JS function (inside payload mapper code) that determines if content-type is text or json according to logic in java class ContentType
• used it to return either textPayload from value as is, or bytePayload as dcodeIO.ByteBuffer.fromBase64(value).toArrayBuffer()

Expected Result

1. I'd like this behavior to be documented, for example under Mapping outgoing messages section on Payload Mapping page.
2. It would be great to have function, e.g., Ditto.isTextContentType/Ditto.isJsonContentType, or Ditto.isPayloadBase64Encoded, that will reflect code from java class ContentType that decides parsing strategy based on content-type.

[thjaeckle]

Sounds good 👍

Remark:
Your use case screams for the "raw" payload mapper.
Together with a header mapping configured in the target of the connection.
However, this logic "use any header starting with x- and cut that off" is unfortunately not doable generically in header mapping.
Only if you would already know all existing headers upfront and configure them explicity.