Transactional Transport

[adamcharnock]

Lightbus includes an experimental (and undocumented) transactional transport. This was an experiment into implementing Reliable Messaging Without Distributed Transactions.

This transport needs rewriting/refatoring.

Learnings are:

1. Database connections cannot be used concurrently. There must be one per event listener or, if we plan for multiple concurrent invocations of the same listener, then each listener will require its own connection
2. Database client APIs are not consistent. Getting lightbus to manage the developer's database connection is both unreliable and magic.
3. The transactional transport will need lightbus to inform it of errors.
4. Managing database tables (ie. migrations) is a faff
5. State becomes duplicated: Redis tracks the IDs it has sent to the client, and the client also tracks what events it thinks it has received. This makes operations such as clearing out messages hard as one must perform a reset in two places (redis and the database)
6. The config system does not have a good way of representing a child transport
7. Perhaps the transactional transport isn't a transport at all, but rather a plugin or decorator of some sort.

1 & 2 can probably be solved together. Perhaps by passing a connection_factory to .listen().

3 & 7 may get solved together

Allowing transports/plugins to add sub commands to the lightbus CLI would help with 6

6 & 7 may get solved together. Perhaps an api config can take a list of transports. Or perhaps transports take a decorators / middleware config option.

Alternative approach

An alternative approach would be to create a database-backed event transport, separate from the Redis transports. This could be db-specific such as a PostgresEventTransport, but with core logic abstracted to a SqlEventTransport to allow for simple implementations for other RDMS.

Get hold of the transaction will still be a faff though.

[adamcharnock]

The config system does not have a good way of representing a child transport

A thought to add – child transports may well also have other uses in future. For example, a transport which supports large data blobs. Similar to Postgres' TOAST system, this transport could move large blobs to an external data store before handing off to the child transport (e.g. the redis transport)

The more I think about this the more it sounds like these 'parent' transports are not transports at all, but more like middleware. So perhaps renaming these to TransportMiddleware could be more accurate. I am hesitant to complicate matters through adding new concepts, but in this case it may be justified.