use EventStore embedded as alternative to FileStorage

[rofr]

In the EventStore project, implement a storage provider that uses an embedded event store node, see https://eventstore.org/docs/dotnet-api/4.0.2/embedded-client/

[goblinfactory]

@rofr Why do we want to do this? I'm not saying we shouldn't, just that this Jira lacks any context for me to contribute, or consider alternative solutions to the desired outcome or problem to be solved. I'm assuming eventstore embedded is a client library? If so, wouldn't that need to be a pluggable option considering your LGPL licence? You'd ideally want memstate to be reliant on as few external libs as possible. If you some information around the "why" then the discussion for that feature can live here and not clutter up the gitter chat.

(some explanation of some of the shortfalls behind the default fileStorage and what the eventStore embedded (client?) offers would be very helpful. This will also help me with documentation as I'd like to consider somewhere to put notes for "best practices" and "recommendations", limitations of default FileStorage etc. Very important information.

Also, is this the lib you're talking about ? : https://eventstore.org/docs/dotnet-api/4.0.2/embedded-client/

A

[rofr]

Eventstore is battle tested, robust, reliable, durable, fault tolerant etc. Our filestorage is currently far from this level of quality, so why reinvent the wheel? Also we already have support for remote eventstore, so adding an embedded option seems like low hanging fruit.

There are some other nice features as well, for example the web based admin UI where we can view the journal entries.

And of course this provider would live in the event store module, so would add a hard dependency to the core memstate package.

[goblinfactory]

Assuming a new dev comes along and just wants to build something with a persistent model, what would his setup look like, and what would his getting started code look like hypothetically if we use eventstore? Would this change his installation and getting started assuming eventstore becomes the new default StorageProvider? If the default storage provider is not reliable, or durable or robust, then perhaps we really shouldnt ship it at all? (Getting started code should be battle tested and usable in live production systems, imho.)

[goblinfactory]

Would it make sense to create separate issues for the current HostFileSystem (IVirtualFileSystem)? I'd be happy to write some tests proving or testing any concerns or assumptions we have about the provider? To be totally honest, it's what I need myself for a bunch of simple prototypes that I don't want a complicated stack for, and having something that scales out of the box, and is stupidly fast is very attractive. I can imagine a lot of projects starting and sticking with the basic fileProvider because "it works", I'd want to be able to show people what it can, and cannot do for you with simple tests.

[rofr]

Using the embedded eventstore would be a simple config settings and a nuget download of the event store module which in turn has a dependency on the eventstore client nuget. Besides the single config setting, it would be entirely transparent to the rest of the users code.

[goblinfactory]

Would be good to have a human friendly error message that's a bit aware of the typical providers, so that if a provider is specified that we know about, we can give an error message with instructions on how to fix, e.g. "You've specified XX provider, and it looks like you've not included the package in your project, please run the following nuget command , " In such a way that a beginner can install just 1 nuget package, write a few lines of code, run his app, see the message, install 1 more nuget package, press F5 and now it works. (a sort of pit of success)?

[rofr]

Testing edge cases and failure conditions for file operations on multiple platforms and with multiple different drivers and hardware is not a trivial task. We've not had a single incidence of file corruption with origo since we added the Packet abstraction which is yet to be implemented in memstate. File storage will still be the default storage and it needs to be sufficiently reliable.

[rofr]

I'm thinking a Memstate.Bundle nuget package with all the important packages bundled would solve this beginner problem. newtonsoft, eventstore, postgres and memstate core. But sure, useful error messages.

[goblinfactory]

We don't need to test everything, just the most common expected use case, and any of our own personal use cases, then the documentation can be explicit about what has been tested, and what has not. We can expand testing if anyone reports a bug, and hopefully our tests can be well enough written that they can easily be adapted if someone wants to test a specific use case we've not covered and of course they're welcome to submit a pull request.

[goblinfactory]

re: memstate.bundle : I did not know that was possible. Are you allowed to do that with an LGPL licence? Just because something is in a nuget package, doesn't mean the combinations of the licences are not complicated as you'd effectively be distributing newtonsoft etc. I thought the purpose of package configurations was to force the user to download the packages, so that you never distribute the actual packages, only the recipe of what packages are needed and the user chooses to accept licence terms when each package arrives.

[goblinfactory]

I might be wrong but I thought that bundling is for internal corporate use, not for publishing to public nuget servers? Something along these lines ? http://www.marcusoft.net/2011/09/creating-local-nuget-repository-with.html

[rofr]

Only our own libraries would be in the package bundle. External dependencies would be downloaded transitively by the user.

[goblinfactory]

Ah, ok ... that makes sense. So if you need eventstore you install the "memstate.core" plus the "memstate.eventstore.xx" (roughly) and you get the eventstore.xx dependant packages.

[rofr]

Maybe we shouldn't use the term "bundle", you are not the first person misled by that wording. The point is to make onboarding easy. The first-timer shouldn't have to figure out which of 12 possible nuget packages they need to get a simple example going. I'll create a separate issue for nuget packaging strategy and implementation.