The future of iodine (vs. plezi)...?

[boazsegev]

Hello dear community members (originally posted on this discussion).

I wanted to ask:

Should we make isomorfeus-iodine the main branch?

I am working on a new version of the facil.io C framework that powers iodine and I was wondering about the future of iodine and where it's going (if at all).

@janbiedermann forked iodine and created the isomorfeus-iodine branch which seems to be a good replacement for the main branch as well as supports Windows.

On one hand it seems that @janbiedermann is more involved and active as a maintainer and so the work they do might superior.

On the other hand, I cannot help but think that the isomorfeus-iodine branch might be following a different roadmap into the future than the one I have in my head.

In the end, if @janbiedermann wants the iodine gem I would happily give them control of the name and the project.

A server or a framework?

Originally I came up with iodine as I needed to write a a real-time app and could find a library I liked with reasonable performance.

This led me down a rabbit hole and it is my current belief that the Ruby community would really benefit from a one-stop evented framework for authoring micro-services and SPAs (which was what I tried to do with plezi.io for a while).

Expending iodine so it becomes that framework could add the need for a remote database access API (i.e, having C bindings to PostgreSQL or other evented remote databases) and making HTTP responses evented so their threads do not block on IO requests (such as database read requests)... and this would require a departure from the current Rack specifications...

Maybe iodine should not be that framework. The iodine server is a fast and performant server. It follows the Rack specification, which is based on CGI and does not allow for an evented design. Perhaps iodine should remain as it is and evolve (if at all) according the current Rack specification.

Maybe it is time I let iodine go and revived my vision for the plezi.io framework instead?

Let me know what you think in the comments below.

Thanks!
Bo.

[janbiedermann]

Hi Bo, Hi @adam12,

the differences between branches are visible in this PR #119

The version bump to 0.8 is because of Rack 3, which changed its API a bit (breaking iodine). I took over the API change to i-iodine, removing code from i-iodine, so it now depends on Rack 3 (previously it could work without Rack). A breaking change, so i bumped the version.

I expect to invest some more time into i-iodine soon again. Also my requirements for iodine changed over time with the advancement of my isomorfeus framework. Most probably it will become a easy to use application server for isomorfeus applications. Since i am using Bo's cstl within isomorfeus, one of the next steps will be to adapt i-iodine to cstl. Also one more fix is waiting for i-iodine 0.8 for upcoming Ruby 3.2. And i think, i need to pull a few fixes from iodine 0.7 into i-iodine 0.8

Users of iodine maybe dont expect it to become a isomorfeus application server, as that also includes removing features, that i dont require (like redis support), but users of iodine may depend on; and adding features i require for isomorfeus, but current user of iodine will never need.

Bo, please feel free to decide.

[fabio]

I do not have precise answers to the questions, however I would like to make the following considerations.

I found Iodine well designed to deploy fast, reliable and scalable Ruby apps. All feature and components are appropriate and well integrated, like Redis support for example. My preceding experience to manage high numbers of concurrent WebSockets connections in Ruby with different application servers and gem stacks was frustrating before adopting Iodine. Moreover, I always found Bo’s support and suggestions on any issue. Last version of Iodine is mature and very stable.

I understand the need to have hands on real project in Ruby to maintain the gem more easily. I understand also the potential of having Iodine working also in Windows--this certainly tend to improve the quality of the codebase-- even if I do not know Isomorfeus purposes.

I think that greater community involvement is crucial for the future of the project, whatever it will be.

[janbiedermann]

@fabio Isomorfeus is a full stack isomorphic Ruby Webapp/SPA framework: https://github.com/isomorfeus , Ruby everywhere and for everything, one Ruby to rule them all, basically

[boazsegev]

Hi @janbiedermann ,

Would you mind sharing the reason you removed the redis support rather than just leave it as an unused feature?

There is something that makes a lot of sense about having the redis support a little separate.

The reason I am asking is that memory restrictions and storage restrictions lead to different solutions, one allowing the redis code to be kept in the same gem *with a require "iodine/redis" to be loaded) and another requiring it to be moved to a separate gem... both are options raising their own questions and challenges.

Kindly,
Bo.

[janbiedermann]

Hi @boazsegev ,
i did not remove it yet, i plan to remove it, rather replace it with a iodine internal solution based on fio_hash. Iodine already has cluster communication, so that can be used to have some internal "redis", well hashmap. Thats my plan.
The reason behind it, isomorfeus is for regional scale distributed services, comes with data store, no need for redis, so i can remove the cost of mainaining it. Reduce cost, effort. I simply want to make things as easy as possible for isomorfeus apps, no need to configure services or use external services, instead start the app and everything just works.

Basically like this.

[janbiedermann]

Well, isomorfeus is a somewhat "complete" framework, it employs two large c-extensions, one for i-ferret and one for iodine and a smaller one for i-preact. They are somewhat self contained and together a huge C code base, that i would like to maintain. So naturally i look for things to exclude to reduce the effort required on my side.

[boazsegev]

Hi @janbiedermann ,

Thank you for your answer and explanation.

I totally understand your reason for wanting to remove Redis - code maintenance is always a burden. If we can decide on a common C and Ruby API for pub/sub "engines" we could probably export the Redis implementation to an external gem which can be shared across iodine flavors.

replace it ["reidis"] with a iodine internal solution based on fio_hash
...

Indeed for version 0.8.x of the C framework (facil.io) I was hoping to add similar features:

1. Auto-discovery for "shared network" app clusters.

   Using UDP broadcasts to discover and connect multiple iodine instances, it will be possible to combine multiple instances into an even bigger pub/sub cluster. This would allow micro-services in the same local-network to share pub/sub messages and events. This will not improve pub/sub performance (msg/sec), as that would require sharding, but it could be helpful in other ways such as failure resistance, event notifications, data sharing and supporting more concurrent clients.

2. Key based in-memory local/cluster cache storage... the complexity and features of which I am still debating. I was thinking of either one of:

   1. Simple String key => String value cache of limited capacity (i.e., 100,000 keys with LRU eviction) with no race condition resolution. This would be the simplest to implement and would provide a "near enough" source of truth for many implementations... but it would require the end-user (developer) to be creative about storying keys and clearing them out.

   2. JSON style storage that allows the constant updating of a JSON tree. However things like Array updates (push / delete / eviction / ordering) and stale data removal are still unclear to me. Deleting data can be as easy as setting a key to nil (NULL), but race conditions may have unclear effects and Array management introduces other challenges.

However, I doubt if these will replace the use-case for Redis completely, as Redis can be used for cross-region messaging, persistent event/task queues and sharding - all cases which require a 3rd party app / service (which can also be implemented in iodine, but why reinvent the wheel).

[janbiedermann]

Hi @boazsegev ,

that sounds very interesting. A common engine framework would be great, maybe thats a chance modularize iodine even a bit more, maybe for protocols.

Please clarify the state/relationship of cstl vs facil.io

[boazsegev]

Hi @janbiedermann ,

A common engine framework would be great...

I agree :)

I think we can come up with a Ruby specification API for pub/sub engines as well as a specification API for attaching protocols to iodine (or any compatible server).

I'm guessing we can also come up with a C API specification, but if we're to use the same C library it is mostly academic (as the library's API will be the de-facto specification).

Please clarify the state/relationship of cstl vs facil.io

TL;DR; - once completed it, the CSTL should become the facil.io version 0.8.0, replacing the core library (fio.h & fio.c).

The CSTL project started as something small, just unifying the Type Templates in fio.h (i.e., here) with the FIOBJ types. I wanted a cleaner and more consistent API for the different dynamic types...

...but it grew to swallow the whole of the facil.io core - including the server, pub/sub, JSON, FIOBJ types, etc'.

Protocol and extension specific logic (i.e., WebSockets, HTTP, Redis) are currently placed in an extras folder, where they are likely to remain.

The way I currently see it, the CSTL + extras folder will together become the facil.io version 0.8.0 (and hopefully move us towards a 1.0 release).

I'm not sure how I'll organize the GitHub repos, but I think I will either move the CSTL into the facil.io repo or dedicate the facil.io repo to the documentation and app builder script.

Obviously suggestions are welcome :)

Thanks for asking,
B.