Akira File Format #90
Comments
|
Did you think about use |
|
Does this mean that Akira will have a dependency on git being installed on the system? Or do you intend to ship a lightweight git implementation with Akira and use that? |
|
@terndrup in that case, we will use libgit2 so no need for git to be installed |
|
if there is some git thing running behind as a save you may end up with very heavy file (especially with one svg with everything inside). Maybe having a Also, the idea of having git behind could be pretty useful to merge (got already some UX idea about how it could be done :D) or to fork a project. be sure that these options don’t get forbidden by your git implementations please :D. |
|
@julientaq since we're managing everything via SVG, which is basically just an XML string, we should be able to keep the file size quite low. If you want to contribute with some design ideas, we'll need a popup window where you can scroll through the commits (probably named with the date and time) and the preview image for that commit.
|
|
You should consider other free alternatives to zip as the algorithm is very outdated (from the 80s) A good alternative: Also free and open. |
|
Despite the primary format being SVGs, presumably bitmap images are going to be supported in Krita? If so, these will make the file size big, GIT was never really designed to manage binary files. One option might be to look at GIT LFS although I don't know how it would work with libgit2. Another thought, if this is basically a git repo, would it be possible to manage it manually? If, as well as a .akira zip file, I would love to be able to open it from an uncompressed directory that I manually manage with git. Lastly, will you facilitate the ability to push the git repository to a remote repository? Obviousl, if we are able to facilitate managing the repo manually, this would be addressed by that. |
|
😄 If you really want to be future-proof and maybe want to support real-time online collaboration in a distant future (with independent offline edits and seamless conflict resolution, without a necessary need for central server), you might want to consider designing your JSON files and document representation as CRDTs (conflict-free replicated data types) and replace Git this way. CRDTs naturally keep track of document state history, but at the same time allow you to specify automatic merging semantics for different subtrees of the JSON for the case of concurrent edits (Example for a list of shapes: You have a rectangle A, one offline user removes it, second offline user changes its dimensions. What happens after online sync? You can specify e.g. that list of shapes should have UPDATE WINS semantics, so that the rectangle gets "unremoved" and the dimensions change (update op.) doesn't get lost.) CRDTs design ensures that after sync everybody ends up with the same document state (usually formally proven). There are interesting implementations in JavaScript, e.g.:
I'm not sure how many native CRDTs implementations are there and yes, reimplementing custom library would require some time... The thing is (in my opinion) it might be hard to add collaboration support into an offline-only app down-the-line, when your data-structures and file-format are already defined and are not designed for collaboration. (See Sketch - it still doesn't support real-time collaboration, does it? I don't have a macOS on hand, sorry...) I hope I didn't disturb you too much! 😄 You know... I had to mention CRDTs, it is the new medicine that cures every problem, right? 😄 I really keep my fingers crossed for your project, I absolutely understand that you have some limited resources, but still, at least for the record, I had to mention this technology. 🙂 |
|
The main issue I would have with the .akira file being a ZIP archive (or whatever kind of non-text file) is that versioning that would be an issue. I mean it's not that it won't work, but I usually make one Git repository for a project (unless it's a really big one) and have everything in there, from the (Akira) design files to notes to code, documentation, etc... It doesn't make any sense for such a use case to have the design file being an archived git repo inside another git repository. A lot of space would be wasted since the outer repo wouldn't be able to manage the internal one efficiently. This gets worse with every Akira file that is inside (I can see having multiple designs on a single repo). I'm not sure what the best option would be. If a user wants to store the Akira file inside a repository, the file should just represent the design with a text format (even SVG is totally fine). For users which won't store the file inside a repo, then something like what @jhrdina described seems really cool. Not sure about having that inside a repo, history would be duplicated anyway... This doesn't seem to be a straightforward issue to solve. Assuming that my use case is shared by other people of course (i.e. if I'm the only one then who cares 😄 ). The best solution I can think of is to have 2 formats ("raw" SVG without versioning and something specific with versioning) and make the user choose or choose automatically (i.e. when saving for the first time, traverse the directories and if a VCS repository is found then save in SVG (after asking the user for confirmation) otherwise use the versioned format. Just sharing my two cents, hope it's appreciated |
|
Thank you all for your insightful suggestions and point of views. @jhrdina you're not disturbing us at all, on the contrary, we need these suggestions since day 1 in order to avoid conflicts or technical issues while trying to implement new features in the future. @gdelazzari thanks for your suggestion as well. The idea here is to offer built-in version control for designers who have no idea what GIT is. Offering a version control already into one single file out of the box is convenient and removes any technical barrier. The user can upload that single file on a cloud storage for backup, knowing that one file contains the full history and versions of the project. You can save that file on a remote GIT repo as well if you want, but that seems quiet an overkill as you won't need to keep a history of versions of that file. If in your case you decide to not use the built-in version control of Akira, you can simply export your design in SVG and set up your own repo. |
|
Thanks for the response and the clarification, I take for granted that you have a way better understanding of Akira's users needs than me, since I'm not a designer myself. In that case the .akira file format sounds good. CRDTs seem to be a really interesting thing though, hope you can do some testing with such a technology and evaluate them.
Not so relevant, but just to be complete ("mettere i puntini sulle i") it's also true that most cloud services already do version control by themselves. I don't think someone would want to disable Akira's built-in versioning because their favorite cloud storage already versions the files, but it's also to realize that (IMHO) we're going more and more into versioned file storage services/systems. Again, not so relevant, but still worth to consider in order to "see the thing in its entirety", I think. Regarding the versioning inside the .akira files, I honestly don't think Git is a great idea. I mean, of course it's freaking battle-tested and everything, but is such a complex and full-featured system really needed? Also, how easy is it to make it manage the repository inside the archive? Does libgit provide user-definable hooks for opening, reading, writing, etc.. files so that they can be overridden with functions that operate inside the archive? I think something ad-hoc could be implemented, since the file format has a relatively simple and known structure which should make it less harder. CRDTs seem indeed the best solution IMHO, also being future-proof for (real-time) collaboration or whatever. Again, just my two cents 🙂 |
I wish I had the time... unfortunately, I'm currently desperately busy with with my Master's Thesis and eventually finishing my Master's Degree. I don't think I will able to help with coding before the end of June. 😒 The general approach would be not to save the final document (graphical objects in your case), but rather single operations/deltas that lead to creation of the document (add rectangle to pos XY, move object with UUID af1b11234 to pos X'Y', remove object with UUID 197dff). These operations then get "replayed" on each file opening to get the current (or older) version of the document. Using these discrete operations, CRDTs can find out a lot about different user's actual intents and therefore can offer much better conflict resolution (in contrast to comparing two static finished documents). I've found a C++ reference implementation of δ-CRDTs: The Automerge JS library mentioned above doesn't allow you to exactly specify the merging semantics, it does, on the other hand, offer some interesting advanced features - undo/redo support, relational JSON tables, optimized collaborative text editing... The underlying implementation might be a bit harder to port, because it uses JavaScript Proxies to "record" changes and also builds upon Immutable datatypes - these are great, as they enable structural sharing (you can have multiple snapshots of a document in time and share data that didn't change between snapshots) - the problem is I don't know any immutable data-structures implementation for Vala. You can however play with the library in codesandbox, I've created, so that you can get the feeling of dev experience I would hope for. If you uncomment the last line, you can see in the console all the generated operations ready to be saved to file. You can also dig into CRDT internals using DevTools. This is probably all I can do for you right now... :/ |
|
I think part of the problem might be mitigated if we have the ability not to "zip up" the file at the end. This would allow anyone with an existing understanding of git to use the akira directory as a sumodule rather than a binary file that cannot easily/efficiently be version controlled in part of a larger repository. |
|
I'm sorry guys but I completely disagree with where this is going. I think the main fact is that I wasn't able to properly communicate the purpose and the implementation of this feature, so, I will try again. First of all, giving the option to save in one way or another, or manually manage the repo, or do other things other than "Click > Save", would be a nightmare. It'd add so much complexity to the code only to support few edge cases. The fact that you read the word "GIT" doesn't mean you will even know there's a GIT repository in your Akira file. It shouldn't be of your concern. Since we're parsing the entire canvas with every single object in a JSON format, we're storing the entire design as text. Images and other assets will be stored inside dedicated folders within the In within Akira, you will be able to visually see the history of your commits (which will not be called commits), and visually see the differences between various saved states. You will be able to see the saved assets, delete them, as well as clear your history from a point in time of your choice. All of this will be handled by a properly defined UI, inside the app itself, removing the necessity of handling these stuff manually with a custom repository. If someone doesn't want to use the built-in version control (we should probably call it something else), he will be able to deactivate it from the UI, but the saved file will always be the Once again, if you guys want to use your own Git repo for whatever reason, you can simply export your design in SVG and commit that file, but this process will not be managed by Akira. I hope this will clear a bit the issue and will help to understand the purpose and objective of this implementation. |
|
With that said, in the future, we have the plan to implement a remote cloud sync option, which will allow the user to use our own cloud VPS, or connect it to their own. |
|
Uhm, not sure if you was also referring to my last comment or not, if you were I think there was a bit of a misunderstanding. As I said, the path you're taking looks fine IMHO (even if that would mean more work for people like me that uses Git or whatever). I was just trying to give a suggestion from a development/technical point of view. The .akira file contains assets, the JSON representation of the design and whatever you have in mind, but I was simply suggesting to look into something else than Git inside the archive to accomplish file versioning, since
With "how easy is it to make it manage the repository inside the archive" I meant "how easy is it to use libgit in Vala to make it work with the .git directory inside the archive", I was not referring to the user managing the Git repository inside the .akira archive. So my suggestion was to look into a simpler alternative, which could simply be to bring in (or implement - if no one has ever done that in Vala) just a text-diff algorithm (instead of the relatively huge Git system) and store the the diff for each file save/change inside the archive. Given the suggestion of CRDTs, looking into them could be an even better option at this point. I hope that it's now clear what I meant and you can evaluate this suggestion. It's just easier IMHO from a development perspective. You'll have to bind to a C library anyway, why bind to the entirety of libgit if you just need to preserve history? Just bind to a C text diffing library, the binary will be way smaller and it simply makes more sense to me, I mean bringing in Git with branches, remote capabilities, merging algorithms, etc.. doesn't make sense. |
Vala offers native bindings for My message was mostly directed to stop the discussion of "Having the option to manage the GIT archive the way we(users) want", not regarding your valid suggestions 😄
Using
Because this is only the first step.
Using GIT may seem an overkill for the simple system we want to ship with v1, but it's a necessity to future proof our app and allow us to implement new feature without rebuilding the save system once a year.
It doesn't make sense now, but the goal of Akira is to offer a complete design system platform capable of following design iterations in a sane way. We want to implement in the future some crazy options that will boost productivity 10x:
Cheers |
|
Clear enough, sounds very good 😎 So to use I'm bringing this up just to understand if this way of working on the currently opened file would have any drawbacks to tackle, such as... what happens if the application crashes or is closed unexpectedly before the contents of the temporary directory are archived in the .akira file? I assume you'll want to update the .akira archive every time there has been a change (i.e. every time the user saves the file/"commits" a change) to minimize this risk. Are you looking into some kind of incremental archival of just the changes in the temporary directory or are you planning to rebuild the entire archive each time? In that case it may be a bit problematic if a lot of external assets are imported by the user (let's say on the order of some hundred MBytes) and every time the file is saved the data needs to be archived (potentially compressed) again. Do you already have some ideas? (note: I'm not here trying to find weaknesses in the way you want to proceed at all, I just want to share my thoughts to ensure the method that is being chosen will perform well and be reliable, as long as we're in time before it stabilizes 😄 , I hope that's appreciated and I'm not disturbing in any way) |
|
@gdelazzari, yes, libgit2 currently only works on physical paths, and we also don't want to hold the whole file in memory since the files can get big. #110 had the initial work to zip and unzip the archive into a hidden folder inside the directory. This way, we can run the git commands, and access files without having to hold it all in memory. Updating the .akira file could be done on closing or saving of the file, with the smaller incremental updates being written to the unarchived dir. That way, if akira were to crash the unarchived contents would still be there and could be restored |
|
@Philip-Scott libgit2 supports backends like memcached, mysql, redis, sqlite and you can implement your own too. https://git-scm.com/book/it/v2/Appendix-B%3A-Embedding-Git-in-your-Applications-Libgit2 You can check that project: https://github.com/libgit2/rugged which allows to create a repository in-memory. The idea putting that case in RAM can be destructive and memory hungry :) IMO creating an incremental file format (with version, or something to identify for the migration process) would be much safer as Akira will grow up. You need to think about backwards compatibility with older formats, it's hard to define only one file format and use it forever. Making a 7z or ZIP or other format IMHO is the fastest way but the ugliest. |
|
Meanwhile we decide how to turn goo canvas into xml (svg) we can play around with serialization https://developer.gnome.org/json-glib/stable/json-glib-GObject-Serialization.html We cannot loose any other paint in Akira:
Credits to @AUNaseef https://twitter.com/AUNaseef/status/1181771257206857728 |
|
I'd love to see a save feature on Akira soon. Akira is a really powerful tool, its simple interface is much easier to work with as I can just focus on what I am designing. With a save feature, more people will start designing stuff with Akira and I believe it'll help the development of the program.
Lost one more right there 👆 ;) |
|
Just from a design/developer perspective who likes to keep things organized in repositories. How would something like this work out if you put your Akira files in a git repository itself? Because if you are planning to use something like a zip format or any other compression mechanism it would not be able to merge. Maybe it's a good idea to look to how https://github.com/kactus-io/kactus/, Abstract or other solutions solve this. Ps. I was really looking for something like this for a long time. Almost wanted to switch to Mac os just for Sketch because there wasn't any similar program providing the same simplicity. So keep up the good work! |
|
@KeizerDev I went through kacktus.io and didn't find really what is there for us. I'm maintainer of gitg and it can compare images in many different ways. Right now the only real file implementation is a JSON file, so it shouldn't be hard to diff it. In the worst case we will provide a diff filter so git knows how to compare an .akira file. I need to reread our original file format design an refactor the JSON serialization. |
|
I just recently read about SQLite being used as an application file format, and I wondered if it would be a good fit for Akira's use case. More information here: https://sqlite.org/aff_short.html |
This is a behemoth and we will probably split this issue into multiple sub-tasks, but let's start collecting our approach here.
ziparchive with ansvgfile and a hidden.gitrepositorysavetriggered command, a new commit should be pushed to the local git repoI'm still not sure if we should deal with undos the same way, or keep a limited amount of undos in memory as any other application.
The text was updated successfully, but these errors were encountered: