[Node] Arbitrary embedding .meta info into a tiddler file, i.e. single-file tiddlers of any type
#8156
Replies: 5 comments 9 replies
-
|
Great idea. I'd also need to look through the specifics of both the utils filesystem code & the filesystem adaptor plugin. I think we would also need to hook into the boot code that loads tiddlers & tracks filesystem paths. Possibly add an MIME type there as well. That all seems like we could abstract it better to support custom metadata. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @CyberFoxar. As you note, this has been discussed a few times in the past which is a clear sign that it is an enduring issue. In TiddlyWiki terms, I think the proposal is to introduce a new file format for representing tiddlers with embedded YAML metadata, and then to use that format in the Node.js configuration in preference to There are a couple of issues:
So, where I am at the moment is that I think we should indeed introduce a new file format that is similar to Could you give some examples of the metadata you'd like to use, and the effect you'd expect them to have? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @CyberFoxar that all makes sense. Some background: boot.js has to know how to read all the different tiddler file types. It has the abstraction of a Lines 1599 to 1745 in a081e58 The mapping of file extension to file type is done here: Lines 2444 to 2492 in a081e58 In terms of writing files of the new format, the serialiser module can be packaged as a plugin. I think the logic here would need to be adjusted: TiddlyWiki5/core/modules/utils/filesystem.js Lines 217 to 286 in a081e58 The core changes needed to satisfy your proposal would make a good grounding for subsequently changing the default file format. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks ! Great to see that I'm not too rusty on my Node-fu. I had arrived more-or-less at the same bits of code you pointed out. Glad to see I didn't totally miss the mark (helps that the code is impeccably documented). I might try and give a shot at implementing my proposal as PR, though I couldn't find a lot of documentation on how to setup a dev environment that uses NodeJS. The setup info on the dev site expects a single-file wiki, and running using I'll try fiddling a bit with the various options and try and take care to document the process to contribute there too. |
Beta Was this translation helpful? Give feedback.
-
Should do the trick. New tiddlers created in the wiki will land in the |
Beta Was this translation helpful? Give feedback.
-
|
Hi @CyberFoxar the scripts in the |
Beta Was this translation helpful? Give feedback.
-
|
I use VSCode, and base much of the setup on the scripts in the ./bin files. Here is my These start "F5 Start Debugging" sessions in the "Debug Console", and are kicked off in the "Run and Debug (Ctrl + Shift D)" Panel-tab on the left side. You can set break-points, use |
Beta Was this translation helpful? Give feedback.
-
I'm OK with your example of listing JPG-files and extract the EXIF data as TW meta data. But I'm against the possibility to handle arbitrary binary files with unknown content for security reasons. |
Beta Was this translation helpful? Give feedback.
-
|
That's a valid concern, but I'm not sure it's a big issue. This, in turn, means that you'd need a specific plugin to handle a specific type of binary file. This "delegate" the security issues to the plugin developper, and the wiki administrator. It's not great, but the separation of concerns still is there. I'm trying to think of scenarios where this might be an issue. I can imagine trying to inject a tiddler with arbitrary binary content in, but it wouldn't be written to disk unless a proper handler for the type is registered. The responsibility would be on the plugin maker to make sure the file written can be read correctly after... Maybe one could try and inject a nefarious payload, but that wouldn't be executed, since the only thing interacting with the file is tiddlywiki and the plugin. Maybe someone could try to create buggy invalid files ? But it's going through nodeJS' fs library anyway. There's also the idea that you're not giving away rights to write to the filesystem either. The proposed api would be that tiddlywiki simply ask for the data to write and the filename to use. This would then be passed to nodeJS, which does some sanity checks on the written binary data. The plugin wouldn't even really have the right to choose where the file is saved, just how it should be. It is, however, entierly possible that I'm missing something. |
Beta Was this translation helpful? Give feedback.
-
|
@CyberFoxar I think abstracting this away from being directly inside the boot.js code is a good idea. The issue with conceptualizing this as a If you'd like to sync on this at some point, let me know. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, that's an issue with the approach. I believe it could work, though, because Another approach might be to just have a config/fileinfo that says "refrain from loading tiddlers here at boot and wait for plugin XYZ" and then, once plugins are loaded, have some logic to "re-discover" those tiddler and add them post-boot. I'm not sure how well this approach would work, since it would mean that we'd add potentially a few hundred of tiddler to the store post-boot... |
Beta Was this translation helpful? Give feedback.
-
|
Alright, an update. My goal is to create a new kind of plugin, that I'm tentatively naming Proposed Mechanism: deferred tiddler loadingI'm trying to work out how a deferred loading of Tiddlers might work. The goal is to add a new, optional property to {
"directories": [
{
"isDeferred": true // {boolean}
"deferredContentType": "bin/test-tiddler" // {string} - defines which plugin to defer to
// Rest of the fields as-is
}
]
}This optional property would signal to I'm a bit iffy on the details, but I believe that registering another This would basically be a lite version of the content of Tiddlers loaded via this method need to be marked in some way (so that TiddlerLoader API proposalMy JS is a bit rusty, but that's what I'm working with for now. Please do tell me improvements ! The most obvious one, that I'm on the fence about is whether or not we should try to make the API more flexible, to allow passing a bit more info that just the filename to the plugin. Maybe /*\
title: Proposed API for deferred tiddler loader module
type: application/javascript
module-type: tiddlerLoader
Example API
\*/
(function(){
/**
* When given binary data, tries to deserialize it into a tiddler.
*
* @param {string} filename - original filename, does not include path
* @param {Buffer} fileBuffer - file as read by NodeJS
* @returns {$tw.Tiddler} - decoded Tiddler object to be added to the store
*/
function loadTiddlerFromBinary(filename, fileBuffer){
throw Error("not implemented")
}
/**
* When given a Tiddler, binarize it however we like and gives
* back a temporary object holding the data.
*
* @param {string} filename
* @param {$tw.Tiddler} tiddler - tiddler to be binarized
* @returns {
* {
* buffer: Buffer,
* filename: string
* fileOptions: {fs.WriteFileOptions | undefined}
* }
* }
*/
function makeBinaryFromTiddler(filename, tiddler){
throw Error("not implemented")
}
TiddlerLoaderPlugin.prototype.load = loadTiddlerFromBinary
TiddlerLoaderPlugin.prototype.save = makeBinaryFromTiddler
function TiddlerLoaderPlugin(){
// init stuff
}
exports["bin/test-tiddler"] = TiddlerLoaderPlugin
})() |
Beta Was this translation helpful? Give feedback.
-
|
More news ! I've made a PR ( #8201 ) to implement the proposed feature ! |
Beta Was this translation helpful? Give feedback.
-
Idea for arbitrary-embedding metadata info inside a tiddler.
The gist
Would it be possible to have the information that's currently in a
.metafile embedded in the tiddler itself, without resorting to a.jsonor.tidtiddler ?Why
I am in the process of migrating a good bunch of my markdown notes from dendron (think obsidian, but more open) to tiddlywiki.
One feature I like about dendron is the use of YAML Frontmatter, which is a block of yaml (or anything, really), delimited by fences, that defines meta-information about the rest of the document. It works more or less exactly how a
.tidfile is constructed, except it's a tad more versatile.Anyhow, my issue stems from the fact that I'd love to reuse/keep the frontmatter that is already in my markdown notes, without having to convert them to json/tid or live with a bunch of
.metafiles. I have a personal gripe with the use of.metafiles for text-based content, as I feel this will create undue clutter in my notes as well as risk me losing even more info next time I try switching systems.How?
I have taken a look as to if a plugin might fix that, but it seems to be a root issue with how the nodeJS filesystem code is implemented, taking a look at this line and this one from
/core/modules/utils/filesystem.jsit seems this behavior is not (currently) overridable via plugins.A tentative fix would be to either have a hook to change the behavior at this place, or allow plugins to inject themselves there. Say, allow a plugin to hook into
filesystem.js.saveTiddlerToFileto handle either the actual writing of the file, or pass on the "compiled" binary data to the call tofs.writeFile. This would also require to patchboot.jsto handle loading tiddler with meta in them, specifically patching/boot/boot.js.[$tw.loadTiddlersFromSpecification, $tw.loadTiddlersFromFile]to be able to read and load meta-in-tiddlers.This would enable a plugin developer like me to use any arbitrary file format to serialize (and deserialize) single-file tiddlers in the filesystem. It'd allow me to call a plugin such as remark-frontmatter to parse frontmatter of text files and use that as the metadata for the tiddler, but it would also enable more enterprising people to use any kind of field for meta info. One could imagine storing the metadata inside of the EXIF data of a picture (see XMP:Description) or the "comment" field of JPEG files.
Similar subjects
The subject of tiddlywiki's handling of
.metafiles has been lightly touched on in the past, see this discussion on Discourse of someone whom a frontmatter-enabled tiddlywiki would've helped.Interest for storing and accessing metadata within EXIF has been mentionned here in the Discourse.
I am not sure if there are more specific examples.
Offer for help
I am really liking TiddlyWiki so far, and I'd be glad to help / contribute to the above feature. I am still learning the ropes a bit, but I was already looking into patching this behavior into my own version of tiddlywiki, but I figure that some of you guys might have a better idea of how best it should be done.
Beta Was this translation helpful? Give feedback.
All reactions