Making posticle portable and other improvements

[ghost]

I'm thinking of how I can make posticle more portable and useful outside of the rustodon, since there are probably cases where this functionality would be useful to others, and besides that, it'd improve the quality of the code.

The first thing that needs work, as far as I can tell, are emoticons. For the utf-8 icon based emoticons there isn't anything additional that needs work, however we need an EntityKind for custom name based emoticons.

Once that is done, I want to move the HTML generation into positcle, I've identified three cases where posticle syntax is used:

1. In user titles (e.g. Measly Twerp :archlinux:)
2. In user biographies
3. In user posts

For titles it doesn't make sense to support hashtags, mentions and URLs, but we would want to include emoticons. To make this possible I'd add a EntityResolver trait that rustodon can implement for the two cases.

The EntityResolver would have the following methods that all return an Option<String> to confirm that the entity is recognised and valid:

• resolve_emoticon(name: String): resolves to the image URL
• resolve_hashtag(name: String): resolves to the hashtag URL
• resolve_mention(name: String, domain: Option<String>): resolves to the mention URL
• resolve_url(url: String): resolves to the URL given (allows blacklisting of URLs for example)

You'd then just pass in an instance of EntityResolver to the parser and it'd spit out the HTML.

What do you think?

[barzamin]

this sounds like a good idea! i do have a few questions, though:

• what if we want to support basic markdown (eg []() links, _xxx_ italics)? where would something like that fit? does it become part of the Pest parser and we just extend the tokens to a fuller AST-y thing? because that sounds fine to me.
• under your proposal, posticle should probably do the sanitization we're doing now. any ideas for how that would look?
• i'd be inclined to make trait EntityResolver be something more like trait EntityResolver<F,T> { fn resolve(x: F) -> T }? something still makes me vaguely uncomfortable with that approach too. i guess i'm still trying to think of what the API should look like on posticle, then

[ghost]

I started working on this last night.

what if we want to support basic markdown (eg links, xxx italics)? where would something like that fit? does it become part of the Pest parser and we just extend the tokens to a fuller AST-y thing? because that sounds fine to me.

Yeah, that'd be the correct way to do it, I've switched to using something like an AST for the minute, although it's completely flat so there's not really a tree as such. When/if we need markdown syntax we will go full AST.

under your proposal, posticle should probably do the sanitization we're doing now. any ideas for how that would look?

At the moment you're using something built into maud, correct? Obviously it doesn't make sense to include all of maud, however there is a html sanitizer library called ammonia that should fit our needs.

i'd be inclined to make trait EntityResolver be something more like trait EntityResolver<F,T> { fn resolve(x: F) -> T }? something still makes me vaguely uncomfortable with that approach too. i guess i'm still trying to think of what the API should look like on posticle, then

In the end this is the trait I settled on:

pub trait Resolver {
    fn resolve(&self, entity: Entity) -> Option<Entity>;
}

It makes more sense to be using match inside of the resolve function than to have a method for every single entity type.

[barzamin]

we're not using maud anymore. we do depend on maud_htmlescape for escaping, and ammonia for sanitization - i was more wondering where these should be hooked into posticle (should it directly dep on them and do everything out-of-the-box? should we pass escapers in or something?)

i don't know how great maud_htmlescape is. ammonia works very well, but they're for two different purposes, in that maud_htmlescape just replaces things like < with <, and ammonia doesn't (cf https://github.com/notriddle/ammonia/issues/76).

i'm a bit confused about what your resolver trait is supposed to do there. it's taking an entity and... returning one? generally, we want to take a username/domain/hashtag/image/whatever and get a URL, right?

[ghost]

we're not using maud anymore. we do depend on maud_htmlescape for escaping, and ammonia for sanitization - i was more wondering where these should be hooked into posticle (should it directly dep on them and do everything out-of-the-box? should we pass escapers in or something?)

Yeah, posticle should handle that since it's responsible for generating HTML in this scenario.

i'm a bit confused about what your resolver trait is supposed to do there. it's taking an entity and... returning one? generally, we want to take a username/domain/hashtag/image/whatever and get a URL, right?

Sorry, yeah, needs documentation!

The resolver allows you to link up mentions and hashtags by returning a different entity (Some(Entity::...), or to render everything as text by returning the same entity (Some(entity)), or to exclude the entity entirely by returning None.

I've set things up this way for a couple of reasons so that you can blacklist/whitelist hashtags, mentioned users or entire mentioned domains by either not linking them, or by removing them entirely.

The following resolver would remove all entities, and therefore all content:

impl Resolver for ResolveNothing {
    fn resolve(&self, entity: Entity) -> Option<Entity> {
        None
    }
}

This resolver would link hashtags and mentions to the example.com domain:

impl Resolver for ResolveNothing {
    fn resolve(&self, entity: Entity) -> Option<Entity> {
        match entity {
            Entity::Hashtag(name) => Some(Entity::Link(
                format!("#{}", name),
                format!("https://example.com/tag/{}", name),
            )),
            Entity::Mention(username, domain) => {
                if let Some(domain) = domain {
                    Some(Entity::Link(
                        format!("@{}@{}", username, domain),
                        format!("https://{}/user/{}", domain, username),
                    ))
                } else {
                    Some(Entity::Link(
                        format!("@{}", username),
                        format!("https://example.com/user/{}", username),
                    ))
                }
            },
            _ => Some(entity),
        }
    }
}

[barzamin]

ahhhhh, so this ties up nicely with the AST/internal representation of formatting, and is almost more like an EntityTransformer than a resolver. i like it 😃

[ghost]

After this bit of work is done I'd like to tackle basic Markdown support, the features I think are most important:

• Bold
• Italic
• Code
• Quote blocks
• Code blocks

Links, and lists are actually quite complicated to implement so I'd like to keep it simple for the time being.

Once that is done I'd like to split posticle into it's own crate so that I can then look into making a ruby package with helix, then make a PR for mastodon to use our implementation.

Then finally I'd like to make a JavaScript version (wasm/transpiled?) that can be used on the frontend and do cool stuff like do a character count based on Text tokens only.

[ashkitten]

wasm would be heckin awesome!! also yeah those are what i'd recommend for basic markdown, it's what i'd use the most anyway

[ghost]

Sounds like a plan then! 🎉

[ashkitten]

for wasm i've seen stdweb recommended, which looks really nice

[ghost]

Just a heads up, I'm delaying doing this until the new year. I've got a lot on my plate at the minute, including moving house and I promised someone I'd take a look at a PeerTube issue.

Do ping me if there are Posticle bugs that need attention in the mean time!

[barzamin]

@measlytwerp don't worry, that's totally okay 💜