How to automatically unshorten certain domain's URLs?

[PacoBell]

I want to always unshorten certain shortened URLs. Am I doing this correctly? It doesn't seem to do it for me in the Pattern Checker, although I do see that it apparently parsed the regex correctly. For example:

URL: https://flip.it/LsPW_X

"Unshorten flip.it links": {
    "regex": "^https?:\/\/flip.it\/(.*)",
    "decode": "true",
    "automatic": "true",
    "enabled": "true"
  }

[PabloOQ]

Short answer, those shortened URLs can't be unshorten by Pattern Checker, and the Unshorten module, which is responsible of unshorting those URLs is not working at this moment.

Long answer:
There are 2 types of shortened URLs, I don't remember the names but:

• First type is the one you posted, for those you need to connect to their server or know where it leads, for that we had the Unshortener module, in which the URL is sent to unshorten.me, then they retrieve the URL, store it and sent it back. That way the owner of the web, flip.it in this case, doesn't know your IP, so it can't track who requested the URL. Sadly unshorten.me has been down for months now, I think it started malfunctioning on april, I don't expect it will be fixed. Thinking about it, maybe we should disable the module.
• Second type can be unshorten just by looking at the URL itself, those URLs tend to be really long. This will be handled mainly by the ClearURLs database used by the Cleaner Module, although you can do it with regex magic on the Pattern Checker module. If you want to see an example of this, go to YouTube, open any video with URLs in their description and copy/tap them (if on phone, URLCheck may apply the Cleaner Module automatically, so go back in the history), somewhere in there you can find the query q, that's the destination URL.

Currently this module doesn't make any connection to the internet, as such is impossible for the module to get the destination URL of the first type of shortened URLs. I think the problem here comes from a misunderstanding of the fields in the Pattern Checker module. Those are explained in the advanced editor of the module, but what decode does is change some characters that are encoded.
For example, if you encode https://github.com/TrianguloY/UrlChecker/discussions/317 you get https%3A%2F%2Fgithub.com%2FTrianguloY%2FUrlChecker%2Fdiscussions%2F317, decoding is the reverse operation (you can test it yourself here https://www.urlencoder.org/). The decode field mainly exists to extract URLs that are embedded in the original (for redirection purposes), as usually, embedded URLs are encoded first.

Let me know if you need more help.

[PacoBell]

Sadly unshorten.me has been down for months now

Is that really the case now? If I manually tap the "Unshort" button, it seems to properly unshorten that URL. Is it not actually using the unshorten.me anonymity service and just using something like a WebView to do the redirect from my device?

[PabloOQ]

Glad I was wrong! Unshorten.me still fails to resolve t.co URLs so I thought it was still broken.

[TrianguloY]

I think t.co were broken even before the "shutdown", maybe due to the restrictions that twitter x has on anonymous accounts, no idea

[TrianguloY]

Great answer Pablo!

Two important distinctions though:

Both the Url cleaner module and the Pattern checker module works offline. As explained the url is already there, but encoded, so both modules will extract and decode it.

If the url only contains an identifier, you need to ask the server for the final url. You have two alternatives here:

The status code module will perform an http GET petition through the network, and will extract any redirection found on the headers. This is similar as using a hidden webview, but there is no webview involved. The page itself is requested, but isn't even read (which means that redirections based on JavaScript won't work).

The other alternative is the Unshortener module, which as Pablo explained does a similar thing but using an external service. The module send a petition to the unshorten.me api to "unshort" the url, and expects a response. The unshorter.me servers will fetch the page, so this is both more secure (your device will not contact the url directly) and less secure (you depend and must trust the third party api).

And yes, this module was broken a few months ago because the free api was shut down.
BUT I can confirm that it works for me now as well...maybe they have restored it? I don't know. I still want to allow other apis though...at some point.

[PacoBell]

Okay, thanks for confirming that the API service works. But that still leaves me wondering why my Pattern Checker config doesn't automatically display the unshortened URL for that domain in the UI. Am I also misunderstanding how that "automatic" field works? Also, when activating that link, I see my rule show up under the Pattern Checker section, but its grayed out and not actionable. Why is that?

[TrianguloY]

There are three parts to answer that question:

1. If you want the Pattern checker module to show the extracted url (and not be gray), you need to add the replacement field. The content of that field will be used for the regex replacement method.

For example, if you write the following)

"Unshorten flip.it links": { 
   "regex": "^https?:\/\/flip.it\/(.*)",
   "replacement": "$1"
}

With the input https://flip.it/LsPW_X after pressing the apply button you will get LsPW_X . Remember that this is a completely offline thing, and that sequence is just an identifier that only the flip servers know where it points to.

2. The decoded parameter will only decode the grouped elements (in this case the LsPW_X) which is already decoded, so it will do nothing. Encoded urls look like the example from Pablo.

For example, with the totally invalid link https://flip.it/https%3A%2F%2Fgithub.com%2FTrianguloY%2FUrlChecker%2Fdiscussions%2F317 the pattern above will output https%3A%2F%2Fgithub.com%2FTrianguloY%2FUrlChecker%2Fdiscussions%2F317. However, if you add the "decode":true property, the output will be https://github.com/TrianguloY/UrlChecker/discussions/317

3. The automatic property will automatically apply the pattern. This is equivalent as pressing the apply button, but automatic.

[PacoBell]

Oh, huh, it sounds like the Pattern Checker module will not perform the task I desire. Is there any current module that can do automatic, online unshortening operations or should that be a Feature Request?

[TrianguloY]

Either use the Status code module, which should unshorten urls by making the petition from your phone itself,

Or use the Unshortener module, which will use unshorten.me (and hope that it doesn't break again before the app is updated to allow other unshortening providers)