importer/twitter: archive the URL shortlinks' destination

[bradfitz]

If you make a tweet containing a URL, it gets short-linkified with Twitter's URL shortener and the URL we get back from Twitter's API only has the Twitter shortlink URL, not the original URL as tweeted.

Example:

We should index those t.co shortlinks and record somewhere where they actually point to, in case that shortlink service goes down.

We could just store new attributes on the permanode, like:

  shortlink:<t.co-suffix> = http://domain.tld/target

[sbshah97]

Hey, could I help with this issue ?
Looks interesting to me and would love to help understand the project as well.

[rjp]

Looks like the expanded information is in the entities block of the API response these days, e.g.

json.entities.Urls[0].Url = "https://t.co/yVlNlAnRtr"; json.user.entities.Url.Urls[0].Display_url = "lawyersgunsmoneyblog.com/author/Scott-L…"; json.user.entities.Url.Urls[0].Expanded_url = "http://www.lawyersgunsmoneyblog.com/author/Scott-Lemieux/";

I can probably have a look at adding these as new attributes (entities also contains links to mentioned users, etc. which might also be handy to store.)

Is there a document or discussion about naming of attributes (such as what to do if there's more than one shortlink, etc.) anywhere? Or any support for hierarchy in the attributes?

The attr search doesn't seem to support wildcards which means attr:shortlink:~ (or attr:~shortlink:~) won't find shortlink.0 (or other variant).