Import content from an authenticated Twitter account

[azurelunatic]

Title:
Import content from an authenticated Twitter account

Area:
importing, Twitter

Summary:
Import the day's tweets from a twitter account under the user's control. This would eliminate the need to share one's Dreamwidth password with an external service, and might be able to have settings tailored to take better advantage of Dreamwidth's features.

Description:
Some people like to import the day's tweets from their Twitter accounts to Dreamwidth. (For archival purposes, so friends who don't do Twitter can read it, for people who use Twitter to collect interesting links, et cetera.) Some people don't care to see Twitter imports showing up on their reading lists.

Having a Dreamwidth-side importer is likely to be useful to both of these parties: people who do import can do so more easily and using more of Dreamwidth's features; having a Twitter-importer that supports tags means more people's imports will be tagged consistently and will therefore let paid users set up their default reading filters more accurately to exclude those tags; if the Twitter-importer consistently wraps the imported entry in <div class="LoudTwitter">, it will remain compatible with most existing make-Twitter-imports-disappear style hacks.


Must authenticate to Twitter properly (I believe Twitter now requires OAuth). While one technically can scrape up an unlocked account, authenticating to the Twitter account means that the user has control over that account, and therefore should have authorization to repost to Dreamwidth. Authentication also means locked accounts are visible.

Must collect all tweets daily, whether from a locked or unlocked account. (The Twitter account being locked should not prevent from public posting, because some people are merely locked to keep out the Twitter spambots.)

Should be able to set a security for all imported-from-Twitter entries that is distinct from the minimum security; some people prefer to post their Twitter imports to a custom security group even if their minimum is public. (Locked twitter account and locked entries would maintain security for people who need it.) For UI, the set-minimum-security setting in the 'Manage' area should link back and forth to the Twitter-imported-entries security setting, in case someone found one but wanted the other.

What should happen if someone switches from public on Twitter to locked on Twitter? Is this something the service should look out for and send a notification about? Probably a notification paired with a "Match the security setting on Twitter (if there are multiple, go with the most restrictive)" setting for entry security (as well as the usual selections) would solve that problem.

Should reproduce the tweet as seen on Twitter as faithfully as possible, including http:// links working clickably, @links working correctly, and #hashtags linking to a Twitter search for the hashtag. (Known incompatibility: mentioning IRC channels on Twitter turns them into hashtags, even though you might prefer to have the IRC channels link usefully.) (It would be nice to optionally include in-reply-to, location, and other doodads, but even though retrieving them through the API may be easy, would they be really wanted in an import?)

Should collect retweets via the most modern method, as using the old method makes the end cut off if it's long.

Options for:
timestamps or no timestamps; timestamp format.
linkback to original tweet, or no link back.

Should be able to set tags to be applied to every imported entry (suggest "twitter"). (This will help with paid users' filtering, but since exclude-a-tag is at this point per-user, as long as the one user keeps consistent tagging, it shouldn't matter whether the tag is 'twitter' or 'chickens shifting in the night'.)

Would be nice to have an option that would try to use any hashtags in the import as tags on the entry (but would fail back to only existing tags if it went over the number of allowable tags, and would send a message that tagging on this automatically posted entry did not work properly, and maybe send a list of the tags that could not be created).

Should be able to set userpic (if they have a userpic called "twitter", maybe have it pre-selected for them?)

Should be able to customize the subject, with variables to insert the date and/or number of tweets if you want to, in your exact preferred configuration.

Should be able to cut tweets, either all of them entirely, or after some small reasonable number. My service does 5; LJ's does 10. Either of these are reasonable.

Should be able to customize the cut text.

What time options should there be for posting? Twittinesis posts sometime in the middle of their night. LiveJournal's posts at noon. My script host posts five minutes before midnight. Would there be a way to allow customization of posting time that does not affect site performance? It seems as though allowing a post to be set at an exact time would possibly set up situations where too many people would try for the same thing and bog things down. Though a reasonable effort should be made to a) have the things post at the same time for the same person, and b) get the tweets of a particular span, most popular likely to be the whole day's twittery.

Should the time on the entry correspond to the last time in the included tweets (and/or the time the user wants the thing to run), or to the actual time it's sent? (If the time it should have been posted and the time it was posted differ, what happens if the journal owner posts in between then? Maybe try again with the actual time?)

Should have the option to use the 'do not show on reading pages' option, in case someone wants that.

Should be robust enough to try again if Twitter failwhales for any reason while attempting to fetch.

Should be able to choose whether to import all tweets, or only tweets that lack an in-reply-to. (without in-reply-to is good for not having half-a-conversation, but bad if what you wanted to do was archive it all for your own use). There's some debate about whether the without-@ should be tweets that don't start with an @-name, or tweets that have an in-reply-to -- various clients handle things differently, and sometimes break in-reply-to when it should be there, or one was starting a conversation.

Might want to set up more than one import (paid feature?), from either multiple accounts (some people have lots) or multiple imports for the same account (one public, cut and without @replies, one private, uncut, with @replies, for example)

Should expand as many as possible shortened URLs (including t.co links), to prevent the failure of a shortening site from making your old links useless. (There are, however, some people who use shorteners on purpose? What about them?)

Since it's an archival under the control of the person who wrote the tweets (the ones that aren't retweets), rather than a fully operational client, I'm not sure how much of this stuff applies, but it's potentially useful: http://dev.twitter.com/pages/display_guidelines

While it would be nice for people to be able to run a Twitter-importer like an importer from other journal sites, and slurp up all the data for past days, there are limits on the Twitter API for how much you can extract from the site: the REST API lets you pull up to approximately 3200 items, the other APIs top out at 800. (So if someone SOMEHOW MANAGES to tweet over 800 times in one day, they'll probably not have all their tweets ship. However, I'm darned chatty on Twitter, and I don't believe I've ever gone over 200.)

Mood, location, music, adult content -- can these be left alone, or should there be settings for them?

Should there be the ability to override the default crosspost settings for this type of entry?

Comments/comment screening?

Poll #6506 Import content from an authenticated Twitter account

Open to: Registered Users, detailed results viewable to: All, participants: 69

View Respondents

This suggestion:

View Answers

Should be implemented as-is.
32 (46.4%)

Should be implemented with changes. (please comment)
9 (13.0%)

Shouldn't be implemented.
4 (5.8%)

(I have no opinion)
24 (34.8%)

(Other: please comment)
0 (0.0%)