Import content from an authenticated Twitter account

[azurelunatic]

Title:
Import content from an authenticated Twitter account

Area:
importing, Twitter

Summary:
Import the day's tweets from a twitter account under the user's control. This would eliminate the need to share one's Dreamwidth password with an external service, and might be able to have settings tailored to take better advantage of Dreamwidth's features.

Description:
Some people like to import the day's tweets from their Twitter accounts to Dreamwidth. (For archival purposes, so friends who don't do Twitter can read it, for people who use Twitter to collect interesting links, et cetera.) Some people don't care to see Twitter imports showing up on their reading lists.

Having a Dreamwidth-side importer is likely to be useful to both of these parties: people who do import can do so more easily and using more of Dreamwidth's features; having a Twitter-importer that supports tags means more people's imports will be tagged consistently and will therefore let paid users set up their default reading filters more accurately to exclude those tags; if the Twitter-importer consistently wraps the imported entry in <div class="LoudTwitter">, it will remain compatible with most existing make-Twitter-imports-disappear style hacks.


Must authenticate to Twitter properly (I believe Twitter now requires OAuth). While one technically can scrape up an unlocked account, authenticating to the Twitter account means that the user has control over that account, and therefore should have authorization to repost to Dreamwidth. Authentication also means locked accounts are visible.

Must collect all tweets daily, whether from a locked or unlocked account. (The Twitter account being locked should not prevent from public posting, because some people are merely locked to keep out the Twitter spambots.)

Should be able to set a security for all imported-from-Twitter entries that is distinct from the minimum security; some people prefer to post their Twitter imports to a custom security group even if their minimum is public. (Locked twitter account and locked entries would maintain security for people who need it.) For UI, the set-minimum-security setting in the 'Manage' area should link back and forth to the Twitter-imported-entries security setting, in case someone found one but wanted the other.

What should happen if someone switches from public on Twitter to locked on Twitter? Is this something the service should look out for and send a notification about? Probably a notification paired with a "Match the security setting on Twitter (if there are multiple, go with the most restrictive)" setting for entry security (as well as the usual selections) would solve that problem.

Should reproduce the tweet as seen on Twitter as faithfully as possible, including http:// links working clickably, @links working correctly, and #hashtags linking to a Twitter search for the hashtag. (Known incompatibility: mentioning IRC channels on Twitter turns them into hashtags, even though you might prefer to have the IRC channels link usefully.) (It would be nice to optionally include in-reply-to, location, and other doodads, but even though retrieving them through the API may be easy, would they be really wanted in an import?)

Should collect retweets via the most modern method, as using the old method makes the end cut off if it's long.

Options for:
timestamps or no timestamps; timestamp format.
linkback to original tweet, or no link back.

Should be able to set tags to be applied to every imported entry (suggest "twitter"). (This will help with paid users' filtering, but since exclude-a-tag is at this point per-user, as long as the one user keeps consistent tagging, it shouldn't matter whether the tag is 'twitter' or 'chickens shifting in the night'.)

Would be nice to have an option that would try to use any hashtags in the import as tags on the entry (but would fail back to only existing tags if it went over the number of allowable tags, and would send a message that tagging on this automatically posted entry did not work properly, and maybe send a list of the tags that could not be created).

Should be able to set userpic (if they have a userpic called "twitter", maybe have it pre-selected for them?)

Should be able to customize the subject, with variables to insert the date and/or number of tweets if you want to, in your exact preferred configuration.

Should be able to cut tweets, either all of them entirely, or after some small reasonable number. My service does 5; LJ's does 10. Either of these are reasonable.

Should be able to customize the cut text.

What time options should there be for posting? Twittinesis posts sometime in the middle of their night. LiveJournal's posts at noon. My script host posts five minutes before midnight. Would there be a way to allow customization of posting time that does not affect site performance? It seems as though allowing a post to be set at an exact time would possibly set up situations where too many people would try for the same thing and bog things down. Though a reasonable effort should be made to a) have the things post at the same time for the same person, and b) get the tweets of a particular span, most popular likely to be the whole day's twittery.

Should the time on the entry correspond to the last time in the included tweets (and/or the time the user wants the thing to run), or to the actual time it's sent? (If the time it should have been posted and the time it was posted differ, what happens if the journal owner posts in between then? Maybe try again with the actual time?)

Should have the option to use the 'do not show on reading pages' option, in case someone wants that.

Should be robust enough to try again if Twitter failwhales for any reason while attempting to fetch.

Should be able to choose whether to import all tweets, or only tweets that lack an in-reply-to. (without in-reply-to is good for not having half-a-conversation, but bad if what you wanted to do was archive it all for your own use). There's some debate about whether the without-@ should be tweets that don't start with an @-name, or tweets that have an in-reply-to -- various clients handle things differently, and sometimes break in-reply-to when it should be there, or one was starting a conversation.

Might want to set up more than one import (paid feature?), from either multiple accounts (some people have lots) or multiple imports for the same account (one public, cut and without @replies, one private, uncut, with @replies, for example)

Should expand as many as possible shortened URLs (including t.co links), to prevent the failure of a shortening site from making your old links useless. (There are, however, some people who use shorteners on purpose? What about them?)

Since it's an archival under the control of the person who wrote the tweets (the ones that aren't retweets), rather than a fully operational client, I'm not sure how much of this stuff applies, but it's potentially useful: http://dev.twitter.com/pages/display_guidelines

While it would be nice for people to be able to run a Twitter-importer like an importer from other journal sites, and slurp up all the data for past days, there are limits on the Twitter API for how much you can extract from the site: the REST API lets you pull up to approximately 3200 items, the other APIs top out at 800. (So if someone SOMEHOW MANAGES to tweet over 800 times in one day, they'll probably not have all their tweets ship. However, I'm darned chatty on Twitter, and I don't believe I've ever gone over 200.)

Mood, location, music, adult content -- can these be left alone, or should there be settings for them?

Should there be the ability to override the default crosspost settings for this type of entry?

Comments/comment screening?

Poll #6506 Import content from an authenticated Twitter account

Open to: Registered Users, detailed results viewable to: All, participants: 69

View Respondents

This suggestion:

View Answers

Should be implemented as-is.
32 (46.4%)

Should be implemented with changes. (please comment)
9 (13.0%)

Shouldn't be implemented.
4 (5.8%)

(I have no opinion)
24 (34.8%)

(Other: please comment)
0 (0.0%)

Comments

[assassinscloak]

Rather than "all tweets" or "all tweets except @replies", I'd like and "all tweets with a #dw tag" (example tag).

Rather in the manner that only my #fb tagged tweets appear on Facebook, but all in one post rather than individually as on Facebook.

[liv]

I think this is kind of a good idea, except that I don't think DW should be trying to act as a full-featured Twitter client. It's partly that you described your suggestion very thoroughly, but this sounds like a lot of effort for something that isn't a core part of what DW does. And if Twitter breaks or changes something, people will get angry at DW / DW will have to tweak stuff to fit with the new system... I'm not convinced it's worth it.

really really really really really want twitter imports

[faithofone]

I voted "with changes" because there were questions in the suggestion that need answers. They are not necessarily in the order in which they were asked.

I believe Twitter now requires OAuth

Yes they do.

What should happen if someone switches from public on Twitter to locked on Twitter?

I don't think it should make any difference if the feed is locked or not. It should respect whatever security settings you have set for the entries on the Dw side.

(It would be nice to optionally include in-reply-to, location, and other doodads, but even though retrieving them through the API may be easy, would they be really wanted in an import?)

Yes for reply to and location. I know that now that I actually user a twitter client that submits reply to data that I find it extremely useful. I've also seen people use the location data as an integral part of the tweets. "I'm here" for example, with location data available. What other doodads are there?

What time options should there be for posting?

Have this sort of work like the current imports. Allow you to specify a time of day for the import to be made, but when the import actually runs it gets stuck into a queue with all the other imports that people have requested. Have it know when you wanted the import done, and have it take all tweets for the past 24 hours from that time.

I don't have an answer to which timestamp to use for posting, the one that you wanted it done at or the one it was actually done at.

Should be robust enough to try again if Twitter failwhales for any reason while attempting to fetch.

This is really important. I'm currently having crossposting issues with LJ because of their DDoS attacks, and it's annoying as poo to have to keep going back and trying to crosspost it again.

There's some debate about whether the without-@ should be tweets that don't start with an @-name, or tweets that have an in-reply-to -- various clients handle things differently, and sometimes break in-reply-to when it should be there, or one was starting a conversation.

Since SMS doesn't have support for "in reply to", I think relying upon that as the sole determiner for if something is an @reply would be bad. All "in reply to" posts start with @someone, so I think that should be the defining feature.

Offer the options of "exclude posts that start '@someone'", "only include posts with #dw" and "import all posts".

Might want to set up more than one import (paid feature?)

Yes paid feature. Analogous to being able to crosspost to more than one external site.

linkback to original tweet, or no link back.

Also, can we have these links labeled clearly what they are please? [source] is much more helpful than #.

Would be nice to have an option that would try to use any hashtags in the import as tags on the entry (but would fail back to only existing tags if it went over the number of allowable tags, and would send a message that tagging on this automatically posted entry did not work properly, and maybe send a list of the tags that could not be created).

Also would be nice if these tags could be prepended with "Twitter:" or something so that those of us who want to keep the twitter tags separate from the Dw tags can.

(There are, however, some people who use shorteners on purpose? What about them?)

We may be getting into too many options, but I think that's a completely reasonable "respect url shorteners" option.

Mood, location, music, adult content -- can these be left alone, or should there be settings for them? Should there be the ability to override the default crosspost settings for this type of entry? Comments/comment screening?

All of this I think should work the same as any other "scheduled entry" or whatever we're calling the future feature that allows us to post the same thing at regular intervals. Because that's what we're talking about here. We're talking about making a mashup between running the importer and making a scheduled entry. I don't know a lot of how that whole thing is going to work, but I think that whatever interface and logic is used for that project can be applied here.

The way I see this working is that you've got a page set up where you start setting up the Twitter imports. This is where you OAuth and set up stuff like if you want to include replies and anything else that applies only to interacting with the twitter site.

Then you get directed to the update page, which is where you can set all your security levels, comment settings etc. The contents of the update page would be something like <twitter import />.

And then it saves it in the same way that it saves other scheduled entries, and when it comes time to post the entry, it throws it over to the import queue, and then when the import gets done it posts it.

Of course, I could be talking out my rear here, since I know nothing of how any of Dw works on the back end.

[delladea]

I would LOVE this so much if it were to happen.

[slyfox]

A little late to the party, but I would love this as well. I believe that Livejournal now has a "Twitter Connect" feature, but it only imports public tweets. I would love to be able to import all of my tweets and then put them under a special filter for only people who opt in/don't use twitter already. Since it's very difficult to go through your old tweets on twitter if they're protected, it would be awesome if I could have a "daily digest" of them on my journal.

[azurelunatic]

Option to attempt to detect media in tweets (YouTube videos, for example) and embed them. (What about images? That could get page-breaky and/or bandwidth-heavy. Below the cut only?)

[azurelunatic]

Until Twitter clarifies its rules about allowing users to initiate backup of their own content to "cloud-based" services, it looks like this is a non-starter. :<

[jss]

It's been a few years; any chance this could get revisited?

Twitter Importer

[olivia_sutton]

OMG - Dreamwidth still doesn't have this? I had it set up in Live Journal to have all my Tweets imported to Live Journal. I did this for archiving purposes. Also, the import would occur once a day, which is preferable to the "every time you post" which happens on Tumblr.
It's been over two years - please fix this so we can at least import Tweets to Dreamwidth. At least for archiving.
Also, PLEASE create an Android app for Dreamwidth.
Also, PLEASE allow importing of images to journals. Again, it was possible to import images to Live Journal as part of a journal. I frankly do not understand why that's impossible with Dreamwidth. I checked the FAQ and it said the "image must be on the Internet" but that really makes no sense - I don't want to link to someone else's picture. There are lots of problems with that - I'd rather post my own content to my journal.
There should be a way to force journal style on imported content. None of my journal entries look the same because they were composed at different times. It looks amateurish and I can't really find a way to fix it.
That's all for now.
Olivia

[nlbarber]

I'll add another comment in the hopes of keeping this idea alive. It's the lack of this feature that keeps me on LJ, and I was reminded of this when LJ needed a reauthorization to access my Twitter. I'm hoping that whatever allows LJ to get around the Twitter TOS on importing content can apply to DW as well, as I would so like to see this feature implemented.
I don't need the hashtag idea though could live with it if implemented, and I only need public tweets for my uses--like others, it's my archive of tweets plus makes them accessible for a few friends who don't use twitter.