azurelunatic: A glittery black pin badge with a blue holographic star in the middle. (Default)
Azure Jane Lunatic (Azz) 🌺 ([personal profile] azurelunatic) wrote in [site community profile] dw_suggestions2011-04-02 06:37 pm

Import content from an authenticated Twitter account

Title:
Import content from an authenticated Twitter account

Area:
importing, Twitter

Summary:
Import the day's tweets from a twitter account under the user's control. This would eliminate the need to share one's Dreamwidth password with an external service, and might be able to have settings tailored to take better advantage of Dreamwidth's features.

Description:
Some people like to import the day's tweets from their Twitter accounts to Dreamwidth. (For archival purposes, so friends who don't do Twitter can read it, for people who use Twitter to collect interesting links, et cetera.) Some people don't care to see Twitter imports showing up on their reading lists.

Having a Dreamwidth-side importer is likely to be useful to both of these parties: people who do import can do so more easily and using more of Dreamwidth's features; having a Twitter-importer that supports tags means more people's imports will be tagged consistently and will therefore let paid users set up their default reading filters more accurately to exclude those tags; if the Twitter-importer consistently wraps the imported entry in <div class="LoudTwitter">, it will remain compatible with most existing make-Twitter-imports-disappear style hacks.


Must authenticate to Twitter properly (I believe Twitter now requires OAuth). While one technically can scrape up an unlocked account, authenticating to the Twitter account means that the user has control over that account, and therefore should have authorization to repost to Dreamwidth. Authentication also means locked accounts are visible.

Must collect all tweets daily, whether from a locked or unlocked account. (The Twitter account being locked should not prevent from public posting, because some people are merely locked to keep out the Twitter spambots.)

Should be able to set a security for all imported-from-Twitter entries that is distinct from the minimum security; some people prefer to post their Twitter imports to a custom security group even if their minimum is public. (Locked twitter account and locked entries would maintain security for people who need it.) For UI, the set-minimum-security setting in the 'Manage' area should link back and forth to the Twitter-imported-entries security setting, in case someone found one but wanted the other.

What should happen if someone switches from public on Twitter to locked on Twitter? Is this something the service should look out for and send a notification about? Probably a notification paired with a "Match the security setting on Twitter (if there are multiple, go with the most restrictive)" setting for entry security (as well as the usual selections) would solve that problem.

Should reproduce the tweet as seen on Twitter as faithfully as possible, including http:// links working clickably, @links working correctly, and #hashtags linking to a Twitter search for the hashtag. (Known incompatibility: mentioning IRC channels on Twitter turns them into hashtags, even though you might prefer to have the IRC channels link usefully.) (It would be nice to optionally include in-reply-to, location, and other doodads, but even though retrieving them through the API may be easy, would they be really wanted in an import?)

Should collect retweets via the most modern method, as using the old method makes the end cut off if it's long.

Options for:
timestamps or no timestamps; timestamp format.
linkback to original tweet, or no link back.

Should be able to set tags to be applied to every imported entry (suggest "twitter"). (This will help with paid users' filtering, but since exclude-a-tag is at this point per-user, as long as the one user keeps consistent tagging, it shouldn't matter whether the tag is 'twitter' or 'chickens shifting in the night'.)

Would be nice to have an option that would try to use any hashtags in the import as tags on the entry (but would fail back to only existing tags if it went over the number of allowable tags, and would send a message that tagging on this automatically posted entry did not work properly, and maybe send a list of the tags that could not be created).

Should be able to set userpic (if they have a userpic called "twitter", maybe have it pre-selected for them?)

Should be able to customize the subject, with variables to insert the date and/or number of tweets if you want to, in your exact preferred configuration.

Should be able to cut tweets, either all of them entirely, or after some small reasonable number. My service does 5; LJ's does 10. Either of these are reasonable.

Should be able to customize the cut text.

What time options should there be for posting? Twittinesis posts sometime in the middle of their night. LiveJournal's posts at noon. My script host posts five minutes before midnight. Would there be a way to allow customization of posting time that does not affect site performance? It seems as though allowing a post to be set at an exact time would possibly set up situations where too many people would try for the same thing and bog things down. Though a reasonable effort should be made to a) have the things post at the same time for the same person, and b) get the tweets of a particular span, most popular likely to be the whole day's twittery.

Should the time on the entry correspond to the last time in the included tweets (and/or the time the user wants the thing to run), or to the actual time it's sent? (If the time it should have been posted and the time it was posted differ, what happens if the journal owner posts in between then? Maybe try again with the actual time?)

Should have the option to use the 'do not show on reading pages' option, in case someone wants that.

Should be robust enough to try again if Twitter failwhales for any reason while attempting to fetch.

Should be able to choose whether to import all tweets, or only tweets that lack an in-reply-to. (without in-reply-to is good for not having half-a-conversation, but bad if what you wanted to do was archive it all for your own use). There's some debate about whether the without-@ should be tweets that don't start with an @-name, or tweets that have an in-reply-to -- various clients handle things differently, and sometimes break in-reply-to when it should be there, or one was starting a conversation.

Might want to set up more than one import (paid feature?), from either multiple accounts (some people have lots) or multiple imports for the same account (one public, cut and without @replies, one private, uncut, with @replies, for example)

Should expand as many as possible shortened URLs (including t.co links), to prevent the failure of a shortening site from making your old links useless. (There are, however, some people who use shorteners on purpose? What about them?)

Since it's an archival under the control of the person who wrote the tweets (the ones that aren't retweets), rather than a fully operational client, I'm not sure how much of this stuff applies, but it's potentially useful: http://dev.twitter.com/pages/display_guidelines

While it would be nice for people to be able to run a Twitter-importer like an importer from other journal sites, and slurp up all the data for past days, there are limits on the Twitter API for how much you can extract from the site: the REST API lets you pull up to approximately 3200 items, the other APIs top out at 800. (So if someone SOMEHOW MANAGES to tweet over 800 times in one day, they'll probably not have all their tweets ship. However, I'm darned chatty on Twitter, and I don't believe I've ever gone over 200.)

Mood, location, music, adult content -- can these be left alone, or should there be settings for them?

Should there be the ability to override the default crosspost settings for this type of entry?

Comments/comment screening?

Poll #6506 Import content from an authenticated Twitter account
Open to: Registered Users, detailed results viewable to: All, participants: 64


This suggestion:

View Answers

Should be implemented as-is.
28 (43.8%)

Should be implemented with changes. (please comment)
9 (14.1%)

Shouldn't be implemented.
4 (6.2%)

(I have no opinion)
23 (35.9%)

(Other: please comment)
0 (0.0%)

[personal profile] assassinscloak 2011-04-05 12:55 pm (UTC)(link)
Rather than "all tweets" or "all tweets except @replies", I'd like and "all tweets with a #dw tag" (example tag).

Rather in the manner that only my #fb tagged tweets appear on Facebook, but all in one post rather than individually as on Facebook.
sofiaviolet: drawing of three violets and three leaves (Default)

[personal profile] sofiaviolet 2011-04-05 02:57 pm (UTC)(link)
I very much like this idea. I don't use the #fb thing (in part because I haven't figured out how to set it up on the Facebook end and in part because my client handles both Twitter and Facebook, so I can directly update both), but I would use a special hashtag to import tweets to Dreamwidth.

[personal profile] assassinscloak 2011-04-05 05:05 pm (UTC)(link)
For the curious: Selective Twitter :-)
thedivinegoat: A sheep dreaming of dreamwidth, Text TheDivineGoat Anti_Spam (Antispam Spring Dreamsheep)

[personal profile] thedivinegoat 2011-04-05 03:07 pm (UTC)(link)
+1
susanreads: my avatar, a white woman with brown hair and glasses (Default)

[personal profile] susanreads 2011-04-05 08:13 pm (UTC)(link)
I don't tweet, but this sounds like the most useful thing.
kyrielle: A photo of kyrielle, in profile, turned slightly toward the viewer (Default)

[personal profile] kyrielle 2011-04-06 02:34 am (UTC)(link)
+1 but it should be smarter than the #fb one and not require the tag to be at the end. :P Otherwise I'd have a very hard time sending to both. :P
liv: Stylised sheep with blue, purple, pink horizontal stripes, and teacup brand, dreams of Dreamwidth (sheeeep)

[personal profile] liv 2011-04-05 02:53 pm (UTC)(link)
I think this is kind of a good idea, except that I don't think DW should be trying to act as a full-featured Twitter client. It's partly that you described your suggestion very thoroughly, but this sounds like a lot of effort for something that isn't a core part of what DW does. And if Twitter breaks or changes something, people will get angry at DW / DW will have to tweak stuff to fit with the new system... I'm not convinced it's worth it.

[personal profile] feathertail 2011-04-05 06:48 pm (UTC)(link)
I'd vote no just because it seems like this isn't a core competency / shouldn't be a priority, but it seems pretty important to you and your friend and I don't see why volunteers shouldn't be able to add it.

really really really really really want twitter imports

[personal profile] faithofone 2011-04-05 08:07 pm (UTC)(link)
I voted "with changes" because there were questions in the suggestion that need answers. They are not necessarily in the order in which they were asked.

I believe Twitter now requires OAuth

Yes they do.

What should happen if someone switches from public on Twitter to locked on Twitter?

I don't think it should make any difference if the feed is locked or not. It should respect whatever security settings you have set for the entries on the Dw side.

(It would be nice to optionally include in-reply-to, location, and other doodads, but even though retrieving them through the API may be easy, would they be really wanted in an import?)

Yes for reply to and location. I know that now that I actually user a twitter client that submits reply to data that I find it extremely useful. I've also seen people use the location data as an integral part of the tweets. "I'm here" for example, with location data available. What other doodads are there?

What time options should there be for posting?

Have this sort of work like the current imports. Allow you to specify a time of day for the import to be made, but when the import actually runs it gets stuck into a queue with all the other imports that people have requested. Have it know when you wanted the import done, and have it take all tweets for the past 24 hours from that time.

I don't have an answer to which timestamp to use for posting, the one that you wanted it done at or the one it was actually done at.

Should be robust enough to try again if Twitter failwhales for any reason while attempting to fetch.

This is really important. I'm currently having crossposting issues with LJ because of their DDoS attacks, and it's annoying as poo to have to keep going back and trying to crosspost it again.

There's some debate about whether the without-@ should be tweets that don't start with an @-name, or tweets that have an in-reply-to -- various clients handle things differently, and sometimes break in-reply-to when it should be there, or one was starting a conversation.

Since SMS doesn't have support for "in reply to", I think relying upon that as the sole determiner for if something is an @reply would be bad. All "in reply to" posts start with @someone, so I think that should be the defining feature.

Offer the options of "exclude posts that start '@someone'", "only include posts with #dw" and "import all posts".

Might want to set up more than one import (paid feature?)

Yes paid feature. Analogous to being able to crosspost to more than one external site.

linkback to original tweet, or no link back.

Also, can we have these links labeled clearly what they are please? [source] is much more helpful than #.

Would be nice to have an option that would try to use any hashtags in the import as tags on the entry (but would fail back to only existing tags if it went over the number of allowable tags, and would send a message that tagging on this automatically posted entry did not work properly, and maybe send a list of the tags that could not be created).

Also would be nice if these tags could be prepended with "Twitter:" or something so that those of us who want to keep the twitter tags separate from the Dw tags can.

(There are, however, some people who use shorteners on purpose? What about them?)

We may be getting into too many options, but I think that's a completely reasonable "respect url shorteners" option.

Mood, location, music, adult content -- can these be left alone, or should there be settings for them? Should there be the ability to override the default crosspost settings for this type of entry? Comments/comment screening?

All of this I think should work the same as any other "scheduled entry" or whatever we're calling the future feature that allows us to post the same thing at regular intervals. Because that's what we're talking about here. We're talking about making a mashup between running the importer and making a scheduled entry. I don't know a lot of how that whole thing is going to work, but I think that whatever interface and logic is used for that project can be applied here.

The way I see this working is that you've got a page set up where you start setting up the Twitter imports. This is where you OAuth and set up stuff like if you want to include replies and anything else that applies only to interacting with the twitter site.

Then you get directed to the update page, which is where you can set all your security levels, comment settings etc. The contents of the update page would be something like <twitter import />.

And then it saves it in the same way that it saves other scheduled entries, and when it comes time to post the entry, it throws it over to the import queue, and then when the import gets done it posts it.

Of course, I could be talking out my rear here, since I know nothing of how any of Dw works on the back end.
matgb: Artwork of 19th century upper class anarchist, text: MatGB (Default)

Re: really really really really really want twitter imports

[personal profile] matgb 2011-04-05 08:13 pm (UTC)(link)
All "in reply to" posts start with @someone

Actually, they don't, at least not from my web client. example

I made sure it kept the reply to there as I was parodying a particularly obtuse comment by a friend.

However, I agree with you that I'd like to exclude @replies that start with the username, but I'd like to include stuff like the above. Because if I reword it, it's to get it more public and visible.

Re: really really really really really want twitter imports

[personal profile] faithofone 2011-04-05 08:47 pm (UTC)(link)
I stand corrected. I hadn't seen this before, and should have known better than to make an absolute statement with such a broad scope. My apologies.

Re: really really really really really want twitter imports

[personal profile] itshiseyes 2011-04-06 10:53 pm (UTC)(link)
That works for me. It's clearly labeled enough. Though I think using the <cite> element in there as well would be nice.

Re: really really really really really want twitter imports

[personal profile] faithofone 2011-04-06 10:54 pm (UTC)(link)
and I did it again... I so rock the multiple account thing.

[personal profile] delladea 2011-04-07 02:54 pm (UTC)(link)
I would LOVE this so much if it were to happen.
slyfox: (Default)

[personal profile] slyfox 2011-04-14 02:06 am (UTC)(link)
A little late to the party, but I would love this as well. I believe that Livejournal now has a "Twitter Connect" feature, but it only imports public tweets. I would love to be able to import all of my tweets and then put them under a special filter for only people who opt in/don't use twitter already. Since it's very difficult to go through your old tweets on twitter if they're protected, it would be awesome if I could have a "daily digest" of them on my journal.
cyprinella: a red octopus on a white background with a red star above it (Default)

[personal profile] cyprinella 2012-03-18 10:47 pm (UTC)(link)
don't know if you're still thinking about this one (I was googing for a way to do just this) but I'd rather just have links to the media.
snippy: Lego me holding book (Default)

Status?

[personal profile] snippy 2012-11-28 11:16 pm (UTC)(link)
I'm still interested in this. Has Twitter "clarified?"
jss: (sixties)

[personal profile] jss 2017-04-24 05:12 pm (UTC)(link)
It's been a few years; any chance this could get revisited?
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2017-04-25 02:41 am (UTC)(link)
The reason it was rejected was due to Twitter's ToS not allowing this kind of wholesale import of account content, I believe! I'll check again to see if that's changed, but last time I looked, doing something like this would result in the revocation of your API key :(
jss: (simpsons)

[personal profile] jss 2017-04-25 09:11 am (UTC)(link)
IIRC the T-to-LJ import was a "run once daily for tweets-not-replies made in the past 24 hours," which isn't IMO a "wholesale import," but I understand the concern. Thanks for checking!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2017-04-26 05:18 am (UTC)(link)

Yeah, I looked at it again, and they've still got the clause in place. (I misremembered: it's not "bans import of content", it's "you can't access moer than 50,000 tweets a day", so with reasonable adoption of the feature, we'd hit that limit really quickly.) Alas. :(

jss: (simpsons)

[personal profile] jss 2017-04-26 11:56 am (UTC)(link)
I haven't investigated their developers' terms in any depth, but my question would be: Do they mean "you" the entity (aka Dreamwidth, and if so, how does LJ handle tweet imports) or "the account" (aka each user)?

Assuming the former, would it be possible to allow the embedded twitter widget, such as
<a class="twitter-timeline"
data-chrome="nofooter"
data-widget-id="
unique_widget_id"
href="https://twitter.com/
twitter_id">Tweets by @twitter_id</a>
<script type="text/javascript">
twitter_javascript_code</script>
in a pinned post so someone's feed would at least be visible? (I have a widget for the Twitter account I tweet from as my employer which is why I know the construction there; I don't yet have one for me-the-individual.) IIRC the use of JavaScript is the deal breaker there.