Full Export Tool

[azurelunatic]

Title:
Full Export Tool

Area:
exporting, interoperability, backups

Summary:
Create an interface/client with an easy way of setting up and later downloading a full backup of one's journal. This would be in a file format of one's choice, with entries, comments, icons, memories, and all.

Description:
Currently, one can back up one's journal through a scattering of mostly-legacy interfaces in different places. There are tools that were never exactly designed to work with Dreamwidth, such as LJ Archive. (While some of them work even most of the time, there are times that they do not, and I suspect Dreamwidth developers may not necessarily prioritize jumping in to third-party client code developed for another site, assuming the code is even open and available to be jumped into.)

There already exists a bug requesting a PDF export of one's journal: http://bugs.dwscoalition.org/show_bug.cgi?id=32

This would broaden the scope of the ability to export, so one could choose between, say, .pdf, HTML, plain text, rich text, comma separated values, tab separated values, and perhaps other formats (WordPress-friendly?) as developers see the need.

One could go to a page similar to the current import journal page, and select the date range (as for the existing spec for .pdf, whole journal or month by month), elements and file type that one wished to back up, and queue a download request. The system would queue the download request, and prepare it in a timely fashion, as other load allowed so as to keep things readable in other journals on the same cluster. (Paid users might be given priority in the queue, if there was a queue worth mentioning.) Similar to the importer, the page would show progress of the export file preparation when revisited, and offer a low-impact refresh option, because you just know that people are going to want to sit there and beam at the progress the first couple times they download. Once the files were prepared, in reasonable-sized chunks for downloading if it were a particularly large journal *cough*azurelunatic*cough*, a notification would be sent with a link to the download page, so the user would not have to sit on the page refreshing if they did not want to. There might also be a password challenge before downloading, to diminish the possibility that someone's theoretical nosy little brother could copy one's whole journal to a USB drive in an unguarded minute alone with the computer.


In addition to the web interface, there should also be an API that allows (at minimum) the same choices as the web interface, complete with full documentation. Perhaps there could even be official clients for a range of different platforms.

Poll #4293 Full Export Tool

Open to: Registered Users, detailed results viewable to: All, participants: 77

View Respondents

This suggestion:

View Answers

Should be implemented as-is.
66 (85.7%)

Should be implemented with changes. (please comment)
10 (13.0%)

Shouldn't be implemented.
0 (0.0%)

(I have no opinion)
1 (1.3%)

(Other: please comment)
0 (0.0%)

Comments

[jumpuphigh]

I like this idea but think we should consider offering it as a paid member option.

[aedifica]

That sounds reasonable, given the resources I expect it would use.

[kerravonsen]

+1

[azurelunatic]

I strongly believe that whatever paid options there are, that there should be a way to make a basic relatively-easy export for an unpaid user functional, even if it is the least-prioritized thing on that server and takes Quite Some Time. I think that it is in the spirit of Dreamwidth to not keep people's data hostage for pay.

It might be month by month, or year by year, or only in a few of the most useful formats, or both, but ... yes, it uses server resources, but it's also important.

[denise]

But there already is the basic export!

[azurelunatic]

So there is; that part of this would probably be filled by slapping a "and see this FAQ about downloading your comments" on that page. (Which, er, appears not to exist yet.)

[jadelennox]

+1. To me one of the most important parts of calling any tool open is the ability to remove your information from it cleanly, and part of that involves documentation. I'm fine with making a better option paid, but we need to be very open and well-documented for free users as well.

[feathertail]

Right now it's pretty lousy where free users are concerned. I mean really lousy.

I'm not even sure server load would be that much of a problem, since most people ignore backups anyway. There might be a run on it when it's announced though.

[jumpuphigh]

I am not suggesting that DW "keep people's data hostage for pay". If someone were to create a program like LJArchive to specifically work with DW and then DW purposefully blocked that program from working, then I would agree that that action would be not in keeping with the spirit of DW. However, I think that saying, "Hey, we have this cool tool which we have created that you might like to use. It uses more of our resources than we have allocated for free accounts so it will be for paid members" is not in the same category as that at all.

[reddragdiva]

You mean that you did not mean to suggest that, but that is in fact what you effectively suggested.

If people don't know they can get their stuff out again, they will be reluctant to put it in.

[reddragdiva]

I should note also that this sort of not-quite-open model is common with pseudo-open-source projects: where functionality that turns out to be quite important is pay-only. This generally leads to people feeling deeply ripped-off and forking. I suggest that would not be a good path for DW to appear to be taking even slightly - just imagine trying to explain to someone why DW's case is special when the model matches this pattern.

[charmian]

Hmm, given the importance of data backup, I'd say that maybe if it's going to be limited, free users might get to do it like every so often, but paid users would be able to do it more frequently?

[jumpuphigh]

Currently, anybody can export their journal at any time with this tool here: http://www.dreamwidth.org/export I don't think that we should remove that functionality at all.

[feathertail]

You have to keep clicking on it for each month, and it doesn't give you the comments.

[reddragdiva]

No, absolutely not. A "free" tool that you need to pay to get your data out of is nothing but a trap. People know this.

[medrin]

I'm not exactly sure how import actually works and this might be a totally obvious thing, but it should be possible to re-import whatever you've just exported (in whatever format). Because isn't this the function of a back up? To be able to bring back material you thought you've lost?

[zvi]

+1

[instantramen]

+1

[ariestess]

THIS.

[feathertail]

+1

[holyschist]

WordPress-friendly?

XML. Definitely XML, and if there's a need for tweaks for different platforms (WP...Blogger?), that would be a spiffy option.

I agree that this should probably be a paid feature.

[dhobikikutti]

Since I know the lack of this feature is keeping at least one friend away from Dreamwidth, I would love for this to happen.

[zulu]

I would really appreciate being able to use lj-sec's functionality with dw. There's already a mass-change tool for post-security, but it's not fine-grained enough for me; it's all or nothing. I don't know if this is something that should be included in an lj-archive type feature, or if I should suggest it on its own (or whether it's already been suggested--I can't find it, but maybe it has).

[(Anonymous)]

I would love this, by the way. I run RPG campaign sessions (2-3 day events) a few times a year rather than a full game, and I want a way to read these all at once -- but the problem is that only the initial "boxed text" starter is given in the post, and then the players all rp in the comments. I'd love to be able to back up these game sessions somewhere (to share easily, to read easily, in case data is lost). I've been running them for over 5 years now, so having an easy reference to the previous sessions is really important. But right now my method of doing so is literally copy-pasting all the comment text into a word doc and then manually cleaning it all up - which takes about a week to do per game session.

[rosefox]

I'd like to bump this up for renewed consideration and higher prioritization. A lot of people were using LJ as a primary data host and Dreamwidth as a backup. Now that LJ is dying, Dreamwidth becomes the primary host and we need a better way of backing up from it than the month-by-month exports (especially those of us who have 15+ years of data).

[makamu]

+1

[denise]

Better journal export tools are totally on The List, and have been steadily working upward in priority!

[rosefox]

That's great to know, thank you!

[denise]

We're limited in how fast we can implement new features, especially big new features or ones that require a lot of work, because we're a Very Tiny Company and lack the resources other sites have. (Mark has to have a dayjob to be able to afford rent in the Bay Area; I'm disabled and crazy and sometimes have to spend like a week in bed with little advance notice, and I'm not the best programmer in the world to begin with; Jen's part-time; everybody else is a volunteer and works on what they want to, not what we tell them to.) So, I don't want to get your hopes up that it will necessarily be SOON! But we're aware of the lack. :)

[lhexa]

A friend of mine recently imported his Livejournal to Dreamwidth simply as a means of backing up a huge amount of text from a decade of his life. Neither of us believes LJ is going to survive, and we were discussing a means of backing up this site too. So I'm glad to see this is a suggestion, and I hope it makes it up the queue quickly. :)