Is there....

A forum about the forums

Moderator: Carleas

Is there....

Postby AutSider » Fri Jul 03, 2015 3:29 pm

some kind of program for saving forum posts?

I want to save all of my old posts, so I can read them when I want and get to know my thinking processes in the past. PhilosophyForums has a nice little function that allows you to download all posts of a user to a .txt file, is there a program which does the similar thing, preferably in word or PDF since the function is absent?

ILP won't be here forever, so in case it shuts down, I'd rather have my posts some place safe. Omnia mea mecum porto is the proverb I think.
User avatar
AutSider
BANNED
 
Posts: 3337
Joined: Mon Jan 20, 2014 9:04 pm

Re: Is there....

Postby The Golden Turd » Sat Jul 04, 2015 5:03 pm

Please prohibit this function for my posts. I pretended to be Taz (Barbarian Horde) in my first 150ish posts, convincing him I was him (and I succeeded), I really don't want people trying to get all my posts together and trying to figure out what my system was.... I don't post that way. I respond.... to a billion different highly varied questions, and have adapted my rhetorical methods over time to fit the forum.

If you just saw my posts alienated from all the other posts active on the site, you wouldn't end up with a book, but a schizophrenic mess. You need context too, sometimes larger than just the thread, but what everyone is talking about in other threads that given day too.... sometimes I tie it all together. My responses target a culture at that time, on the forums.

I would feel icky if someone got the bright idea of only looking at my posts, spread out over years. They are 2000 separate letters to many people.

If you want a book, email me, and we can have a book long, topic specific discussion. I just don't want to see all my posts stored on another site congealed together. I never surrendered my copyright, always insisted I owned it. I'm comfortable with it being in threads, but not in a file given merged into a congealed lump. It will misrepresent me out of context. It changes the nature of the work. Much easier to misconstrue. I prefer having context, the people I am conversing with.... dated to compare to other threads.... with access to those other threads.
Support "The Angels of East Africa" on smile.Amazon.com it is free to do, they donate 0.05% of your purchase cost to them, or give donations directly via:

http://www.machinegunpreacher.org

Image

Recently hidden post:
viewtopic.php?f=6&t=192227&p=2649513#p2649513
User avatar
The Golden Turd
Fucking Unflushable
 
Posts: 9450
Joined: Wed Nov 23, 2011 6:37 am
Location: Apparently Well Up Manical Mongoose's Ass

Re: Is there....

Postby AutSider » Sun Jul 05, 2015 12:13 pm

I just want to preserve my own posts you narcissist, relax :lol:
User avatar
AutSider
BANNED
 
Posts: 3337
Joined: Mon Jan 20, 2014 9:04 pm

Re: Is there....

Postby Carleas » Sun Jul 05, 2015 3:39 pm

I agree there should be a way to do this, but currently there isn't. I'd like to implement that function, although a possible issue is that, as the database grows, users downloading their entire posing history could actually put more strain on the server than it can handle (or bump us over our outbound traffic limit). We're a pretty rinky-dink little operation here.

I promise that I'll implement something like that well before ILP shuts down (unless I die unexpectedly, in which case all bets are off). I'm a packrat for my own writings, so I appreciate the desire to have a backup of anything you've spent some time on.
User Control Panel > Board preference > Edit display options > Display signatures: No.
Carleas
Magister Ludi
 
Posts: 5670
Joined: Wed Feb 02, 2005 8:10 pm
Location: Washington DC, USA

Re: Is there....

Postby phyllo » Sun Jul 05, 2015 3:56 pm

This retrieves posts from a PHPBB forum:
http://creativeandcritical.net/fups/ent ... type=phpbb

As Alex revealed not long ago, I had written a few years back at his request (and expense) a little script to extract and retrieve a user's posts from a phpBB-powered board such as Genius Forum, and as I explained in that thread, it's basically a web scraper. The problem was, it was a command line script, which made it awkward to share with others, because a user had to have PHP installed on their machine, and had to fiddle around with a settings file. So, I finally got around to spinning off and offering a web version.

You can now retrieve all of the posts of any specified (non-banned) user through the phpBB-extract page on my website. Other than Alex and Katy (now going by "Dani"), you are the first people I've revealed this URL to - I haven't even linked to it from my home page yet. I'm very interested in feedback - what you think works, what doesn't work, what could be worded/worked better, etc. After following up on your feedback, I will feel more confident in the "advertising" of this resource.

I do hope that this script is useful to the phpBB community.

http://www.theabsolute.net/phpBB/viewto ... =17&t=7230
phyllo
ILP Legend
 
Posts: 10901
Joined: Thu Dec 16, 2010 1:41 am

Re: Is there....

Postby phyllo » Sun Jul 05, 2015 4:08 pm

:-k That script doesn't seem to work.
phyllo
ILP Legend
 
Posts: 10901
Joined: Thu Dec 16, 2010 1:41 am

Re: Is there....

Postby Carleas » Sun Jul 05, 2015 4:15 pm

Please don't use scripts like that. They will have an effect on the serve similar to a denial of service attack, because they work by making repeated requests for posts from the server. 6800 odd requests will make the site unusable for quite some time.


If you must use that kind of script, please use it between 3 am and 4 am Eastern time, preferably on a Sunday or Monday, as that is when traffic to the site is lowest.
User Control Panel > Board preference > Edit display options > Display signatures: No.
Carleas
Magister Ludi
 
Posts: 5670
Joined: Wed Feb 02, 2005 8:10 pm
Location: Washington DC, USA

Re: Is there....

Postby phyllo » Sun Jul 05, 2015 4:29 pm

Please don't use scripts like that. They will have an effect on the serve similar to a denial of service attack, because they work by making repeated requests for posts from the server. 6800 odd requests will make the site unusable for quite some time.
The programmer thought of that :
How long will the process take?

It depends on how many posts are to be retrieved, and how many pages they are spread across. You can expect to wait roughly one hour to extract and output 1,000 posts.
Why is this script so slow?

So as to avoid hammering other people's web servers, the script pauses for five seconds between each page retrieval.
phyllo
ILP Legend
 
Posts: 10901
Joined: Thu Dec 16, 2010 1:41 am

Re: Is there....

Postby Laird » Mon Jul 06, 2015 4:55 am

Hi,

I'm the author of the FUPS script which phyllo linked to. I found my way to this thread by following my web server's referrer log after receiving emails notifying me of FUPS errors. Hopefully, those errors are now fixed. The problem was that for some reason (I've not seen this before), this forum was returning pages to FUPS using a "mobile" skin, which up until now FUPS hadn't supported. After adding support, it seems to be working. If you have any more problems, please feel free to let me know either in this thread or through my website.

Best wishes,
Laird

P.S. In case anyone's interested (I'm not sure why you would be), you can see the changes I made to get things working in commit 6cc21778277578a8c07dec13ebb35c645cad85b5 in the project's GitHub records.
Laird
 
Posts: 2
Joined: Mon Jul 06, 2015 2:55 am

Re: Is there....

Postby Carleas » Tue Jul 07, 2015 4:31 am

Hey Laird, thanks for chiming in. Our mobile skin is a bit out of date, that may be part of the problem. But it should still be the case for any phpBB board that the mobile skin is served based on useragent, spoofing a common desktop browser should get you the regular site.

And I appreciate that steps are taken to space out requests, that makes me much more open to the use of this script. I'd still request that it be used in off hours if possible (any morning before about 10am eastern), because the server creaks under a normal load and a thousand requests, even spaced out, is much higher than normal load. But it is totally unreasonable that a thousand requests over an hour would tax the server, so I will put the blame on myself if anyone is unable to schedule around it.
User Control Panel > Board preference > Edit display options > Display signatures: No.
Carleas
Magister Ludi
 
Posts: 5670
Joined: Wed Feb 02, 2005 8:10 pm
Location: Washington DC, USA

Re: Is there....

Postby phoneutria » Tue Jul 07, 2015 4:42 am

Carleas, are you self-hosted?
Have you considered aws, or is that too mmuchout of your own pocket?
Speaking of that, you should do a little fund raiser. I'm sure at least a handful of folks here would be willing to send some money your way.
phoneutria
purveyor of enchantment, advocate of pulchritude AND venomously disarming
 
Posts: 2733
Joined: Fri May 23, 2014 5:37 am

Re: Is there....

Postby Carleas » Thu Jul 09, 2015 3:48 pm

ILP runs on a VPS, I don't know if that qualifies as self hosted: I don't touch the hardware, but I control all the software.

I looked at AWS, but the VPS we use is awesome, their prices are reasonable and their support is great. Plus, since it's a smaller provider, they have a great mailing list of much more experienced server admins who love to tell me all the things I don't know, and that's invaluable. For a site this size and an admin of my skill level, AWS doesn't compare.

And cost isn't a problem. We do have a Paypal account for donations, but it's mostly demands on my time (and my own laziness and fear of breaking things) that holds up maintenance and development. I'm getting some more time as I settle in to my schedule, so expect a little more activity on that front.
User Control Panel > Board preference > Edit display options > Display signatures: No.
Carleas
Magister Ludi
 
Posts: 5670
Joined: Wed Feb 02, 2005 8:10 pm
Location: Washington DC, USA

Re: Is there....

Postby AutSider » Tue Jul 21, 2015 8:06 pm

I have not tried the application yet, but since I am unsure about Carleas's honesty, I will not rest my posting history on his promise of making such a function before the end of ILP.

An alternative easy and quick way of saving your posts is to download the program "DropBox" and then you just go over your post and click the screenshot button (PrintScreen). You can do 1 post per 1-2 seconds this way.

I'll try to use the application before doing that tho, has anybody had any success with it so far?
User avatar
AutSider
BANNED
 
Posts: 3337
Joined: Mon Jan 20, 2014 9:04 pm

Re: Is there....

Postby Carleas » Tue Jul 21, 2015 8:14 pm

Arbiter of Change wrote:I have not tried the application yet, but since I am unsure about Carleas's honesty, I will not rest my posting history on his promise of making such a function before the end of ILP.

Sorry, I think my last post was unclear for trying too hard to be clever:

Feel free to use the script. If possible use it mornings before ~10 and avoid afternoons ~14-19 (all times eastern US, UTC-4). But if you can't work in that schedule, you have my blessing to use it whenever.

I do intend to add such a function, but there's no need to wait on me.
User Control Panel > Board preference > Edit display options > Display signatures: No.
Carleas
Magister Ludi
 
Posts: 5670
Joined: Wed Feb 02, 2005 8:10 pm
Location: Washington DC, USA

Re: Is there....

Postby AutSider » Wed Jul 22, 2015 2:57 pm

There has been few members active today, so I did it. Took 2h35min10s overall for my number of posts (about 2200).

I could have simply tried it out by downloading from a user with 1-10 posts... but I only remembered that half-way through, so I let it complete.

It's arranged in an alphabetical order of topics you've posted in.

A screenshot of how it looks like once you're done:

ScrScreen.png
ScrScreen.png (168.53 KiB) Viewed 1369 times
User avatar
AutSider
BANNED
 
Posts: 3337
Joined: Mon Jan 20, 2014 9:04 pm

Re: Is there....

Postby phyllo » Wed Jul 22, 2015 9:01 pm

Does this mean that you are happy with the result?

Would you recommend it to another user who wants to download his posts?
phyllo
ILP Legend
 
Posts: 10901
Joined: Thu Dec 16, 2010 1:41 am

Re: Is there....

Postby AutSider » Wed Jul 22, 2015 9:37 pm

Yeah it's decent, one drawback IMO: - It doesn't allow the option to sort your posts by date

Another thing which I don't like, but is entirely my fault, is that I don't know who posted the posts which I replied to and quoted, but like

this


, without the name of who I am quoting. So I see what I was responding to, but I don't know to whom I was responding to. But, as I said, this is entirely my fault and I only mention this as a precaution for other, newer users, not to screw themselves over if they intend to use this nifty little gadget.
User avatar
AutSider
BANNED
 
Posts: 3337
Joined: Mon Jan 20, 2014 9:04 pm

Re: Is there....

Postby phyllo » Wed Jul 22, 2015 9:48 pm

Okay, something for Laird to consider.

Otherwise, a satisfied customer.


Thank you to Laird for coming here and fixing some problems with the program. :D
phyllo
ILP Legend
 
Posts: 10901
Joined: Thu Dec 16, 2010 1:41 am

Re: Is there....

Postby Carleas » Wed Jul 22, 2015 10:32 pm

Not sure if Laird can help. The quoting isn't actually a pointer to the quoted post, it's just formatting markup. You can attribute a quote box to anyone:
Ace Ventura's ass wrote:Something ridiculous
User Control Panel > Board preference > Edit display options > Display signatures: No.
Carleas
Magister Ludi
 
Posts: 5670
Joined: Wed Feb 02, 2005 8:10 pm
Location: Washington DC, USA

Re: Is there....

Postby phyllo » Wed Jul 22, 2015 10:37 pm

Not sure if Laird can help.
I meant a different order besides alphabetical. It could be ordered by original post date.
phyllo
ILP Legend
 
Posts: 10901
Joined: Thu Dec 16, 2010 1:41 am

Re: Is there....

Postby AutSider » Thu Jul 23, 2015 8:11 pm

Looking back, I sounded somewhat nitpicky, far more than I intended to. The program is excellent, and would recommend it to anybody wishing to save their posts.

The quote thing, as I said, is entirely my fault, it was a mere warning for new members who might use the program later on.
And the ordering by threads is in many ways superior to ordering by date. Maybe since every post contains a date, there is another way to order them by date not necessarily included in this particular program. But that doesn't really matter.
User avatar
AutSider
BANNED
 
Posts: 3337
Joined: Mon Jan 20, 2014 9:04 pm

Re: Is there....

Postby Laird » Fri Jul 24, 2015 11:04 pm

Carleas wrote:But it should still be the case for any phpBB board that the mobile skin is served based on useragent, spoofing a common desktop browser should get you the regular site.


I figured it was the user-agent causing that, yes - FUPS provides a URL for its user-agent. I'd prefer not to spoof because I think it's good etiquette to let webmasters know that a bot's accessing their site, along with a link to more info on the bot. As I said, I've not seen this behaviour on any other phpBB board before.

Carleas wrote:But it is totally unreasonable that a thousand requests over an hour would tax the server


I wouldn't presume to know why your web server's so sluggish, but one thing you can do if you haven't already to dramatically improve PHP speed is to enable opcode caching. Chances are high that you've already done that though. :-)

Arbiter of Change wrote:Yeah it's decent, one drawback IMO: - It doesn't allow the option to sort your posts by date


Thanks for the feedback. Yes, there really should be other output options, I've been aware of that for a while now, and your comment gave me the kick in the pants that I needed to finally implement them. I wrote the original script out of which this one evolved for a friend who paid me to write it, so I simply coded the output as he personally wanted it. Now that the script is publicly available, it should cater to different personal preferences.

Here's the new success screen:

Image

Again, anyone who for some strange reason wants to see the changes I made to the code to get this working can view the commit on GitHub; it's commit f85211594d7ecd4c71fd3aa70c981ce2845d40cf.

Arbiter of Change wrote:Another thing which I don't like, but is entirely my fault, is that I don't know who posted the posts which I replied to and quoted


One way to solve this problem (and offer a useful feature in general) would be to provide an option to download the entirety of each thread in which the user was involved. Someone else has asked for this feature already, I just haven't gotten around to implementing it yet.

The other missing feature of which I'm keenly aware is image scraping.

phyllo wrote:Thank you to Laird for coming here and fixing some problems with the program.


No worries. If I'm going to offer a script, I might as well offer one that works, otherwise I'm not really offering anything at all.

Arbiter of Change wrote:Looking back, I sounded somewhat nitpicky, far more than I intended to. The program is excellent, and would recommend it to anybody wishing to save their posts.

The quote thing, as I said, is entirely my fault, it was a mere warning for new members who might use the program later on.
And the ordering by threads is in many ways superior to ordering by date. Maybe since every post contains a date, there is another way to order them by date not necessarily included in this particular program. But that doesn't really matter.


Hey, no need to second-guess yourself, you were totally right about the need for more output options. But I do appreciate your positive comments, so, thanks.

If anyone has any further feedback, or any other suggestions, please let me know.
Laird
 
Posts: 2
Joined: Mon Jul 06, 2015 2:55 am


Return to Meta



Who is online

Users browsing this forum: No registered users