Search Issues

[obsrvr524]

Sorry if this is a common question but is there a way to download all of the posts of a chosen member? I am doing a study and the search facilities on this board make it very difficult to examine and measure discussed topics and phrases.

Thank you

[Carleas]

There is no official way. A third-party option was found and discussed here, and the creator popped in to discuss. I didn't make it and have never used it, so I can't vouch for it or provide support.

[obsrvr524]

Thank you for the response.

So I take it that you do not prefer that such a script be used due to server load issues. The member I am focused on currently has over 20,000 posts, hence the need for a good search engine. Is there a way to download a portion each day? If so, how could I go about that?


And why do I keep getting logged out in the middle of my posting?

[Carleas]

So I take it that you do not prefer that such a script be used due to server load issues. The member I am focused on currently has over 20,000 posts, hence the need for a good search engine. Is there a way to download a portion each day? If so, how could I go about that?

I bless your use of that script without modifications thereto. I can't guarantee that the server won't hang and leave you with a half-finished download, but you are free to try. I'd request that you run the script on a Monday morning, as that is usually the slowest time and traffic picks up as the week progresses.

And why do I keep getting logged out in the middle of my posting?

This shouldn't happen if you have ticked the 'Keep me logged in' option at login, and have cookies enabled in your browser. Other users have reported this issue, but it is not an issue for most users and seems related to user-side configurations.

[obsrvr524]

Morning between 4-7AM your time, US Eastern, would be easy for me. It might take quite a few Mondays. It constitutes a lot of reading for me and I don't have a lot of time to spend on it anyway. I'll try it. Thanks


And I don't see a "keep me logged in" option. if there is a timer involved, it would be nice if it could be lengthened a bit.

It also seems odd that I don't appear on the "registered users" list even when I am logged in.

[Carleas]

If you start early on a Monday, you can let it run, or run it several mornings in a row. I appreciate your flexibility on this, and any accommodation you make is a courtesy, not an obligation on you. If the site can't handle a request every five seconds, that's on me.

And I don't see a "keep me logged in" option. if there is a timer involved, it would be nice if it could be lengthened a bit.

Sorry, I meant the 'log me in automatically' option. With that checked, it should just ask you to resubmit if the login session expires while you're posting, rather than asking you to login again.

As for timing, the current session length is 1 hour, and I would expect (though I don't know that this is the case) that to reset every time you interact with the site, so e.g. previewing a post.

It also seems odd that I don't appear on the "registered users" list even when I am logged in.

This is set to 5 minutes, i.e. users who have interacted with the site within the past 5 minutes. There is a box on the login page to hide your presence, which would of course hide you from the list.

Another possibility is that, for your first 5 posts, you're in a different group, 'newly registered users', and they may not be included in the list. You would be a normal user now, but if you noticed this earlier, that may be why.

[obsrvr524]

Thank you for your indulgence. I'll see how it goes and hope for the best.

[obsrvr524]

That script seems to require a "Forum ID" and explains:
"Set this to a comma-separated list of IDs of any forums that you wish for FUPS to scrape. You can find a forum's ID by hovering your cursor over a forum hyperlink and taking note of the integer that appears after "&f=" in the URL in the browser's status bar."

That method doesn't seem to reveal any "&f=" notation. Any ideas?

[phyllo]

I think that's supposed to be ?f=

Philosophy Forums f=27

Philosophy f=1
Society ... f=3
Science ... f=4
Psychology ... f=15
Religion ... f=5
Sandbox f=25

[obsrvr524]

Thank You phyllo. That seemed to have worked.

Now my next problem is that there seems to be only a "start from" with no "end at". I didn't want to set it off downloading every post since his first all in one fell swoop.

[phyllo]

I suggest doing one forum at a time, starting where he posted the least ... like Sandbox or Psychology. Merge the results later.

[obsrvr524]

Thanks again. i think I'll just let it run for a while, stop it, see how far it got, then start next time where it left off.

[obsrvr524]

I decided to just run it very slowly. After 80 hours, I finally had a 630M file. It took 25 minutes just to download the smilies. Now I get to parse, correlate, categorize, and classify it all. This is going to take quite a while.