Search Issues

A forum about the forums

Moderator: Carleas

Search Issues

Postby obsrvr524 » Sun Jul 21, 2019 9:27 am

Sorry if this is a common question but is there a way to download all of the posts of a chosen member? I am doing a study and the search facilities on this board make it very difficult to examine and measure discussed topics and phrases.

Thank you
obsrvr524
 
Posts: 109
Joined: Thu Jul 11, 2019 9:03 am

Re: Search Issues

Postby Carleas » Tue Jul 23, 2019 3:18 pm

There is no official way. A third-party option was found and discussed here, and the creator popped in to discuss. I didn't make it and have never used it, so I can't vouch for it or provide support.
User Control Panel > Board preference > Edit display options > Display signatures: No.
Carleas
Magister Ludi
 
Posts: 5823
Joined: Wed Feb 02, 2005 8:10 pm
Location: Washington DC, USA

Re: Search Issues

Postby obsrvr524 » Tue Jul 23, 2019 4:56 pm

Thank you for the response.

So I take it that you do not prefer that such a script be used due to server load issues. The member I am focused on currently has over 20,000 posts, hence the need for a good search engine. Is there a way to download a portion each day? If so, how could I go about that?


And why do I keep getting logged out in the middle of my posting?
obsrvr524
 
Posts: 109
Joined: Thu Jul 11, 2019 9:03 am

Re: Search Issues

Postby Carleas » Tue Jul 23, 2019 7:04 pm

obsrvr524 wrote:So I take it that you do not prefer that such a script be used due to server load issues. The member I am focused on currently has over 20,000 posts, hence the need for a good search engine. Is there a way to download a portion each day? If so, how could I go about that?

I bless your use of that script without modifications thereto. I can't guarantee that the server won't hang and leave you with a half-finished download, but you are free to try. I'd request that you run the script on a Monday morning, as that is usually the slowest time and traffic picks up as the week progresses.

obsrvr524 wrote:And why do I keep getting logged out in the middle of my posting?

This shouldn't happen if you have ticked the 'Keep me logged in' option at login, and have cookies enabled in your browser. Other users have reported this issue, but it is not an issue for most users and seems related to user-side configurations.
User Control Panel > Board preference > Edit display options > Display signatures: No.
Carleas
Magister Ludi
 
Posts: 5823
Joined: Wed Feb 02, 2005 8:10 pm
Location: Washington DC, USA

Re: Search Issues

Postby obsrvr524 » Wed Jul 24, 2019 10:32 am

Morning between 4-7AM your time, US Eastern, would be easy for me. It might take quite a few Mondays. It constitutes a lot of reading for me and I don't have a lot of time to spend on it anyway. I'll try it. Thanks


And I don't see a "keep me logged in" option. if there is a timer involved, it would be nice if it could be lengthened a bit.

It also seems odd that I don't appear on the "registered users" list even when I am logged in.
obsrvr524
 
Posts: 109
Joined: Thu Jul 11, 2019 9:03 am

Re: Search Issues

Postby Carleas » Wed Jul 24, 2019 3:06 pm

If you start early on a Monday, you can let it run, or run it several mornings in a row. I appreciate your flexibility on this, and any accommodation you make is a courtesy, not an obligation on you. If the site can't handle a request every five seconds, that's on me.

obsrvr524 wrote:And I don't see a "keep me logged in" option. if there is a timer involved, it would be nice if it could be lengthened a bit.

Sorry, I meant the 'log me in automatically' option. With that checked, it should just ask you to resubmit if the login session expires while you're posting, rather than asking you to login again.

As for timing, the current session length is 1 hour, and I would expect (though I don't know that this is the case) that to reset every time you interact with the site, so e.g. previewing a post.

obsrvr524 wrote:It also seems odd that I don't appear on the "registered users" list even when I am logged in.

This is set to 5 minutes, i.e. users who have interacted with the site within the past 5 minutes. There is a box on the login page to hide your presence, which would of course hide you from the list.

Another possibility is that, for your first 5 posts, you're in a different group, 'newly registered users', and they may not be included in the list. You would be a normal user now, but if you noticed this earlier, that may be why.
User Control Panel > Board preference > Edit display options > Display signatures: No.
Carleas
Magister Ludi
 
Posts: 5823
Joined: Wed Feb 02, 2005 8:10 pm
Location: Washington DC, USA

Re: Search Issues

Postby obsrvr524 » Thu Jul 25, 2019 11:09 am

Thank you for your indulgence. I'll see how it goes and hope for the best.
obsrvr524
 
Posts: 109
Joined: Thu Jul 11, 2019 9:03 am

Re: Search Issues

Postby obsrvr524 » Fri Jul 26, 2019 9:00 am

That script seems to require a "Forum ID" and explains:
"Set this to a comma-separated list of IDs of any forums that you wish for FUPS to scrape. You can find a forum's ID by hovering your cursor over a forum hyperlink and taking note of the integer that appears after "&f=" in the URL in the browser's status bar."

That method doesn't seem to reveal any "&f=" notation. Any ideas?
obsrvr524
 
Posts: 109
Joined: Thu Jul 11, 2019 9:03 am

Re: Search Issues

Postby phyllo » Fri Jul 26, 2019 2:42 pm

I think that's supposed to be ?f=

Philosophy Forums f=27

Philosophy f=1
Society ... f=3
Science ... f=4
Psychology ... f=15
Religion ... f=5
Sandbox f=25
phyllo
ILP Legend
 
Posts: 11235
Joined: Thu Dec 16, 2010 1:41 am

Re: Search Issues

Postby obsrvr524 » Sun Jul 28, 2019 10:23 am

Thank You phyllo. That seemed to have worked.

Now my next problem is that there seems to be only a "start from" with no "end at". I didn't want to set it off downloading every post since his first all in one fell swoop.
obsrvr524
 
Posts: 109
Joined: Thu Jul 11, 2019 9:03 am

Re: Search Issues

Postby phyllo » Sun Jul 28, 2019 4:31 pm

I suggest doing one forum at a time, starting where he posted the least ... like Sandbox or Psychology. Merge the results later.
phyllo
ILP Legend
 
Posts: 11235
Joined: Thu Dec 16, 2010 1:41 am

Re: Search Issues

Postby obsrvr524 » Mon Jul 29, 2019 9:30 am

Thanks again. i think I'll just let it run for a while, stop it, see how far it got, then start next time where it left off.
obsrvr524
 
Posts: 109
Joined: Thu Jul 11, 2019 9:03 am

Re: Search Issues

Postby obsrvr524 » Thu Aug 08, 2019 10:25 am

I decided to just run it very slowly. After 80 hours, I finally had a 630M file. It took 25 minutes just to download the smilies. Now I get to parse, correlate, categorize, and classify it all. This is going to take quite a while.
obsrvr524
 
Posts: 109
Joined: Thu Jul 11, 2019 9:03 am


Return to Meta



Who is online

Users browsing this forum: No registered users