Page 1 of 1
http on non-standard ports - defeat spiders ?
Posted: Mon Feb 05, 2007 5:42 pm
by DaveQB
Hi guys,
Would running apache on a non standard http port allow it to not be indexed by google or yahoo indexing bots ?
Posted: Mon Feb 05, 2007 5:49 pm
by Vogateer
Google indexes my site, and it's on a non-standard port, though I have no clue about how it might impact one's Page Rank.
Posted: Mon Feb 05, 2007 6:22 pm
by DaveQB
Hmm interesting.
Is it port 8080 or something totally random like 1234 ?
Posted: Mon Feb 05, 2007 6:28 pm
by CptnObvious999
You could always block them in the robots.txt file or you could require authentication to view it.
Posted: Mon Feb 05, 2007 6:41 pm
by DaveQB
Yes, am aware of that. More curious then anything. Was talking to a mate about setting up a private blog and the layers of security you can have and wondered if that would be an added extra help. Guess not though.
Posted: Mon Feb 05, 2007 7:02 pm
by allix
DaveQB wrote:Yes, am aware of that. More curious then anything. Was talking to a mate about setting up a private blog and the layers of security you can have and wondered if that would be an added extra help. Guess not though.
you could set up a vpn
http://openvpn.net/ , whereby nobody that you have not authorized can view it.
Posted: Mon Feb 05, 2007 7:22 pm
by DaveQB
Good call. But then you need VPN clients on everyones machine that needs to see it. Considering they are remote and using Windows and have no technical skills.....could get sticky.
Posted: Mon Feb 05, 2007 9:09 pm
by chuck
DaveQB wrote:Yes, am aware of that. More curious then anything. Was talking to a mate about setting up a private blog and the layers of security you can have and wondered if that would be an added extra help. Guess not though.
Putting it on a random higher port would help. Password protecting the page/directory would be the best option in addition to the robots.txt file.
Keep in mind that crawlers respect the robots.txt file because they are told to do so. Some spammers and crackers do use crawlers that ignore the file. There is nothing inherently secure about using the robots.txt file.
Posted: Mon Feb 05, 2007 9:52 pm
by DaveQB
Yeah, robots.txt is not a security mechanism. Its just a courtesy.
I know many ways to secure it. Was more curious.
Posted: Tue Feb 06, 2007 5:00 pm
by Vogateer
Whoa, I didn't know you were trying to hide it from google. I actually submitted my site through google's webmaster tools because I wanted it to be found. I have no idea if a spider would have picked it up otherwise...
Posted: Tue Feb 06, 2007 7:12 pm
by DaveQB
Of course it does.
Thats the point of googlebot.
My blog keeps updated on google every 2-3 weeks it seems. And its only a very small site.
Posted: Wed Feb 07, 2007 9:22 am
by Gomer_X
Vogateer wrote:Whoa, I didn't know you were trying to hide it from google. I actually submitted my site through google's webmaster tools because I wanted it to be found. I have no idea if a spider would have picked it up otherwise...
I have never submitted my site to anybody, but Google found it just fine. I mirror podcasts (which means lots of links to my site) which is probably how I was found. There are sites that will spider any new domain that is registered, though.
I've found that the bigger engines respect my robots.txt, but I still don't keep anything sensitive and publically accessible in those directories. Anyone malicious can pull my robots.txt and specifically look there if they want.
Once I saw that I was getting Google hits on the search term "pictures of my wife," I cleaned up my pics directory and locked some things down. There was never anything compromising, but my wife believes the internet will steal her soul.
