http on non-standard ports - defeat spiders ?

DaveQB · Post by **DaveQB** » Mon Feb 05, 2007 5:42 pm

Hi guys,

Would running apache on a non standard http port allow it to not be indexed by google or yahoo indexing bots ?

Vogateer · Post by **Vogateer** » Mon Feb 05, 2007 5:49 pm

Google indexes my site, and it's on a non-standard port, though I have no clue about how it might impact one's Page Rank.

DaveQB · Post by **DaveQB** » Mon Feb 05, 2007 6:22 pm

Hmm interesting.

Is it port 8080 or something totally random like 1234 ?

CptnObvious999 · Post by **CptnObvious999** » Mon Feb 05, 2007 6:28 pm

You could always block them in the robots.txt file or you could require authentication to view it.

DaveQB · Post by **DaveQB** » Mon Feb 05, 2007 6:41 pm

Yes, am aware of that. More curious then anything. Was talking to a mate about setting up a private blog and the layers of security you can have and wondered if that would be an added extra help. Guess not though.

allix · Post by **allix** » Mon Feb 05, 2007 7:02 pm

DaveQB wrote:Yes, am aware of that. More curious then anything. Was talking to a mate about setting up a private blog and the layers of security you can have and wondered if that would be an added extra help. Guess not though.

you could set up a vpn http://openvpn.net/ , whereby nobody that you have not authorized can view it.

DaveQB · Post by **DaveQB** » Mon Feb 05, 2007 7:22 pm

Good call. But then you need VPN clients on everyones machine that needs to see it. Considering they are remote and using Windows and have no technical skills.....could get sticky.

chuck · Post by **chuck** » Mon Feb 05, 2007 9:09 pm

DaveQB wrote:Yes, am aware of that. More curious then anything. Was talking to a mate about setting up a private blog and the layers of security you can have and wondered if that would be an added extra help. Guess not though.

Putting it on a random higher port would help. Password protecting the page/directory would be the best option in addition to the robots.txt file.

Keep in mind that crawlers respect the robots.txt file because they are told to do so. Some spammers and crackers do use crawlers that ignore the file. There is nothing inherently secure about using the robots.txt file.

DaveQB · Post by **DaveQB** » Mon Feb 05, 2007 9:52 pm

Yeah, robots.txt is not a security mechanism. Its just a courtesy.

I know many ways to secure it. Was more curious.

Vogateer · Post by **Vogateer** » Tue Feb 06, 2007 5:00 pm

Whoa, I didn't know you were trying to hide it from google. I actually submitted my site through google's webmaster tools because I wanted it to be found. I have no idea if a spider would have picked it up otherwise...

DaveQB · Post by **DaveQB** » Tue Feb 06, 2007 7:12 pm

Of course it does.
Thats the point of googlebot.

My blog keeps updated on google every 2-3 weeks it seems. And its only a very small site.

Gomer_X · Post by **Gomer_X** » Wed Feb 07, 2007 9:22 am

Vogateer wrote:Whoa, I didn't know you were trying to hide it from google. I actually submitted my site through google's webmaster tools because I wanted it to be found. I have no idea if a spider would have picked it up otherwise...

I have never submitted my site to anybody, but Google found it just fine. I mirror podcasts (which means lots of links to my site) which is probably how I was found. There are sites that will spider any new domain that is registered, though.

I've found that the bigger engines respect my robots.txt, but I still don't keep anything sensitive and publically accessible in those directories. Anyone malicious can pull my robots.txt and specifically look there if they want.

Once I saw that I was getting Google hits on the search term "pictures of my wife," I cleaned up my pics directory and locked some things down. There was never anything compromising, but my wife believes the internet will steal her soul.