Web crawlers

This forum is for technical questions related to the message board. Now for registered users only.

Web crawlers

Postby Dr. V » Wed Nov 14, 2007 10:24 pm

Does anybody know anything about web crawlers? Using them to trawl websites and automatically download specific information into a database.

I've tried a couple of things - mirroring the website / mod_oia - some kind of Apache thingy (needless to say I wasn't in charge of that one!) / web scraper / wget - but so far nothing is really working.

Apparently its a cutting edge research method - so cutting edge nobody can get it to work!

thanks,
Last edited by Dr. V on Wed Nov 14, 2007 11:04 pm, edited 1 time in total.
User avatar
Dr. V
Academikus Seductresus
 
Posts: 1567
Joined: Thu Aug 18, 2005 2:03 pm
Location: UK

Postby Kurt » Wed Nov 14, 2007 10:52 pm

and I was of no help to her at all.
User avatar
Kurt
In Manus Manus
 
Posts: 21998
Joined: Mon Mar 08, 2004 6:29 am
Location: New York City

Re: Web crawlers

Postby phasmoid » Fri Jul 04, 2008 8:16 pm

I work with search engine web crawlers. At least until my side job hijacking cigarette trucks gains some steam.

If i understand your question correctly you are using your own crawler to scrape content from other sites, right? Or do you want to get your own site crawled - not sure.

If you can give me a little more info i can possible answer your question.

Sorry for no profile yet. I am a newbie here.
You don't need a weatherman to know which way the wind blows
User avatar
phasmoid
BFCus Regularus
 
Posts: 164
Joined: Thu Jul 03, 2008 7:53 pm
Location: Dixie

Re: Web crawlers

Postby coldharvest » Fri Jul 04, 2008 8:23 pm

phasmoid wrote:I work with search engine web crawlers. At least until my side job hijacking cigarette trucks gains some steam.

If i understand your question correctly you are using your own crawler to scrape content from other sites, right? Or do you want to get your own site crawled - not sure.

If you can give me a little more info i can possible answer your question.

Sorry for no profile yet. I am a newbie here.

You behave better than most of them so feel free to stay.
I know the law. And I have spent my entire life in its flagrant disregard.
User avatar
coldharvest
Abdul Rahman
 
Posts: 25677
Joined: Thu Mar 25, 2004 2:36 am
Location: Island of Misfit Toys

Re: Web crawlers

Postby phasmoid » Sat Jul 05, 2008 3:46 pm

Thanks, will do. I can help with questions regarding search engines, survivalism, kalashnikovs, and drinking. Cross trained, you see.
You don't need a weatherman to know which way the wind blows
User avatar
phasmoid
BFCus Regularus
 
Posts: 164
Joined: Thu Jul 03, 2008 7:53 pm
Location: Dixie

Re: Web crawlers

Postby Dr. V » Sun Jul 06, 2008 9:32 pm

phasmoid wrote:I work with search engine web crawlers. At least until my side job hijacking cigarette trucks gains some steam.

If i understand your question correctly you are using your own crawler to scrape content from other sites, right? Or do you want to get your own site crawled - not sure.

If you can give me a little more info i can possible answer your question.

.


Thanks for the offer but its too late - I gave it up as a bad job (having done the rounds of every techie person I could find and a few who weren't!) I found another way of gathering the data - which actually in hindsight is more accurate (which is always good) but much lengthier (which is bad) however if I decide to have another go in the future I'll be in touch - I'm kindof curious to know how it works anyway so when I get a minute or two spare (which could be long in the future as I'm moving house) you might get an incoming PM which outlines the 'master plan' in more detail.
User avatar
Dr. V
Academikus Seductresus
 
Posts: 1567
Joined: Thu Aug 18, 2005 2:03 pm
Location: UK

Re: Web crawlers

Postby phasmoid » Sun Jul 06, 2008 10:00 pm

My pleasure vlindsay. I can blab on about search engines and spiders etc so if i can be of help just give me a holler.
You don't need a weatherman to know which way the wind blows
User avatar
phasmoid
BFCus Regularus
 
Posts: 164
Joined: Thu Jul 03, 2008 7:53 pm
Location: Dixie


Return to Technical Questions

Who is online

Users browsing this forum: No registered users and 5 guests