cURL , how to scrape multiple page ?

**fordvb** · 11-10-2009 05:41 PM

Hello everyone,
i want to use cUrl to scrape content from another website and store them to a database , like in www.filestube.com .
i know how to get a page html code and use preg_match to get links for example . but i want to entre every link i find using preg_match and go to that link and get specific things on that page.

**Mr. DOS** · 11-10-2009 06:11 PM

preg_match is probably the wrong way to go about identifying URL's, seeing as it only identifies whether or not a match exists, not where in the string it exists. I don't have the time to look for the exact code to do what you need right now, but here's the steps your code needs to take:
1. Load the external page into a string.
2. Loop through the string looking for instances of href=". For each instance, get a substring from its position plus its length to the position of the next ". Add each substring to an array.
3. Once you've got the array populated, you'll need to loop through it (tip: foreach) and then process each item as necessary.

The big step is #2. I hope I've described it clearly enough. There's a bunch of built-in PHP functions to help.

--- Mr. DOS

**descalzo** · 11-10-2009 10:26 PM

Do you use cURL to get the base page (the one with the links)?
Scrape the links, put them in an array
Loop through the array, using cURL to grab the secondary pages.
Scrape each of those pages, put the info into the DB.

How much code do you have already?
Which part of the above is the part you cannot do?

**fordvb** · 11-11-2009 04:39 PM

The problem is not the first step . i can get all links , ihave used foreach and cUrl to get content but i can't get all links content, it stop on the secound link and show an error message .

**Mr. DOS** · 11-11-2009 08:48 PM

Well that's different then! What code do you have already, and what's the error?

--- Mr. DOS

**descalzo** · 11-12-2009 12:31 PM

Originally Posted by fordvb

and show an error message .

And the error message was.........?

Did you call

PHP Code:


curl_close($curlHandle);

at the bottom of the loop?

Were you using the curl_multi_xxx family of functions?

**JenniC** · 11-24-2009 10:42 AM

This is just an alternate approach.

To get the contents of all web pages referred in www.filestube.com, you can try the following code.

Code:

 
# Script urlcontents.txt
# Get the URLs referred in the referring URL.
var str URLList, nextURL
script ss_URLs.txt URL("http://www.filestube.com") > $URLList
echo -e "DEBUG: Found the following URLs."
echo -e $URLList
 
# Go thru URL List one by one and print contents using cat.
while ($URLList <> "")
do
 
    # Get the next referred URL.
    lex "1" $URLList > $nextURL
 
    # Show this URL's contents.
    cat $nextURL
done

Script is in biterscripting. Save the script in file C:/Scripts/urlcontents.txt, run it with the following command.

Code:

 
script "C:/Scripts/urlcontents.txt"

The documentation for the SS_URLs script and cat command are at http://www.biterscripting.com/SS_URLs.html and http://www.biterscripting.com/helppages/repro.html . I just tested this - my output is showing the following ( I am posting only the beginning portion.)

Thread: cURL , how to scrape multiple page ?

LinkBack

Thread Tools

Display

cURL , how to scrape multiple page ?

Re: cURL , how to scrape multiple page ?

Re: cURL , how to scrape multiple page ?

Re: cURL , how to scrape multiple page ?

Re: cURL , how to scrape multiple page ?

Re: cURL , how to scrape multiple page ?

Re: cURL , how to scrape multiple page ?

Similar Threads

Unique php page

Need Help: Cpanel-Apache page displaying on my site

Joomla Blank Page

Slow loading of page

cURL Login And Cookies and Regex

Posting Permissions