Latest Announcemnent
New thread

CurlAxel - PHP file download accelerator

Status
Not open for further replies.
J

jokerhacker

Active Member
415
2010
45
0
e5b0e69371.png

hey, i thought many people will like that.
I just wanted to help, so this PHP class may help some people who need it.
it's not a script, neither a software, it's a class used by developers to be integrated in their scripts.
How about cookies, proxies and other curl option??
well, everybody know curl_setopt_array(), you need to create an array, put all your configs there and pass it to the class using
setCurlOpts(). But keep in mind, these settings will be overwritten:
  • CURLOPT_RETURNTRANSFER
  • CURLOPT_FOLLOWLOCATION
  • CURLOPT_BINARYTRANSFER
  • CURLOPT_FRESH_CONNECT
  • CURLOPT_CONNECTTIMEOUT
  • CURLOPT_FILE
  • CURLOPT_RANGE

Included Functins
this is a list of function made in the class

  • activeLog() - true or false - to desactivate activate log - log file is created in tempdir
  • setUrl() - sets the url to download
  • setParts() - number of connection (parts) to be created
  • setCookies() - set the cookies string to be passed with the curl request
  • setTempDir() - must be absolute parth, if not found, it will be created
  • setProgressCallback() - enable progressbars
  • setDownloadDir() - must be absolute parth, if not found, it will be created
  • setCurlOpts() - this is the function allowing you to set your cookies, proxies and other option as an array of curl options.
  • getFileSize() - you may use it to get the file size of a url
  • isMT() - check if the given url supports multithreaded connection
  • download() - the most important function lol
  • slow_download() - force CurlAxel to download file using 1 connection
  • fast_download() - force CurlAxel to download file using multiple connection

Version
curent version is 0.5 30/10/11
still have lots of work to do, but i need your feedback!

Example how to use it
It's pretty simple!
PHP: 
$fileurl = "++ url here ++";
$curlaxel = new CurlAxel;
$curlaxel->setUrl($fileurl);
$curlaxel->activeLog(true);
$curlaxel->download();
and you will get your file with the fastest speed ever :)

Want to see progressbars?
It's pretty simple! (see test.php also)
PHP: 
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	<head>
		<script src="js/jquery.js"></script>
		<script src="js/jquery.progressbar.min.js"></script>
	</head>
	<body>
<?php
ini_set('error_reporting', E_ALL);
ini_set('display_errors', 1);
include 'CurlAxel.php';
//$fileurl = "http://cachefly.cachefly.net/100mb.test";
$fileurl = "http://93.190.137.8/1000mb.bin";

echo "downloading $fileurl <br>";
$curlaxel = new CurlAxel;
$curlaxel->setUrl($fileurl);  
$curlaxel->setProgressCallback(); 
$curlaxel->setBufferSize(32*1024*1024);
$curlaxel->activeLog(true); 
$curlaxel->download();

to use the progress bar, you need to integrate jquery and jquery progressbar plugin into your page. Here is what to do:
Code: 
<head>
	<script src="js/jquery.js"></script>
	<script src="js/jquery.progressbar.min.js"></script>
</head>

[SLIDE]http://i.imgur.com/b0SS0.png[/SLIDE]

I'd like to see people using this and providing feedback, if you see an error just activate the log and make a reply here, or contact me directly.
pleas keep it away from commercial use.

Special thanks to CuraHack who motivated me and gave me a great development environment :D


changelog
version 0.5 30/10/11
  • improved link checking method
  • added 100% working detailed progress bar (see docs for more infos)
  • some code cleanup
version 0.2 16/10/11
  • added support for various server responses
  • fixed filesize issue when requested server doesn't allow download accelerating
  • added support for cookies (passed as string)
  • added support for standard curl progress callback functon
version 0.1 29/08/11
  • small bug fixes
  • added GNU GPL 3 licence
version 0.1b beta 27/08/11
  • added optional buffersize to merge files. can be set though setBufferSize (in bytes).the default buffersize is 64Mb
version 0.1a beta 27/08/11
  • updated useragent choice as Robin H requested. now mozilla is the default user agent, but can be changed using user curl options array :)

 
Last edited:
73 comments
Noticed the useragent is just mozilla..
so why not let people decide which useragent they want to set?
And add error handling.

Anyway, nice piece of code.
That's one of the things I can delete of my 'I wanna code this:' list :p

edit:
This worry's me:
PHP: 
<?php
...
$contents = fread($bh[$i], filesize($partpath));
...
?>
Doesn't it create massive memory usage when you download a (very) large file?
Not sure how PHP would handle it tho, just a guess.
Fix would be to do a while loop and only concatenate x mbyte at once?

But once again, very very useful class.
 
Last edited:
Nice to see you're willing to work on it :)
And be sure to check out the concatenate issue.

You'll want to have something like this:
PHP: 
<?php
...
        $finalpath = $this->downdir . $this->filename;
        $i = 0;
        while(file_exists($finalpath)){
            $finalpath = $this->downdir . substr($this->filename,0,-4) . $i . substr($this->filename,-4);
            $i++;
        }
        $final = fopen($finalpath, "a");
        $chunksize = 5 * 1024 * 1024; // 5MB, I have no idea what a decent amount would be
        for ($i = 0; $i < sizeof($this->splits); $i++) {
            $partpath = $this->tempdir . $this->partnames[$i];
            fseek($bh[$i], 0, SEEK_SET);
            while(!FEOF($bh[$i])){
                $contents = fread($bh[$i], $chunksize);
                fwrite($final,$contents);
                flush();
                fflush();        
            }
            fclose($bh[$i]);
            unlink($partpath);
        }
        fclose($final);
...
?>
The while loop that checks the filename and changes it in case it exists might not be necessary, but never hurts.
I also assume there is a 3 char extension, which you will need to verify first. ( might also been wrongfully done, I always fuckup with substr :p )
I didn't test it myself.
 
Last edited:
I got this error:
Code: 
PHP Parse error:  syntax error, unexpected '}' in /home/curahack/public_html/axel/CurlAxel.php on line 141
 
jokerhacker said:
are you sure you have the last verson?
+ you didn't edit the file?
Click to expand...

I'm running 0.1b, I've only added a testfile to download, and modded the download location:

Code: 
class CurlAxel {
	private $url = "http://93.190.137.8/1000mb.bin";
	private $optarray = array();
	private $megaconnection;
	private $partnames = array();
	private $tempdir = "./temp/";
	private $downdir = "./files/";
	private $partcount = 5;
	private $log = true;
	private $buffersize = 67108864;
	public $version = "0.1b beta 27/08/11";
 
CuraHack, you don't need to put dots, just check the __construct function :)
try to revert the mods, and use it like that : create another file in the same dir and put its contents like that
PHP: 
<?php
require_once "CurlAxel.php";
$fileurl = "http://93.190.137.8/1000mb.bin"; 
$curlaxel = new CurlAxel; 
$curlaxel->setUrl($fileurl); 
$curlaxel->activeLog(true); 
//$curlaxel->setTempDir(somewhere);
//$curlaxel->setDownloadDir(somewhere);
$curlaxel->download();
aishamontreal, soft2050, praveer : i'm glad you like it :)
 
Good script, and most definitely good effort right there. But a quick look through it, and the way you're accelerating downloads (using Curl) is not efficient at all. As it stands, curl/php is not good with file streaming/handling. And curl was most definitely not built with multi threaded downloading as it's pioneer or trump feature; it's just there.

I'll go through it in detail and suggest improvements. But beyond that, good effort.

Google 'Aria2' and then Google 'Aria2Web'. Then look through Aria2Web and see if you can create a simpler class based on that for users of WJ. Aria2 is an excellent command line download accelerator for Linux; and coding a PHP XML-RPC gateway for it, which is simpler (and works) would be a great idea.
 
Lifetalk said:
Good script, and most definitely good effort right there. But a quick look through it, and the way you're accelerating downloads (using Curl) is not efficient at all. As it stands, curl/php is not good with file streaming/handling. And curl was most definitely not built with multi threaded downloading as it's pioneer or trump feature; it's just there.

I'll go through it in detail and suggest improvements. But beyond that, good effort.

Google 'Aria2' and then Google 'Aria2Web'. Then look through Aria2Web and see if you can create a simpler class based on that for users of WJ. Aria2 is an excellent command line download accelerator for Linux; and coding a PHP XML-RPC gateway for it, which is simpler (and works) would be a great idea.
Click to expand...

Hi, Lifetalk,

JokerHacker's idea was originated from my thread here, we already had this discussion, as mentioned in that thread Aria's XML-RPC is very badly documented so it'll be very hard to build as scripts as such with Aria's API, if you're able to prove us wrong, feel free to do so ;)
 
Status
Not open for further replies.
Back
Top