Latest Announcemnent
New thread

IP.Board 3 Linkchecker bot by Nihalz needs to be fixed. Anyone can help?

Status
Not open for further replies.
NewEraCracker

NewEraCracker

Active Member
1,335
2010
203
10
PHP: 
<?php

/**
 * LINK BOT v2.1 for IPB3
 * Build 16 by NewEraCracker
 * 
 * Author: nihalz
 * Modified by NewEraCracker
 * 
 * Web (nihalz): http://nihalz.forums-free.com/
 * Web (NewEraCracker): http://planet-dl.org/
 **/

if ( ! defined( 'IN_IPB' ) )
{
	print "<h1>Incorrect access</h1>You cannot access this file directly. If you have recently upgraded, make sure you upgraded all the relevant files.";
	exit();
}

class task_item
{
	/**
	* Parent task manager class
	*
	* @access	protected
	* @var		object
	*/
	protected $class;

	/**
	* This task data
	*
	* @access	protected
	* @var		array
	*/
	protected $task			= array();
	
	/**
	* Registry Object Shortcuts
	*/
	protected $registry;
	protected $settings;
	
	/**
	* Constructor
	*
	* @access	public
	* @param 	object		ipsRegistry reference
	* @param 	object		Parent task class
	* @param	array 		This task data
	* @return	void
	*/
	public function __construct( ipsRegistry $registry, $class, $task )
	{
		/* Make registry objects */
		$this->registry	= $registry;
		$this->settings =& $this->registry->fetchSettings();		
		$this->class	= $class;
		$this->task		= $task;
	}
	
	/**
	* Run this task
	*
	* @access	public
	* @return	void
	*/
	public function runTask()
	{
		
		//-----------------------------------------
		// ATTEMPT TO CONNECT TO DB
		//-----------------------------------------
				
		$con = mysql_connect($this->settings['sql_host'], $this->settings['sql_user'], $this->settings['sql_pass']) or die('Could not connect: ' . mysql_error());
		mysql_select_db($this->settings['sql_database'], $con);
		
		if(!mysql_query("SELECT bot_msg FROM " . $this->settings['sql_tbl_prefix'] . "posts LIMIT 0"))
		mysql_query("ALTER TABLE " . $this->settings['sql_tbl_prefix'] . "posts ADD COLUMN bot_msg varchar (20)");
		
		//-----------------------------------------
		// CHECK AND GET BOT SETTINGS
		//-----------------------------------------
		
		if($this->settings['linkbot_member_id']=="" OR $this->settings['linkbot_scanfirstpost']=="" OR $this->settings['linkbot_forumids']=="" OR $this->settings['linkbot_trashcan_id']=="" OR $this->settings['linkbot_threshold']=="" OR $this->settings['linkbot_reply_msg']=="" OR !is_numeric($this->settings['linkbot_member_id']) OR !is_numeric($this->settings['linkbot_threshold']))
		{
			echo "Please configure the bot's settings completely and correctly. Hit your browser's back button to go back to the ACP.";
			mysql_query("UPDATE " . $this->settings['sql_tbl_prefix'] . "task_manager SET task_locked=0 WHERE task_id = " . $this->task['task_id']);
			mysql_close($con);
			exit;
		}
		$get_bot_name = mysql_query("SELECT members_l_display_name FROM " . $this->settings['sql_tbl_prefix'] . "members WHERE member_id=".$this->settings['linkbot_member_id']);
		$bot_name = mysql_result($get_bot_name,0);
		if($this->settings['linkbot_scanfirstpost'] == 1)
		$scan_first_option = " AND p.new_topic=1";
		
		$hosts = explode("\n", $this->settings['linkbot_filehosts']);
		$total_urls = 0;
		$total_dead_urls = 0;
		$total_errors = 0;
		$i = 0;
			
		//-----------------------------------------
		// BOT START
		//-----------------------------------------
		
		mysql_query("INSERT INTO " . $this->settings['sql_tbl_prefix'] . "task_logs (log_title,log_date,log_ip,log_desc) VALUES ('" . $this->task['task_title'] . "','" . time() . "','127.0.0.1','Started Scanning Forums')");
		
		//-----------------------------------------
		// GET POSTS
		//-----------------------------------------
		
		$get_post_query = mysql_query("SELECT t.forum_id,t.title,t.title_seo,p.post,p.pid,p.topic_id,p.new_topic FROM " . $this->settings['sql_tbl_prefix'] . "topics AS t INNER JOIN " . $this->settings['sql_tbl_prefix'] . "posts AS p WHERE t.forum_id IN (".$this->settings['linkbot_forumids'].") AND t.forum_id<>".$this->settings['linkbot_trashcan_id']." AND t.tid=p.topic_id".$scan_first_option);
		
		if ( mysql_num_rows( $get_post_query ) )
		{
			while( $post = mysql_fetch_assoc( $get_post_query ) )
			{	
				
				$i++;
				
				//-----------------------------------------
				// FOR DEBUGGING
				//-----------------------------------------
								
				if($i%$this->settings['linkbot_postinterval'] == 0)
				mysql_query("INSERT INTO " . $this->settings['sql_tbl_prefix'] . "task_logs (log_title,log_date,log_ip,log_desc) VALUES ('" . $this->task['task_title'] . "','" . time() . "','127.0.0.1','Scanned " . $i . " Posts')");
				
				$rs = ""; $mu = "";
				$total_links = 0;
				$total_dead = 0;
				
				//-----------------------------------------
				// GET POST CONTENT AND SCAN FOR LINKS
				//-----------------------------------------
				
				$subject = $post['post'];
				$pattern = "|http://.*/.+|";
				preg_match_all($pattern, $subject, $matches);
				foreach($matches[0] as $link)
				{
                
                    //-----------------------------------------
                    // INCREASE VARIABLE THAT STORES LINKS COUNT
                    //-----------------------------------------                    

                    $total_links++;	
				
					//-----------------------------------------
					// ADD RS AND MU LINKS TO BATCH ARRAY
					//-----------------------------------------  
					
					if(strpos($link,"rapidshare.com/files"))
					{
						$rs .= $link . "\n";
					}
					if(strpos($link,"megaupload.com"))
					{
						$mu .= $link . "\n";
					}
					
					//-----------------------------------------
					// CHECK HOSTS OTHER THAN RS AND MU
					//-----------------------------------------

                    /* Create temporary variables*/
                    foreach($hosts as $h)
                    {
                        $filehosts = explode("|",$h);
                        if(strpos($link,$filehosts[0])) { $keyword = $filehosts[1]; break; }
                    }
                    
                    /* Check other hosts defined in ACP */
                    if($hosts != "" && $keyword != "")  
                    {
					
						// @todo: limit sizes to avoid memory leaks
						
                        $ch = curl_init();
                        curl_setopt($ch, CURLOPT_URL,$link);
                        curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.10');
                        curl_setopt($ch, CURLOPT_HEADER, true);
                        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
                        curl_setopt($ch, CURLOPT_FAILONERROR, 1);
                        curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
                        curl_setopt($ch, CURLOPT_TIMEOUT, 8);
                        curl_setopt($ch, CURLOPT_COOKIEFILE, IPSLib::getAppDir( 'forums' ) . '/tasks/mscookie');
                        $result = curl_exec($ch);
                        $header  = curl_getinfo( $ch );
                        curl_close($ch);
                        if($header['http_code'] == "200")
                        {
                            if(!strpos($result,$keyword))
                            $total_dead++;
                        }
                        else
                        $total_errors++;
						
                    }
                    
                    /* Destroy temporary variables */
                    unset($keyword,$h,$filehosts);  
                }                    	
				
				//-----------------------------------------
				// BATCH CHECK RS LINKS
				//-----------------------------------------
				
				if($rs != "")
				{
					$ch = curl_init();
					curl_setopt($ch, CURLOPT_URL,"http://rapidshare.com/cgi-bin/checkfiles.cgi");
					curl_setopt($ch, CURLOPT_FAILONERROR, 1);
					curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
					curl_setopt($ch, CURLOPT_TIMEOUT, 5);
					curl_setopt($ch, CURLOPT_POST, 1);
					curl_setopt($ch, CURLOPT_POSTFIELDS, "toolmode=1&urls=$rs");
					$result = curl_exec($ch);
					$header  = curl_getinfo( $ch );
					curl_close($ch);
					if($header['http_code'] == "200")
					{
						$total_dead += substr_count($result,",-1");
					}
					else
					$total_errors++;
				}
				
				//-----------------------------------------
				// BATCH CHECK MU LINKS
				//-----------------------------------------
				
				if($mu != "")
				{
					$pattern = '|\?d=\w{8}|';
					preg_match_all($pattern, $mu, $matches);
					$mu = "";
					for($j=0;$j<count($matches[0]);$j++)
					{
						$mu .= "id" . $j . "=" . substr($matches[0][$j],3) . "&";
					}
					$ch = curl_init();
					curl_setopt($ch, CURLOPT_URL,"http://megaupload.com/mgr_linkcheck.php");
					curl_setopt($ch, CURLOPT_FAILONERROR, 1);
					curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
					curl_setopt($ch, CURLOPT_TIMEOUT, 5);
					curl_setopt($ch, CURLOPT_POST, 1);
					curl_setopt($ch, CURLOPT_POSTFIELDS, "$mu");
					$result = curl_exec($ch);
					$header  = curl_getinfo( $ch );
					curl_close($ch);
					if($header['http_code'] == "200")
					{
						$pattern = '|id\d{1}=1|';
						preg_match_all($pattern, $result, $matches);
						$total_dead += count($matches[0]);
					}
					else
					$total_errors++;
				}

				//-------------------------------------------------
				// IF TOTAL LINKS > 0, TAKE SOME ACTION
				//-------------------------------------------------

				if($total_links > 0)
				{
					
					//-------------------------------------------------
					// IF DEAD LINKS % > THRESHOLD, TAKE SOME ACTION
					//-------------------------------------------------
					
					if(($total_dead/$total_links)*100 >= $this->settings['linkbot_threshold'])
					{
						if($this->settings['linkbot_action'] == "1")
						{	
							//-----------------------------------------
							// POST NOT FIRST POST OF TOPIC, UNAPPROVE
							//-----------------------------------------
							
							if($post['new_topic'] != "1")
							{
								mysql_query("UPDATE " . $this->settings['sql_tbl_prefix'] . "posts SET queued=1 WHERE pid = " . $post['pid']);
								$posts = mysql_query("SELECT COUNT(pid) as posts FROM " . $this->settings['sql_tbl_prefix'] . "posts WHERE queued!=1 AND topic_id=".$post['topic_id']);
								$pcount = mysql_result($posts,0) - 1;
								$qposts = mysql_query("SELECT COUNT(pid) as posts FROM " . $this->settings['sql_tbl_prefix'] . "posts WHERE queued=1 AND topic_id=".$post['topic_id']);
								$qpcount = mysql_result($qposts,0);
								mysql_query("UPDATE " . $this->settings['sql_tbl_prefix'] . "topics SET posts=" . $pcount . ",topic_queuedposts=" . $qpcount . " WHERE tid = " . $post['topic_id']);
								unset($qposts,$qpcount,$posts,$pcount);
							}
							
							//----------------------------------------------------------
							// POST IS FIRST POST OF TOPIC, MOVE TOPIC AND ADD BOT REPLY
							//----------------------------------------------------------
					
							else
							{
								mysql_query("UPDATE " . $this->settings['sql_tbl_prefix'] . "topics SET forum_id='" . $this->settings['linkbot_trashcan_id'] . "' WHERE tid = " . $post['topic_id']);
								mysql_query("UPDATE " . $this->settings['sql_tbl_prefix'] . "polls SET forum_id='" . $this->settings['linkbot_trashcan_id'] . "' WHERE tid = " . $post['topic_id']);
								mysql_query("INSERT INTO " . $this->settings['sql_tbl_prefix'] . "posts (author_id,author_name,ip_address,post_date,post,topic_id) VALUES ('" . $this->settings['linkbot_member_id'] . "','" . $bot_name . "','127.0.0.1','" . time() . "','" . htmlentities($this->settings['linkbot_reply_msg'],ENT_QUOTES) . "','" . $post['topic_id'] . "')");
								mysql_query("UPDATE " . $this->settings['sql_tbl_prefix'] . "topics SET posts=posts+1,last_poster_id='" . $this->settings['linkbot_member_id'] . "',last_post='" . time() . "',last_poster_name='" . $bot_name . "' WHERE tid = " . $post['topic_id']);
								mysql_query("UPDATE " . $this->settings['sql_tbl_prefix'] . "members SET posts=posts+1 WHERE member_id = " . $this->settings['linkbot_member_id']);
							}
						}
						
						//-----------------------------------------
						// REPORT POST
						//-----------------------------------------

						if($this->settings['linkbot_action'] == "2")
						{
						
							// @todo: Check if topic was reported
						
							$cacheupdate['last_updated'] = time();
							$report_content = "I found " . $total_dead . " dead link(s) in a post from the topic: ". $post['title'] ."<br/>[url=\"" . $this->settings['board_url'] ."/index.php?showtopic=". $post['topic_id'] ."&amp;view=findpost&amp;p=". $post['pid'] ."\"]Click here to view the post[/url]";
							mysql_query("INSERT INTO " . $this->settings['sql_tbl_prefix'] . "rc_reports_index (title,status,url,rc_class,updated_by,date_updated,date_created,exdat1,exdat2,exdat3,num_reports,num_comments,seoname,seotemplate) VALUES ('". htmlentities($post['title'],ENT_QUOTES) ."','1','/index.php?showtopic=". $post['topic_id'] ."&amp;view=findpost&amp;p=". $post['pid'] ."','2','". $this->settings['linkbot_member_id'] ."',". time() .",". time() .",'". $post['forum_id'] ."','". $post['topic_id'] ."','". $post['pid'] ."','1','0','". $post['title_seo'] ."','showtopic')");
							$rid = mysql_insert_id();
							mysql_query("INSERT INTO " . $this->settings['sql_tbl_prefix'] . "rc_reports (rid,report,report_by,date_reported) VALUES ('". $rid ."','". $report_content ."','". $this->settings['linkbot_member_id'] ."',". time() .")");
							mysql_query("INSERT INTO " . $this->settings['sql_tbl_prefix'] . "cache_store (cs_key,cs_value,cs_array,cs_updated) VALUES ('report_cache','". serialize($cacheupdate) ."','1',". time() . ") ON DUPLICATE KEY UPDATE cs_value='". serialize($cacheupdate) ."',cs_updated=". time());
							unset($cacheupdate,$rid,$report_content);
						}
					}
					
					//-----------------------------------------
					// ADD BOT CHECK TIME TO POST
					//-----------------------------------------
					
					if($total_links>0)
					mysql_query("UPDATE " . $this->settings['sql_tbl_prefix'] . "posts SET bot_msg='" . time() . "," . $total_dead . "' WHERE pid = " . $post['pid']);
				}
				
				//-----------------------------------------
				// UPDATE GLOBAL VARIABLES AND CLEAR MEMORY
				//-----------------------------------------
				
				$total_urls += $total_links;
				$total_dead_urls += $total_dead;
				unset($total_links,$total_dead,$subject,$pattern,$matches,$link,$keyword,$ch,$result,$header,$post,$rs,$mu,$j);
			}
		}
		
		//-----------------------------------------
		// BOT FINISHED CHECKING
		//-----------------------------------------
		
		mysql_query("INSERT INTO " . $this->settings['sql_tbl_prefix'] . "task_logs (log_title,log_date,log_ip,log_desc) VALUES ('" . $this->task['task_title'] . "','" . time() . "','127.0.0.1','Finished! Total Posts Scanned: " . $i . " Total Links Checked: " . $total_urls . " Total Dead: " . $total_dead_urls . " Errors: " . $total_errors . "')");
		
		//-----------------------------------------
		// UPDATE FORUM STATISTICS
		//-----------------------------------------
		
		if($this->settings['linkbot_action'] == "1")
		{
			$allforums = explode(",",$this->settings['linkbot_forumids'].",".$this->settings['linkbot_trashcan_id']);
			foreach($allforums as $fid)
			{
				$topics = mysql_fetch_assoc(mysql_query("SELECT COUNT(tid) as count FROM " . $this->settings['sql_tbl_prefix'] . "topics WHERE approved=1 AND forum_id=" . $fid));
				$queued_topics = mysql_fetch_assoc(mysql_query("SELECT COUNT(tid) as count FROM " . $this->settings['sql_tbl_prefix'] . "topics WHERE approved=0 AND forum_id=" . $fid));
				$posts = mysql_fetch_assoc(mysql_query("SELECT SUM(posts) as replies FROM " . $this->settings['sql_tbl_prefix'] . "topics WHERE approved=1 AND forum_id=" . $fid));
				$queued_posts = mysql_fetch_assoc(mysql_query("SELECT SUM(topic_queuedposts) as replies FROM " . $this->settings['sql_tbl_prefix'] . "topics WHERE forum_id=" . $fid));
				$last_post = mysql_fetch_assoc(mysql_query("SELECT tid, title, last_poster_id, last_poster_name, last_post FROM " . $this->settings['sql_tbl_prefix'] . "topics WHERE approved=1 AND forum_id=" . $fid . " ORDER BY last_post DESC LIMIT 1"));
				if(!$last_post) { $last_post['last_poster_id'] = 0; $last_post['last_poster_name'] = ''; $last_post['last_post'] = 0; $last_post['title'] = ''; $last_post['tid'] = 0; }
				if($queued_posts['replies']=="") $queued_posts['replies']=0; if($posts['replies']=="") $posts['replies']=0;
				mysql_query("UPDATE " . $this->settings['sql_tbl_prefix'] . "forums SET last_poster_id=" . $last_post['last_poster_id'] . ",last_poster_name='" . $last_post['last_poster_name'] . "',last_post=" . $last_post['last_post'] . ",last_title='" . $last_post['title'] . "',last_id=" . $last_post['tid'] . ",topics=" . $topics['count'] . ",posts=" . $posts['replies'] . ",queued_posts=" . $queued_posts['replies'] . ",queued_topics=" . $queued_topics['count'] . " WHERE id=" . $fid);
				unset($topics,$queued_topics,$posts,$queued_posts,$last_post);
			}
		}
		
		//--------------------------------------------------
		// UNLOCK TASK, CLOSE DB CONNECTION AND CLEAR MEMORY
		//--------------------------------------------------
		
		mysql_query("UPDATE " . $this->settings['sql_tbl_prefix'] . "task_manager SET task_locked=0 WHERE task_id = " . $this->task['task_id']);
		mysql_close($con);
		
		unset($total_urls,$total_dead_urls,$total_errors,$i,$get_bot_name,$bot_name,$scan_first_option,$get_post_query,$post,$con,$this->settings,$this->registry,$this->task,$allforums,$fid,$hosts);
	
	}

}

This is the link_bot.php and currently needs two todos. I've been modifying it yet I don't have enough skills to complete job.

This bot was designed from IPB 3.0.5 by the way.

Link to LCB:
Code: 
http://rapidshare.com/files/403838512/LinkBotv2.1_IPB3_nihalz_Build_16.7z
OR: http://www.multiupload.com/CWQ6EXCFLI

Also creating a new version compatible with IPB 3.1 would be gr8.


Regards,
NewEraCracker.
 
5 comments
The rapidshare part is running fine, all other downloadservers are screwed up.

Anyone can fix this ????

I think a lot of IPB users will be happy if this is fixed
 
Rapidshare and Megaupload checker uses their API so it works.

I can help fixing the string searches in settings and add more cookies to sort some issues.

I still need someone to fix bugs in php file (and add more features).
 
strangely enough i'm using this on IPB 3.1 and it hasn't dumped a bad post yet. the problem is it seems to be checking old un-formatted posts from IPB 2.3.5 (leftovers from when I made the upgrade to IPB3).

sure, RS and MU maybe are easy to fix, but so are other hosts, as all that needs fixing (i would assume) are the regexes.

as to new features i wouldn't be able to do that as i've never really figured out how IPB registry works. not only that, but i wanted to make an IMDB grabber for IPB3+ but I can't figure out how the hell their bbcode system works, and nobody seems to care either.

to make matters even worse, i don't know what changes happened from IPB3.0.x -> 3.1.x so I don't understand why it's not working to begin with -_- ; otherwise I should be able to help.

a new feature we do need is the ability to check multiple link mirrors inside the same post, or even inside a thread, if posting mirrors in a 'reply-with-a-mirror' style is used.
 
Status
Not open for further replies.
Back
Top