Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

crawler on appengine

i want to run a program continiously on appengine.This program will automatically crawl some website continiously and store the data into its database.Is it possible for the program to continiously keep doing it on appengine?Or will appengine kill the process?

Note:The website which will be crawled is not stored on appengine

like image 745
xyz Avatar asked Jun 21 '10 19:06

xyz


2 Answers

i want to run a program continiously on appengine.

Can't.

The closest you can get is background-running scheduled tasks that last no more than 30 seconds:

Notably, this means that the lifetime of a single task's execution is limited to 30 seconds. If your task's execution nears the 30 second limit, App Engine will raise an exception which you may catch and then quickly save your work or log process.

like image 172
Ben S Avatar answered Oct 03 '22 05:10

Ben S


A friend of mine suggested following

  • Create a task queue
  • Start the queue by passing some data.
  • Use an Exception handler and handle DeadlineExceededException.
  • In your handler create a new queue for same purpose.

You can run your job infinitely. You only need to consider used CPU Time and storage.

like image 29
Manjoor Avatar answered Oct 03 '22 05:10

Manjoor