crawler on appengine

Question

i want to run a program continiously on appengine.This program will automatically crawl some website continiously and store the data into its database.Is it possible for the program to continiously keep doing it on appengine?Or will appengine kill the process?

Note:The website which will be crawled is not stored on appengine

Answer 1

i want to run a program continiously on appengine.

Can't.

The closest you can get is background-running scheduled tasks that last no more than 30 seconds:

Notably, this means that the lifetime of a single task's execution is limited to 30 seconds. If your task's execution nears the 30 second limit, App Engine will raise an exception which you may catch and then quickly save your work or log process.

Answer 2

A friend of mine suggested following

• Create a task queue
• Start the queue by passing some data.
• Use an Exception handler and handle DeadlineExceededException.
• In your handler create a new queue for same purpose.

You can run your job infinitely. You only need to consider used CPU Time and storage.