Skip to content

Commit 3f63a5d

Browse files
MSevenMSeven
committed
Updated Unittests
* added grouping test * made _findProgress testable Updated config doc in Readme
1 parent 0398879 commit 3f63a5d

File tree

5 files changed

+163
-85
lines changed

5 files changed

+163
-85
lines changed

‎README

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,9 @@ $config['queue']['workermaxruntime'] = 0;
5656
#Seconds after which finished jobs may be cleaned from the jobTable
5757
$config['queue']['cleanuptimeout'] = 2000;
5858

59+
#Should a Workerprocess quit when there are no more tasks for it to execute (true = exit, false = keep running)
60+
$config['queue']['exitwhennothingtodo] = false
61+
5962
The values above are the default settings which apply, when no configuration is found.
6063

6164
Usage:

‎README.textile

Lines changed: 83 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -1,80 +1,83 @@
1-
2-
h1. CakePHP Queue Plugin
3-
4-
5-
h2. Background:
6-
7-
This is a very simple and minimalistic job Queue (or deferred-task) system for CakePHP.
8-
9-
Overall functionality is inspired by systems like Gearman, Beanstalk or dropr, but without any illusion to compete with these more advanced Systems.
10-
11-
The Plugin is an attempt to provide a basic, simple to use method to enable deferred job execution, without the hassle of setting up or running an extra queue daemon, while integrating nicely into CakePHP and also simplifying the creation of worker scripts.
12-
13-
h3. Why use deferred execution?
14-
15-
deferred execution makes sense (especially in PHP) when your page wants' to execute tasks, which are not directly related to rendering the current page.
16-
For instance, in a BBS-type system, a new users post might require the creation of multiple personalized email messages, notifying other users of the new content.
17-
Creating and sending these emails is completely irrelevant to the currently active user, and should not increase page response time.
18-
Another example would be downloading, extraction and/or analyzing an external file per request of the user.
19-
The regular solution to these problems would be to create specialized cronjobs which use specific database states to determine which action should be done.
20-
21-
The Queue Plugin provides a simple method to create and run such non-user-interaction-critical tasks.
22-
23-
While you can run multiple workers, and can (to some extend) spread these workers to different machines via a shared database, you should seriously consider using a more advanced system for high volume/high number of workers systems.
24-
25-
h2. Installation:
26-
27-
* Copy the files in this directory into _yourapp_/plugins/queue
28-
* Run the following command in the cake console to create the tables:
29-
on Cakephp 1.2:
30-
@cake schema run create -path plugins\queue\config\sql -name queue@
31-
on Cakephp 1.3
32-
@cake schema create -path plugins\queue\config\sql -name queue@
33-
34-
If you Installed the Queue Plugin in your global plugins directory instead of the app specific one, you will have to modify the -path parameter to point to the correct directory.
35-
36-
h2. Configuration:
37-
38-
The plugin allows some simple runtime configuration.
39-
You may create a file called "queue.php" inside your 'APP/config' folder (NOT the plugins config folder) to set the following values:
40-
41-
#seconds to sleep() when no executable job is found
42-
@$config['queue']['sleeptime'] = 10;@
43-
44-
#Propability in percent of a old job cleanup happening
45-
@$config['queue']['gcprop'] = 10;@
46-
47-
#Default timeout after which a job is requeued if the worker doesn't report back
48-
@$config['queue']['defaultworkertimeout'] = 120;@
49-
50-
#Default number of retries if a job fails or times out.
51-
@$config['queue']['defaultworkerretries'] = 4;@
52-
53-
#Seconds of runnig time after which the worker will terminate (0 = unlimited)
54-
@$config['queue']['workermaxruntime'] = 0;@
55-
56-
The values above are the default settings which apply, when no configuration is found.
57-
58-
h2. Usage:
59-
60-
Run the following using the cakephp shell:
61-
62-
@cake queue help@
63-
* Display Help message
64-
65-
@cake queue add <taskname>@
66-
* Try to call the cli add() function on a task
67-
* tasks may or may not provide this functionality.
68-
@cake queue runworker@
69-
* run a queue worker, which will look for a pending task it can execute.
70-
* the worker will always try to find jobs matching its installed Tasks
71-
72-
*Notes:*
73-
_<taskname>_ may either be the complete classname (eg. queue_example) or the shorthand without the leading "queue_" (eg. example)
74-
75-
Use '@cake queue help@' to get a list of installed/available tasks.
76-
77-
Custom tasks should be placed in _yourapp_/vendors/shells/tasks.
78-
Tasks should be named 'queue_something.php' and implement a "queueSomethingTask", keeping Cakephp Naming conventions intact.
79-
80-
A detailed Example task can be found in /vendors/shells/tasks/queue_example.php inside this folder.
1+
2+
h1. CakePHP Queue Plugin
3+
4+
5+
h2. Background:
6+
7+
This is a very simple and minimalistic job Queue (or deferred-task) system for CakePHP.
8+
9+
Overall functionality is inspired by systems like Gearman, Beanstalk or dropr, but without any illusion to compete with these more advanced Systems.
10+
11+
The Plugin is an attempt to provide a basic, simple to use method to enable deferred job execution, without the hassle of setting up or running an extra queue daemon, while integrating nicely into CakePHP and also simplifying the creation of worker scripts.
12+
13+
h3. Why use deferred execution?
14+
15+
deferred execution makes sense (especially in PHP) when your page wants' to execute tasks, which are not directly related to rendering the current page.
16+
For instance, in a BBS-type system, a new users post might require the creation of multiple personalized email messages, notifying other users of the new content.
17+
Creating and sending these emails is completely irrelevant to the currently active user, and should not increase page response time.
18+
Another example would be downloading, extraction and/or analyzing an external file per request of the user.
19+
The regular solution to these problems would be to create specialized cronjobs which use specific database states to determine which action should be done.
20+
21+
The Queue Plugin provides a simple method to create and run such non-user-interaction-critical tasks.
22+
23+
While you can run multiple workers, and can (to some extend) spread these workers to different machines via a shared database, you should seriously consider using a more advanced system for high volume/high number of workers systems.
24+
25+
h2. Installation:
26+
27+
* Copy the files in this directory into _yourapp_/plugins/queue
28+
* Run the following command in the cake console to create the tables:
29+
on Cakephp 1.2:
30+
@cake schema run create -path plugins\queue\config\sql -name queue@
31+
on Cakephp 1.3
32+
@cake schema create -path plugins\queue\config\sql -name queue@
33+
34+
If you Installed the Queue Plugin in your global plugins directory instead of the app specific one, you will have to modify the -path parameter to point to the correct directory.
35+
36+
h2. Configuration:
37+
38+
The plugin allows some simple runtime configuration.
39+
You may create a file called "queue.php" inside your 'APP/config' folder (NOT the plugins config folder) to set the following values:
40+
41+
#seconds to sleep() when no executable job is found
42+
@$config['queue']['sleeptime'] = 10;@
43+
44+
#Propability in percent of a old job cleanup happening
45+
@$config['queue']['gcprop'] = 10;@
46+
47+
#Default timeout after which a job is requeued if the worker doesn't report back
48+
@$config['queue']['defaultworkertimeout'] = 120;@
49+
50+
#Default number of retries if a job fails or times out.
51+
@$config['queue']['defaultworkerretries'] = 4;@
52+
53+
#Seconds of runnig time after which the worker will terminate (0 = unlimited)
54+
@$config['queue']['workermaxruntime'] = 0;@
55+
56+
#Should a Workerprocess quit when there are no more tasks for it to execute (true = exit, false = keep running)
57+
@$config['queue']['exitwhennothingtodo] = false@
58+
59+
The values above are the default settings which apply, when no configuration is found.
60+
61+
h2. Usage:
62+
63+
Run the following using the cakephp shell:
64+
65+
@cake queue help@
66+
* Display Help message
67+
68+
@cake queue add <taskname>@
69+
* Try to call the cli add() function on a task
70+
* tasks may or may not provide this functionality.
71+
@cake queue runworker@
72+
* run a queue worker, which will look for a pending task it can execute.
73+
* the worker will always try to find jobs matching its installed Tasks
74+
75+
*Notes:*
76+
_<taskname>_ may either be the complete classname (eg. queue_example) or the shorthand without the leading "queue_" (eg. example)
77+
78+
Use '@cake queue help@' to get a list of installed/available tasks.
79+
80+
Custom tasks should be placed in _yourapp_/vendors/shells/tasks.
81+
Tasks should be named 'queue_something.php' and implement a "queueSomethingTask", keeping Cakephp Naming conventions intact.
82+
83+
A detailed Example task can be found in /vendors/shells/tasks/queue_example.php inside this folder.

‎models/queued_task.php

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -245,9 +245,9 @@ protected function _findProgress($state, $query = array(), $results = array()) {
245245
if ($state == 'before') {
246246

247247
$query['fields'] = array(
248-
'QueuedTask.reference',
249-
'(CASE WHEN QueuedTask.notbefore > NOW() THEN \'NOT_READY\' WHEN QueuedTask.fetched IS NULL THEN \'NOT_STARTED\' WHEN QueuedTask.fetched IS NOT NULL AND QueuedTask.completed IS NULL AND QueuedTask.failed = 0 THEN \'IN_PROGRESS\' WHEN QueuedTask.fetched IS NOT NULL AND QueuedTask.completed IS NULL AND QueuedTask.failed > 0 THEN \'FAILED\' WHEN QueuedTask.fetched IS NOT NULL AND QueuedTask.completed IS NOT NULL THEN \'COMPLETED\' ELSE \'UNKNOWN\' END) AS status',
250-
'QueuedTask.failure_message'
248+
$this->alias . '.reference',
249+
'(CASE WHEN ' . $this->alias . '.notbefore > NOW() THEN \'NOT_READY\' WHEN ' . $this->alias . '.fetched IS NULL THEN \'NOT_STARTED\' WHEN ' . $this->alias . '.fetched IS NOT NULL AND ' . $this->alias . '.completed IS NULL AND ' . $this->alias . '.failed = 0 THEN \'IN_PROGRESS\' WHEN ' . $this->alias . '.fetched IS NOT NULL AND ' . $this->alias . '.completed IS NULL AND ' . $this->alias . '.failed > 0 THEN \'FAILED\' WHEN ' . $this->alias . '.fetched IS NOT NULL AND ' . $this->alias . '.completed IS NOT NULL THEN \'COMPLETED\' ELSE \'UNKNOWN\' END) AS status',
250+
$this->alias . '.failure_message'
251251
);
252252

253253
if (isset($query['conditions']['exclude'])) {
@@ -262,10 +262,9 @@ protected function _findProgress($state, $query = array(), $results = array()) {
262262
);
263263
}
264264
if (isset($query['conditions']['group'])) {
265-
$query['conditions'][]['QueuedTask.group'] = $query['conditions']['group'];
265+
$query['conditions'][][$this->alias . '.group'] = $query['conditions']['group'];
266266
unset($query['conditions']['group']);
267267
}
268-
269268
return $query;
270269

271270
} else {

‎tests/cases/models/queued_task.test.php

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -347,5 +347,61 @@ public function testRequeueAfterTimeout() {
347347
$this->assertEqual($tmp['failed'], '1');
348348
}
349349

350+
public function testRequestGroup() {
351+
$capabilities = array(
352+
'task1' => array(
353+
'name' => 'task1',
354+
'timeout' => 1,
355+
'retries' => 2,
356+
'rate' => 0
357+
)
358+
);
359+
360+
// create an ungrouped task
361+
$this->assertTrue($this->QueuedTask->createJob('task1', 1));
362+
//create a Grouped Task
363+
$this->assertTrue($this->QueuedTask->createJob('task1', 2, null, 'testgroup'));
364+
365+
// Fetching without group should completely ignore the Group field.
366+
$tmp = $this->QueuedTask->requestJob($capabilities);
367+
$this->assertEqual($tmp['jobtype'], 'task1');
368+
$this->assertEqual(unserialize($tmp['data']), 1);
369+
$tmp = $this->QueuedTask->requestJob($capabilities);
370+
$this->assertEqual($tmp['jobtype'], 'task1');
371+
$this->assertEqual(unserialize($tmp['data']), 2);
372+
373+
// well, lets tra that Again, while limiting by Group
374+
// create an ungrouped task
375+
$this->assertTrue($this->QueuedTask->createJob('task1', 3));
376+
//create a Grouped Task
377+
$this->assertTrue($this->QueuedTask->createJob('task1', 4, null, 'testgroup', 'Job number 4'));
378+
$this->assertTrue($this->QueuedTask->createJob('task1', 5, null, null, 'Job number 5'));
379+
$this->assertTrue($this->QueuedTask->createJob('task1', 6, null, 'testgroup', 'Job number 6'));
380+
381+
// we should only get tasks 4 and 6, in that order, when requesting inside the group
382+
$tmp = $this->QueuedTask->requestJob($capabilities, 'testgroup');
383+
$this->assertEqual($tmp['jobtype'], 'task1');
384+
$this->assertEqual(unserialize($tmp['data']), 4);
385+
$tmp = $this->QueuedTask->requestJob($capabilities, 'testgroup');
386+
$this->assertEqual($tmp['jobtype'], 'task1');
387+
$this->assertEqual(unserialize($tmp['data']), 6);
388+
389+
// use FindProgress on the testgroup:
390+
$progress = $this->QueuedTask->find('progress', array(
391+
'conditions' => array(
392+
'group' => 'testgroup'
393+
)
394+
));
395+
396+
$this->assertEqual(count($progress), 3);
397+
398+
$this->assertNull($progress[0]['reference']);
399+
$this->assertEqual($progress[0]['status'], 'IN_PROGRESS');
400+
$this->assertEqual($progress[1]['reference'], 'Job number 4');
401+
$this->assertEqual($progress[1]['status'], 'IN_PROGRESS');
402+
$this->assertEqual($progress[2]['reference'], 'Job number 6');
403+
$this->assertEqual($progress[2]['status'], 'IN_PROGRESS');
404+
}
405+
350406
}
351407
?>

‎tests/fixtures/queued_task_fixture.php

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,18 @@ class QueuedTaskFixture extends CakeTestFixture {
2929
'null' => true,
3030
'default' => NULL
3131
),
32+
'group' => array(
33+
'type' => 'string',
34+
'length' => 255,
35+
'null' => true,
36+
'default' => NULL
37+
),
38+
'reference' => array(
39+
'type' => 'string',
40+
'length' => 255,
41+
'null' => true,
42+
'default' => NULL
43+
),
3244
'created' => array(
3345
'type' => 'datetime',
3446
'null' => false
@@ -54,6 +66,11 @@ class QueuedTaskFixture extends CakeTestFixture {
5466
'default' => '0',
5567
'length' => 3
5668
),
69+
'failure_message' => array(
70+
'type' => 'text',
71+
'null' => true,
72+
'default' => NULL
73+
),
5774
'workerkey' => array(
5875
'type' => 'string',
5976
'null' => true,

0 commit comments

Comments
 (0)