Skip to content

Commit d1ce223

Browse files
committed
Version bumped and added a proper changelog.
1 parent 1ec59a7 commit d1ce223

File tree

2 files changed

+34
-5
lines changed

2 files changed

+34
-5
lines changed

README.rst

+33-4
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,22 @@
11
Redis-based components for Scrapy
22
=================================
33

4-
This is a initial work on Scrapy-Redis integration, not production-tested.
5-
Use it at your own risk!
4+
This project attempts to provide Redis-backed components for Scrapy.
65

76
Features:
87

98
* Distributed crawling/scraping
9+
You can start multiple spider instances that share a single redis queue.
10+
Best suitable for broad multi-domain crawls.
1011
* Distributed post-processing
12+
Scraped items gets pushed into a redis queued meaning that you can start as
13+
many as needed post-processing processes sharing the items queue.
1114

1215
Requirements:
1316

14-
* Scrapy >= 0.13 (development version)
17+
* Scrapy >= 0.14
1518
* redis-py (tested on 2.4.9)
16-
* redis server (tested on 2.2-2.4)
19+
* redis server (tested on 2.4-2.6)
1720

1821
Available Scrapy components:
1922

@@ -149,6 +152,32 @@ Then:
149152
redis-cli lpush myspider:start_urls http://google.com
150153

151154

155+
Changelog
156+
---------
157+
158+
0.5
159+
* Added `REDIS_URL` setting to support Redis connection string.
160+
* Added `SCHEDULER_IDLE_BEFORE_CLOSE` setting to prevent the spider closing too
161+
quickly when the queue is empty. Default value is zero keeping the previous
162+
behavior.
163+
164+
0.4
165+
* Added `RedisSpider` and `RedisMixin` classes as building blocks for spiders
166+
to be fed through a redis queue.
167+
* Added redis queue stats.
168+
* Let the encoder handle the item as it comes instead converting it to a dict.
169+
170+
0.3
171+
* Added support for different queue classes.
172+
* Changed requests serialization from `marshal` to `cPickle`.
173+
174+
0.2
175+
* Improved backward compatibility.
176+
* Added example project.
177+
178+
0.1
179+
* Initial version.
180+
152181

153182
.. image:: https://d2weczhvl823v0.cloudfront.net/darkrho/scrapy-redis/trend.png
154183
:alt: Bitdeli badge

setup.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77

88
setup(name='scrapy-redis',
9-
version='0.4',
9+
version='0.5',
1010
description='Redis-based components for Scrapy',
1111
long_description=LONG_DESC,
1212
author='Rolando Espinoza La fuente',

0 commit comments

Comments
 (0)