Home

Welcome to the storm-crawler wiki!

Getting Started

• Introduction
• Configuration: how to configure the storm-crawler
• Registering Metadata for Serialization: If your topology doesn't extend ConfigurableTopology, you will need to manually register storm-crawler's Metadata class for serialization in Storm.

Components

• Protocols: Network protocols that are usable in storm-crawler
• SiteMapParserBolt: how to handle sitemaps
• JSoupParserBolt: parse HTML documents
• URLFilters: how to filter or normalise outlinks

Resources

• Presentations
• Powered By
• External plugins