Details

Applies to
1.7.0
Purpose
Capable to define spider allowed actions
Description
Allow admin to set a list of allowed actions for spiders

Patch

   1 diff -r 5afc0a9e3fc3 MoinMoin/config/multiconfig.py
   2 --- a/MoinMoin/config/multiconfig.py	Wed Jun 25 21:01:58 2008 +0200
   3 +++ b/MoinMoin/config/multiconfig.py	Sat Jun 28 07:15:36 2008 +0800
   4 @@ -605,6 +605,7 @@
   5                    'intelix|jeeves|larbin|leech|libwww-perl|linkbot|linkmap|linkwalk|litefinder|mercator|'
   6                    'microsoft.url.control|mirror| mj12bot|msnbot|msrbot|neomo|nutbot|omniexplorer|puf|robot|scooter|seekbot|'
   7                    'sherlock|slurp|sitecheck|snoopy|spider|teleport|twiceler|voilabot|voyager|webreaper|wget|yeti')
   8 +    spider_allowed_actions = []
   9  
  10      unzip_single_file_size = 2.0 * 1000 ** 2
  11      unzip_attachments_space = 200.0 * 1000 ** 2
  12 diff -r 5afc0a9e3fc3 MoinMoin/request/__init__.py
  13 --- a/MoinMoin/request/__init__.py	Wed Jun 25 21:01:58 2008 +0200
  14 +++ b/MoinMoin/request/__init__.py	Sat Jun 28 07:15:36 2008 +0800
  15 @@ -1036,7 +1036,8 @@
  16              # allow spiders to get attachments and do 'show'
  17              not (action == 'AttachFile' and 'do=get' in qs) and
  18              action != 'show' and
  19 -            action != 'sitemap'
  20 +            action != 'sitemap' and
  21 +            action not in self.cfg.spider_allowed_actions
  22              ):
  23              forbidden = self.isSpiderAgent
  24  

spider_allowed_actions.patch

Discussion

I have a customized action which generates blog-like RSS and FeedBurner grabs it, I realized that spider will be blocked after added FeedBurner to ua_spiders. So I made this patch to allow admin to set the list of allowed actions. Admin can add

spider_allowed_actions = [u'rss_blog_root']

The name of this setting is not appropriate and rss_rc, sitemap or AttachFile are default allowed action since they already are programmed. This patch is just an idea that some actions (customized actions) may need to be allowed for (good) spiders.

Plan


CategoryMoinMoinPatch

MoinMoin: MoinMoinPatch/SpiderAllowedActions (last edited 2008-06-27 23:36:12 by YuJieLin)