Details

Applies to
Moin 1.9
Purpose
Add Bingbot to the list of spiders
Description

Bing uses "bingbot" in their UA string: http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2010/09/03/bingbot-is-coming-to-town.aspx

Patch

   1 exporting patch:
   2 # HG changeset patch
   3 # User Gerald Combs <gerald@wireshark.org>
   4 # Date 1384881950 28800
   5 #      Tue Nov 19 09:25:50 2013 -0800
   6 # Node ID 072577e05c0c11de8e5ffe64ae38e02129fb8bad
   7 # Parent  77533e162654eb09faee8115c8c1884ffd85f864
   8 Add bingbot to ua_spiders.
   9 
  10 Bing has been crawling our wiki with the user agent "Mozilla/5.0
  11 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)".  A blog
  12 entry about the UA string can be found at
  13 http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2010/09/03/bingbot-is-coming-to-town.aspx
  14 
  15 diff -r 77533e162654 -r 072577e05c0c MoinMoin/config/multiconfig.py
  16 --- a/MoinMoin/config/multiconfig.py	Tue Aug 13 12:17:47 2013 +0200
  17 +++ b/MoinMoin/config/multiconfig.py	Tue Nov 19 09:25:50 2013 -0800
  18 @@ -877,7 +877,7 @@
  19      # and receive a FORBIDDEN for anything except viewing a page
  20      # list must not contain 'java' because of twikidraw wanting to save drawing uses this useragent
  21      ('ua_spiders',
  22 -     ('archiver|cfetch|charlotte|crawler|gigabot|googlebot|heritrix|holmes|htdig|httrack|httpunit|'
  23 +     ('archiver|bingbot|cfetch|charlotte|crawler|gigabot|googlebot|heritrix|holmes|htdig|httrack|httpunit|'
  24        'intelix|jeeves|larbin|leech|libwww-perl|linkbot|linkmap|linkwalk|litefinder|mercator|'
  25        'microsoft.url.control|mirror| mj12bot|msnbot|msrbot|neomo|nutbot|omniexplorer|puf|robot|scooter|seekbot|'
  26        'sherlock|slurp|sitecheck|snoopy|spider|teleport|twiceler|voilabot|voyager|webreaper|wget|yeti'),

bingbot_ua_spiders.patch

Discussion

Plan


CategoryMoinMoinPatch

MoinMoin: MoinMoinPatch/BingbotUaSpider (last edited 2013-11-20 12:53:22 by ThomasWaldmann)