New Arg Parser

This should become a new argument parser for parsers, macros and other plugins requiring external parameters. Something like wikiutil.parseAttributes with positional arguments, type conversion, defaults, etc.

When this is finished, it should be recommended to plugin implementors. That should save us stupid parsing bugs like the ones in the Include macro.

Specification

• It should be able to parse a python like function call.
• arguments are defined as tuple of tuples (varname, type, default), where default is optional.

  • pattern = (('name',  str),
               ('from',  str,  None),
               ('to',    str,  None),
               ('text',  bool, false),
               ('title', bool, false))

• arguments without default are required - arguments with a default value are optional

  • PatternParser(pattern).parse('"FrontPage"')
    => ok
    PatternParser(pattern).parse('')
    => error!

• all arguments can be used positional (like in python). No argument must be left out between positional used arguments.

  • PatternParser(pattern).parse('"FrontPage", "^----$", "end", True, False')

• or as keyword arguments and can apear in any order after the positional args

  • PatternParser(pattern).parse('"FrontPage", "^----$", "end", title=True')
    => ok
    PatternParser(pattern).parse('"FrontPage", "^----$", "end", from=""')
    => error!

• a special boolean flag mode is possible: if the type is bool, one can just use the name of this kwarg to make it True. A prefix of - changes this to False. Example:

  • PatternParser(pattern).parse('"FrontPage", text, from="123"')
    => text == True
    PatternParser(pattern).parse('"FrontPage", to="end", title, text)
    => text == True, title == True
    PatternParser(pattern).parse('"FrontPage", -text, from="123"')
    => text == False

• it is required to write string within quotes. The old style unquoted strings are not supported.

Implementation

A basic working version using tokenize is ready. I will now write the tests to check all options. -- OliverGraf 2004-10-24 21:37:15

FullSearch in my testwiki now uses the new parser, feel free to test: http://fp2.sky.rhein-zeitung.de/moin.cgi/SearchTest -- OliverGraf 2004-10-25 07:37:57

• parse pattern

• string parameters

• int parameters

• float parameters

• bool/flag parameters

• tristate (on, off, none) (for colorizer numbering)

• write unittests

Discussion

I think this could be easily build using tokenize.

Example:

   1 import StringIO, tokenize
   2 def tok(s):
   3   s=StringIO.StringIO(s)
   4   for t in tokenize.generate_tokens(s.readline): 
   5     print tokenize.tok_name[t[0]]+'\t'+t[1]

Output:

>>> tok('FrontPage, -1, "test", from="^----$", 333')
NAME    FrontPage
OP      ,
OP      -
NUMBER  1
OP      ,
STRING  "test"
OP      ,
NAME    from
OP      =
STRING  "^----$"
OP      ,
NUMBER  333
ENDMARKER
>>> tok('start=1 step=22 numbering=on')
NAME    start
OP      =
NUMBER  1
NAME    step
OP      =
NUMBER  22
NAME    numbering
OP      =
NAME    on
ENDMARKER

I already started an argument parser used for the search macros. It is located at MoinMoin/util/parameterparser.py -- FlorianFesti 2004-10-22 06:44:55

• Looks nice! Especially the pattern string for specifying what arguments to expect looks good. -- OliverGraf 2004-10-22 06:54:43

More ideas:

• don't make it too complicated or people will rather hack their own
• return a dict so one can do locals().update(retdict)

  • this seems not to work all the time: on the commandline it works fine, but within the execute function of a macro locals gets updated, but using on eof the new vars still throws a UnboundLocalVariable exception.

• wikimacro could do the parsing before calling the macro, if the pattern is in some module global var like ArgumentPattern

  • as we process multiple wikis and multiple requests at about the same time, no globals can be used for that

    • it's not about caching an object, its just getting the defs from a readonly global (like Dependencies)

• defaults:

  • val1 = None
    val2 = 1
    val3 = 'dummy'
    pattern = '%(val1)s|%(val2)d%(val3)s'
    result = PatternParser.parse(pattern, args)
    locals().update(result)

• tuples as config (more or less pseudo-code):

  • pattern = (('val1', str,), 
               ('val2', int, optional, default=1),
               ('val3', str, optional, default='dummy'),
              )
    result = PatternParser(pattern).parse(args)

• The #FORMAT PI and #!parser hash-bang should be changed from:

  #FORMAT parser arg arg
  #!parser arg arg

  to:

  #FORMAT parser(arg, arg)
  #!parser(arg, arg)

  I added the arguments in 1.3 for the colorizers as whitespace separated arguments. To keep consistency, this should be changed to a brakened, comma-separated syntax, too. -- OliverGraf 2004-10-24 21:37:15

• perhaps add docstrings to each parameter, so the plugin can describe on its own what it expects? After all: I still think self-documenting plugins are a really good idea. -- OliverGraf 2004-10-25 08:32:26

I don't think we need python like arguments. We need much simpler and verbose way, just like html attributes.

[[Macro(arg="value" another="value")]]

Looks like easy to use and easy to parse.

• To me it looks like a bigger change, cause this would only allow keyword arguments. I like the pseudo-python way (note the subtle differences above!). -- OliverGraf 2004-10-26 05:44:23

How about this?

• [[Macro(page:xyz justincludeheadings:yes really:true)]]
  [[Macro(pagename:This page name contains spaces depth:4)]]

  • I like python syntax very much, cause its very clear. This thing won't allow flags and it is very unclear because this space string is not delimited. -- OliverGraf 2004-11-19 06:43:26