Description

Many csv files generated with Microsoft Office products only have \r (0x0d) newlines. Trying to view these kinds of csv attachments results in a traceback:

Steps to reproduce

1. Test file

2. Include the csv file to the page with {{attachment:test.csv}}

Example

Example file

Component selection

I couldn't think of other components to show the csv than:

• formatter/__init__.py

• action/AttachFile.py

Details

formatter/__init__.py

2014-11-03 20:24:10,810 MoinMoin.wsgiapp ERROR An exception has occurred [https://localhost/collab/24070518].
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/wsgiapp.py", line 282, in __call__
    response = run(context)
  File "/usr/local/lib/python2.7/dist-packages/graphingwiki/__init__.py", line 476, in patched_run
    return orig_wsgiapp_run(context)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/wsgiapp.py", line 88, in run
    response = dispatch(request, context, action_name)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/wsgiapp.py", line 136, in dispatch
    response = handle_action(context, pagename, action_name)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/wsgiapp.py", line 195, in handle_action
    handler(context.page.page_name, context)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/action/edit.py", line 192, in execute
    pg.send_page()
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/Page.py", line 1217, in send_page
    start_line=pi['lines'])
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/Page.py", line 1317, in send_page_content
    self.execute(request, parser, code)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/Page.py", line 1342, in execute
    exec code
  File "24070518", line 43, in <module>
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/formatter/__init__.py", line 138, in attachment_inlined
    colorizer = Parser(content, self.request, filename=filename)
  File "/usr/local/lib/python2.7/dist-packages/graphingwiki/plugin/parser/text_csv.py", line 10, in __init__
    _Parser.__init__(self, raw, *args, **keys)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/parser/text_csv.py", line 128, in __init__
    cols = map(lambda x: x.decode('utf-8'), r.next()) + staticcols
Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

action/AttachFile.py

2014-11-03 20:24:32,837 MoinMoin.wsgiapp ERROR An exception has occurred [https://localhost/collab/24070518?action=AttachFile&do=view&target=test.csv].
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/wsgiapp.py", line 282, in __call__
    response = run(context)
  File "/usr/local/lib/python2.7/dist-packages/graphingwiki/__init__.py", line 476, in patched_run
    return orig_wsgiapp_run(context)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/wsgiapp.py", line 88, in run
    response = dispatch(request, context, action_name)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/wsgiapp.py", line 136, in dispatch
    response = handle_action(context, pagename, action_name)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/wsgiapp.py", line 195, in handle_action
    handler(context.page.page_name, context)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/action/AttachFile.py", line 522, in execute
    msg = handler(pagename, request)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/action/AttachFile.py", line 1153, in _do_view
    send_viewfile(orig_pagename, request)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/action/AttachFile.py", line 1070, in send_viewfile
    colorizer = Parser(content, request, filename=filename)
  File "/usr/local/lib/python2.7/dist-packages/graphingwiki/plugin/parser/text_csv.py", line 10, in __init__
    _Parser.__init__(self, raw, *args, **keys)
  File "/usr/local/lib/python2.7/dist-packages/MoinMoin/parser/text_csv.py", line 128, in __init__
    cols = map(lambda x: x.decode('utf-8'), r.next()) + staticcols
Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

This Wiki.

Workaround

This patch works for me:

--- action/AttachFile.py~       2012-12-29 21:06:54.000000000 +0200
+++ action/AttachFile.py        2014-11-03 20:24:38.869102126 +0200
@@ -1065,7 +1065,7 @@ def send_viewfile(pagename, request):
         Parser = wikiutil.getParserForExtension(request.cfg, ext)
         if Parser is not None:
             try:
-                content = file(fpath, 'r').read()
+                content = file(fpath, 'rU').read()
                 content = wikiutil.decodeUnknownInput(content)
                 colorizer = Parser(content, request, filename=filename)
                 colorizer.format(request.formatter)
--- formatter/__init__.py~      2012-12-29 21:06:54.000000000 +0200
+++ formatter/__init__.py       2014-11-03 20:24:17.588996604 +0200
@@ -131,7 +131,7 @@ class FormatterBase:
         Parser = wikiutil.getParserForExtension(self.request.cfg, ext)
         if Parser is not None:
             try:
-                content = file(fpath, 'r').read()
+                content = file(fpath, 'rU').read()
                 # Try to decode text. It might return junk, but we don't
                 # have enough information with attachments.
                 content = wikiutil.decodeUnknownInput(content)

Discussion

I am wondering a bit about this... - the usual line endings on dos/windows used to be crlf, while linux/posix systems use lf. So since when / why is MS Office generating cr-only line ends?

About the change you propose: please check the compatibility with python 2.5, 2.6 and 2.7 and document it here.

Plan

• Priority:
• Assigned to:

• Status: fixed: https://github.com/moinwiki/moin-1.9/pull/10

CategoryMoinMoinBugFixed