MoinMoin: MoinMoinChat/Logs/moin-dev/2008-05-11

2008-05-11T00:07:25  <ThomasWaldmann> btw, some Twisted and mod_python stuff still needs testing, if anybody cares about this stuff :)
2008-05-11T00:10:12  <dreimark> I've run them several times without seeing this timing problem. pagesaving sends the event to get wikidictsrescan
2008-05-11T00:12:18  <dreimark> if there is in between these a timing problem then it can cause an update problem of the dict
2008-05-11T00:16:06  <ThomasWaldmann> you maybe need a faster machine or slower clock :)
2008-05-11T00:16:46  <dreimark> do you think it depends on the length of the page too.
2008-05-11T00:16:58  <dreimark> default is 3000 users and one test with 30000
2008-05-11T00:18:17  <ThomasWaldmann> i just ran py.test in the toplevel dir
2008-05-11T00:18:45  * dreimark tests finished: 635 passed, 1 failed, 64 skipped in 104.89 seconds
2008-05-11T00:18:56  <ThomasWaldmann> (my machine is rather slow 1GHz/1GB, but has a flash based disk)
2008-05-11T00:21:16  <ThomasWaldmann>  tests finished: 617 passed, 2 failed, 67 skipped in 78.94 seconds
2008-05-11T00:21:55  <ThomasWaldmann> E       return self.groupdict[name]
2008-05-11T00:21:55  <ThomasWaldmann> >       KeyError: u'AnotherGroup'
2008-05-11T00:21:55  <ThomasWaldmann> [/home/tw/eclipse-ws/moin-1.7/src/MoinMoin/wikidicts.py:236]
2008-05-11T00:23:47  <dreimark> that is an older test. Is always this failing?
2008-05-11T00:23:59  <ThomasWaldmann> running it again: only 1 failure
2008-05-11T00:25:32  <ThomasWaldmann> E       assert result == "This is a brand new example"
2008-05-11T00:25:32  <ThomasWaldmann> >       assert u'This is an example' == 'This is a brand new example'
2008-05-11T00:25:44  <ThomasWaldmann> now 2 failures
2008-05-11T00:25:53  <ThomasWaldmann> [/home/tw/eclipse-ws/moin-1.7/src/MoinMoin/macro/_tests/test_GetVal.py:65]
2008-05-11T00:26:59  <dreimark> the first calls  page.renampePage os.rename(oldpath, newpath)
2008-05-11T00:27:50  <dreimark> flash r/w is asynchronous so it can be slower in the renaming
2008-05-11T00:28:53  <ThomasWaldmann> no
2008-05-11T00:29:11  <ThomasWaldmann> now 1 failure total again
2008-05-11T00:34:33  <dreimark> why does my system think I do like to have only 1GHz
2008-05-11T00:38:14  <dreimark> ok now with 2.2GHz
2008-05-11T00:38:16  <dreimark> tests finished: 635 passed, 1 failed, 64 skipped in 54.97 seconds
2008-05-11T00:39:25  <dreimark> tests finished: 635 passed, 1 failed, 64 skipped in 55.30 seconds
2008-05-11T00:42:20  <ThomasWaldmann> do you have a ramdisk? tmpfs?
2008-05-11T00:43:58  <dreimark> currently not, can add one
2008-05-11T00:44:31  <dreimark> we can add for a test  time.sleep(0.2) after page.renamePage('AnotherGroup')
2008-05-11T00:48:46  * ThomasWaldmann will do self.sleep() now :)
2008-05-11T00:48:53  <dreimark> good night
2008-05-11T00:48:59  <ThomasWaldmann> gn
2008-05-11T00:49:03  <dreimark> seen 65MB is the default size here
2008-05-11T01:09:32  <dreimark> ThomasWaldmann: confirmed
2008-05-11T01:27:57  <dreimark> gn
2008-05-11T09:37:54  <dreimark> mmoin
2008-05-11T09:37:58  <dreimark> I think we should not backport 1.6.3FullSearchWithXapianForCategory
2008-05-11T09:38:00  <dreimark> we can provide a patch for 1.6 on the page so only people who ran into this and wanted it fixed can get it
2008-05-11T09:46:38  * dreimark works on wikidictsrescan
2008-05-11T11:13:33  <CIA-53> Reimar Bauer <rb.proj AT googlemail DOT com> default * 3574:0b7eb697e952 1.7/MoinMoin/_tests/test_wikidicts.py: test_wikidicts: PEP8 whitespace fix
2008-05-11T11:15:17  <ThomasWaldmann> moin
2008-05-11T11:16:19  <ThomasWaldmann> dreimark: seems that the sourcecodechecker does not fail on these
2008-05-11T11:17:26  <dreimark> http://moinmo.in/ReimarBauer?action=AttachFile&do=view&target=wikidictsrescan_patch.patch
2008-05-11T11:17:43  <dreimark> ThomasWaldmann: yeah I was wondering too
2008-05-11T11:18:03  <dreimark> ThomasWaldmann: I have no failing dict test with that patch
2008-05-11T11:18:16  <dreimark> on my ramdisk too
2008-05-11T11:21:33  <ThomasWaldmann> ...because it can be interrupted?
2008-05-11T11:22:07  <dreimark> The actual suspension time may be less than that requested because any caught signal will terminate the sleep() following execution of that signal's catching routine.
2008-05-11T11:22:32  <dreimark> explaination from http://docs.python.org/lib/module-time.html
2008-05-11T11:23:10  <ThomasWaldmann> what signals do you expect?
2008-05-11T11:27:21  <ThomasWaldmann> and also please explain why and how this patch fixes the problem
2008-05-11T11:27:25  <dreimark> http://blog.nickvonhollen.com/2007/05/sleep-in-python-should-never-be-trusted.html
2008-05-11T11:29:39  <ThomasWaldmann> that doesnt answer my question
2008-05-11T11:30:03  <dreimark> I know I tried before with sleep currently it works too
2008-05-11T11:31:40  <ThomasWaldmann> the code in you patch might run around in circles for up to a second, consuming 100% cpu, that is not acceptable
2008-05-11T11:32:11  <dreimark> yeah, it is currently only an idea to get closer to the real solution
2008-05-11T11:32:18  <ThomasWaldmann> but let's first discuss about how it works at all
2008-05-11T11:32:25  <ThomasWaldmann> so why does that fix anything?
2008-05-11T11:33:50  <dreimark> it seems to me that there is something doing 1sec steps
2008-05-11T11:34:15  <dreimark> it can be from the filesystem too
2008-05-11T11:34:31  <dreimark> my ramdisk is of fstype ext2
2008-05-11T11:37:11  <ThomasWaldmann> did you try insert a 2s sleep into the test (between changing stuff and testing effect)?
2008-05-11T11:38:37  <dreimark> I did after every test 1s sleep and this doesn't change
2008-05-11T11:38:52  <dreimark> I have had 5-6 failing tests always
2008-05-11T11:39:05  <ThomasWaldmann> not after, but within
2008-05-11T11:39:29  <dreimark> right, after create, append ...
2008-05-11T11:55:10  <ThomasWaldmann> btw, create_random_string does not do what the function name tells it does
2008-05-11T11:56:39  <ThomasWaldmann> and why does it create 3000 random strings by default?
2008-05-11T11:59:19  <dreimark> may be it is a bit to excessive I do wanted to have a larger number
2008-05-11T11:59:55  <dreimark> if there are timing problems I don't wanted to test with to small list
2008-05-11T12:02:36  <ThomasWaldmann> creating slower timing by using a bigger list is broken
2008-05-11T12:03:07  <ThomasWaldmann> so please: change the name to something sane, and default count 10 or so
2008-05-11T12:04:34  <dreimark> create_name_list ?
2008-05-11T12:05:53  <ThomasWaldmann> create_random_string_list
2008-05-11T12:06:23  <dreimark> :)
2008-05-11T12:06:34  <ThomasWaldmann> and also fix param names and comments
2008-05-11T12:06:54  <ThomasWaldmann> it is just strings, not restricted to "names"
2008-05-11T12:07:34  <ThomasWaldmann> remove the # long list comment
2008-05-11T12:07:51  <dreimark> heh too much details now
2008-05-11T12:09:50  <ThomasWaldmann> the nuke_user code should be moved upwards to the stuff dealing with users
2008-05-11T12:14:31  <dreimark> how long can I make the list of strings in testAppendingGroupPage
2008-05-11T12:17:26  <ThomasWaldmann> what's the point of that test? testing scalability?
2008-05-11T12:17:33  <dreimark> right
2008-05-11T12:19:50  <ThomasWaldmann> ok, then add a comment telling this
2008-05-11T12:20:33  <ThomasWaldmann> and please, use more sane variable names
2008-05-11T12:20:45  <ThomasWaldmann>         name =  [u" * %s" % text for text in create_random_string(name_len=20, count=30000)]
2008-05-11T12:20:52  <dreimark> changed len to length already
2008-05-11T12:21:05  <ThomasWaldmann> this is not obfuscated python contest :P
2008-05-11T12:21:59  <dreimark> create_random_string_list(length=20, count=1) ?
2008-05-11T12:22:34  <ThomasWaldmann> maybe explain to yourself what your code is about and then choose speaking names
2008-05-11T12:23:20  <ThomasWaldmann> "name" is a bad name for page content
2008-05-11T12:23:37  <ThomasWaldmann> "text" is a bad name for a group member
2008-05-11T12:25:32  <ThomasWaldmann> and if the default length of 14(?) is no problem, you don't need to explicitely change it to 20
2008-05-11T12:29:39  <dreimark> i don't want to test for a repeated value
2008-05-11T12:32:11  <ThomasWaldmann> 52^14 is unlikely to have dupes
2008-05-11T12:36:50  <dreimark> unlikly but that one should be different by length too. (15) is long enough
2008-05-11T12:38:29  * dreimark will continue after lunch 
2008-05-11T17:43:03  <dreimark> re
2008-05-11T17:49:22  <ThomasWaldmann> wb dreimark :)
2008-05-11T17:49:41  * dreimark tries to do all the wanted changes
2008-05-11T17:50:28  <dreimark> (sometimes I'm to generic with names)
2008-05-11T17:58:21  <ThomasWaldmann> hmm, we have wikiserverconfig.py, so we could also mv moin.py wikiserver.py
2008-05-11T18:03:49  <dreimark> that's a good idea easier to get/explain dependencies too
2008-05-11T18:04:25  * ThomasWaldmann renames it
2008-05-11T18:06:12  <CIA-53> Reimar Bauer <rb.proj AT googlemail DOT com> default * 3575:baa5bfe1b8e1 1.7/MoinMoin/_tests/ (__init__.py test_wikidicts.py): test_wikidicts: refactored tests to more useful names, changed defaults for the string list
2008-05-11T18:20:40  <dreimark> ThomasWaldmann: caching uses os.path.getmtime
2008-05-11T18:20:54  <dreimark> which does not work for eventlog
2008-05-11T18:23:52  <dreimark> http://hg.moinmo.in/moin/1.7/rev/d62fd466d810
2008-05-11T18:35:38  <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3576:08a8965162fe 1.7/ (docs/CHANGES wikiserver.py wikiserverlogging.conf moin.py): rename ./moin.py to wikiserver.py, add wikiserverlogging.conf to configure its logging
2008-05-11T18:35:39  <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3577:6b1b444b45f9 1.7/MoinMoin/_tests/ (__init__.py test_wikidicts.py): merged main
2008-05-11T18:37:42  <ThomasWaldmann> dreimark: caching is not related to eventlog
2008-05-11T18:38:27  <dreimark> I know but the time in eventlog was better as mtime of the file
2008-05-11T18:39:59  <ThomasWaldmann> btw, comments as  # don't use event_log.date()   are pretty useless if you don't tell why
2008-05-11T18:56:05  <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3578:9cb46c423384 1.7/wikiserverlogging.conf: logging: change the stderr handler to log at DEBUG level (we control the logging with the logger's level)
2008-05-11T19:05:07  <ThomasWaldmann> hmm, moin script command misses logging configuration
2008-05-11T19:10:09  <ThomasWaldmann> hmm, maybe we should rather use PYTHONPATH and MOINLOGGING than editing script files
2008-05-11T19:15:00  * dreimark has a solution for the caching problem
2008-05-11T19:15:19  <dreimark> (of course for discussion)
2008-05-11T19:15:32  <dreimark> or not
2008-05-11T19:15:55  <dreimark> sh.t
2008-05-11T19:16:19  <dreimark> (should run tests several times)
2008-05-11T19:43:27  <dreimark> grzywacz: hi, do you have added your changes to the changes file?
2008-05-11T19:43:31  <dreimark> http://moinmo.in/MoinMoinTodo/Release%201.7.0
2008-05-11T19:57:31  <grzywacz> dreimark, jabber notification is mentioned there
2008-05-11T19:57:56  <grzywacz> However, the HelpOnNotification page seems to have disappeared.
2008-05-11T19:58:04  <grzywacz> Or am I targeting a wrong wiki instance?
2008-05-11T19:58:56  <dreimark> http://master17.moinmo.in/HelpOnNotification
2008-05-11T19:58:57  <grzywacz> Ah, master17.
2008-05-11T19:59:22  <grzywacz> dreimark, what other "xmpp work" do you need mentioned in changes?
2008-05-11T19:59:43  <grzywacz> The GHOP tasks were technical and not really visible to end users.
2008-05-11T20:00:07  <ThomasWaldmann> i meant ghop when writing that
2008-05-11T20:00:40  <ThomasWaldmann> but we also have a section for developers :)
2008-05-11T20:01:02  <grzywacz> GHOP was basically 2 tasks: determining installation requirements for the jabber bot running on windows and implementing entity capabilities, which was split into 3 tasks IIRC.
2008-05-11T20:02:29  <ThomasWaldmann> and what do we do with entity capabilities? :)
2008-05-11T20:03:18  <dreimark> "Interactive Jabber bot allows to per..." is not described
2008-05-11T20:03:24  <grzywacz> We store them to avoid resending service discovery requests to every client that gets online.
2008-05-11T20:04:18  <ThomasWaldmann> ok, so it was kind of a fix
2008-05-11T20:04:31  <grzywacz> More like an enhancement to make the bot more robust.
2008-05-11T20:04:32  <ThomasWaldmann> then just edit the todo page and remove those lines
2008-05-11T20:04:48  <ThomasWaldmann> and see also the notification help page todo
2008-05-11T20:04:59  <dreimark> the hints for "Jabber servers" is missing too
2008-05-11T20:06:10  <grzywacz> dreimark, sorry..?
2008-05-11T20:06:31  <grzywacz> dreimark, you mean traffic limits?
2008-05-11T20:07:26  <grzywacz> ThomasWaldmann, http://moinmo.in/RicardoFang/Issue84 do you want this copied to master17 or somehow merged with HelpOnNotification?
2008-05-11T20:09:48  <dreimark> grzywacz: http://hg.moinmo.in/moin/1.7-jabber-knowak/file/f5896c05c7ea/docs/CHANGES.jabber
2008-05-11T20:10:09  <dreimark> I was refering to the entries below New Features there
2008-05-11T20:11:43  <dreimark> ThomasWaldmann: we still have the detailed description mode of CHANGES and will compress it later to a less verbose one
2008-05-11T20:11:48  <dreimark> as we did for 1.6
2008-05-11T20:11:53  <ThomasWaldmann> grzywacz: well, at least it should not stay at that url, but moved to a global page
2008-05-11T20:12:14  <ThomasWaldmann> i guess moinmoin wiki is better than master as usually not many people update win32 stuff
2008-05-11T20:12:55  <dreimark> so everything related to jabber should go into the main CHANGES file
2008-05-11T20:12:58  <grzywacz> First of all I can't edit HelpOnNotification on master17 even though I seem to be in the Editors group.
2008-05-11T20:13:55  <grzywacz> dreimark, CHANGES.jabber ended up in HelpOnNotification
2008-05-11T20:14:01  <grzywacz> dreimark, it's not even merged in 1.7
2008-05-11T20:14:18  <dreimark> ah ok
2008-05-11T20:49:02  <dreimark> ThomasWaldmann: new idea to fix the dict caching trouble
2008-05-11T20:49:12  <dreimark> and it is only a proof of concept
2008-05-11T20:49:16  <dreimark> http://paste.pocoo.org/show/49439/
2008-05-11T20:49:33  <dreimark> btw. I have had trouble to upload this file to MM
2008-05-11T20:52:09  <dreimark> 0 dict failures on my (ram)disk
2008-05-11T20:57:59  * dreimark dinner
2008-05-11T20:58:01  <ThomasWaldmann> dreimark: if you read it anyway, checking for its mtime is pointless
2008-05-11T21:02:20  <gizmach> moin
2008-05-11T21:04:16  <ThomasWaldmann> hi melita
2008-05-11T21:06:49  <gizmach> hi thomas :)
2008-05-11T21:10:01  <dreimark> hi melita
2008-05-11T21:10:10  <dreimark> ThomasWaldmann: scan_end_time: 1210532961.911472. mtime: 1210532961.953721.
2008-05-11T21:11:34  <dreimark> the problem is mtime is slightly wrong or less precision digits of file mtime
2008-05-11T21:15:59  <gizmach> hi Reimar
2008-05-11T21:18:22  <ThomasWaldmann> dreimark: that's ok
2008-05-11T21:19:04  <dreimark> if i do return mtime after reading the whole file it is going wrong too
2008-05-11T21:19:17  <dreimark> (the old mtime)
2008-05-11T21:20:21  <dreimark> I mean it looks like we need the "scan_end_time" entry of the file
2008-05-11T21:21:01  <dreimark> gizmach: how was your birthday party? did you have nice weather too?
2008-05-11T21:21:02  <ThomasWaldmann> we usually do mtime checks to avoid reading a file if it has nothing new
2008-05-11T21:21:06  * dreimark is sunburned
2008-05-11T21:21:22  <ThomasWaldmann> so if you read the file, you don't need mtime any more...
2008-05-11T21:21:25  <gizmach> dreimark: it was great weather and we had a great time
2008-05-11T21:22:00  <gizmach> I'm still trying to face myself that tomorrow is monday and I need to start working
2008-05-11T21:22:10  <ThomasWaldmann> esselle on #moin has package pages problems
2008-05-11T21:22:31  <ThomasWaldmann> gizmach: no free day tomorrow?
2008-05-11T21:22:53  <ThomasWaldmann> or do you mean moin work? :)
2008-05-11T21:22:58  <dreimark> mtime is more often used as only for the wikidicts files
2008-05-11T21:23:07  <gizmach> ThomasWaldmann: no I have college and some laboratories, also I mean moin work :)
2008-05-11T21:23:21  <gizmach> ThomasWaldmann: do you have some holliday or something tomorrow?
2008-05-11T21:23:40  <ThomasWaldmann> dreimark: that makes it even worse if your read the content there
2008-05-11T21:24:14  <dreimark> it is  not sane currently only proof of concept I told before
2008-05-11T21:27:59  <dreimark> do you know which precision file mtime has for the different platforms?
2008-05-11T21:28:03  <dreimark> ThomasWaldmann:
2008-05-11T21:30:50  <ThomasWaldmann> can be as worse as 2s on win32 iirc
2008-05-11T21:33:50  <dreimark> I think for dict pages we need then the  stored "scan_end_time"
2008-05-11T21:34:42  <dreimark> Are there other critical page/ache files
2008-05-11T21:34:48  <dreimark> +c
2008-05-11T21:37:50  <ThomasWaldmann> i first want that a definite bug is recognized, before stuff is "fixed"
2008-05-11T21:38:16  <ThomasWaldmann> or rather the root cause of the bug
2008-05-11T21:55:27  <dreimark> I think it happens at the path of caching.update
2008-05-11T21:55:52  <dreimark> we do create a tmp file do write all the stuff we need into that
2008-05-11T21:55:58  <dreimark> and we close it
2008-05-11T21:56:12  <dreimark> then it is renamed to the right file
2008-05-11T21:56:37  <dreimark> at that time it has the wrong mtime already an older one as the one written into the file
2008-05-11T21:57:22  <dreimark> s/older/newer/
2008-05-11T21:57:26  <dreimark> (bigger)
2008-05-11T22:05:34  <ThomasWaldmann> so it looks fresher than it is
2008-05-11T22:09:20  <dreimark> yeah, currently i try if i can set mtime after  renaming
2008-05-11T22:10:07  <dreimark> but it seems os.utime likes only integers
2008-05-11T22:38:07  <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3579:ff4afe930cb2 1.7/ (MoinMoin/log.py docs/CHANGES wikiserverlogging.conf): logging configuration: you can use MOINLOGGINGCONF env var now to override from where your logging conf is read, fix wikiserverlogging.conf to not create empty moin.log files
2008-05-11T22:42:05  <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3580:fc2f05f2b0bd 1.7/ (9 files in 2 dirs): server adaptor scripts: make sys.path / logging configuration / comments consistent
2008-05-11T22:46:40  <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3581:357cea7b7780 1.7/wiki/server/ (wikiserver.py moin moin.py): remove old moin daemonizing script (replaced by moin server standalone --start/stop), renamed wiki/server/moin.py to wikiserver.py and updated its content from the toplevel clone of it
2008-05-11T22:49:07  <dreimark>  os.stat_float_times(True) should switch to float but do I have to set it everywhere?
2008-05-11T22:50:41  <ThomasWaldmann> if the filesystem mtime granularity is rather big, it doesn't help to use floats
2008-05-11T22:59:24  <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3582:27ba7445b480 1.7/MoinMoin/log.py: logging: fix typos
2008-05-11T23:16:32  <dreimark> ThomasWaldmann: true. ext2 has 1sec that's an explaination for the time.sleep(1) too
2008-05-11T23:17:30  <dreimark> but i think getting that info from the file itself is better
2008-05-11T23:20:42  <dreimark> good night
2008-05-11T23:37:34  <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3583:347dff3b8197 1.7/ (contrib/mailimport/wikimailimport wiki/server/moin): provide moin script command for people not using setup.py, use it for mailimport script
2008-05-11T23:37:37  <ThomasWaldmann> gn dreimark