MoinMoin: MoinMoinChat/Logs/moin-dev/2006-08-17

2006-08-17T09:11:03  <ThomasWaldmann> moin
2006-08-17T09:13:28  <ThomasWaldmann> xorAxAx: we have to test some stuff on win32
2006-08-17T09:14:21  <ThomasWaldmann> waldi had a good idea how to avoid much of the locking I/O we currently have
2006-08-17T10:15:25  <ThomasWaldmann> xorAxAx: i pushed a new test to 1.6, please fix and test on windows *.*
2006-08-17T10:23:30  <Kepplar> moin
2006-08-17T10:36:23  <xorAxAx> moin
2006-08-17T10:36:46  <xorAxAx> ThomasWaldmann: let me check ...
2006-08-17T10:44:40  <ThomasWaldmann> soon another push
2006-08-17T10:44:55  <ThomasWaldmann> wait a minute before hacking it
2006-08-17T10:46:26  <ThomasWaldmann> we might have to drop 95/98/ME support (if that stuff still worked)
2006-08-17T10:49:53  <xorAxAx> ThomasWaldmann: umm
2006-08-17T10:50:01  <xorAxAx> ThomasWaldmann: that would require an external module
2006-08-17T10:50:59  <ThomasWaldmann> the pywin32 extension is rather often needed for windows python stuff, isnt it?
2006-08-17T10:51:06  <xorAxAx> no
2006-08-17T10:51:41  <ThomasWaldmann> is there another way calling that api function without needed a C compiler at install time?
2006-08-17T10:51:45  <Kepplar> ThomasWaldmann: think ive nailed this None bug
2006-08-17T10:51:45  <xorAxAx> windows ME is just 5 years old ...
2006-08-17T10:51:58  <xorAxAx> ThomasWaldmann: hehe, by requiring ctypes %-)
2006-08-17T10:52:06  <ThomasWaldmann> yeah, and it sucks since 5 years :)
2006-08-17T10:52:07  <Kepplar> ThomasWaldmann: btw, I've cleared this weekend for a 24x7 codeathon =)
2006-08-17T10:52:54  <ThomasWaldmann> Kepplar: please code when being awake |-)
2006-08-17T10:53:14  <Kepplar> ThomasWaldmann: IrnBru32 + Redball + Mocha's == awake =D
2006-08-17T10:53:26  <xorAxAx> Kepplar: i sense severe miscalculation of the days with regards to the "weekend", mine has just 2
2006-08-17T10:53:26  <Kepplar> bull*
2006-08-17T10:53:50  <Kepplar> xorAxAx: well yes
2006-08-17T10:53:53  <ThomasWaldmann> i guess he added that 5 days before :)
2006-08-17T10:54:00  <Kepplar> I'm working up until then but I can't 24x7 it atm
2006-08-17T10:54:15  <xorAxAx> ThomasWaldmann: but another idea would be to abstract this locking-less code and the lock.py into a highlevel interface
2006-08-17T10:54:25  <xorAxAx> ThomasWaldmann: and, furthermore, often locking is needed
2006-08-17T10:54:39  <xorAxAx> ThomasWaldmann: have you seen my metadict changes for example? you would still need locking there
2006-08-17T10:54:56  <xorAxAx> (so a highlevel interface would ensure data integrity on both platforms)
2006-08-17T10:55:04  <xorAxAx> by either using locks or direct renames
2006-08-17T10:55:29  * ThomasWaldmann pushed again
2006-08-17T10:56:46  <ThomasWaldmann> i guess I rather take performance improvements on current platforms than 95/98/ME compat or 1 dependency less on win32
2006-08-17T10:56:57  <xorAxAx> ThomasWaldmann: btw, if i understand MSDN correctly, that MOVEFILE_REPLACE_EXISTING flag does not work if the file is still open
2006-08-17T10:56:58  <ThomasWaldmann> see I/O load here
2006-08-17T10:57:07  <xorAxAx> which is the case that should be avoided
2006-08-17T10:57:32  <xorAxAx> i think that moin is used very often on 9x
2006-08-17T10:57:35  <xorAxAx> because of MMDE
2006-08-17T10:57:35  <ThomasWaldmann> you can just run that test
2006-08-17T10:57:44  <xorAxAx> yeah
2006-08-17T10:58:16  <xorAxAx> and i dont see any problem to disable locking on the posix platform as long as its used on windows
2006-08-17T10:58:31  <xorAxAx> (in those cases where its just to ensure integrity in atomic writes)
2006-08-17T10:59:34  <ThomasWaldmann> it's not only about locking, that stuff depends on doing data updates different
2006-08-17T10:59:47  <xorAxAx> ?
2006-08-17T10:59:50  <ThomasWaldmann> (by rename)
2006-08-17T10:59:54  <xorAxAx> yep, test fails
2006-08-17T11:00:17  <xorAxAx> ThomasWaldmann: it is possible to abstract this on a higher level and allow compatiblity on both platforms this way
2006-08-17T11:00:33  <ThomasWaldmann> add it at bottom of ReducingIO page
2006-08-17T11:00:35  <xorAxAx> where do you need this rename, just in caching, right?
2006-08-17T11:00:43  <xorAxAx> add what? :)
2006-08-17T11:00:54  <ThomasWaldmann> how you suggest doing it
2006-08-17T11:01:10  <xorAxAx> first we need to talk about the problem
2006-08-17T11:01:16  <xorAxAx> what do you want to solve by a rename? :)
2006-08-17T11:01:19  <ThomasWaldmann> it is just the difference between file name (dir entry) and file content (inode)
2006-08-17T11:01:31  <xorAxAx> i currently see caching.py as the only place where it could be used
2006-08-17T11:02:45  <ThomasWaldmann> well, at many places we read and lock - have to look at them still
2006-08-17T11:03:24  <xorAxAx> i doubt that we have such many places :)
2006-08-17T11:03:29  <ThomasWaldmann> and the places we do the corresponding writes have to be checked for possibility of rename stuff
2006-08-17T11:03:47  <ThomasWaldmann> what error did you get on windows?
2006-08-17T11:04:02  <xorAxAx> IOerror on a remove call
2006-08-17T11:04:20  <ThomasWaldmann> remove?
2006-08-17T11:04:34  <xorAxAx> currently, just metadict and the search lock
2006-08-17T11:04:43  <xorAxAx> and my tagstore
2006-08-17T11:04:54  <xorAxAx> (besides caching)
2006-08-17T11:04:57  <ThomasWaldmann> I am talking about the "new locking" test
2006-08-17T11:05:01  <xorAxAx> me too!
2006-08-17T11:05:21  <xorAxAx>     shutil.rmtree(self.test_dir)
2006-08-17T11:05:21  <xorAxAx>   File "C:\Programme\Python24\lib\shutil.py", line 168, in rmtree
2006-08-17T11:05:21  <xorAxAx>     onerror(os.remove, fullname, sys.exc_info())
2006-08-17T11:05:21  <xorAxAx>   File "C:\Programme\Python24\lib\shutil.py", line 166, in rmtree
2006-08-17T11:05:21  <xorAxAx>     os.remove(fullname)
2006-08-17T11:05:24  <xorAxAx> lock_gmebvq\\readtest'
2006-08-17T11:05:26  <xorAxAx>  OSError: [Errno 13] Permission denied: 'c:\\dokume~1\\admini~1\\lokale~1\\temp\\
2006-08-17T11:05:41  <ThomasWaldmann> this is test cleanup
2006-08-17T11:05:44  <xorAxAx> and, earlier:
2006-08-17T11:05:52  <xorAxAx>     ret = MoveFileEx(oldname, newname, MOVEFILE_REPLACE_EXISTING)
2006-08-17T11:05:54  <xorAxAx>  error: (5, 'MoveFileEx', 'Zugriff verweigert')
2006-08-17T11:05:57  <xorAxAx> i.e. still open
2006-08-17T11:06:28  <xorAxAx> on a sidenote, it is possible to delete the file first using win32file
2006-08-17T11:06:59  <xorAxAx> so, what do you think about rewriting caching so that it doesnt use locking on posix?
2006-08-17T11:07:21  <ThomasWaldmann> yeah, I will do that
2006-08-17T11:07:48  <xorAxAx> i think windows admin can cope with slow systems :)
2006-08-17T11:07:52  <ThomasWaldmann> can you try to find out how to do a atomic rename like posix does on win32?
2006-08-17T11:08:03  <xorAxAx> i doubt that it works
2006-08-17T11:08:18  <xorAxAx> mercurial deletes and moves then
2006-08-17T11:08:58  * xorAxAx checks MSDN
2006-08-17T11:09:58  <ThomasWaldmann> or have some code around it making it kind of uninterruptable?
2006-08-17T11:10:31  <xorAxAx> heh, then we would need a global mutex %-)
2006-08-17T11:10:36  <xorAxAx> for all moin processes
2006-08-17T11:10:46  <ThomasWaldmann> (i guess we could do that for other threads of python process, but not for all processes)
2006-08-17T11:12:19  <xorAxAx> oh, would be possible
2006-08-17T11:12:25  <xorAxAx> using some win32 function
2006-08-17T11:12:38  <xorAxAx> but i dislike such platform specific code :)
2006-08-17T11:12:49  <ThomasWaldmann> btw, sync tests still failing
2006-08-17T11:13:03  <xorAxAx> hmm, then i need to push my branch
2006-08-17T11:13:09  <xorAxAx> because its working there on py 2.3
2006-08-17T11:13:11  <ThomasWaldmann> if would be within a platform specific section anyway
2006-08-17T11:13:26  <xorAxAx> i always like some fallback
2006-08-17T11:13:35  <xorAxAx> because moin would not run on my pocket pc otherwise :)
2006-08-17T11:17:13  <ThomasWaldmann> we can use del/ren for windows compatible behaviour (works sometimes, maybe often :)
2006-08-17T11:17:15  <xorAxAx> hmm, it might be possible to rename the file if the target file was opened using FILE_SHARE_DELETE
2006-08-17T11:17:23  <xorAxAx> ThomasWaldmann: locking is ok
2006-08-17T11:18:10  <ThomasWaldmann> that isnt supported by file(), so we need special code there, too?
2006-08-17T11:18:18  <xorAxAx> but that would need some 50 lines of compatiblity code that ensures that all related open calls go through the wrappers
2006-08-17T11:18:22  <xorAxAx> yes
2006-08-17T11:19:15  <ThomasWaldmann> maybe you could try making the test work by that method
2006-08-17T11:19:44  <ThomasWaldmann> I'll check caching.py stuff later
2006-08-17T11:20:32  <ThomasWaldmann> (in the end, I need stuff for production wikis, so either a rather small change for 1.5 or migrating them to 1.6)
2006-08-17T11:21:25  <xorAxAx> customers say that their wiki is too slow?
2006-08-17T11:21:38  <xorAxAx> then sell them new hardware :)
2006-08-17T11:22:40  <ThomasWaldmann> I am talking about thinkmo.de hosted stuff, and that definitely is too slow.
2006-08-17T11:23:03  <xorAxAx> yep, but thinkmo is not running windows
2006-08-17T11:23:14  <ThomasWaldmann> i guess it usually is not a problem for a "1 wiki on a machine" installation.
2006-08-17T11:23:36  <xorAxAx> yes, locking is not a problem speedwise
2006-08-17T11:23:44  <xorAxAx> in that scenario
2006-08-17T11:28:28  <ThomasWaldmann> caching's copyto is used nowhere
2006-08-17T11:47:58  <ThomasWaldmann> xorAxAx: i guess it is also not possible to remove a opened file on win32?
2006-08-17T11:48:22  <ThomasWaldmann> caching -> .remove
2006-08-17T11:48:23  <xorAxAx> ThomasWaldmann: if it was opened using FILE_SHARE_DELETE, it is possible on windows nt
2006-08-17T11:49:38  <ThomasWaldmann>     def remove(self):
2006-08-17T11:49:38  <ThomasWaldmann>         # TODO add wlock.acquire or it might fail on win32
2006-08-17T11:50:13  <ThomasWaldmann> (until we maybe have that special open() stuff)
2006-08-17T11:50:14  <xorAxAx> umm
2006-08-17T11:50:34  <xorAxAx> wasnt there any locking before?
2006-08-17T11:51:00  <xorAxAx> hmm, no, indeed
2006-08-17T11:51:12  * xorAxAx wonders where remove is used at all :)
2006-08-17T11:51:18  <xorAxAx> i guess i am using it
2006-08-17T11:51:52  <ThomasWaldmann> I add that call
2006-08-17T12:13:04  * ThomasWaldmann pushed rename-style caching update code (still uses locking as before)
2006-08-17T12:36:41  <ThomasWaldmann> I have made a LazyReadLock(ReadLock) class doing nothing on != win32
2006-08-17T13:04:04  * ThomasWaldmann pushed
2006-08-17T13:05:06  <ThomasWaldmann> caching.py on test.wikiwikiweb.de uses that code now
2006-08-17T16:02:05  * ThomasWaldmann is glad we have noone in the US for SOC, so the form demanded by google was short and easy.
2006-08-17T16:04:07  <xorAxAx> yeah
2006-08-17T16:07:57  * ThomasWaldmann is also glad that thinkmo.de is not in the US, as they include "equipment in the US".
2006-08-17T16:13:38  <xorAxAx> so if people get an IP route via the US, the IRS will arrest them
2006-08-17T16:14:30  <ThomasWaldmann> yeah, better never visit the US :)
2006-08-17T17:42:07  <ThomasWaldmann> Kepplar: your last commit was almost 2 days ago
2006-08-17T17:42:54  <ThomasWaldmann> fpletz: not much better
2006-08-17T17:45:47  <ThomasWaldmann> mvirkkil: 4 days ago
2006-08-17T18:00:59  <Kepplar> two days O_o
2006-08-17T18:01:05  <Kepplar> did i forget to push
2006-08-17T18:01:25  <Kepplar> mm
2006-08-17T18:14:56  * ThomasWaldmann .oO(bah, outside someone is training with a model racing car at 10.000rpm+ 8-* )
2006-08-17T18:39:11  <ThomasWaldmann> http://upload.wikimedia.org/wikipedia/wikimania2006/4/47/BV1_-_MediaWiki_Goodies.pdf
2006-08-17T18:43:11  <xorAxAx> tagging sounds nice
2006-08-17T19:45:57  <xorAxAx> ..\MoinMoin\caching.py:53: RuntimeWarning: tempnam is a potential security risk
2006-08-17T19:46:01  <xorAxAx> to your program return os.tempnam(self.arena_dir, self.key)
2006-08-17T19:46:01  <xorAxAx> ThomasWaldmann:
2006-08-17T19:47:40  <ThomasWaldmann> you know a better one? the hint in the docs is not useful.
2006-08-17T19:48:11  <xorAxAx> it says that tmpfile is better
2006-08-17T19:48:25  <ThomasWaldmann> yeah, because it's useless for that case :)
2006-08-17T19:48:26  <xorAxAx> but you cannot move tmpfiles :)
2006-08-17T19:48:35  <ThomasWaldmann> useless == secure :)
2006-08-17T19:48:58  <xorAxAx> hmm, tmpnam is not suitable for sensible moves on posix either
2006-08-17T19:49:08  <xorAxAx> because /tmp is seldomly on the same volume
2006-08-17T19:49:29  <xorAxAx> so some random filename in the same directory is good enough
2006-08-17T19:49:55  <ThomasWaldmann> >>> os.tempnam("/home/tw", "aaa")
2006-08-17T19:49:59  <ThomasWaldmann> '/home/tw/aaaNc9iC1'
2006-08-17T19:50:27  <xorAxAx> oh, there are two entries
2006-08-17T19:50:37  <xorAxAx> hmm
2006-08-17T19:50:49  <xorAxAx> umm, symlink attacks ...
2006-08-17T19:51:03  <xorAxAx> i guess not really possible if you dont have write access
2006-08-17T19:51:10  <xorAxAx> so moin needs to mute that warning
2006-08-17T19:51:14  * xorAxAx adds code for that
2006-08-17T19:59:45  <xorAxAx> ok, pushed
2006-08-17T20:38:19  <ThomasWaldmann> did you see any problems in the caching/locking code (except that now muted warning)?
2006-08-17T20:39:03  <ThomasWaldmann> xorAxAx: did you try it on win32?
2006-08-17T20:39:42  <xorAxAx> it seems to work, yes
2006-08-17T20:39:53  <xorAxAx> i didnt review it
2006-08-17T20:40:14  <Kepplar> hmm
2006-08-17T20:43:41  <ThomasWaldmann> ok, I make a branch to backport it to 1.5
2006-08-17T20:44:22  <xorAxAx> a branch?
2006-08-17T20:52:50  <ThomasWaldmann> a clone
2006-08-17T21:06:56  * xorAxAx added the GetText2 macro
2006-08-17T21:07:30  <xorAxAx> [[GetText2(|%s is a green house|Palast)]] renders as Palast ist ein grünes Haus
2006-08-17T21:20:19  <xorAxAx> ThomasWaldmann: ldap is not documented
2006-08-17T21:20:24  <xorAxAx> on HelpOnAuthentication
2006-08-17T21:40:59  <Kepplar> AHHHHHHHHHHHHH
2006-08-17T21:41:07  <Kepplar> NOW ive nailed the bug
2006-08-17T21:41:44  <Kepplar> took me long enough i had to read though god knows how manymodules with paper and manually tracing it!
2006-08-17T21:41:47  <Kepplar> took hours!
2006-08-17T22:11:11  <Kepplar> Dudes
2006-08-17T22:11:20  <Kepplar> this is seriously one strange bug
2006-08-17T22:11:43  <Kepplar> Basically its fetching from data not underlay
2006-08-17T22:11:54  <Kepplar> dispite accessing the underlay engine
2006-08-17T22:27:31  <xorAxAx> have you written a test case?
2006-08-17T22:32:26  <Kepplar> its not that
2006-08-17T22:32:32  <Kepplar> its integration
2006-08-17T22:32:40  <xorAxAx> a test case for the bug ...
2006-08-17T22:32:53  <Kepplar> its not storage code i mean :)
2006-08-17T22:32:59  <xorAxAx> so you can track it down
2006-08-17T22:33:09  <xorAxAx> nothing else is related to underlay
2006-08-17T22:33:14  <Kepplar> im there mostly
2006-08-17T22:34:21  <xorAxAx> can you try to merge the branches in your repo?
2006-08-17T22:37:19  <Kepplar> i have i think
2006-08-17T22:37:20  <Kepplar> not pushed
2006-08-17T22:37:31  <xorAxAx> and you still havent set a correct username :-/
2006-08-17T22:38:07  <Kepplar> ill try and do it, but when i have my mind on a problem i have to solve it first :P
2006-08-17T22:39:17  <xorAxAx> yeah, you seem to have had your mind on problems for 3 months
2006-08-17T22:39:27  <Kepplar> well no i did change my name
2006-08-17T22:39:35  <Kepplar> its only been in the last 4 weeks its reverted
2006-08-17T22:39:46  <Kepplar> i don't deem it important
2006-08-17T22:39:51  <Kepplar> sorry
2006-08-17T22:42:18  <xorAxAx> yeah, one month
2006-08-17T23:03:37  <Kepplar> i think borg could be broken
2006-08-17T23:05:08  <Kepplar> AHA
2006-08-17T23:07:59  <Kepplar> which is nicer
2006-08-17T23:08:11  <Kepplar> trying an attribute and catching attributeerror
2006-08-17T23:08:12  <Kepplar> or
2006-08-17T23:08:22  <Kepplar> has_key from __dict__
2006-08-17T23:08:23  <Kepplar> ?
2006-08-17T23:08:57  <xorAxAx> getattr
2006-08-17T23:11:14  <Kepplar> does it return false if itdoesnt exist?
2006-08-17T23:11:58  <xorAxAx> help(getattr)