1 2010-01-07T00:48:34 *** grzywacz
2 2010-01-07T01:41:37 <ronny> ThomasWaldmann: hmm, can consistency stuff like hashes be removed from the rest of the metadata? it creates unnecessary data for some kinds of backend
3 2010-01-07T03:39:02 <ThomasWaldmann> moin needs the hash
4 2010-01-07T03:44:49 <ronny> ThomasWaldmann: but it fscks up storing metadata in a normal vcs history, too many useless diffs
5 2010-01-07T03:54:56 <ronny> and the even worse fact, it kills editability by users
6 2010-01-07T03:55:12 <ronny> there is a whole class if backends one cant do if that data is stored together
7 2010-01-07T03:55:45 <ronny> now the question is, should the backend just rip it out/reinsert it, or should it be a generally different concept
8 2010-01-07T04:26:43 <ThomasWaldmann> (03:44) < ronny> ThomasWaldmann: but it fscks up storing metadata in a normal vcs history, too many useless diffs
9 2010-01-07T04:26:46 <ThomasWaldmann> why?
10 2010-01-07T04:31:34 <ThomasWaldmann> maybe the problem is that you think about trying editing moin items without moin.
11 2010-01-07T04:32:25 <ThomasWaldmann> there is not just that hash moin creates, it also adds some other metadata like editor info
12 2010-01-07T07:38:35 *** xjjk
13 2010-01-07T08:39:52 <johill> :(
14 2010-01-07T08:40:00 <johill> my moin processes seem to be growing only
15 2010-01-07T08:40:08 <johill> they are now 100 mb after less than an hour
16 2010-01-07T08:43:00 <dennda> 1.9?
17 2010-01-07T08:43:10 <johill> yeah
18 2010-01-07T08:43:16 <dennda> http://dev.pocoo.org/projects/flickzeug/
19 2010-01-07T08:43:47 <johill> heh
20 2010-01-07T08:43:50 <johill> "# Memory Lead Finder "
21 2010-01-07T08:44:09 <dennda> yeah, should've a typo finder also
22 2010-01-07T08:44:36 <johill> seems ... empty
23 2010-01-07T08:44:40 <dennda> never tried that though. comes from the werkzeug people and isn't released yet
24 2010-01-07T08:44:48 <dennda> seen some nice screenshots
25 2010-01-07T08:45:30 <johill> it says 'mockups' here ;)
26 2010-01-07T08:45:35 <johill> http://dev.pocoo.org/~mitsuhiko/leakfinder.png
27 2010-01-07T08:46:37 <dennda> too bad :)
28 2010-01-07T08:52:13 <johill> does seem to work actually
29 2010-01-07T08:53:20 <dennda> cool
30 2010-01-07T08:54:16 <johill> http://wireless.kernel.org/_leakfinder/
31 2010-01-07T08:55:03 <dennda> oh
32 2010-01-07T08:55:05 <dennda> nice
33 2010-01-07T08:55:17 <dennda> happy debugging :)
34 2010-01-07T08:55:27 <johill> I'll let it run for a while I guess
35 2010-01-07T08:59:19 <johill> this thing needs a filter for < 10 objects
36 2010-01-07T08:59:22 <johill> some are _thousands_
37 2010-01-07T09:00:09 <dennda> ?
38 2010-01-07T09:01:08 <johill> it's displaying "I have 2 generators"
39 2010-01-07T09:01:19 <johill> like I care, when there are 27000 tuples
40 2010-01-07T09:04:04 <johill> and 15000 dicts
41 2010-01-07T09:04:59 <johill> something's wrong
42 2010-01-07T09:05:06 <johill> none of those dicts should have to stick around for long
43 2010-01-07T09:05:13 <johill> dict of len 5: {'read': True, 'write': False, 'revert': False, 'admin': False, 'delete': False}
44 2010-01-07T09:05:25 <johill> thousands of those, for example
45 2010-01-07T09:06:49 <dennda> some append but no release/delete perhaps?
46 2010-01-07T09:12:34 <johill> hmm
47 2010-01-07T09:13:10 <johill> it seems this thing is buggy
48 2010-01-07T09:13:50 <johill> ah no n/m, it only samples periodicially
49 2010-01-07T09:16:49 <johill> hmm
50 2010-01-07T09:16:56 <johill> some things still never go down, while others do
51 2010-01-07T09:17:07 <dennda> sounds like a bug in moin then, no?
52 2010-01-07T09:17:21 <dennda> i mean, that's out of question anyway :)
53 2010-01-07T09:18:41 <johill> no idea
54 2010-01-07T09:18:54 <xjjk> licensing question... is it ok to use Creative Commons 3.0 content in themes?
55 2010-01-07T09:19:14 <xjjk> moin's modernized uses Tango, which was CC, but I'm not sure whether it's used under CC or public domain
56 2010-01-07T09:19:36 <johill> oh, dumb, I have multiple moin processes
57 2010-01-07T09:25:20 <johill> no wonder it confuses me
58 2010-01-07T09:32:36 <johill> ah. you can add a number at the end
59 2010-01-07T09:34:33 <johill> http://wireless.kernel.org/_leakfinder/1500
60 2010-01-07T09:34:42 <johill> there you can see it created page objects for all pages too :)
61 2010-01-07T09:34:58 <dennda> nice
62 2010-01-07T09:35:08 <dennda> that's not 1.9-storage, is it?
63 2010-01-07T09:35:28 <johill> plain 1.9
64 2010-01-07T09:35:35 <dennda> ok not my fault then :)
65 2010-01-07T09:37:01 <johill> heh
66 2010-01-07T09:49:09 <johill> so
67 2010-01-07T09:49:16 <johill> moin seems to keep a dict with all underlay pages
68 2010-01-07T09:49:33 <johill> containing 'layer_auto', 'layer_underlay' and 'layer_normal' keys
69 2010-01-07T09:49:34 <johill> ?
70 2010-01-07T09:49:46 <johill> with pretty long strings
71 2010-01-07T09:49:55 <johill> with a shitload of pages like I have ....
72 2010-01-07T09:52:51 <dennda> not my area of expertise
73 2010-01-07T09:53:32 <johill> and the acl for each page too
74 2010-01-07T09:54:22 <johill> MoinMoin.Page.ItemCache
75 2010-01-07T09:57:09 <dennda> missing a cache size?
76 2010-01-07T09:57:59 <johill> missing any kind of limitation
77 2010-01-07T09:58:17 <dennda> yeah well that might also be a feature X)
78 2010-01-07T10:01:46 <johill> and the itemcache also opens all pages
79 2010-01-07T10:01:56 <johill> when refreshed
80 2010-01-07T10:02:02 <johill> it reads teh entire editlog
81 2010-01-07T10:02:08 <johill> and ThomasWaldmann claims to know nothing about such behaviour :P
82 2010-01-07T10:07:15 <johill> the page spikes are cute
83 2010-01-07T10:07:16 <johill> http://wireless.kernel.org/_leakfinder/1500
84 2010-01-07T10:07:42 <johill> wonder what that was, but at least they got gc'ed
85 2010-01-07T10:10:05 <johill> yay
86 2010-01-07T10:10:14 <johill> at least now it's limited to like 55M per process
87 2010-01-07T10:10:28 <johill> with the item cache completely disabled
88 2010-01-07T10:10:38 <johill> so far no adverse side effects either ...
89 2010-01-07T11:05:13 <dreimark> moin
90 2010-01-07T11:05:30 <johill> hey
91 2010-01-07T11:09:59 <dreimark> johill: the page isn't accessible
92 2010-01-07T11:10:23 <dreimark> johill: can that reproduced with the standalone server ?
93 2010-01-07T11:10:49 <johill> I removed the debug stuff again
94 2010-01-07T11:10:56 <johill> I don't know how the itemcache is triggered
95 2010-01-07T11:11:10 <johill> if you can trigger an itemcache refresh, then you should be able to reproduce it
96 2010-01-07T11:18:46 <dreimark> ok. i try to look at it
97 2010-01-07T11:19:06 <johill> there's a todo item
98 2010-01-07T11:19:13 <johill> in item cache
99 2010-01-07T11:19:17 <johill> that might help, not sure
100 2010-01-07T12:25:02 <ThomasWaldmann> johill: yoo don't want to disable the item cache completely, otherwise the next issue you'll have is disk access
101 2010-01-07T12:25:35 <johill> I'd rather access disk when needed than swap continuously
102 2010-01-07T12:25:37 <ThomasWaldmann> but you could add some code to do LRU on the last 50 items or so
103 2010-01-07T12:26:51 <ThomasWaldmann> looks like your server has less ram than some mobile phones nowadays :D
104 2010-01-07T12:27:15 <johill> not really, but it's not dedicated to running wikis
105 2010-01-07T12:28:38 <ThomasWaldmann> btw, that itemcache is not new in 1.9, that has been there since ... 1.6(?)
106 2010-01-07T12:28:50 <johill> must be doing something different now though
107 2010-01-07T12:28:55 <johill> at least so it seems
108 2010-01-07T12:50:15 <ThomasWaldmann> gtg/bbl
109 2010-01-07T15:15:18 <ThomasWaldmann> re
110 2010-01-07T15:20:15 <ThomasWaldmann> johill: if you got any insights about how to reproduce strange memory usage / disk access behaviour, please update the wiki page
111 2010-01-07T15:20:49 <johill> I'm guessing itemcache.refresh() will cause that?
112 2010-01-07T15:22:05 <ThomasWaldmann> that just empties the cache
113 2010-01-07T15:22:40 <johill> it also goes through the entire editlog
114 2010-01-07T15:22:47 <johill> since old_pos will be None
115 2010-01-07T15:22:48 <johill> no?
116 2010-01-07T15:22:58 <johill> ah but it just removes things
117 2010-01-07T15:23:29 <johill> hm well, no idea
118 2010-01-07T15:23:39 <johill> and it still seems to create up to 2k page objects anyway
119 2010-01-07T15:23:44 <johill> but at least they get removed again
120 2010-01-07T15:24:29 <ThomasWaldmann> old_pos shouldn't be None in the usual case
121 2010-01-07T15:24:56 <johill> but it starts out as None before the first refresh
122 2010-01-07T15:26:07 <ThomasWaldmann> yeah, maybe someone(tm) should do that TODO :D
123 2010-01-07T15:26:56 <johill> in any case I doubt that's the problem
124 2010-01-07T15:26:59 <johill> I see it accessing all pages
125 2010-01-07T15:27:09 <johill> but the memory consumption seems to come from a lot of junk being in ItemCache
126 2010-01-07T15:27:16 <johill> the accessing all pages appears to be harmless to that
127 2010-01-07T15:28:37 <ThomasWaldmann> "junk" means?
128 2010-01-07T15:30:28 <ThomasWaldmann> it is known that ItemCache grows over time, it doesn't have a limit or LRU strategy
129 2010-01-07T15:30:37 <johill> yeah but I never even accessed those pages
130 2010-01-07T15:30:47 <johill> yet that crawling run, whatever it was, put them all into the cache
131 2010-01-07T15:31:05 <johill> which explains why the cache wasn't a problem for me before
132 2010-01-07T15:31:08 <johill> I didn't see the crawling
133 2010-01-07T15:31:21 <johill> but I can't explain the crawling, yet anyway
134 2010-01-07T15:31:46 <johill> but if it doesn't result in everything being cached, it seems /fairly/ harmless
135 2010-01-07T15:32:08 <ThomasWaldmann> if you can reproduce that crawling somehow, that would be helpful
136 2010-01-07T15:32:38 <johill> it always happens on my server eventually, but I don't know why
137 2010-01-07T15:32:55 <ThomasWaldmann> is it at first request?
138 2010-01-07T15:33:00 <johill> I don't think so
139 2010-01-07T15:33:05 <ThomasWaldmann> or first edit?
140 2010-01-07T15:33:09 <johill> maybe
141 2010-01-07T15:33:38 <johill> but I don't get lots of edits, so even that's unlikely
142 2010-01-07T15:34:56 <ThomasWaldmann> btw, do you think that is reproduceable with ./wikiserver.py?
143 2010-01-07T15:35:05 <johill> I don't know
144 2010-01-07T15:35:10 <johill> I don't see why not
145 2010-01-07T15:35:16 <johill> but I have a fairly complex farm config
146 2010-01-07T15:36:02 <ThomasWaldmann> btw did you check whether your custom stuff maybe could cause it
147 2010-01-07T15:36:11 <ThomasWaldmann> like unusual group usage?
148 2010-01-07T15:36:19 <ThomasWaldmann> theme? other stuff?
149 2010-01-07T15:36:39 <johill> hm patches should be ok
150 2010-01-07T15:36:43 <johill> group usage ..?
151 2010-01-07T15:37:44 <johill> I just do WikiGroup(request, pg, WikiGroups(request)) once a while
152 2010-01-07T15:38:05 <johill> well, for every request, I guess
153 2010-01-07T15:38:15 <johill> but 'pg' is always from a small set of strings
154 2010-01-07T15:57:38 <ThomasWaldmann> i mean the new group/dict code is one of the bigger changes that got into 1.9
155 2010-01-07T15:59:23 <johill> but why would it, occasionally, access all pages?
156 2010-01-07T15:59:32 <ThomasWaldmann> it shouldn't
157 2010-01-07T15:59:40 <ThomasWaldmann> it even was designed to not do that
158 2010-01-07T16:19:39 <ronny> ThomasWaldmann: i decided that it might be best for human-editable backends to strip out the metadata thats costly/inferable for them
159 2010-01-07T16:19:51 <ronny> i.e. stuff like author/hash
160 2010-01-07T16:53:53 <dreimark> we now have request.groups and request.dicts
161 2010-01-07T17:57:43 <dreimark> can we check while processing something if the user still requests it?
162 2010-01-07T17:58:12 <dreimark> e.g. if you initiate a search operation and you click stop at the browser
163 2010-01-07T17:58:39 <dreimark> then the python process continues but won't be able to deliver the result
164 2010-01-07T17:59:41 <dreimark> (instead of stop the browser could be closed too)
165 2010-01-07T18:00:17 <TheSheep> I suppose the browser closes the connection
166 2010-01-07T18:00:33 <dreimark> it doesn't at least not with the standalone server
167 2010-01-07T18:00:33 <TheSheep> I don't know if the web servers communicate that somehow
168 2010-01-07T18:00:47 <TheSheep> it does if you close the browser
169 2010-01-07T18:00:51 <dreimark> no
170 2010-01-07T18:01:13 <TheSheep> then what process has the socket opened?
171 2010-01-07T18:02:01 <dreimark> i tried with a bad regex search and closed the browser after sending it
172 2010-01-07T18:02:14 <dreimark> now it eats 2.3GB and it still increases
173 2010-01-07T18:02:50 <TheSheep> maybe the standalone server doesn't check if the socket is closed from the other side
174 2010-01-07T18:03:00 <johill> web servers in general don't tell you
175 2010-01-07T18:03:04 <johill> and can't even
176 2010-01-07T18:03:23 <TheSheep> johill: I thought that tcp is stateful?
177 2010-01-07T18:03:30 <johill> tcp sure
178 2010-01-07T18:03:41 <johill> but the fcgi, wsgi etc APIs don't provide that kind of option
179 2010-01-07T18:04:04 <johill> besides, the only way it could possibly work was if the app regularly checked the socket status
180 2010-01-07T18:07:10 <TheSheep> not that expensive searches are that useful anyways
181 2010-01-07T18:07:42 <TheSheep> at least regular expressions don't have the stop dillema
182 2010-01-07T18:14:06 <dreimark> there are two points a) if one searches for .* we don't need to search
183 2010-01-07T18:14:11 <dreimark> it is everything
184 2010-01-07T18:14:44 <dreimark> b) that should not consume any memory
185 2010-01-07T18:15:41 <dreimark> and a search with . should not crash your server
186 2010-01-07T18:16:02 <dreimark> (xapian search seems not to have this problem)
187 2010-01-07T18:19:39 <dreimark> gtg/bbl
188 2010-01-07T19:56:22 *** grzywacz
189 2010-01-07T20:16:28 <TheSheep> ronny: hey!
190 2010-01-07T20:16:44 <TheSheep> ronny: I've got an idea that will make you want to kill me on the spot
191 2010-01-07T20:17:33 <TheSheep> ronny: microbugging: twitter meets bugzilla
192 2010-01-07T20:17:53 * TheSheep runs
193 2010-01-07T20:20:51 <ronny> TheSheep: there is no place for you to hide
194 2010-01-07T21:33:37 <ThomasWaldmann> hehe, microbugging
195 2010-01-07T21:33:49 <ThomasWaldmann> isn't that what's IRC for? :)
196 2010-01-07T23:55:22 *** grzywacz
197
