1 2011-06-16T00:24:35 *** RogerHaase
2 2011-06-16T00:31:18 *** sinha
3 2011-06-16T00:50:59 * ThomasWaldmann made a new doodle, with weekdays and international tz support
4 2011-06-16T00:56:58 *** brunomartin
5 2011-06-16T01:21:24 *** m4k3r
6 2011-06-16T01:36:42 *** moinBot
7 2011-06-16T01:55:04 *** moinBot`
8 2011-06-16T02:38:15 *** MattMaker
9 2011-06-16T06:23:55 *** Marchael
10 2011-06-16T07:28:25 *** m4k3r
11 2011-06-16T08:17:01 *** gwork
12 2011-06-16T08:19:10 *** MattMaker
13 2011-06-16T08:21:55 *** MattMaker
14 2011-06-16T08:50:14 <dreimark> moin
15 2011-06-16T08:58:12 <ThomasWaldmann> moin
16 2011-06-16T09:01:44 *** hardy89
17 2011-06-16T09:35:43 *** hardy89
18 2011-06-16T09:52:42 *** pkumar
19 2011-06-16T09:56:23 *** pkumar
20 2011-06-16T10:45:01 *** ronny
21 2011-06-16T10:57:06 *** greg_f
22 2011-06-16T10:58:11 <ThomasWaldmann> Marchael: you're having progress?
23 2011-06-16T10:59:50 *** pkumar
24 2011-06-16T11:00:40 <Marchael> ThomasWaldmann: why I can't write code in http://paste.pocoo.org/show/408711/ ?
25 2011-06-16T11:00:44 <Marchael> 13-17
26 2011-06-16T11:01:14 <Marchael> AttributeError: 'NoneType' object has no attribute 'app'
27 2011-06-16T11:01:18 <Marchael> again
28 2011-06-16T11:03:05 <ThomasWaldmann> those lines look rather inconsistent regarding usage of "self"
29 2011-06-16T11:03:19 <ThomasWaldmann> and you do not use app there
30 2011-06-16T11:04:48 <Marchael> I use app in run()
31 2011-06-16T11:05:05 <Marchael> but if I put it outside the run() it doesn't make sense
32 2011-06-16T11:05:30 <Marchael> in general I want to share some values between 3 functions
33 2011-06-16T11:06:16 <ThomasWaldmann> ehrm, the code you put directly into the class is run at the time when the class is evaluated, which is at import time
34 2011-06-16T11:07:24 <Marchael> so that's why I can't access to app at this time
35 2011-06-16T11:07:50 <ThomasWaldmann> i can't comment one code I don't see
36 2011-06-16T11:08:18 <Marchael> http://paste.pocoo.org/show/408715/
37 2011-06-16T11:08:50 <Marchael> traceback http://paste.pocoo.org/show/408717/
38 2011-06-16T11:11:32 <ThomasWaldmann> it looks rather simple to split the code into those 2 smaller methods. and then call both or just one of them from run()
39 2011-06-16T11:11:58 <Marchael> ye
40 2011-06-16T11:12:17 <Marchael> I tried to do so, but also tried to avoid duplication
41 2011-06-16T11:12:26 <Marchael> and then I put this code outside
42 2011-06-16T11:12:34 <Marchael> I mean 13..17
43 2011-06-16T11:13:08 <ThomasWaldmann> well, that obviously doesn't work. but there won't be much duplication if you do it right.
44 2011-06-16T11:14:32 <ThomasWaldmann> you of course will have 2 copies of the loops to iterate over all items
45 2011-06-16T11:15:18 <ThomasWaldmann> and you will read the latest rev twice from the backend (once for indexing all revs, once for indexing latest rev)
46 2011-06-16T11:15:31 <ThomasWaldmann> but that's not an issue
47 2011-06-16T11:17:29 <ThomasWaldmann> http://paste.pocoo.org/show/408715/ copy 37..51 to build_all and build_latest and remove it from run
48 2011-06-16T11:17:52 <ThomasWaldmann> then remove all you don't need from build_all, similar for build_latest
49 2011-06-16T11:18:26 <ThomasWaldmann> move stuff from 29..33 to build_all and build_latest, but only what you really need
50 2011-06-16T11:18:57 <ThomasWaldmann> use run() to decide what methods to call, depending on commandline options
51 2011-06-16T11:30:14 <Marchael> you reading my mind :)
52 2011-06-16T11:58:27 <eSyr> Marchael: self is an argument (which points to class instance) which is passed to class methods implicitly (as first argument), so you have no self in class outside methods. You can use super(RebuildIndexes) to get access to super attributes or so, i can't remember exactly -_-
53 2011-06-16T12:00:26 <ThomasWaldmann> there's not much there for super() yet
54 2011-06-16T12:01:56 <eSyr> there is some solution to get super of current class, i don't like to do it via __mro__ -_-
55 2011-06-16T12:02:27 <ThomasWaldmann> eSyr: we better keep it simple :)
56 2011-06-16T12:03:37 <eSyr> yes, i wonder why it can't be inside __init__() :)
57 2011-06-16T12:10:45 <pkumar> ThomasWaldmann : hi, as you had said yesterday, the tests will only fail if it goes over the resource limit.
58 2011-06-16T12:11:02 <pkumar> ThomasWaldmann : What exactly did you mean with resource limit?
59 2011-06-16T12:11:13 <pkumar> ThomasWaldmann : can you please explain?
60 2011-06-16T12:12:08 <ThomasWaldmann> if the process uses more open files then given to it by the OS
61 2011-06-16T12:12:13 <ThomasWaldmann> than*
62 2011-06-16T12:12:32 <ThomasWaldmann> iirc on linux this usually is 1024
63 2011-06-16T12:13:03 <ThomasWaldmann> but it can be changed and also it might be different on other OSes
64 2011-06-16T12:14:16 <ThomasWaldmann> that's why i suggested watching lsof -p PID. if you see a rising use of open files (>50), there is very likely something wrong.
65 2011-06-16T12:14:35 <pkumar> lsof will show the files being used by the process right?
66 2011-06-16T12:15:38 <pkumar> or will it show the files given by os?
67 2011-06-16T12:17:40 <pkumar> for me it is currently showing 35-40 files. though some of them are deleted(as they are intermediate temp files)
68 2011-06-16T12:21:39 <ThomasWaldmann> LiSt of Open Files
69 2011-06-16T12:22:19 <pkumar> ok
70 2011-06-16T12:23:03 <pkumar> So I think I need to keep checking in between while the test is running if the list grows too long
71 2011-06-16T12:23:11 <pkumar> ?
72 2011-06-16T12:33:06 <ThomasWaldmann> that's what I said :)
73 2011-06-16T12:36:34 <pkumar> ok
74 2011-06-16T12:36:38 <pkumar> :)
75 2011-06-16T13:19:53 <Marchael> ThomasWaldmann: please, review http://codereview.appspot.com/4539114/
76 2011-06-16T13:39:12 <ThomasWaldmann> Marchael: review done
77 2011-06-16T13:39:44 <Marchael> thx
78 2011-06-16T13:41:47 *** pkumar
79 2011-06-16T13:44:27 *** ronny
80 2011-06-16T13:58:44 *** andialbrecht_
81 2011-06-16T14:02:27 *** andialbrecht
82 2011-06-16T14:02:32 *** andi_
83 2011-06-16T14:02:37 *** andi
84 2011-06-16T14:03:19 *** andialbrecht_
85 2011-06-16T14:07:21 *** andialbrecht
86 2011-06-16T14:07:21 *** andialbrecht
87 2011-06-16T14:15:58 <Marchael> ThomasWaldmann: self.procs = procs or cpu_count()
88 2011-06-16T14:16:21 <Marchael> cpu_count() from multiprocessing
89 2011-06-16T14:17:52 <Marchael> procs = None by default
90 2011-06-16T14:17:55 *** andialbrecht
91 2011-06-16T14:18:01 <Marchael> so I think I can just leave it
92 2011-06-16T14:51:11 *** m4k3r
93 2011-06-16T14:52:50 *** m4k3r
94 2011-06-16T14:54:47 <Marchael> ThomasWaldmann: please, review http://codereview.appspot.com/4539114
95 2011-06-16T14:55:26 <Marchael> I'm go back in hostel for dinner and get back to university, it's take about 1-2 hours
96 2011-06-16T14:55:40 * Marchael bbl
97 2011-06-16T15:09:51 <ThomasWaldmann> Marchael: you please keep that procs option as it was
98 2011-06-16T15:10:11 <ThomasWaldmann> if you have 8 cores in your machine, noone says that you want use all just for indexing
99 2011-06-16T15:34:12 *** pkumar
100 2011-06-16T15:36:51 <eSyr> ThomasWaldmann: why not?
101 2011-06-16T15:41:00 <ThomasWaldmann> because something else might also want some cpu?
102 2011-06-16T15:41:35 <ThomasWaldmann> one may want to allocate all cores, if the machine is otherwise idle, but there must be also the option to not catch them all
103 2011-06-16T15:42:41 <waldi> it is always up to the user to decide that
104 2011-06-16T15:48:39 <eSyr> ThomasWaldmann: all cores + renice?
105 2011-06-16T15:48:53 <eSyr> (and ionice)
106 2011-06-16T15:55:53 <dreimark> http://www.ohloh.net/p/moinmoin
107 2011-06-16T15:55:58 <dreimark> style has changed
108 2011-06-16T15:56:55 <dreimark> and it analyses much better
109 2011-06-16T15:59:50 <waldi> eSyr: normal nice level is okay. ionice is coupled and you don't want to use the idle priority
110 2011-06-16T16:08:19 * Marchael back
111 2011-06-16T16:09:14 <Marchael> ThomasWaldmann: you ask me to use default whoosh value and I just done it :)
112 2011-06-16T16:11:11 <ThomasWaldmann> Marchael: i understood it like you want to remove that procs option, if you don't want that, fine.
113 2011-06-16T16:11:25 <ThomasWaldmann> Marchael: review done
114 2011-06-16T16:11:32 * Marchael looks
115 2011-06-16T16:11:47 <Marchael> no, I don't want to remove it
116 2011-06-16T16:12:01 <Marchael> bit it seems like whoosh has default value for it
117 2011-06-16T16:12:10 <Marchael> then why just don't use their?
118 2011-06-16T16:12:28 <ThomasWaldmann> if that None works, it is fine
119 2011-06-16T16:12:39 <ThomasWaldmann> but better check that, None is not an int
120 2011-06-16T16:12:42 <Marchael> yes, it works. at least in 2.7.1 L)
121 2011-06-16T16:12:43 <Marchael> :)
122 2011-06-16T16:12:57 <ThomasWaldmann> and I am not sure if it wants only int type, even for the default
123 2011-06-16T16:13:36 <ThomasWaldmann> in know your code doesn't work on 2.6, you don't need to point that out
124 2011-06-16T16:16:15 <Marchael> ok, I try to test in under 2.6
125 2011-06-16T16:18:37 <Marchael> ThomasWaldmann: how can I change default interpreter?
126 2011-06-16T16:18:43 <Marchael> by default it 2.7
127 2011-06-16T16:19:29 *** pkumar
128 2011-06-16T16:19:38 <Marchael> just replace python to env/bin ?
129 2011-06-16T16:19:44 <Marchael> s/to/in/
130 2011-06-16T16:21:29 <ThomasWaldmann> python2.6 moin ...
131 2011-06-16T16:21:43 <ThomasWaldmann> usually you do not want to change the system default
132 2011-06-16T16:22:07 <ThomasWaldmann> because some scripts on that system might rely on it being 2.7, esp. stuff from linux distribution
133 2011-06-16T16:22:37 <Marchael> but if I want to use stuff like py.test?
134 2011-06-16T16:22:46 <ThomasWaldmann> (and this is especially true for py2 vs py3)
135 2011-06-16T16:22:58 <ThomasWaldmann> there is an option
136 2011-06-16T16:23:33 <ThomasWaldmann> also, if you work in a virtualenv (as you do, hopefully), you have that python interpreter there which you used for creating that virtual env
137 2011-06-16T16:23:57 <ThomasWaldmann> see virtualenv options
138 2011-06-16T16:24:03 <Marchael> ok
139 2011-06-16T16:24:38 * ThomasWaldmann did that even for pypy :)
140 2011-06-16T16:27:53 * Marchael compiling 2.6
141 2011-06-16T16:33:03 <Marchael> ThomasWaldmann: where I can find virtualenv options?
142 2011-06-16T16:33:44 *** brunomartin
143 2011-06-16T16:35:33 <Marchael> I don't see any options file in env/bin/activate
144 2011-06-16T16:43:51 *** m4k3r
145 2011-06-16T16:44:03 *** m4k3r
146 2011-06-16T16:50:47 *** m4k3r
147 2011-06-16T16:51:24 *** m4k3r
148 2011-06-16T16:59:39 <ThomasWaldmann> Marchael: virtualenv --help
149 2011-06-16T17:00:03 <ThomasWaldmann> (as usual)
150 2011-06-16T17:01:14 <Marchael> AssertionError: Filename /home/marchael/Downloads/python2.6/Python-2.6/Lib/os.py does not start with any of these prefixes: ['/usr/local']
151 2011-06-16T17:01:27 <Marchael> I don't want to make install
152 2011-06-16T17:02:54 <ThomasWaldmann> make altinstall
153 2011-06-16T17:03:56 <ThomasWaldmann> (if you put it into /usr/local and you do not install a link from "python" to it, that's rather harmless and easily removed again later)
154 2011-06-16T17:05:02 <Marchael> ok
155 2011-06-16T17:21:06 <Marchael> installed
156 2011-06-16T17:24:14 <Marchael> I rename python2.6 to python in env/bin but it doesn't make sense
157 2011-06-16T17:26:09 <Marchael> oh, seems like I'm need run quickinstall again fr 2.6
158 2011-06-16T17:26:17 <ThomasWaldmann> yes
159 2011-06-16T17:27:44 <Marchael> and it'll use 2.6?
160 2011-06-16T17:27:58 <Marchael> if python = python2 = python2.6 ?
161 2011-06-16T17:29:41 *** dreimark2
162 2011-06-16T17:30:07 <ThomasWaldmann> Marchael: in your virtualenv, you can have that as "python", but don't do it system-wide
163 2011-06-16T17:31:45 <Marchael> $ python
164 2011-06-16T17:31:45 <Marchael> Python 2.6 (r26:66714, Jun 16 2011, 20:26:55)
165 2011-06-16T17:31:53 <Marchael> I think it's ok :)
166 2011-06-16T17:36:51 <Marchael> metadata = {key: value for key, value in revision.items() if key in all_rev_field_names}
167 2011-06-16T17:36:51 <Marchael> ^
168 2011-06-16T17:36:51 <Marchael> SyntaxError: invalid syntax
169 2011-06-16T17:36:56 <Marchael> o_0
170 2011-06-16T17:40:26 <ThomasWaldmann> told you so :P
171 2011-06-16T17:41:26 <Marchael> ThomasWaldmann: but you recomends me to use this code instead of that 2 fors :)
172 2011-06-16T17:42:29 <Marchael> should i back to old code?
173 2011-06-16T17:44:21 <ThomasWaldmann> i told you twice that this does not work with 2.6
174 2011-06-16T17:44:38 <ThomasWaldmann> but you can use this for 2.6:
175 2011-06-16T17:45:16 <ThomasWaldmann> ... = dict([(key, value) for ... if ...])
176 2011-06-16T17:45:59 <Marchael> ok
177 2011-06-16T17:46:40 <ThomasWaldmann> 2.7 way is of course prettier, but as long as we don't have 2.7 as minimum requirement, we must use 2.6
178 2011-06-16T17:57:27 <ThomasWaldmann> btw, such stuff is especially a pain if it goes unnoticed and a moin user would notice it after release that it does not work with 2.6
179 2011-06-16T17:58:15 <ThomasWaldmann> we sometimes had that with 2.4 in the past, that's why I am so strict about it
180 2011-06-16T17:58:24 <Marchael> seems like Matt still doesn't add your patch to release...
181 2011-06-16T18:00:44 <Marchael> seems like whoosh has troubles with 2.6
182 2011-06-16T18:00:59 <Marchael> http://paste.pocoo.org/show/408890/
183 2011-06-16T18:04:55 <ThomasWaldmann> Marchael: matt didn't release yet since i submitted that patch, but it is committed to repo, so it will be in next release
184 2011-06-16T18:05:55 <ThomasWaldmann> that doesn't look like a py2.6 issue
185 2011-06-16T18:08:19 <Marchael> but what is it?
186 2011-06-16T18:08:30 <Marchael> that stuff working with 2.7
187 2011-06-16T18:12:10 <Marchael> may be contact with whoosh author
188 2011-06-16T18:13:23 *** grzywacz
189 2011-06-16T18:13:23 *** grzywacz
190 2011-06-16T18:14:08 <ThomasWaldmann> Marchael: there must be at least 2 issues
191 2011-06-16T18:14:24 <ThomasWaldmann> a) it calls cancel() - why?
192 2011-06-16T18:14:48 <ThomasWaldmann> b) there seems to be a bug in whoosh with handling cancel() in MultiSegmentWriter
193 2011-06-16T18:15:14 <ThomasWaldmann> it is very likely not "self.lock", but "self.writelock"
194 2011-06-16T18:19:44 <Marchael> hm,yes writelock rocks
195 2011-06-16T18:22:00 <Marchael> but now it like http://paste.pocoo.org/show/408901/
196 2011-06-16T18:23:38 <ThomasWaldmann> https://bitbucket.org/mchaput/whoosh/issue/148/with-fix-attributeerror-multisegmentwriter
197 2011-06-16T18:24:34 <ThomasWaldmann> well, print repr(metadata) before doing the call
198 2011-06-16T18:24:55 <ThomasWaldmann> likely there is something in there that is not quite as expected
199 2011-06-16T18:25:11 * ThomasWaldmann suspects it does not like unicode as keyword
200 2011-06-16T18:31:51 <Marchael> I tried different combination with repr, it doesn't make sense
201 2011-06-16T18:32:08 <Marchael> s/it/repr/
202 2011-06-16T18:32:15 <ThomasWaldmann> show output
203 2011-06-16T18:32:38 <ThomasWaldmann> btw, that TypeError was the reason why you found that whoosh bug
204 2011-06-16T18:33:14 <ThomasWaldmann> if there is an exception within the "with" scope, it calls __exit__() with that exception
205 2011-06-16T18:33:29 <Marchael> http://paste.pocoo.org/show/408912/
206 2011-06-16T18:33:32 <ThomasWaldmann> if exit detects it is called with an exception, in cancels the transaction
207 2011-06-16T18:34:23 <Marchael> cool
208 2011-06-16T18:34:25 <ThomasWaldmann> ehrm, you misunderstood
209 2011-06-16T18:34:53 <ThomasWaldmann> i mean you add a line before the add_document call and put print repr(metadata) in it
210 2011-06-16T18:35:02 <Marchael> ah
211 2011-06-16T18:36:31 <Marchael> http://paste.pocoo.org/show/408914/
212 2011-06-16T18:36:39 <ThomasWaldmann> in general, if some call blows up, always make sure you know what you give it
213 2011-06-16T18:39:32 <ThomasWaldmann> what python version is that?
214 2011-06-16T18:40:14 <Marchael> Python 2.6 (r26:66714, Jun 16 2011, 20:26:55)
215 2011-06-16T18:40:46 <ThomasWaldmann> ok, you found a feature of 2.6(.0), 2.6.5 behaves better :)
216 2011-06-16T18:41:01 <ThomasWaldmann> but that's ok, we must live with that
217 2011-06-16T18:41:24 <ThomasWaldmann> you just need to make sure that keywords are str and not unicode
218 2011-06-16T18:42:01 <Marchael> but it won't unicode whoshh blows up with exception...
219 2011-06-16T18:42:11 <Marchael> *if it
220 2011-06-16T18:42:15 <ThomasWaldmann> the keyword, not the value :)
221 2011-06-16T18:42:27 <Marchael> ah
222 2011-06-16T18:42:34 <ThomasWaldmann> you are having a python issue, not a whoosh issue
223 2011-06-16T18:43:02 <Marchael> aghh
224 2011-06-16T18:43:05 <Marchael> *argh
225 2011-06-16T18:44:57 <Marchael> ThomasWaldmann: so I should repr each key?
226 2011-06-16T19:22:22 <Marchael> In [58]: repr(u"string")
227 2011-06-16T19:22:23 <Marchael> Out[58]: "u'string'"
228 2011-06-16T19:22:44 <Marchael> We'll pur "u'sting'" to index?
229 2011-06-16T19:23:46 <Marchael> ThomasWaldmann: ping
230 2011-06-16T19:35:41 <ThomasWaldmann> no, use str()
231 2011-06-16T19:35:57 <ThomasWaldmann> for the keywords it is fine, we do not have non-ascii stuff
232 2011-06-16T19:38:17 <Marchael> ok
233 2011-06-16T19:40:43 *** greg_f
234 2011-06-16T19:46:43 *** pkumar
235 2011-06-16T20:27:37 <Marchael> https://github.com/MrMEEE/bumblebee/commit/a047be85247755cdbe0acce6 epic win
236 2011-06-16T20:31:00 <ThomasWaldmann> lol
237 2011-06-16T20:31:15 <ThomasWaldmann> you still have your /usr? :)
238 2011-06-16T20:32:09 <Marchael> make backups /usr @ install bumblbee :D
239 2011-06-16T20:33:37 <ThomasWaldmann> but nice pictures :)
240 2011-06-16T20:33:45 <ThomasWaldmann> (in teh comments)
241 2011-06-16T20:38:13 <Marchael> they may take our lives @ but they'll never take our /usr :D
242 2011-06-16T20:39:46 *** grzywacz
243 2011-06-16T20:42:24 <pkumar> ronny : you there?
244 2011-06-16T20:46:15 <pkumar> ronny: I had to discuss regarding an issue in my codes.
245 2011-06-16T21:05:17 *** sinha
246 2011-06-16T21:09:31 <ronny> pkumar: ping?
247 2011-06-16T21:10:18 <pkumar> ronny : hi
248 2011-06-16T21:11:10 <pkumar> ronny : As suggested by ThomasWaldmann, I was working on the idea of making the tests faster.
249 2011-06-16T21:11:53 <pkumar> ronny : since for current tests, app is initialized form scratch for every test item
250 2011-06-16T21:12:16 <pkumar> in init_test_app() function
251 2011-06-16T21:13:32 <pkumar> so, I made some changes in the code,
252 2011-06-16T21:13:40 <pkumar> of conftest.py
253 2011-06-16T21:14:30 <ronny> so what does it do now and whats broken
254 2011-06-16T21:15:31 <ThomasWaldmann> pkumar: use codereview?
255 2011-06-16T21:15:43 <pkumar> though, it works for many of the tests(almost all) but there are still some problems I guess. It would be great if you would have a look at it.
256 2011-06-16T21:16:06 <Marchael> ThomasWaldmann: seems now, my stuff working on 2.6 :)
257 2011-06-16T21:16:40 <ronny> pkumar: please more code && more concise
258 2011-06-16T21:17:18 <pkumar> ronny : just a mom
259 2011-06-16T21:18:04 <pkumar> ThomasWaldmann : can you please give me the link for the code review, I haven't used it
260 2011-06-16T21:19:03 <pkumar> I mean place where I can put my codes
261 2011-06-16T21:19:09 <pkumar> ThomasWaldmann : ^
262 2011-06-16T21:22:41 <ThomasWaldmann> 4th link on google
263 2011-06-16T21:23:42 <pkumar> ronny: here is the raw code(for testing purpose) http://paste.pocoo.org/show/409020/
264 2011-06-16T21:24:20 <Marchael> ThomasWaldmann: please, review http://codereview.appspot.com/4539114
265 2011-06-16T21:27:10 <ThomasWaldmann> Marchael: mostly ok, one minor thing
266 2011-06-16T21:27:16 <ronny> pkumar: those globals dont look tasty
267 2011-06-16T21:27:18 <Marchael> thx
268 2011-06-16T21:27:31 <ronny> pkumar: the way previous configs are managed looks like a mess
269 2011-06-16T21:27:45 <pkumar> ronny : yeah, currently they are
270 2011-06-16T21:27:57 <ronny> i cant make out if its correct or not from just taking a look, so its probably broken in some substele way
271 2011-06-16T21:28:30 <Marchael> ThomasWaldmann: but this code has some on major issue :)
272 2011-06-16T21:28:37 <Marchael> s/some/one/
273 2011-06-16T21:28:49 <pkumar> ronny : can you please check the algorithm, if it is logically correct?
274 2011-06-16T21:29:17 <ronny> why is there a TRY/FINALLY for function defintions
275 2011-06-16T21:29:19 <Marchael> it doesn't checks already commited documetns
276 2011-06-16T21:29:38 <Marchael> so it'll be duplicate all stuff
277 2011-06-16T21:29:38 <ronny> pkumar: also using the string 'none'
278 2011-06-16T21:29:55 <ThomasWaldmann> yes, you have only code for initial index building
279 2011-06-16T21:30:14 <Marchael> I see 2 ways:
280 2011-06-16T21:30:14 <Marchael> 1) checks for duplication
281 2011-06-16T21:30:14 <Marchael> 2) rm -rf wiki/index
282 2011-06-16T21:30:18 <ThomasWaldmann> maybe add another option that kills an existing index
283 2011-06-16T21:30:48 <ronny> pkumar: it looks rather confusing
284 2011-06-16T21:30:49 <pkumar> ronny : yes, that's still raw and need to be modified like variable and stuffs
285 2011-06-16T21:31:06 <pkumar> ronny : ok, I'll add some comments if it helps.
286 2011-06-16T21:31:06 <ThomasWaldmann> (that should not happen automatically and by default, though, it can make sense to have multiple wikis indexed into same index files)
287 2011-06-16T21:31:35 <pkumar> ronny : just a moment.
288 2011-06-16T21:31:39 <ThomasWaldmann> Marchael: and iirc whoosh has an api function to clear an index, which is better than bumblebee :)
289 2011-06-16T21:31:51 <Marchael> :D
290 2011-06-16T21:32:31 <ronny> pkumar: also please explain the basic caching idea in detail
291 2011-06-16T21:32:42 *** sinha
292 2011-06-16T21:33:04 *** sinha
293 2011-06-16T21:33:16 <pkumar> ok.
294 2011-06-16T21:34:50 <Marchael> ThomasWaldmann: so will we search current document in index before add_documents() it? And if we'll find something them just ignore it.
295 2011-06-16T21:35:13 <pkumar> ronny: most of the tests uses wikiconfig.Config and some its own config i.e. cls.Config.
296 2011-06-16T21:35:14 <Marchael> s/them/then/
297 2011-06-16T21:36:38 <pkumar> ronny : now, in init_test_app only thing is passed is given_config. So, the idea is if they have same given_config, previous app can work for the current test item
298 2011-06-16T21:37:01 <ronny> afk for half an hour
299 2011-06-16T21:37:14 *** ronny
300 2011-06-16T21:45:32 <Marchael> ThomasWaldmann: ping
301 2011-06-16T21:56:19 <ThomasWaldmann> you need to ask more precisely. which use case do you mea?
302 2011-06-16T21:56:21 <ThomasWaldmann> n
303 2011-06-16T21:56:41 <Marchael> all
304 2011-06-16T21:56:56 <Marchael> what we'll do with duplication?
305 2011-06-16T21:57:11 <Marchael> I mean when building indexes
306 2011-06-16T21:57:11 <ThomasWaldmann> you do not need to "search current document in index" for the "build index" case
307 2011-06-16T21:57:28 <Marchael> so I just purge old indexes and build new>
308 2011-06-16T21:57:28 <Marchael> ?
309 2011-06-16T21:57:42 <ThomasWaldmann> as I said, you add an option --clear or so that clears the index
310 2011-06-16T21:58:08 <ThomasWaldmann> and please read what i write, i dislike to repeat all the time
311 2011-06-16T21:58:13 <Marchael> but we should do by default?
312 2011-06-16T21:58:37 <ThomasWaldmann> (21:30) < ThomasWal> maybe add another option that kills an existing index
313 2011-06-16T21:58:44 <ThomasWaldmann> (21:31) < ThomasWal> (that should not happen automatically and by default, though, it can make sense to have multiple wikis indexed into same index files)
314 2011-06-16T21:58:50 <ThomasWaldmann> (21:31) < ThomasWal> Marchael: and iirc whoosh has an api function to clear an index, which is better than bumblebee :)
315 2011-06-16T21:59:42 <Marchael> you don't ask on my question
316 2011-06-16T21:59:57 <Marchael> I ask not about option
317 2011-06-16T22:00:01 <ThomasWaldmann> if you like it better, you could also add an option --noclear for the farm case
318 2011-06-16T22:00:37 <Marchael> then what I should do now?
319 2011-06-16T22:01:17 <ThomasWaldmann> i have no idea what you want to know
320 2011-06-16T22:01:31 <ThomasWaldmann> maybe try asking precise questions
321 2011-06-16T22:01:41 * ThomasWaldmann puts that on an F-key soon
322 2011-06-16T22:02:37 <pkumar> ThomasWaldmann : my tests are still running (they are very slow) and any file leak is not encountered yet.
323 2011-06-16T22:02:53 <pkumar> for fs2
324 2011-06-16T22:03:37 <pkumar> It is on test_backend_memory now
325 2011-06-16T22:07:49 <ThomasWaldmann> read the bug reports on bb, reproduce the problem. this is the usual beginning of how to fix a bug.
326 2011-06-16T22:07:59 <ThomasWaldmann> if you can't reproduce it, you can't fix it.
327 2011-06-16T22:08:38 <ThomasWaldmann> pkumar: ^^
328 2011-06-16T22:09:00 <Marchael> ThomasWaldmann: I can leave this code as is and go to next step or I need to add updating/cleaning?
329 2011-06-16T22:09:18 <ThomasWaldmann> and if you run tests on fs2 backend, yes, then they are much slower than with memory backend. that's why I suggested tmpfs or ramdisk a few days ago.
330 2011-06-16T22:10:17 <ThomasWaldmann> next step should be that "clear index" option, next afterwards the "make sure an existing index is uptodate" functionality.
331 2011-06-16T22:11:04 <ThomasWaldmann> iirc, both is well described in the whoosh docs.
332 2011-06-16T22:11:48 <ThomasWaldmann> (if the first thing is not, maybe have a quick look into the src, iirc there is something)
333 2011-06-16T22:12:29 <pkumar> ThomasWaldmann : What steps can I take to reproduce the problem. on the bb it just says on running using fs2 it'll run out of files?
334 2011-06-16T22:13:37 <ThomasWaldmann> well, considering that I didn't make that up, start by going back in the repo to that time, use py.test 1.3.4 and do like described in the issue
335 2011-06-16T22:14:29 <ThomasWaldmann> if you have a file leak then, update to default branch, current stuff and try again.
336 2011-06-16T22:16:05 <pkumar> ThomasWaldmann : I was wondering if pytest2 or little changes in conftest.py could have taken care of that problem?
337 2011-06-16T22:16:52 <pkumar> I'm not sure though.
338 2011-06-16T22:18:49 <ThomasWaldmann> well, find out
339 2011-06-16T22:19:21 <pkumar> ok, i'll try the tests using py.test 1.3.4 also :)
340 2011-06-16T22:20:10 <ThomasWaldmann> 1efaef3eec9c please go back to that changeset for first try
341 2011-06-16T22:20:35 <pkumar> ok
342 2011-06-16T22:20:52 <ThomasWaldmann> (and see the log, i fixed some fs2 stuff, but i thought it was still happening afterwards, too [otherwise i'ld have closed it])
343 2011-06-16T22:21:00 <ThomasWaldmann> hg log i mean
344 2011-06-16T22:23:01 <ThomasWaldmann> pkumar: https://bitbucket.org/thomaswaldmann/moin-2.0/changeset/8135eeb9ab9a this was one fix maybe making a difference
345 2011-06-16T22:24:03 <pkumar> looking
346 2011-06-16T22:24:10 <ThomasWaldmann> pkumar: https://bitbucket.org/thomaswaldmann/moin-2.0/changeset/989ada5c5cbb not sure about that one
347 2011-06-16T22:25:22 <ThomasWaldmann> pkumar: https://bitbucket.org/thomaswaldmann/moin-2.0/changeset/76a6dd2fae57 that one might make a difference if exceptions happen
348 2011-06-16T22:26:18 <pkumar> ThomasWaldmann : I think so, because when I used lsof some more files were created every time and are deleted too.
349 2011-06-16T22:27:08 <ThomasWaldmann> are you sure you know what you are looking for?
350 2011-06-16T22:27:13 <pkumar> so, as a result number do not increase.
351 2011-06-16T22:27:35 <pkumar> i think so
352 2011-06-16T22:27:49 *** ronny
353 2011-06-16T22:27:50 <pkumar> am I missing something?
354 2011-06-16T22:27:55 <ronny> re
355 2011-06-16T22:27:59 <ThomasWaldmann> no, i was just wondering :)
356 2011-06-16T22:28:46 <ThomasWaldmann> pkumar: you have a tmpfs meanwhile or does each try cost you hours?
357 2011-06-16T22:29:29 <pkumar> no, I don't have tmpfs as for now.
358 2011-06-16T22:30:21 <ThomasWaldmann> you're wasting lots of time
359 2011-06-16T22:30:36 * ronny is currently hardware-fixing his mobile phone
360 2011-06-16T22:31:55 <pkumar> ThomasWaldmann : though I'm also working on other testing stuff meanwhile, this delay is surely unpleasant
361 2011-06-16T22:32:38 <ronny> pkumar: you can run tests on single files/directories to cut it short
362 2011-06-16T22:33:18 <ronny> pkumar: you should use None/real objects to compare wikiconfigs instead of strings
363 2011-06-16T22:33:22 <ronny> its utterly confusing
364 2011-06-16T22:33:59 <ThomasWaldmann> ronny: the bug report for the file leak refers to running tests from workdir, so all tests are run
365 2011-06-16T22:34:17 <ronny> ThomasWaldmann: got details for me?
366 2011-06-16T22:34:50 <ThomasWaldmann> https://bitbucket.org/thomaswaldmann/moin-2.0/issue/10/pytest-run-with-fs2-shows-file-leak
367 2011-06-16T22:35:49 <ThomasWaldmann> https://bitbucket.org/thomaswaldmann/moin-2.0/issue/24/fs-backend-may-run-out-of-files this is a similar one, but the reason for this one is known
368 2011-06-16T22:36:33 <ronny> both on pypy?
369 2011-06-16T22:36:41 <ThomasWaldmann> xorAxAx: ronny: btw, if you're hacking on pypy: moin2 is much slower on pypy than on cpython for unknown reasons.
370 2011-06-16T22:36:55 <ThomasWaldmann> ronny: read the bug please
371 2011-06-16T22:37:06 <ronny> ThomasWaldmann: the first one isnt clear
372 2011-06-16T22:37:22 <ThomasWaldmann> if it does not tell it is on pypy, then it isn't
373 2011-06-16T22:37:29 <ronny> ok
374 2011-06-16T22:37:47 * ThomasWaldmann drives home, brb
375 2011-06-16T22:38:02 <pkumar> ronny : http://paste.pocoo.org/show/409045/
376 2011-06-16T22:38:14 <pkumar> ronny : I've added some comments.
377 2011-06-16T22:38:57 <pkumar> ronny : it is a little messy so please tell me if it is unclear at any point.
378 2011-06-16T22:41:22 <ronny> pkumar: its prety much still confusing as a whole
379 2011-06-16T22:41:31 <ronny> brb,
380 2011-06-16T22:43:42 <ronny> pkumar: its absolutely not clear if its correctly deciding when to make a new app init
381 2011-06-16T22:44:21 <pkumar> ronny : Actually, the main idea is to check previous cls configuration, get the previous app, if same assign to current app
382 2011-06-16T22:44:56 <ronny> pkumar: you dont do that correctly
383 2011-06-16T22:45:13 <ronny> you need to check very carefull if the previous applying config IS the same as the current one
384 2011-06-16T22:45:19 <ronny> and really just that
385 2011-06-16T22:45:32 <ronny> no poking around in the class if you have the previous config as object
386 2011-06-16T22:45:52 <ronny> in fact, keep it around, cause the poking around in maybe pevious class makes it broken for sure from my pov
387 2011-06-16T22:47:07 <pkumar> do we have any pytest hooks or some other better way in which we can do this thing?
388 2011-06-16T22:47:51 <pkumar> i.e. we don't need to call init_test_app again and again?
389 2011-06-16T22:48:10 <pkumar> for each and every test item
390 2011-06-16T22:48:15 <pkumar> ronny : ^
391 2011-06-16T22:48:16 <ronny> there is cached_setu for funcargs, i suppose you could try and use that
392 2011-06-16T22:48:28 <ronny> (you can use the config as extra key for the setup
393 2011-06-16T22:48:34 <ronny> but the teardown would be weird
394 2011-06-16T22:51:00 <pkumar> any link where I can find more about 'cached_setu'
395 2011-06-16T22:51:07 <pkumar> ronny : ^
396 2011-06-16T22:51:17 <ronny> its documented with funcargs
397 2011-06-16T22:51:22 <pkumar> ok
398 2011-06-16T22:52:10 <pkumar> though the current code works for most of the tests, it fails for a few in storage I guess
399 2011-06-16T22:52:16 <pkumar> ronny : ^
400 2011-06-16T22:53:49 <ronny> pkumar: can you test with just those tests
401 2011-06-16T22:54:06 <ronny> if it suddenly works, then you kno you need to rewire checking how the config is grabbed
402 2011-06-16T22:55:40 <pkumar> yeah, right
403 2011-06-16T22:56:50 <pkumar> ronny : ok, first I'll try using cached_setu first and make required changes
404 2011-06-16T22:58:09 <ronny> pkumar: i dont think it will fit the current code that well
405 2011-06-16T22:59:04 <ronny> funcargs and cached_setup are not part of test items atm
406 2011-06-16T22:59:06 <pkumar> ronny : so, any other suggestions about how to proceed on this?
407 2011-06-16T22:59:41 <pkumar> ronny : so, any other suggestions about how to proceed on this
408 2011-06-16T23:02:14 <ronny> pkumar: cirrectly cheching which wikiconf applies is a start, this is done by objec identity to previous wikiconfig
409 2011-06-16T23:03:32 <Marchael> gn/bbl
410 2011-06-16T23:05:49 <ronny> pkumar: also i think all non-class tests dont need an app setup (afair all app using tests are in classes
411 2011-06-16T23:07:56 <pkumar> ronny: how do we find object identity to previous wikiconfig?
412 2011-06-16T23:09:13 <ThomasWaldmann> btw, recreating app is not only needed for config change, it might be also needed to get a fresh backend
413 2011-06-16T23:09:36 <ronny> pkumar: remeber the previous one, use the is operator
414 2011-06-16T23:10:44 <pkumar> ThomasWaldmann : yes, it is. Thats why some tests in storage module are failing
415 2011-06-16T23:10:46 <pkumar> ItemAlreadyExistsError
416 2011-06-16T23:11:54 <pkumar> ronny : ok
417 2011-06-16T23:13:57 <pkumar> so, now the question is, do we need to do this or not. or have another set of rules for some of the tests?
418 2011-06-16T23:14:36 <pkumar> because creating a new app is needed at many tests
419 2011-06-16T23:14:47 <pkumar> ThomasWaldmann : ^
420 2011-06-16T23:14:54 <pkumar> ronny : ^
421 2011-06-16T23:15:53 <pkumar> actually they are not many but a few tests.
422 2011-06-16T23:16:06 <ThomasWaldmann> if you detect config changes, we could just change config when needed :)
423 2011-06-16T23:18:01 <ronny> pkumar: as a important check, dont make configs for tests that dont need one anyway
424 2011-06-16T23:18:19 <ronny> (afair there should be quite a few
425 2011-06-16T23:18:30 <ronny> after all before the class collector did the hard work
426 2011-06-16T23:18:49 <pkumar> ThomasWaldmann : and those changes will be in individual tests I guess?
427 2011-06-16T23:18:57 <pkumar> ronny: yeah :)
428 2011-06-16T23:24:33 <pkumar> gn everyone
429 2011-06-16T23:24:34 <ThomasWaldmann> ThomasWaldmann: where needed
430 2011-06-16T23:24:41 <ThomasWaldmann> ehrm pkumar
431 2011-06-16T23:24:50 <pkumar> ThomasWaldmann : ok
432 2011-06-16T23:30:22 <pkumar> bbl
433 2011-06-16T23:30:27 *** pkumar
434 2011-06-16T23:39:30 *** MattMaker
435 2011-06-16T23:41:55 <dreimark> re
436 2011-06-16T23:47:50 <sinha> dreimark: that next page is functionality is working fine
437 2011-06-16T23:48:04 <sinha> just stuck in finding the previous link
438 2011-06-16T23:48:21 <sinha> i mean i tried something, but didnt get something useful
439 2011-06-16T23:53:12 <dreimark> sinha: have you something to show?
440 2011-06-16T23:57:04 <sinha> dreimark: this is something i tried http://paste.pocoo.org/show/409128/
441 2011-06-16T23:57:13 <sinha> i can send parent to next pages
442 2011-06-16T23:57:25 <sinha> but for previous pages not able to get it
443 2011-06-16T23:58:41 <sinha> you were telling to use some function to find out the previous page offset
444 2011-06-16T23:59:09 <sinha> i tried figuring out, but wasn't getting anything optimal and useful
445 2011-06-16T23:59:39 <dreimark> hmm
