MoinMoin: MoinMoinChat/Logs/moin-dev/2011-09-16

   1 2011-09-16T01:13:46  <ThomasWaldmann> works :)
   2 2011-09-16T02:03:02  <ThomasWaldmann> tw@x300:~/w/storage-ng/storage/_tests$ py.test -k test_perf test_sqlite.py
   3 2011-09-16T02:03:27  <ThomasWaldmann> 2 passed, 20 deselected in 26.84 seconds
   4 2011-09-16T02:03:33  <ThomasWaldmann> tw@x300:~/w/storage-ng/storage/_tests$ py.test -k test_perf test_kc.py
   5 2011-09-16T02:03:43  <ThomasWaldmann> 1 passed, 10 deselected in 0.10 seconds
   6 2011-09-16T02:05:43  <ThomasWaldmann> (sqlite-value must be divided by 2 to compare)
   7 2011-09-16T03:50:03  *** xjjk has quit IRC
   8 2011-09-16T03:54:23  *** xjjk has joined #moin-dev
   9 2011-09-16T04:57:01  *** xjjk has quit IRC
  10 2011-09-16T05:06:04  *** xjjk has joined #moin-dev
  11 2011-09-16T08:04:58  *** Marchael has quit IRC
  12 2011-09-16T08:42:53  *** raignarok has joined #moin-dev
  13 2011-09-16T09:05:48  *** Marchael has joined #moin-dev
  14 2011-09-16T10:03:43  <ronny> moin
  15 2011-09-16T10:03:55  <ronny> ThomasWaldmann: i figured what broke with limitedstream
  16 2011-09-16T10:05:20  *** Marchael has quit IRC
  17 2011-09-16T10:10:17  <ThomasWaldmann> moin
  18 2011-09-16T10:10:33  <ThomasWaldmann> ronny: ah, great, you fixed it / filed a bug?
  19 2011-09-16T10:13:09  <ronny> ThomasWaldmann: werkzeug pull request in 30 seconds, pushed a workaround to storage-ng
  20 2011-09-16T10:19:40  <ronny> ThomasWaldmann: i really need to refactor the tests
  21 2011-09-16T10:20:12  <ronny> am i correct in asuming that all but the storage tests just use memory store?
  22 2011-09-16T10:20:14  <ThomasWaldmann> ronny: that would be great, you know pytest better :)
  23 2011-09-16T10:20:33  <ThomasWaldmann> but please add comments when using uncommon magic of pytest
  24 2011-09-16T10:20:37  <ronny> ThomasWaldmann: btw, please comment on serialize
  25 2011-09-16T10:20:55  <ThomasWaldmann> yes, memory store is faster
  26 2011-09-16T10:21:53  <ronny> ThomasWaldmann: we have a bug in store_revision
  27 2011-09-16T10:22:02  <ThomasWaldmann> kyoto cabinet is quite fast also, btw
  28 2011-09-16T10:22:05  <ronny> it will always make a new dataid and new data stuff
  29 2011-09-16T10:22:31  <ThomasWaldmann> new data is known
  30 2011-09-16T10:23:10  <ronny> ThomasWaldmann: it basically means that a unserialize will completely shifle data id's
  31 2011-09-16T10:23:39  *** raignarok has quit IRC
  32 2011-09-16T10:23:50  <ronny> wich means restoring a backup from serialized data will make sane wikisync completely impossible
  33 2011-09-16T10:24:41  <ronny> *shuffle
  34 2011-09-16T10:26:36  <ronny> ThomasWaldmann: i'll inline store_data into store_revison and add a check
  35 2011-09-16T10:28:14  <ThomasWaldmann> you use writelines to write binary data?
  36 2011-09-16T10:29:28  <ThomasWaldmann> maybe just compare to the serialize code i posted to pastebin for some suggestions
  37 2011-09-16T10:30:29  <ronny> ThomasWaldmann: writelines is really writeall
  38 2011-09-16T10:30:33  <ronny> ThomasWaldmann: its stupid naming
  39 2011-09-16T10:30:47  <ronny> ThomasWaldmann: yours is kind of worse to create and deserialize
  40 2011-09-16T10:31:03  <ronny> i should probably add a version header or something like that
  41 2011-09-16T10:34:58  <ThomasWaldmann> it is important to also prefix the data with len
  42 2011-09-16T10:35:04  <ThomasWaldmann> as it might be compressed
  43 2011-09-16T10:35:13  <ThomasWaldmann> so meta size > compress len
  44 2011-09-16T10:35:21  <ronny> ThomasWaldmann: apply compression always to the whole stream
  45 2011-09-16T10:35:32  <ThomasWaldmann> also, add magics / version
  46 2011-09-16T10:36:37  <ThomasWaldmann> i need to leave and deal with some emergency...
  47 2011-09-16T10:36:44  <ronny> k
  48 2011-09-16T10:43:47  *** raignarok has joined #moin-dev
  49 2011-09-16T11:03:03  *** greg_f has joined #moin-dev
  50 2011-09-16T12:56:16  *** Marchael has joined #moin-dev
  51 2011-09-16T13:26:58  *** raignarok has quit IRC
  52 2011-09-16T13:27:26  <ThomasWaldmann> re
  53 2011-09-16T13:36:45  <ronny> ThomasWaldmann: im starting to reiterate the tests, i think we can factor many things into a few simple mixings
  54 2011-09-16T13:37:02  <ronny> (i.e. compression for streams/bytes, mapping between stream and byte apis and the like)
  55 2011-09-16T13:40:00  <ronny> bbl, lunch
  56 2011-09-16T13:48:54  <ThomasWaldmann> yeah. otoh, some backends like kyoto cabinet also offer compression
  57 2011-09-16T13:49:07  <ThomasWaldmann> not sure whether it makes a speed difference
  58 2011-09-16T13:54:38  *** raignarok has joined #moin-dev
  59 2011-09-16T14:00:46  *** raignarok has quit IRC
  60 2011-09-16T14:15:14  <ronny> ThomasWaldmann: i think for the unittests it makes sense to wrap all stream io stores into something that wraps/unwaps using read/StringIO
  61 2011-09-16T14:15:52  <ronny> (then the same test funtion can be used for bytes and for streams when it makes sense)
  62 2011-09-16T14:17:20  <ThomasWaldmann> yeah
  63 2011-09-16T14:18:01  <ThomasWaldmann> ronny: does py.test have some timing measurement except the total timer?
  64 2011-09-16T14:18:15  <ronny> ThomasWaldmann: i think it has per test timings
  65 2011-09-16T14:18:40  <ronny> ThomasWaldmann: for perf tests, we chould aslo implement a pytest_pyfunc_call hook that makes a very acccurate measurement
  66 2011-09-16T14:20:30  <ThomasWaldmann> it should be something controlled from within the test
  67 2011-09-16T14:20:47  <ThomasWaldmann> so you can first fill a storage, then start timer, then read, then stop
  68 2011-09-16T14:21:09  <ThomasWaldmann> or at least that should be possible
  69 2011-09-16T14:21:26  <ronny> hmm
  70 2011-09-16T14:21:34  <ronny> i suppose a timer funcarg would be nice
  71 2011-09-16T14:22:07  <ThomasWaldmann> "Development for other platworms including Windows is now work-in-progress."
  72 2011-09-16T14:22:13  <ThomasWaldmann> yeah, a plat worm!
  73 2011-09-16T14:22:24  * ThomasWaldmann LOLs
  74 2011-09-16T14:23:35  <ronny> please put emphathis around worms :P
  75 2011-09-16T14:37:22  <dreimark> moin
  76 2011-09-16T14:37:52  <ThomasWaldmann> hi dreimark
  77 2011-09-16T14:42:46  * dreimark looks like good progress on storage-ng
  78 2011-09-16T14:52:03  <ThomasWaldmann> looks like we have one storage now that is faster than indexing :D
  79 2011-09-16T14:53:50  <ronny> ThomasWaldmann: memorystorage?
  80 2011-09-16T14:55:26  *** raignarok has joined #moin-dev
  81 2011-09-16T15:03:47  *** raignarok has quit IRC
  82 2011-09-16T15:07:53  <ronny> ThomasWaldmann: can we implement iter for memcached?
  83 2011-09-16T15:08:03  <ronny> ThomasWaldmann: and does a memcached backend really make sense?
  84 2011-09-16T15:10:20  <ronny> ThomasWaldmann: care to comment on http://paste.pocoo.org/show/476986/
  85 2011-09-16T15:13:42  <ThomasWaldmann> ronny: kc is almost as fast as memory storage
  86 2011-09-16T15:14:00  <ronny> ThomasWaldmann: i see, whats the best way to install?
  87 2011-09-16T15:14:02  <ThomasWaldmann> and memcached has no iter, it is even in their FAQ
  88 2011-09-16T15:14:28  <ThomasWaldmann> i downloaded kc and compiled it, same for the python interface
  89 2011-09-16T15:14:43  <ronny> i see
  90 2011-09-16T15:14:52  <ronny> ThomasWaldmann: does a memcached backend make sense?
  91 2011-09-16T15:15:28  <ThomasWaldmann> no packages for it in ubuntu. only the predecessor "tokyo cabinet", but i rather wanted the newer stuff as they recommend it on their homepage also.
  92 2011-09-16T15:15:55  <ThomasWaldmann> ronny: not for storing real data, but i also thought about using our "storage" for cache data storage
  93 2011-09-16T15:16:12  <ThomasWaldmann> and then we want memcached, memory backend etc.
  94 2011-09-16T15:16:20  <ThomasWaldmann> kc also has in-memory modes
  95 2011-09-16T15:16:23  <ronny> "cache data storage" ?
  96 2011-09-16T15:16:35  <ThomasWaldmann> thumbnails for example
  97 2011-09-16T15:16:52  <ThomasWaldmann> or other non-original stuff
  98 2011-09-16T15:16:55  <ronny> ThomasWaldmann: dont we want flask/werkzeug caches for such things?
  99 2011-09-16T15:17:11  <ThomasWaldmann> we currently use that yes
 100 2011-09-16T15:17:23  <ronny> so why a memcached backend?
 101 2011-09-16T15:17:26  <ThomasWaldmann> i can also write a plugin there, doesn't matter much in the end
 102 2011-09-16T15:18:05  <ThomasWaldmann> yeah, maybe it doesn't make sense then to have it in our "storage"
 103 2011-09-16T15:18:16  <ThomasWaldmann> it was so easy to do :D
 104 2011-09-16T15:18:26  <ronny> yes
 105 2011-09-16T15:18:34  <ronny> it seemed exactly like that
 106 2011-09-16T15:19:12  <ronny> ThomasWaldmann: comment on the diff i posted?
 107 2011-09-16T15:19:13  <ThomasWaldmann> http://fallabs.com/kyototycoon/spex.html that's what I am looking at today
 108 2011-09-16T15:19:19  * ThomasWaldmann looks
 109 2011-09-16T15:21:33  <ThomasWaldmann> maybe don't use "type" as a name, it is a builtine
 110 2011-09-16T15:21:34  <ThomasWaldmann> -e
 111 2011-09-16T15:21:58  <ThomasWaldmann> cls?
 112 2011-09-16T15:24:51  <ThomasWaldmann> dont -> don't
 113 2011-09-16T15:25:05  <ThomasWaldmann> + more spaces in same line
 114 2011-09-16T15:25:23  <ThomasWaldmann> its -> it's
 115 2011-09-16T15:25:55  <ThomasWaldmann> requeset typo
 116 2011-09-16T15:27:31  <ThomasWaldmann> otherwise looks ok. I need to read more about test, some of the magic there I am not familiar with.
 117 2011-09-16T15:29:34  <ronny> ThomasWaldmann: i finished moving over the rest of the base class code
 118 2011-09-16T15:32:05  <ThomasWaldmann> can you show the full code?
 119 2011-09-16T15:38:25  <ThomasWaldmann> codereview.appspot.com is nice btw :)
 120 2011-09-16T15:45:12  *** raignarok has joined #moin-dev
 121 2011-09-16T15:46:01  *** raignarok has quit IRC
 122 2011-09-16T15:52:30  *** raignarok has joined #moin-dev
 123 2011-09-16T16:01:21  *** xorAxAx has quit IRC
 124 2011-09-16T16:02:09  *** xorAxAx has joined #moin-dev
 125 2011-09-16T16:07:45  *** jarretinha has joined #moin-dev
 126 2011-09-16T16:07:47  <ronny> ThomasWaldmann: it is done
 127 2011-09-16T16:08:25  <ronny> ThomasWaldmann: please run the tests on your box
 128 2011-09-16T16:09:07  <jarretinha> Hi Hacklab guys! Hi Thomas!
 129 2011-09-16T16:09:26  <ThomasWaldmann> hi jarretinha :)
 130 2011-09-16T16:10:39  * ThomasWaldmann runs py.test
 131 2011-09-16T16:11:04  <ronny> ThomasWaldmann: on the plus side, we can now add more tests per backend for different modes
 132 2011-09-16T16:11:27  <ThomasWaldmann> ronny: py.test results look ok, but i don't have all stuff installed here, will test again later on my laptop
 133 2011-09-16T16:11:53  * ThomasWaldmann merged and pushed
 134 2011-09-16T16:13:36  <ronny> ThomasWaldmann: see py.test -v for more details
 135 2011-09-16T16:26:25  <ThomasWaldmann> ah, ok
 136 2011-09-16T17:02:54  *** raignarok has quit IRC
 137 2011-09-16T18:16:49  *** raignarok has joined #moin-dev
 138 2011-09-16T18:28:43  <ThomasWaldmann> re
 139 2011-09-16T18:34:24  <ThomasWaldmann> ronny: is .kc a valid extension?
 140 2011-09-16T18:41:15  <ronny> ThomasWaldmann: it was wat was used by the subclass for testing it
 141 2011-09-16T18:42:56  * ThomasWaldmann used .kch
 142 2011-09-16T18:43:35  <ronny> ThomasWaldmann: then i did missreat that by accident
 143 2011-09-16T18:46:39  <ThomasWaldmann> i fix it
 144 2011-09-16T18:51:19  <ThomasWaldmann> ronny: see latest cs
 145 2011-09-16T18:51:59  <ThomasWaldmann> it needs the extension to know the db format
 146 2011-09-16T18:54:03  <ronny> i see
 147 2011-09-16T19:08:53  * ThomasWaldmann pushed
 148 2011-09-16T19:10:23  <ThomasWaldmann> ronny: how about serializing a terminating 0 int, so we now it is really the official end?
 149 2011-09-16T19:14:15  <ronny> ThomasWaldmann: makes sense for finding odd cases of corruption
 150 2011-09-16T19:15:22  * ThomasWaldmann adds it
 151 2011-09-16T19:21:07  *** raignarok has quit IRC
 152 2011-09-16T19:21:13  *** raignarok has joined #moin-dev
 153 2011-09-16T19:25:49  *** raignarok has quit IRC
 154 2011-09-16T19:40:59  *** greg_f has quit IRC
 155 2011-09-16T19:53:19  *** Marchael has quit IRC
 156 2011-09-16T20:00:01  <ThomasWaldmann> https://bitbucket.org/thomaswaldmann/storage-ng/issue/4/trackingfilewrapper-should-check-pre-and
 157 2011-09-16T20:00:05  <ThomasWaldmann> ronny: waldi:
 158 2011-09-16T20:00:31  <ThomasWaldmann> i could add a check for realfile.tell() == 0, but that assumes we have tell()
 159 2011-09-16T20:13:15  <ThomasWaldmann> ok, did it, pushed
 160 2011-09-16T20:14:21  <ThomasWaldmann> https://bitbucket.org/thomaswaldmann/storage-ng/issue/5/router-split-between-ro-and-rw-part comments on that?
 161 2011-09-16T20:16:06  <ronny> ThomasWaldmann: i think router doesnt have to be split, it should simply behave as readonly if one writes to a readonly part
 162 2011-09-16T20:16:19  <ronny> i still like the idea of a immutable backend
 163 2011-09-16T20:16:37  <ronny> its a lot safer from accidentially messing it up
 164 2011-09-16T20:20:03  <ThomasWaldmann> readonly backends do not even have set/delete
 165 2011-09-16T20:20:09  <ThomasWaldmann> currently
 166 2011-09-16T20:20:25  <ThomasWaldmann> ah, i misread
 167 2011-09-16T20:20:35  <ThomasWaldmann> so we only have 1 router class in future
 168 2011-09-16T20:23:57  <ThomasWaldmann> ronny: can you fix it while i try kyoto tycoon? :)
 169 2011-09-16T20:35:33  <ronny> hmk, i'll take a look
 170 2011-09-16T20:54:25  *** jarretinha has quit IRC
 171 2011-09-16T21:00:11  <ronny> ThomasWaldmann: first iteration on it is done
 172 2011-09-16T21:02:59  * ThomasWaldmann looks
 173 2011-09-16T21:04:41  <ronny> ThomasWaldmann: we should probably add a readonly backend in the mix as well
 174 2011-09-16T21:07:09  <ronny> btw, shouldnt router go to backend as well?
 175 2011-09-16T21:08:20  <ThomasWaldmann> well, it does not store stuff
 176 2011-09-16T21:08:28  <ThomasWaldmann> so not really at the end
 177 2011-09-16T21:08:49  <ThomasWaldmann> otoh, it has backend api at top and at bottom
 178 2011-09-16T21:09:03  <ThomasWaldmann> cs looks good at first looks
 179 2011-09-16T21:24:16  <ronny> ThomasWaldmann: ok, added some readonly behaviour tests
 180 2011-09-16T21:45:00  <ThomasWaldmann> hmm, looks like tycoon has no "list keys"
 181 2011-09-16T21:50:40  <ronny> ThomasWaldmann: does it have some kind of list type?
 182 2011-09-16T21:53:10  <ronny> ThomasWaldmann: there should be a cursor concept (iterating keys and values
 183 2011-09-16T21:58:22  *** MattMaker has quit IRC
 184 2011-09-16T21:59:10  *** MattMaker has joined #moin-dev
 185 2011-09-16T21:59:35  <ThomasWaldmann> ronny: it is in the backend, but not on the http interface
 186 2011-09-16T22:01:21  <ThomasWaldmann> http://fallabs.com/kyototycoon/spex.html
 187 2011-09-16T22:03:45  <ThomasWaldmann> maybe via the rpc interface...
 188 2011-09-16T22:05:27  <ronny> ThomasWaldmann: seems like rpc + lua may be a way
 189 2011-09-16T22:30:02  <ThomasWaldmann> http://paste.pocoo.org/show/477160/ current state
 190 2011-09-16T22:34:48  <ronny> ThomasWaldmann: so only listing/iteration missing?
 191 2011-09-16T22:35:52  *** MattMaker has quit IRC
 192 2011-09-16T22:36:48  <ThomasWaldmann> yes
 193 2011-09-16T22:41:04  <ronny> good luck with that
 194 2011-09-16T22:41:11  <ronny> brb, grabing some food
 195 2011-09-16T23:46:55  *** raignarok has joined #moin-dev
 196 2011-09-16T23:51:25  *** MattMaker has joined #moin-dev
 197