MoinMoin: EugeneSyromyatnikov/EventLogIndexing

(some ideas i'd like to implement)

Now, nearly every access to event log consumes o(n) operations during no indexing schemas implemented --- just raw read until needed entry is found. I want to implement some changes to make most operations o(1) instead of o(n):

• Create file *log.idx which contains offsets of log entries in log file. It's ok because log files are append-only, and already written offsets are not changed. Every record in index file should have constant width (and may be still human-readable - in case of using hex with zero padding and line breaks), thus making time access nth entry o(1), not o(n).
• (optional) Add number of entry in log file - just additional check that index file is correct.

As fallback we can left old schema with file scanning.

These changes make using log files operations not heavy (like now, especially with big logs with size of tens and hundrets megabytes) and enables implementing some features like event log access by offset, filtering and so on. Nearly every basic operation — finding nth entry, get log entry count — will have o(1) complexity.

Also this schema is rather generic and may be used in fs backend in moin2.

Note: no big changes in moin 1.x. moin2 works quite differently anyway.