Thinking Aloud Usability Testing

by James Sun (crptone)

Introduction

For the usability testing of a Moin site, I chose to use the "thinking aloud" protocol on two friends: both of them were moderately computer literate users, but only the first had prior experience on editing a Wiki site and using forums. These two students were completely new to MoinMoin as I had not introduced it to them before. During these separate testing sessions for usability, which lasted slightly less than an hour each, I gave each person specific tasks to complete. I made sure that they would achieve their goals as independently as possible, wording my statements carefully and only providing comments instead of hints. Whenever they saw something particularly intriguing or unusual, they were quick to comment on it and explain their reasoning. By the end of each session, both users were particularly impressed at its simplicity and efficiency to use, but at the same time surprised with the large amount of available features in this Wiki engine.

Account Creation

The first task that I delegated to each of the users was to create an account. Having taken a quick glance at the FrontPage, the first friend was able to easily create an account right away. He located the login button at the top-left corner of the screen, redirected himself to the UserPreferences page, and then typed an acceptable username and password. However, my friend was slightly amused by the TextCha, which he thought was strangely out of place and unprofessional. He felt that a regular image Captcha would serve better in its place, as he would rather not spend time actively thinking about “which vehicle would travel quickest from Europe to Australia”, no matter how trivial the question was. In some other task, he even managed to answer a question wrong ("what software do you use to surf the web?" "Mozilla Firefox" was wrong)

However, the second user had more problems with creating an account. He saw the login button at the top of the screen, but scanned through the entire page for a "register" option. As he did not know what to do, he reluctantly clicked on the login button in order to register his account. He noted that it was an expected feature to have a "register" button appear next to the login if no user was logged in. Afterwards, he attempted several times to come up with a suitable password, but he found always found it "too short" or "too easy". He was slightly annoyed that he was unable to determine the specific definition of "short" or "easy". He also found the accompanying TextCha very strange, and preferred to get rid of it altogether (I had to explain that it was used to discourage bots). In later tasks he commented on how he kept getting repeat questions, so it wouldn’t really help much against bots anyways. Although he felt happy that registering an account had few requirement fields, he noted that the process should be made clearer to new MoinWiki users.

Using the Search Function

The next task, finding the Sandbox page, was accomplished fairly easily by both users by searching for the term "sandbox". While the first preferred to use the quick search box at the top-right of the page, the second (even though both users knew about each method already). The second user enjoyed that the "advanced search" on FindPage was straightforward and did not have any extraneous, unnecessary options for a Wiki site. Out of curiosity the second user decided to misspell his query next, and upon seeing that no spell-checking was available, he suggested it would be a nice feature to implement spelling corrections if the user was unsure, denoted by a "use spell-check" checkbox.

User Preferences

I told each user to try to "change the page layout", after I explained that the Moin site had implemented such a feature. The first user was able to immediately locate the theme option in UserPreferences. The second user took slightly longer, before he realized that "theme" was what he wanted to change. After trying out the classic and sidebar themes, the second user said that he still preferred the look of the modern theme. However, the first user enjoyed the right sidebar theme because it looked more pleasing to him and that it had a more compact style. Each time they changed their user preferences, both users were becoming annoyed at the TextCha: they asked, "why do you need this here if you are just simply changing your preferences?"

Page Creation and Editing

I also instructed both users to create a homepage and accomplish various tasks on that. Each user instinctively clicked on their usernames (however the second user had already clicked his before by accident) and easily managed to notice the various options they could use on the navigation bar. When asked to create a simple hyperlink on their page, each user experimented with both the text and GUI edit modes. The first user realized that special syntax would be required to create a link, and mentioned the convenience of the link to the HelpOnEditing page while editing his homepage. During the editing process, he would frequently refer to that help page, and remarked that the markup language was very simple and adequate. After doing this, he fooled around with the GUI editor and was even more satisfied with its user-friendly interface. Accomplishing a few editing tasks in much less time, he determined that this was the greatest feature he had seen so far on the site.

The second user clicked on both modes before attempting any editing, citing his nervousness that it would be a difficult task. However, upon seeing that the material in the GUI editor closely resembled the current content on his page, he found it quite easy and painless to add stuff (like using the “create link” button) and format text. Nevertheless, I asked him to try at least a few tasks using the text edit mode for experience. After attempting to add an external link by simply typing it on the page (which did not render it clickable), he reluctantly sifted through the HelpOnEditing page to find the correct syntax. He did praise the convenience and straightforwardness of the format listings, although he generally took more time locating and implementing the desired items than the first user. Noting the existence of “hidden” options in the raw text of the page (such as language), he told me that although he recognized the importance of the text edit mode, he felt that the GUI editor was simply all that any user would need for common editing tasks.

Integrating Macros into Pages

After testing out various page actions, I gave them a task which required them to search through the HelpContents: Implement a search box into their page. Surprisingly, both users were able to find what they needed relatively quickly; both correctly navigated the HelpContents index to the HelpOnMacros page. Both users had to click through some help pages to see if they could find what they were looking for, but they didn’t mind too much since the help pages were organized well and not at all time-consuming to go through. Both users were very impressed when they saw a “titlesearch” box on their page (by the way, advanced search macro didn’t seem to work for them), and praised the fact that anyone could be able to use these “building blocks” (macros) to create a fairly sophisticated page whenever a particular one was needed at some point.

Conclusion

Overall, I received mostly positive feedback from the two testers, and the complaints were directed only at minor troubles they encountered, such as the lack of a “register” button, and the strange use of the TextChas. They were successfully able to complete their tasks without too much difficulty, and overall they both felt that MoinMoin’s simple layout and clear system for navigating pages made it very good software with many potential uses. The second user even suggested that it could be effectively used by community groups and schools. When asked their opinion of the usability of MoinMoin, they agreed that MoinMoin is both easy to learn and easy to use; the second user claimed that this short introductory session to MoinMoin enabled him to already become an “expert” user since its features are so simple and efficient.

Suggestions for Improving Usability

After asking them for suggestions to improve usability, besides any aforementioned problems, each came up with their own unique suggestions: the first user mentioned that there were very few layouts to choose from. I told him that themes can easily be built and added, but he insisted that it would be a wise choice for the developers to include more default themes, to suit the needs of anyone who would want to immediately and satisfactorily create their own wikis. He also very much enjoyed using the quick links feature, citing its many possible uses, but however he thinks that people like him may have many pages they would like to have handy (assuming that the used wiki has lots of content). Therefore, in addition to the added quick link tabs on the navigation bar, he suggests creating an option for a “bookmarks” page that can be easily accessed from any page (either on a tab or in “more actions”)

The second user was particularly impressed with the GUI Editor, and suggested that it would only get better if more functionality was provided for it (although he could not specifically state what actions he would like to be put in). He at least suggested that anything the “raw text” edit mode could do, he should be able to configure with the GUI Editor as well. He would also like to see the addition of a “talk page” which would present itself as a tab in the navigation bar, if some user created one for it. Although he realizes that most pages themselves are actually used for discussing a topic, there are some pages (such as some of the immutable pages) which may require discussion of clarification of the topic or update requests by users. Lastly, he suggests that users should be able to choose their own font related preferences, and the dimensions of their pages. He believes this would be a trivial change and easier to implement than the above features, although not as important. These were all of the suggestions that both users came up with to improve MoinMoin’s usability.

Comments from moin developers

James, thanks for doing this test with your friends! A few comments:

TextCha

• We ask about the "kind of software" (not "the software"), thus "Firefox" is not the right answer
  • I think we remove that question sooner or later, because many people don't read it exactly enough to give the right answer (which is "browser", btw.).
• It is no problem repeating the questions as long as a machine can not compute the answer and no human helps the spamming. Of course it is important that not all moin sites use the same questions - this is the reason why we do not distribute a set of questions, but advise wiki admins to make up their own (and not share them).

• You only need to answer the question if you save a page (not when previewing) or if you create an account (not when changing settings). This is not obvious, but documented. For the page editor, the simple reason for this is that we don't know in advance whether the user will klick preview or save, but we have to emit the form html for the textcha beforehand. In case of the user preferences, it is just some code that needs to improve.

Registering

• Yes, a register link would be nice. There has been some work in 1.7 for better userform modularization, so I guess, sooner or later, we'll have one.

Preferences

There are lots of themes on ThemeMarket. We do not include lots of them in the default distribution because that would mean we would have to test and fix all of the included one for every release, putting a lot of burden on moin development. Therefore we only provide some very basic themes to show that switching themes is possible.

Search

Currently, there is quite some stuff broken or needing improvement, we know that.

Editors

Both editors have their pros and cons.

While newcomers like the GUI editor, users used to wiki usually use the text editor because it is faster for them (and it also has less bugs because it is very simple).

Suggestions

• bookmarks page instead of extending navibar links: yes, that's a good idea (I am one of those people having lots of links there).
  • Noone implemented it yet (we have lots of ideas, but someone needs to do the code).

  • If you make a bookmarks page, e.g. UserName/BookMarks, you need one click more than with the navibar quicklinks.

  • One could also but those links into a dropdown menu, but it's similar: you need one click more.
• improving GUI editor:
  • we need help for doing this!
• talk pages: we have them, they are called "supplementation pages" and you need to enable them in your config.
• fonts: at least the builtin themes give the browser only very basic hints about what font it has to use to do the rendering (e.g. monospace or sans-serif). If you like your browser to use some great font by default, you can configure your browser to do so (and any site not fixing the fonts will afterwards use the fonts you like).