During this spring I have taken part in the following:
Grieved over what Navigation 2.0 does to the Quiz editing UI I designed and usability tested in summer 2008 (discussion, tracker item), throwing away much of the tested design without testing the changes (update, later the same day: this is still being discussed; see the tracker item). How I wish I could be there when core moodlers design and decide about big UI-related changes like this Navigation 2.0! (spec for the Quiz change)
- Worked with Eloy, Sam and others to get Moodle’s first proper Wizard in. Yay! Sam did a great job of reacting to my feedback quite late in the process, though apparently he did not know of the specifications I had made for the wizard last autumn.
- Discussed Database module exporting UI
- Discussed Forum search UI with Anthony
- Commented on The database module’s UI on Anthony’s request – hope to have a chance to do some research and help in redesigning the Database module UI at some point
- Added MediaWiki categories to Moodle UI guidelines for faster browsing, cleaned up the guidelines so unfinished things are less in the way of usage
- In the Moodle sprint, took a look at the toolbar of the rich text editor; reviewed all the comments the question had received since last summer, and created a new patch. Petr told me it is okay to assign the bug to him, as he will be working on the editor before Moodle 2.0 release.
- Commented on issues, such as MDL-6820 MDL-20461, found on the CANnect Accessibility report by Randall Hansen (MDL-20409)
- Commented on the new Moodle 2.0 Dock navigation’s interactive behaviours: MDL-21529
Also, jotted some usability related bugs down:
Part II to What is a course & the tools for having a great one, with exciting new UI design for an old challenge is coming out at the beginning of June as soon as I can find any free time. Stay tuned!
Tim Hunt 5:30 pm on May 20, 2010 Permalink |
Err… I’m confused. We asked for help, then wrote a public spec, which you commented on. In other words, you were there “when core moodlers design[ed a] big UI-related change”. What are you complaining about?
Really, if you want people to work with you on usability (which is good) then stop bloody whining the whole time, and concentrate on working constructively with people. (If you like, think about the interface you present to developers ;-))
Olli 7:24 pm on May 20, 2010 Permalink |
Thanks Tim for your quick response. And Sorry – I was being a bit dramatic. It is a stressful time for me is all I can say.
Whether I participated in Navigation 2.0 is not an easy question, though. The work did not seem very accessible to me as someone who does not actively engage with the development/architectural side of it. I really did not anticipate the changes that were coming to the Quiz UI, so it seemed awful lot was happening quickly without really being thought through.
And I do understand that making it easier to approach from anything other than a systems point of view is not easy, since that is the world you think in, and as such, that is good, of course.
Tim Hunt 9:55 pm on May 20, 2010 Permalink |
I think you have to distinguish the different ways that closed and open source development happen. Clearly you have been taught that when doing closed development, you should do user testing with a carefully selected group of real users before you release anything.
In the Moodle world, we are having a number of preview releases in the lead up to the final Moodle 2.0 release, and we are inviting a wide range of volunteer users to do all sorts of testing. Usability testing is absolutely one of the things we want right now, as is expert review of the UI. There is still time to act on the findings.
On a lighter note (and I am not sure whether this counts as a usability thing or not!) did you see; http://www.seointern.com/blog/jack-russell-terrier-raises-conversions.html
Olli 10:39 pm on May 20, 2010 Permalink |
You have just touched an area, on which I have been digging HCI articles for the past 18 months. :) Sure, user feedback is important – however, it is an old truism of usability (since back in the 80′s-90′s when the same discussions were had in the closed source world) that users can not describe their own behaviour accurately, but user observation is crucial. See, for instance, the top articles of http://www.google.fi/search?q=dont+listen+to+your+users .
The question has nothing to do with open/closed source per se – developers have for a long time tended to fool themselves they can just ask people to comment, and then imagine their designs have reached *any kind* of satisfactory level of usability. Even if you get feedback from the users, you are still needlessly exposing them to horrific designs just because you did not want to go to actual users and see what actually happens when they use your product.
This is not to say that HCI does not have a lot to learn from open source, and many late articles seem to agree that user centered design methods need to get quite an adjustment to suit the informal model of open source development. Nonetheless, it is clear that no amount of user feedback is going to compensate for the lack of actual design driven by user observation/research.
That blog post touches another very interesting topic indeed. Getting quantitative data on user behaviour, I believe, can help in doing design, like this EuroIA presentation from User Intelligence I saw last spring suggests. Quant can describe what happens but not why; qualitative (usability testing etc.) can explain why it happens but is a lot of work if you try to do it with a lot of people. So you look at the web analytics data and see a trend, identify a problem users are having; then you usability test the feature that seems to be problematic to understand the problem.
http://www.euroia.org/Programme.aspx#combining_methods (apparently a similar presentation in this year’s EuroIA)
On the other hand, Spool labels web analytics as a voodoo method without further explanation – well, it’s not the only way in which the video is a bit controversial, but it is still well worth the while.
http://www.youtube.com/watch?v=WCLGnMdBeW8
Part 2 is interesting since there it says what might actually work (starting 6:12)
http://www.youtube.com/watch?v=Zpy7PKIhXgU
And then, using web analytics in Moodle is hard due to privacy issues… but perhaps, someday, if we asked the users permission whose anonymized data we want to look at…
Appreciate the discussion, as always! (And this commenting box is too small.)
Olli 12:56 pm on May 21, 2010 Permalink |
(sorry for the lecture. it’s just so great when someone is listening. obviously my motives here are entirely egotistic. :)
And, well. I am a bit surprised that you don’t see the benefits of usability testing, though you have been commenting on the data gained from it for at least for one summer now (not sure how much you followed last year’s testing). Formative testing that just aims to find bugs, like the stuff that I have been doing for now, is next to miraculous itself: your eyes get opened again and again on how the other person’s point of view is different from your own in ways you could not have imagined. Because those things are so obvious in those people’s worlds, they don’t mention them in their feedback: after all, they do not see any other way *they* could see things? There is a wall of perspective between users and developers, it just does not get crossed unless users are observed in the tasks we want to understand, reaching the goals we want to understand.
Seems to me http://www.pilpi.net/software/moodle/2009/08/19/quickie-usability-testing-file-upload-and-forgotten-password/ is filled with such understanding, most of which you just would not find without observing users. And Martin’s reaction seems to confirm that in the tracker.
There was also plenty of embarrassment, as there always is, during the tests: users think they are stupid for not understanding the UI. Why would they come and tell developers there is something wrong with the UI, when they feel that it is actually their fault? Yet it is this very thought even the users have that has to be fought if we care to have a UX of any decent quality. This is also the reason you need to tell users that we are testing the program, not them, so they get to relax.
Tim Hunt 1:45 pm on May 21, 2010 Permalink |
Who says that I don’t have an appreciation of the benefits of usability testing? I do.
What I also have is a huge list of development tasks. And I know that doing these things will make the Moodle better. Not perfect, just better.
I also have a belief that making Moodle better, one step at a time, is the only practical way to proceed. No one can tell you exactly what ‘perfect’ is. It is only as we get closer to it, one step at a time, that we can refine the direction of our steps.
I don’t have ready access to users. I can only do corridor usability testing. (Well, open-plan-office usability testing, and I do.) (And Moodle HQ has almost no access to users, although Martin seems to be very good at reviewing and improving the work of his other developers, at least as a first pass of user testing, and Martin recruited Tomaz, in part to have someone who is more a Moodle ‘user’ than ‘developer’ in the office.)
The Open University does have a usability lab, and invites students in for real usability and accessibility testing from time to time, but that is done by other people. We just get the results.
Moodle has a large community of dedicated users and developers. Some others of them have access to users and can do usability tests, and report back.
So, anyway, I have concluded that the most productive use of my time is to concentrate on doing development. I think that usability is one of the key non-functional requirements for software (along with performance, security, accessibility and internationalizabiliy – the other non-functional requirements listed on http://docs.moodle.org/en/Development:Coding). Therefore, as a developer it is my responsibility to know and use the basic usability heuristics.
But, beyond that, I don’t have time. So I will rely on other in the community to actually do user testing; provide more expertise when I hit a tricky usability problem; and so on.
Olli 12:47 pm on July 5, 2010 Permalink |
Thanks, Tim.
I absolutely agree with you on that developers should have a chance to work on their main tasks. Someday, I hope to have the same privilege to usability practitioners – so we can work one step at a time also in terms of the user experience. The goal should be to reach an even remotely similar pace to the steps of the software development – at the moment, usability thinking about user goals on the level of the actual UI is seriously lagging behind the actual development. And man, does that show.
I understand that the OU is doing usability testing. It seems tricky to me though that most usability work done on Moodle is not visible to the community. There seems to be a danger of moving the actual decisions from the community to committees in closed rooms. To me, saying Moodle is open source is less and less credible, the more of the thinking and research is behind closed doors. I guess you agree about this though anyway.