Third report from the Norvig/Thrun/Stanford/Know Labs Artificial Intelligence course

(Other posts tagged ai-course.)

Here is my third participant's report from the Stanford Introduction to Artificial Intelligence course.

For more than half my working life I have been engrossed by on-line distance learning (yes - since 1992). I've been unable to resist giving the course organisers a piece of my mind about aspects of the underlying course design, having been given active encouragement to do so. I've pasted at A below an excerpt from some feedback provided to Know It!'s David Stavens earlier this week.

The rest of this report concerns my experience of the course as a learner over the last week.

1. Nearly ~46,000 of the >100,000 (?) students enrolled to the "advanced track" of the course submitted homework assignments by the deadline. Perhaps predictably the course web site struggled with the pressure, and the homework deadline was extended by 24 hours. In parallel with this there have been times when the course web site has been unreachable.

2. During the week some biggish improvements to the course infrastructure seem to have been made and it will be interesting to see if things unravel in the run up to this week's homework deadline. My guess is that the number of submitters will drop quite sharply owing to the relative difficulty of this week's material. I hope to be proved wrong. (As an aside, this reminds me of the "healthy worker effect" in epidemiology, as a result of which a survivor population is wrongly judged not to be being harmed by the work environment. The aiclass team must take care not to rely mainly or only on the views or successes of the "survivors" in any judgements they draw about the success or otherwise of the course. )

3. The relative ease of the first week's content lulled a lot of students (including me) into a false sense of security. The past week's content - about Probability in AI and Probabilistic Inference - has been more challenging: scores will deteriorate from now on.

4. A number of well-implemented features of the still obviously prototype "LMS" that supports the course are now apparent. Each logged-in user gets a simple "progress bar" view of their activity to date, from which they can drill down to review their answers to all the quiz questions undertaken within the materials, with links to the materials relating to each question. Similarly a user can learn where they "went wrong" in their homework.

5. The impression I have so far given is of the lone distance learner "battling through" the course. But alongside the official materials and tools, there is a burgeoning substructure of informal discussion about and peer support for the course "out there" on the Internet. Some of this peer support is of very high quality. For example, here is a link to a well written piece about Bayes' Theory on the Reddit site for the AI class, with a slew of equally thoughtful responses below it.

6. I believe that the value of this peer support will rise as the course progresses; and because the course has so many participants, a (small?) proportion of whom really are particularly committed and talented, and because of the way recommendations can help "good stuff" rise to the top, to then be built upon, it looks as if a wide range of high quality user-generated support content will have been written by the time the course finishes. This example of Shirky's Cognitive Surplus in action will probably help smooth out some of the unevenness in the level and quality of the some of the course's explanatory content.

22/10/2011

[A - Narrative Feedback on AI Class to David Stavens of Know It! - 20/10/2011]

1. Longer and thinner works better than shorter and fatter. Thus if you provide by OLDL a course that normally takes n weeks delivered F2F, you need to allow, say, 1.5 to 2n weeks to do it by OLDL. This is because a big proportion of your learners are trying to combine the course with everyday life, and will struggle with the pace. Retention rates increase substantially if you take this approach.  A slower pace also allows user-generated corrections to content to be acted in leaving time for students to avail of the corrections. (Less of an issue once a course is established.)

2. Allow scheduled "catch up" weeks. That is, weeks when no assignments are expected. This allows those with other commitments to know that they can devote time to the other commitments (e.g. their kids) from time to time. Along with #1 it also allows for occasional glitches at your end (such as we've had in the last few days) without knocking the course off balance.

3. Provide the text of the homework questions alongside the video-delivered questions, rather than requiring the learner to repeatedly watch/listen to the questions being spoken by the tutor.

4. Express deadlines in GMT not UTC (minor point).

5. Put announcement Tweets into an announcements page on the course site. If the site is up the easiest way to check what announcements have been made is on the site. Searching on the tweet tag is a bit more fiddly and should be used when the site is down.

6. Have some kind of "pre-course filtering", to encourage those enrolling to commit to the course in a realistic way. In my world we did this with a human reviewed "pre-course assessment", but a check-boxes based "statement of commitment" would work pretty well. Obviously if (when!) you start charging fees you will lose some uncommitted learners, but there is separately a widely held view that on-line learning is not hard work, and, it will be probably even more in your interests to "blow away" monied timewasters than it now is to avoid the system being "overloaded by the uncommitted curious". (This is not expressed with any finesse, but you will know what I am driving at.)

7. Do not change the quirky but engaging feel of the videos.

Comments

[Clive Shepherd]

Thanks for the updates, Seb. There's so much to be learned from this venture and your comments are going to be really helpful to those who have similar projects in mind.

[Sebastian Thrun]

This is very insightful, thanks for your careful analysis and suggestions.Obviously, we are still learning how to best do this, and we are working hard on the technology and haven't gotten around some of the things we wish to do. We made a deliberate decision not to sacrifice Stanford quality or speed, so I agree this may result in more drop-outs. But at the same time we really want this to be a Stanford-level class.
Please keep your suggestions coming. They are being heard.