Fortnightly Mailing

David Hu's use of AI - including Bayes networks - in the Khan Academy's assessment systems

Some weeks ago I posted a link to an interesting post by David Hu about how the Khan Academy is beginning to use machine learning and AI in its assessment systems. David Hu was an intern with the Khan Academy, and here is a video he made reflecting on the work he did during his internship. Do not be put off by the first few seconds of the video, or the later slightly breathless sections. There is real science, clearly expressed, about the "journey towards better assessment practices".

You can also read Hu's fuller post-mortem on his internship.

Lessons learned from using Khan Academy content in a blended learning pilot

Final sentence added 3/1/2012.

In this December 2011 report [23 page PDF] Brian Greenberg, Leonard Medlock and Darri Stephens report on the "performance and engagement of low‐performing high school algebra students receiving a mix of traditional teacher-­led instruction and self‐guided instruction through the Khan Academy website".

The authors seek to compare the performance of two groups of "summer school" learners, one group taught traditionally and one group teacher-supported in largely individual use of Khan Academy content (thus "blended learning"). Both groups made substantial progress (as judged by the increase in percentage questions answered correctly on the MDTP Algebra II Readiness Exam).

The Khan Academy group is reported - with plenty of caveats, and with the rather disconcerting bar chart above - to have made very slightly more progress.

Along the way, the authors make interesting and useful observations about:

• classroom management under a "blended learning" model;
• the use of Google Chromebooks (cheapish quick-to-boot laptops that require a Wi-Fi connection to function fully) in a classroom context;
• how teachers in a blended learning environment can use data about learner performance to inform how, when, and with which learners to intervene;
• how having a plethora of systems in use at the same time or in the same school would prove problematic from a data-management point of view:

"One obvious challenge is that currently each online course or software uses its own data reporting system. For isolated pilots such as ours, these kinds of “walled gardens” can work fine. The teacher simply needs to learn how Khan Academy reports data. However, as teachers start using multiple products in a class or as schools blend technology into multiple courses, it becomes increasingly hard to make sense of all the data."

A striking paragraph in the report - which I can personally relate to having recently finished a course based on very short (40 second to 300 second) videos - concerns the value of Khan Academy videos:

"A final interesting perspective on Khan involves the value of the site’s videos. Most people are drawn to Khan based on its massive video library and Sal’s own charming and engaging teaching style. Like many, we assumed the videos would be the predominant learning mechanism for students tackling new material. In fact, the students rarely watched the videos. This result is consistent with some of the observations in the Los Altos pilot. The students greatly preferred working through the problem sets to watching the videos. Students turned to their peers, the hint, and the classroom teacher much more often than they did the linked Khan video. One possible reason is that the videos are aligned to the broader concept, but do not link directly to the problem students are struggling with. A second hypothesis is that the videos may be too long at eight to ten minutes. If students have 60‐90 minutes to work through multiple concepts in a class period, an investment of ten minutes for a single video feels like a lot. The badges and stars within Khan may also be a disincentive, as there is no immediate reward for watching videos as there is when completing streaks. Lastly, we wonder how many of us really enjoy watching instructional videos for extended periods of time. We are left curious about whether Khan’s videos need to be even more modular and shorter in duration and also about the value of video based instruction."

But overall, as Al Essa and Stephen Downes have indicated a somewhat frustrating and inconclusive study.

Current "line" on ICT and Computer Science in the National Curriculum Review

Here is a link to The Framework for the National Curriculum - A report by the Expert Panel for the National Curriculum review by Mary James, Tim Oates, Andrew Pollard, and Dylan Wiliam. Of particular interest to some readers of Fortnightly Mailing is this intriguing recommendation relating to ICT and Computer Science:

We recommend that: Information and communication technology is reclassified as part of the Basic Curriculum and requirements should be established so that it permeates all National Curriculum subjects. We have also noted the arguments, made by some respondents to the Call for Evidence, that there should be more widespread teaching of computer science in secondary schools. We recommend that this proposition is properly considered.

Two recent Association for Learning Technology documents relate to this:

• An ALT contribution to the current discussion about the place of Computer Science in the National Curriculum for England - December 2011;
• An ALT response to the review of the English National Curriculum - April 2011.

[Disclosure: I work half time for ALT.]

Final report from the Norvig/Thrun/Stanford/Know Labs Artificial Intelligence course

Word-cloud by Sea Otter

(Other posts tagged ai-course. Post originally published on 20/12/2011, with paragraph one updated to incorporate completion numbers kindly provided by David Stavens of Know Labs, and a new concluding sentence to the final paragraph.)

Along with just over 20,000 others (some 3,000 fewer than had taken the midterm exam) I completed the final examination for the free online Introduction to Artificial Intelligence course taught by Sebastian Thrun and Peter Norvig. Here is my final participant's  report from the course.

1. The final section of the course concerned Natural Language Processing. I've had an interest in machine translation for some years [e.g.]: and it was this interest that initially made me aware of Peter Norvig's work.) So for me this meant that the best part of the course came last, and if you want to gain an underlying appreciation of the science of natural language processing, it will take you a couple of hours to work through the courses 42 short videos about NLP, starting here. It is probably worth doing despite a certain amount of dependency on earlier sections of the AI course.

2. The course has been mercifully free from programming assignments: being capable of completion using pen and paper, a calculator (and on a couple of occasions a slide rule unused since 1973).  To conclude the NLP unit there were two optional programming problems, both of which could be tackled without programming. I did the second of the problems (recovering a message from a shredded version) using scissors and adhesive tape:

Moe and Chubb - Liberating Learning

These two short videos from the Hoover Institution will give you a sense of the line of argument advanced by Terry Moe and John Chubb, who are right-of-centre, union-hostile US academics with trenchantly positive views about the value of on-line learning, and about the need to make the same kinds of technology/labour substitution in education as has already taken place in large swathes of the rest of the economy.

Terry Moe

John Chubb

For more on/from both, go to Liberating Learning - Technology, Politics, and American Education.

Apples, screwdrivers and desks: a comparative review of three Stanford free online computer science courses

Guest Contribution by Gundega Dekena

[Update posted by Seb Schmoller on 12 July 2012. Note that Gundega now works for Udacity, the company that developed from the AI course. Read how she became part of the Udacity team on the Udacity blog.]

Gundega Dekena is a self taught Linux administrator and web programmer, based in Riga, Latvia. She has been studying all three of the October to December Stanford online computer science courses in parallel - Introduction to Artificial Intelligence (AI), Introduction to Machine Learning (ML), and Introduction to Databases (DB) - putting her in a good position to compare and contrast them. Gundega can contacted at gundega.dekena [AT] gmail.com, or through Google+.

Overview

Comparing these three courses feels a bit like comparing apples, screwdrivers and desks, yet I see a lot of students doing that, usually without much thought about the differences. So, this is my look (from the perspective of a student) atthe things that can be compared, and that can be learned from all three courses, for the benefit of the next batch of courses that is going to come from Stanford next year.

Food for thought. Video recording of Q&A session between Sal Khan, Peter Norvig and Sebastian Thrun

There is plenty of food for thought and a great deal to identify with - from a very diverse set of angles - in yesterday's 45 minute "Google+ Hangout" discussion between Sal Khan, Peter Norvig and Sebastian Thrun. Examples include:

• the authenticity and hence superiority of unscripted, tentative explanations, with "low" production values;
• how lectures empower lecturers not students;
• data analysis as an "underpinning process" for real personalisation;
• why their is such a mismatch between the value that a university thinks its courses provide what students think they provide;
• the challenge of combining individual study with group interaction.

 

Ninth report from the Norvig/Thrun/Stanford/Know Labs Artificial Intelligence course

Screen shot of Filip Wasilewski's Chrome plug-in explained in paragraph 7 below.

(Other posts tagged ai-course.)

Here is my ninth participant's  report from the Stanford Introduction to Artificial Intelligence course.

1. I wrote last week that the units on Games, Game Theory, and Advanced Planning had been hard work; and that the associated homework had felt demanding and been very time-consuming. Securing an adequate mark on that homework despite too many  "stupid errors" means I probably learned more than I had realised. But I also know that I was answering some of the questions from the "stored fat" of knowledge I already had, rather than from what the course had been teaching me.

Eighth report from the Norvig/Thrun/Stanford/Know Labs Artificial Intelligence course

(Other posts tagged ai-course.)

Here is my eighth participant's report from the Stanford Introduction to Artificial Intelligence course.

Last week's report focused mainly on the midterm exam, and on a conversation with Sebastian Thrun. This report conments on the midterm exam and reflects on the way the course has run this week.

First, some comments on the midterm exam

1. According to information just received from Sebastian Thrun, 23k students passed the midterm, with 85% currently falling into the B+ range.

2. There was a surprising amount of post-exam discussion between students about the substance of the exam. There was some whingeing, but not much given the very large numbers on the course.  The interesting discussion fell into three categories:

The shape of the next phase of online learning

Stephen Downes points to a 9 November 2011 Educause piece by Jarret Cummings and writes:

"I think the shape of the next phase of online learning is being defined: reliance on open educational resources, supported by some sort of automated, analytics based and competency based testing mechanism."

Yes.