Here is use a student interchange rubric. That is, rather than aim for how a faculty might use IQs, consider how a student might wonder if IQs could help them. This is a student who already had Vector Calculus but realizes -- since it is a mind-bogglingly difficult course -- that she needs to rethink what she should have gotten from it. 

Yes, she is probably a student to die for, but such students exist. Also, I'm explaining to her -- and to you -- about the technology, too. So, I will assume that she has some computer science background. Finally, I assume she is interested in hearing my motivation for having done the IQs. 

Here is the formal look of the rubric. 

KASIA SAYS: [something she said]

MIKE RESPONDS: [then, something with which I responded]

KASIA GIVES HER GOALS: In addition to helping teachers assess their students' understanding on class topics and efficiently provide relevant follow-up comments/explanations, I can see how IQs could benefit students like myself who would like a way to analyze their own understanding, or lack thereof, of current topics. Not to mention to demonstrate a way to systematically approach solving problems when stuck. 

MIKE RESPONDS: You already have intertwined the two components. Suppose we momentarily take as given that the IQ question format has effective ways of parsing significant pieces of student responses, across the class. 

Then, you can employ the IQ technology to fine-tune where the tough topics stumped students. This is with the IQ reader, portfolios of interaction, and archiving tools. It is a fairly elaborate system. Yet, mostly the tools are cobbled from standard grep, cat, ls, cut, ... programs. 

Since you have computer science training you can imagine that using modularity well can be effective in giving tools to do what people now call data mining. 

Especially, I was seeking those points where texts have small treatments because the issues are conceptual. Yet, most students will have difficulty engaging them at the level they require.

In math/science/engineering/computer science -- at least on the tougher courses -- it is easy to give examples where texts haven't quite turned the topic into something that will stick in the students' minds. It is as if the topic has layers -- three or four -- and the connections between the layers isn't in the text. How did you go from here to there? 

Now I say some of my background that shows what motivated me to develop IQs and their related technology. It is a digression. 

Between the time I was 18 and 21 I was an electrical engineer working in aerospace companies. This was me being patriotic, influenced by the recent launch of Sputnik. Today even the president looks back on that period with nostalgia. Yet, it wasn't the unmitigated success he imagines it was.  

What I found was that those around me -- working engineers -- with rare exception, were afraid of crucial pieces of material they had in courses. So afraid, they didn't want to see it again; to the extent they denied it had any relevance. My response was "Right, who would need to understand an output dependent on 2 or more inputs." 

That phrase was the point of vector calculus. In physics, vector calculus is the core of the study of electricity and magnetism, and many other subjects. Those are both sophomore courses. Extremely difficult sophomore courses, without proper forewarning from the courses prior to them. So, that background is why I picked Vector Calculus. 

KASIA CONTINUES: Is it possible that I could help make IQs available for my college? 

MIKE DOES A BRIEF VERSION OF THE IQ PHILOSOPHY: Let us start with the format philosophy. 

What IQs do is to deliver questions in the style of teaching students about "step-thinking." They showed how an "expert" relies on the basics: 

by tying some gain in knowledge on a problem to some basic statement from the course. [If I could have hi-lighted that I would have.]

The philosophy was based on the following idea. Consider a serious problem in a course as like a 400 lb weight that must be moved from the floor to a higher platform. Students often think that "if you are smart" you just lift it up. 

Of course, that's ridiculous. What makes "geniuses" is a recognition that you need mental organizing and cutting tools. Cutting tools to slice the 400 lb weight into smaller pieces, that can be lifted, and organizing tools that reasonably efficiently allow you to put the object back together when its pieces are in place on the platform. 

Some geniuses actually made those tools. When I was a kid I thought of Lagrange, a genius, who at the end of the 1700s gave us the tools to do the gravity slings that NASA and JPL now use to fling satellites out to Pluto. My actual mathematics research plays on the work of other geniuses, but Lagrange still has a place in my heart. It pains me that so few understand what he did 200+ years ago.  

There is, however, a problem for almost all students.  Even if they memorize the "steps" as given by the text, they can't stop thinking of all the steps at one time. What makes the steps effective is doing them one-at-a-time. Then think of the next step, mentally disconnected from the last or from any others. 

So, I get to the heart of what the IQ question format does. It leads students to develop enough mental energy to disconnect the steps; they are doing them one-at-a-time. This requires  training. The value of it is that even students can attain \ an experts' approach to solving problems.  You get to relax, on the various steps to your goal, long before you have completely solved the problem. 

By saying this I've tried to separate the crucial IQ question philosophy -- a fit subject for psychological/educational research -- from the IQ technology. The latter is there to capture the places where students get overwhelmed. 

KASIA NOTES THE COURSE EVALUATION PROBLEM: However, I've often considered what you pointed out in your article. There seems to be a correlation with challenging courses and less happy course evaluations. 

I suspect this might pressure teachers to aim for short term satisfaction at the expense of more demanding courses (to the detriment of the minority of students who value the latter).

MIKE NOTES THIS WAS ZUCKER'S ISSUE: This came up while Steven Zucker and I were discussing my initial response to his article. He kept on asking me, "What happened to your course evaluations?" 

He saw that I was hiding that point, and when I told him what happened -- they dropped precipitously -- he insisted he wanted me to respond to his article, and to emphasize that point. 

Kasia, your statement is an understatement. For example, my colleagues often gave up with their classes. Some of them -- three of the best of the younger ones, actually, and I would be willing to name-names on this -- came to me pretty assured they were doing everything right. They were certain their class problems were because their students had a bad attitude. 

They were asking me what I would do about that. (This is an aside, and it sounds like I am bragging, but it is crucial to the point I'm making. They went to me not because of the IQs, but because I was well-known as the guy most likely to follow even very difficult colloquia or seminar talks. I had also praised certain talks they had given.)

Here is what I did and then what I discovered. 

I went to their classes -- I only had to go to one apiece. At colloquia I usually took at most a few resonant notes. Unlike that, when listening to their -- undergraduate -- classes, I took detailed notes. On  how they phrased their topics, how they solicited student responses, and then -- in each case -- why it was impossible to get a satisfactory response from the class. 

Now, here is THE crucial point. In each case, after the class, I was able to show them they went so fast, that several of their actual statements and questions were mathematically incorrect. 

It was like they almost couldn't hear what they were asking. The best lecturer of those three was teaching a subject -- a junior course -- in which he wasn't sufficiently expert himself. He was actually feeding the students' misconceptions. Not continually, but at the crucial places. 

All instructors are inundated by their jobs, and if they are researchers, at a place like UCI, they realize they get little credit for teaching well anyway. The administration doesn't care if their students do well. Only that they don't complain about the courses. 

Faculty research counts much more. Add to that, that teaching is hard, but most faculty don't realize just how hard it is. 

After I -- as nicely as I could -- explained these serious difficulties to my colleagues. (I had the goods in my notes, and two of them accepted them and were stunned) I asked myself why didn't that happen to me? 

Recall the business in my article called P(roblem) O(f the) D(ay). After each class I echoed back what I had said, and what students -- usually naming them by name -- had responded in a semi-personalized e-mail to each of them. (Those semi-personalized e-mails are part of the IQ technology. They were done long before businesses learned how to put people's names into their e-mails.) 

Zucker asked me how I tied delivering IQs to what happened in class. It was through the feedback to the PODs. They were variants on the IQ question format. Though they required energy to make them, they had a certain rhythm that became easier with time. Also,  they engaged students, and elicited telling responses about student difficulties that went into each student's portfolio of their interaction with me. 

Responding collectively to the POD responses kept my classes (me) exceptionally honest. It showed me where -- at times -- I was using higher knowledge, or vocabulary, than legitimately suited what I had taught the students. 

KASIA CONTINUES: One question point in your article I'd like to understand further: "Compatible with Zucker's experience, the IQ reports led students to see they could work harder. Many of my students (certainly not all) interpreted that as a negative." 

MIKE LAMENTS: By showing them the standard to the course, the material that made the course really work, I was also revealing that they had some bad mental habits. 

For example, in introducing a new subject, I would be feeding them how something works, and going sufficiently slowly to guide them into the right channels. 

Yet, it is like giving directions to someone. When you do that you aren't at that time screening out the irrelevant issues (signposts) that an expert will know to dismiss when searching the right path. When students try something similar, they cannot at first know there is considerable structure to those little nudges. The small comments I made with pen at the whiteboard to get someone to "help me" with a possible next step. 

That is, they had not planted in their minds how they retrieve from memory the step thinking that engages them as I was engaged. This is the hardest part and it applies the most to the hardest topics. There is no way, even when I am extremely expert on a topic, that I reconstruct it from scratch. 

I go back to the principles I learned, say, from Lagrange. Principles used when I  practiced my understanding. That understanding is about extracting information using resonant basics, versus the possibility that I can't extract sufficient information to solve the problem using those basics. 

In undergraduate courses, having students do this requires students finding a way to recall the basic principles and then to decide if they are relevant. The PODs and  IQ question format emphasized that. 

That meant students had to go back to the book; to realize that -- without me reminding them with my several mnemonically phrased devices -- they hadn't actually found a good way to recall the basics. 

This called for them, long before a test was coming up, to restudy. Yes, with help, but something few were accustomed to doing. To restudy the material they actually (mistakenly) thought they already understood. 

KASIA ASKS AGAIN ABOUT STUDENTS WORKING HARDER: Does this mean that the students realized that they probably could stand to put in more effort. Or,  did they rather learn how they could work harder through questions and help options that kept them working on a problem longer than they would have done alone? 

MIKE RESPONDS: I aimed at the latter approach (Zucker at the former). The point of the PODs and IQ question format was to not have the students spinning their mental wheels for days. 

That would have had many (maybe most) going over and over the same territory. I avoided that by taking them -- using IQs and PODs -- back to the text's basics and reexamining how they interpreted it. 

Those who actually did this, realized it was forcing them to work harder and longer. Often, however, those who came to solving a problem didn't realize they likely couldn't have solved it on their own even if they had worked on it for days. 

That is because much of that kind of work is like being in a rut, and not knowing it. I think many  gave themselves credit, and not the IQ philosophy and PODs. 

Both of those tools were engaging, as many students admitted.  Yet, ultimately, as I said in the article, the students who had the IQs, no longer had this  professor who they viewed as commiserating with them about how hard the course was. instead, they got this guy who insisted -- without actually saying it -- that they could understand these things if they engaged the PODs and IQs. (The latter were mandatory, the former not.) 

In class, when I showed that many students were getting things -- these were tough topics -- those who either weren't engaging with the PODs or hadn't really believed they would work, might have resented such demonstrated student success. 

KASIA'S FINAL TOPIC ON STUDENT ATTITUDE: I ask specifically because I am trying to understand how/what they interpreted that as a negative. This is interesting to me both from a teaching point of view and from a student's point of view, attempting to understand the way the majority of my classmates seem to feel.

MIKE'S RESPONSE: After each class was over, I did get testimonial from students of various kind. UCI is not a small liberal arts school, so it was not common for students to respond to teachers later. I think I got more of my fair share of this. 

Some of that was flattering, but still anecdotal; my published article gives an example. IQs could use a dose of followup -- hopefully better than the after class evaluations typically given at a place like UCI. 

I cannot confidently predict what would happen if someone were to use them at a school where backgrounds are weaker, and students might have lower goals. 

These later e-mails to me had one common theme: "I was able to use [phrase that I had repeated in class a hundred times] and I showed [some person -- often a classmate or the teaching fellow] how to solve this problem." 

The particular curricular issues that would come up (like, find a function that allows you to express the English sentence as an equation) is relevant. Still, not so relevant as this. 

What I was saying was that someone a long time ago taught the world to understand changing qualitative to quantitive. You [the student] didn't think of this. Every time you use this, you are honoring a long tradition. Please humbly do that, because your understanding isn't just about you. 

KASIA CHANGES DIRECTION:  You didn't answer one thing. Suppose I assuming you that the IQ format is effective. How can I use it for myself? 

MIKE EXPLAINS HOW IT WORKED FOR HIM: As a professor I was thinking how I could bring to the students an experience that was similar to how I educated myself to "get into" problem solving. Since, however, I had always done that as a faculty member, I recognized after many years -- often despite excellent evalutions -- that I couldn't turn students into me, without revving up their motivation. 

That was where the IQs -- technology -- came in. But there is one extra aspect of it. What the IQs and PODs do is give every student a private office hour with less pressure. Without the technology the faculty couldn't do it. Yes, if you provide anything like what IQs and PODs give, without the technology, you will kill yourself. (I emphasized that when I gave public lectures on IQs.)  

KASIA INTERJECTS: How would I get the actual technology working on my own computer? 

MIKE TELLS ABOUT THE TECHNOLOGY: It is possible, though I haven't yet written the manual for it. Each program has good documentation within it. Still, I haven't written an overall interface for the system, and guide to using it. Each of the individual programs work on any machine (including any modern Mac) running UNIX. 

I learned many things from IQs AFTER they were instigated. No doubt, however, the idea of making individual office hours was part of it. As a student I was fearless in asking what might have been stupid questions. Even when some instructors would intimate they were (stupid), many would later -- shamefacedly -- admit they had made a mistake. By contrast I saw how few students could easily ask questions in class and then, hear the answer. 

What I figured is that IQs and PODs would break the ice. Students would get into question asking modes -- not always the same style as mine -- and get over the stage-fright of asking the professor questions. 

That turned out to be absolutely correct. After introducing IQs and PODs, questions in class improved. Part of my documentation of that was the PODs. I felt I needed to document the effect of PODs and IQs. 

In that case, you want a reasonably nuanced record of what went on between the student and faculty that you can share briefly and elegantly with other students (or faculty). The technology was excellent at that. 

Also, the technology increased assessment aspects -- grading is neither fun nor easy when it comes to reading student writing -- by an order of magnitude. (That could still use many further improvements.) 

Yet, one aspect you might not get from the article was the significance of the e-mail personalization. That was what had students believing it was like their own office hour. 

It took little to give that impression. Nowadays even on-line advertisements do something like it, interspersing a use of a person's name inside their text as I just did. Mine, however, had other serious personalization's. 

My knowledge of students and data for personalization came from student responses to the PODS, and that in my classes they did individual class projects. For Vector Calculus with 80+ students, they were small projects, but still individualized. For classes with no more than 25 students, they were serious projects, with classroom presentations, but always done in teams. 

Communicating between those teams was the tour-de-force of personalization. The work of putting those together, including the personalization, without technology would have been beyond overwhelming, since these were ordinary classes in which I had to cover a good portion of a curriculum. 

The e-mail technology and the personalization that went with it, had exciting interclass communication elements.    

Add to that how I referred to what happened in class daily -- including using student questions from class -- in the PODs. My guess is that the PODs enhanced tremendously the classroom experience for engaged students. Of course, I was available for students in office hours. Yet, if they came, they would have found that I was using that time for writing the PODs.   

There were, however, serious cognitive lessons on the dynamics of learning and recalling that I hadn't anticipated. No, I should say, that astounded me. I gave one in my published article on IQs. I have two graphics on my web site illustrating those. 

KASIA CONCLUDES: What should be done with the IQs now?

MIKE'S SUGGESTION: Weaving the two -- IQs and PODs; cognitive and interpersonal -- together in a manual/book might add human elements rare for the usual programming manual. If I were to start someone on using the programs, I would start with the personalization and e-mailing behind the PODs. 

The IQs should have an html interface, as I said in my notices article. It should have such an interface for three pieces. 

a. The student taking the IQ -- such a student would click on a URL in the e-mail, and the results would be handled by CGI programs. 

b. The faculty creating IQs -- curricular expertise is needed to make the IQs, but it would transparently handle the technology putting it together, including help files for extra student help. 

c. For an enterprising faculty who wants to learn about IQs. 

Though I can't see they would be more difficult than what is there already, none of those interfaces have been written. The IQ readers and other helper programs are crucial. I'd have to think for awhile how they would relate to html pieces.