[return to Telelearning home page]


Curriculum Evaluation : A Case Study

Hugo Dufort, Esma Aïmeur, Claude Frasson, Michel Lalonde

Université de Montréal

Département d'informatique et de recherche opérationnelle

2920, chemin de la Tour, Montréal, Québec, Canada H3C 3J7

E-mail: {dufort, aimeur, frasson, lalonde}@iro.umontreal.ca

Fax: 1-514-343-5834

Tel: 1-514-343-7019

Abstract: In the field of Intelligent Tutoring Systems (ITS) the organisation of the knowledge to be taught (curriculum) plays an important role. Educational theories have been used to organise the information and tools have been developed to support it. These tools are very useful but not sufficient to edit a large curriculum. We need rules to help preventing coherency mistakes. In this paper, we propose a set of rules in order to reduce these errors.

In this paper, we report on two experiments. The first one seeks of determining some rules which we shall use to improve an existing curriculum. The second experiment uses these rules during the construction of a new curriculum in order to prevent initial mistakes.

Key words: Curriculum validation, knowledge structure and representation, design issue, authoring shells and tools.

  1. Introduction



    Surprisingly, it is difficult to find in the ITS (Intelligent Tutoring Systems) literature in-depth discussions about the success or failure of ITS. It is even more difficult to find about the reasons (or the criterions) that motivate their outcomes [McArthur et al, 1993; Koedinger & Anderson, 1993]. Although some studies have been conducted on the efficiency of the pedagogical strategies [Aïmeur & Frasson, 1996], and on the use of ITS in a class [Anderson et al, 1990] or in laboratories [Mark & Greer, 1991], we remark, that the motto in the literature seems to be: develop, and later evaluate.

    Nevertheless, as we can see in most software engineering books, it has been proved for a long time that the cost for correcting an error committed while developing software increases linearly in time [Pressman, 1997]. Keeping that in view, it would pay (or at least it would be logical) to implement software quality control techniques in the development cycle of ITS. This problematic has been raised before; in the knowledge transfer process, it is possible to use automated evaluation techniques to make sure that the computer scientist does not diverts from what the pedagogue stated [Millet & Gouardères, 1996]. But before going any further, we need to ask ourselves: what are we searching for, exactly? Is it possible to define what we mean by "quality in ITS"?

    In this article, while focusing on the curriculum aspect of an ITS, we define three axes upon which a curriculum can be evaluated. We then pinpoint particular properties and restrictions of our sample curriculum that, when defined in a formal way, are useful to its evaluation. Using the results of the evaluation, we propose a construction methodology that permits a better quality control and we validate it by building a new curriculum from scratch.

  3. Evaluation of ITS
In our search for a generic framework for curriculum quality evaluation, we faced a difficult question: are there some aspects that are present, and important, in any curriculum model? There are almost as many curriculum models as there are ITS, and each one uses a different instructional theory [Murray, 1996] for the knowledge representation. We chose to deal with this issue by using an indirect approach: instead of imposing strict guidelines, we defined the quality of a curriculum as: the conformity to the developer's original intentions. Even with this broad definition, though, we still need to classify these intentions.

In linguistics, and in natural language treatment [Obermeier, 1989], it is common to see a text analysed upon three axes: syntactic, semantic and pragmatic; in our opinion, it is possible to use them when analysing a curriculum. We classify the developer's initial intentions on three axes (figure 1a), which are: teaching goals, the respect of a pedagogical model and the data structure recognised by the ITS. Quality is measured in terms of conformity to each one of these axes.

Figure 1. Three aspects of curriculum quality.

The three axes (or aspects) are defined as follows:

Obviously, errors in the data structure will affect the implantation of the pedagogical model, and similarly errors in the implantation of the pedagogical model will have an influence on the organisation of the material to be teached. The three axes can be seen as three levels of a hierarchy, too (figure 1b). The curriculum we have developed had the following three axes of intentions:
  1. Curriculum Model
The curriculum in an ITS is a structured representation of the material to be taught. Today, the curriculum is often seen as a dynamical structure, that should adapt itself to student needs, subject matter and pedagogical goals [McCalla, 1990]. The CREAM model [Nkambou, 1996; Nkambou et al., 1996] structures this material into three networks: the capabilities network, the objectives network and the didactic resources network. These three networks can be combined in a transition network structure called CKTN (Curriculum Knowledge Transition Network), shown in figure 2.

Figure 2. Architecture of CREAM.

For each of the networks, the curriculum editor leaves choice of the pedagogical model free to the developer. This flexible tool proposes Bloom's taxonomy as the default choice, but other taxonomies can be added. In this article, we pay particular attention to the objectives network. These can belong to each of Bloom’s six cognitive categories. More specifically, an objective describes a set of behaviours that a learner must be able to demonstrate after a learning session [Nkambou, 1996]. The emphasis is therefore placed on the results expected rather than the process.

There are several forms of links between objectives:

CREAM does not propose a method for the construction nor any restrictions on the structure of the objective network. Such freedom is given to the developer in order to make the model flexible. The experiment presented in section 5 is an exploration of the effects of this freedom on the development of a curriculum and the consequences on the quality of the curriculum thus obtained.
  1. Bloom’s Taxonomy of Objectives



    Bloom [Bloom, 1956] sought to construct a taxonomy of the cognitive domain. He had multiple pedagogical goals: identify behaviours that enhance teaching, help pedagogues to determine objectives, analyse situations in which cognitive categories are used. He separates the cognitive domain into six broad categories organised in a hierarchy. The objectives belonging to a category are based on and can incorporate the behaviours of the previous categories. We will show later in this article that this detail has a great impact when one constructs a network of objectives. Figure 3 illustrates the categories of Bloom’s taxonomy of objectives.

    Figure 3. Bloom’s taxonomy of objectives

    The taxonomy is described below:

    Knowledge Acquisition: This category includes the behaviours where memory plays an essential role. The expected behaviour during the recall of knowledge is very similar to that during the preliminary acquisition of that knowledge. A likely objective of this level is knowledge of the base vocabulary of the domain, of the laws, of the criteria and of the theories.

    Comprehension: When one presents some material to a group of students, one expects that they will know the material and will be able to use it or some of the ideas within it. Three basic behaviours relate to comprehension: transpose (translate the communication into another form or other terms), interpret (judge the importance of, and draw conclusions form the ideas and of the links which unite them), and extrapolate (appreciation or predictions based on the comprehension of tendencies, currents and conditions described in the material).

    Application: The application category is situated on a step beyond comprehension. Placed in front of a new problem, a learner shows having mastered this category, if he can determine the resolution method unaided. For example, if one asks the learner to solve a complex physics problem which includes gears and pulleys, and he has never seen a similar problem then, he will have to establish parallels with previously solved problems, and therefore apply laws and formulas.

    Analysis: In the analysis category, emphasis is placed on the capacity to decompose the material in its components, to understand the links that exist between these parts and to understand their organisation. There is no exact boundary between analysis and comprehension since the latter deals with the content of the material and the former deals with both the content and the form. Analysis can take several forms such as research into components, research into relationships, research into the organising principles.

    Synthesis: Synthesis is presented as an operation in which one combines elements so as to construct a structure which did not exist previously. Both the capacity to judge, and creativity are put to the test during synthesis. The production of a personal work, the conception of a plan, and the development of abstract relationships are all part of the activity of synthesis.

    Evaluation: Evaluation is defined as the formulation of judgements on the value of ideas, of works, etc. It implies the use of personal criteria, standards, explicit, and free criteria. The activity of criticism is a form of evaluation.

    Bloom’s theory also describes for each category of objectives a list of sub-categories that permit an even more precise classification. Even though this part of the theory is defined for the curriculum model used, we have omitted it during the analysis. In this article we will content ourselves with examining the base categories.

  3. Evaluation of a curriculum



    In this section, we present a curriculum based on the CREAM model which has been developed between February and May 1997. Systematic methods are then presented in order to evaluate the quality of the objectives network and to correct the detected errors. A fragment of the curriculum thus obtained is also presented.

    1. Development framework



      In order to discover which were the most likely errors during the development of the curriculum we have worked with a high-school teacher to construct a course for the teaching of the Microsoft Excel spreadsheet. We have chosen Excel because it is a well-known tool and is often used by university students and it starts to be teached at high-school level. In addition, the supervising researcher had ample experience in the domain. For four months the teacher used the curriculum editor to construct a capability network, an objective network and a pedagogical network that makes relation between capability and objective. We have used the objective network for our analysis since it was the most complete at the time of this study.

    3. Produced Curriculum



      The curriculum produced by the teacher was not complete at the end of the given time period; it covered only certain aspects of the material. Table 1 shows the essential aspects of the course and their completion level (we have regrouped the objectives into six important aspects for reasons that we will clarify in section 5.3).

      Table 1. Completion levels.

      Aspect Completion level Objectives kept Objectives deleted
      Base tools high 21 5
      Graphics medium 17 4
      Edition tools low 12 2
      Format/Attribute medium 27 1
      Database tools high 13 0
      90 12

      Some objectives were cut from the curriculum since they had not been yet linked to the other objectives (about 11% of the total objective network). This rate was higher in the other two networks and this impeded the analysis.

      We noticed that there was no correlation between the number of objectives cut and the total number of objectives in a given area of the material. This is probably du to the student’s methods which are discussed in section 6.

      Figure 4 shows a fragment of the curriculum before the first step of the corrections. This fragment contains eight objectives, some of the application category and others of the acquisition category. We noticed that there are no is-composed-of relations in this fragment. In its original form the curriculum contained mostly prerequisite links.

      Figure 4. Fragment of the initial curriculum.

      This fragment contains numerous errors that are not easy to identify at a the first glance. It is easy to get confused by the high number of nodes and links, so the only way to manage to find error is to built it again in part, in this case it is possible to make as errors as the first time, but not necessarily at the same place, this will cause more confusion. So, if we want to see things more clearly, we need to use new tools or new rules. In the following sections, we will show some principles which permit the systematic detection of errors, the evaluation of the quality of the work and therefore, the validation of the curriculum.

    5. Entry points
Teaching any material must be done in a hierarchical manner. The structure of the objective network seems to suggest this since some objectives are at the peak of the composition relationships. What are the characteristics of a final objective? Intuitively we can describe a final objective as an objective which: If we examine a curriculum from the data structure point of view, it appears to be an oriented multi-graph. In this case, the objectives are points where one can begin an exploration. We have therefore called them "Entry points". In order to experiment this characteristic in a more formal manner we use first order logic to define the following rule:

Rule(1) " x : (entryPoint(x) Û (Ø$ y : (is-prerequisite-to(x,y) Ú is-composed-of(y,x))))

The identification of the entry points in the initial curriculum has allowed us to notice that the lack of formal development methods has been directly translated into a lack of coherence in the structure. We must ask ourselves of each entry point if it is really a final objective. For example, in Figure 4, the objectives masked and locked are identified by rule (1) as entry points but they should not be so. On the contrary, the objective attribute [ac] should probably be a entry point but a prerequisite link stops it from being so. Of course this process is somewhat subjective but it is a guide that makes the developer asking questions. The developer must constantly ask if the structure being constructed corresponds to what he is trying to express.

The objectives which have been retained as entry points in the curriculum base are: Base tools, Graphics, Edition tools, Format/Attribute, Formula, Database tools. Table 2 shows the extent of the changes made to the curriculum just after taking the entry points into account.

Table 2. Changes in the first phase.

Type of change Quantity Ratio of change (on 90)
Relation changed destination
Relation changed type
Relation was inverted
Relation was added
Relation was removed


The addition of a relationship was the most frequent chance. This may be surprising at the first glance, but we have discovered that the curriculum lacked coherence is usually caused by missing relationships. Of the 17 added relationships 14 were of the is-composed-of type. This is du to the fact that when the structure of the curriculum lacks coherence, it is easier to trace the prerequisite links than the composition links (this shows a lack of a global vision).

Often there existed relationships but they were not of the right type. It is not always easy to determine if a relationship is one of composition or of the prerequisite type. A difficult introspection is often needed. The introduction of entry points makes this choice easier but the difference remains quite subtle. The other changes (changed destination and removed) target more serious errors, often due to a lack of attention. For example, a prerequisite link may has been drawn in a way it short-circuits a composition link.

    1. Semantic Restrictions
As we have seen previously, the objectives in Bloom’s taxonomy are organised in six levels and the competence specific to each level can necessitate competence at previous levels. One may ask the following question: does this fact influence the structure of the curriculum? We answer affirmatively. Let us observe the two types of relationships presented in the objective network: The following rules which purport to the semantic aspects of the curriculum were obtained:

Rule(2) " (x,y) : (isComposedOf(x,y) Þ category(x) ³ category(y))

Rule(3) " (y,x) : (isPrerequisiteTo(y,x) Þ category(x) ³ category(y))

It is important to keep in mind that the second restriction must generally be respected; in the case of a violation, a warning should be issued not an error. Table 3 shows the modifications made in the curriculum by the application of these rules.

Table 3. Changes in the second phase.

Type of change Quantity Ratio of change (on 105)
Relation changed destination
Relation changed type
Relation was inverted
Relation was added
Relation was removed

At this point of the validation process, it is important to be cautious so that the modifications made do not invalidate those made during the first phase. We observe that despite the extent of the changes being less important here, the changes do affect near 10% of the links in the curriculum. The most frequent modification was the inversion of a link. In all cases the change concerned an objective of the application category which was placed prerequisite to an objective of the acquisition category. Most of these errors were detected during the prior step.

If we add the total number of corrections made to the curriculum during both phases we obtain 42.8%, which is much higher than what we expected (we expected an error rate of approximately 20%). Figure 5 shows the fragment of the curriculum after the two phases.

Figure 5. Curriculum fragment after the corrections.

In this curriculum fragment the main changes were additions: four is-composed-of relationships were added. These changes were justified by respecting rule (1) concerning the entry points and by the desire to complete the attribute definition. The two attribute objectives had their relationships changed in order to respect rule (3), and in order to make the attribute [ap] a entry point.

  1. Lessons learned



    In order to understand why so many errors were found in the developed curriculum, we studied the method used by the high-school teacher. Several comments on the curriculum editor seem to suggest that a method should be proposed or even imposed on the developer: "The use of these tools tends to segment the teaching material to the point where one loses sight of the global picture" [Lalonde & Robert, 1997].

    1. The high-school teachers' method



      It seems that most of the errors were du to the fact that the tool allows the curriculum developer to work in a messy way. After having analysed the protocol used [Newell & Simon, 1972] we have determined that the teacher tended to use a bottom-up approach [Gaines, 1990]. Here is his method:

      1- Find the pertinent objectives

      2- Identify their types and regroup them

      3- Enter the objectives in the editor

      4- Trace the relationships

      5- Add objectives if needed and repeat steps 2 to 5

      This method, named content-driven, favours the exhaustive identification of the objectives. This will cover more of the subject material. It could be useful to build a general course on a specific domain without knowing the characteristics of the learners. This approach prevents also the overlapping of low level capabilities or objectives. One of the biggest inconvenient of this approach is that some elements are defined and then put to side since they will not be connected to anything.

      With this approach, there is a loss of both global vision and coherence (therefore of the global quality). The introduction of entry points addresses partially this lack. The semantic restrictions permits the detection of some errors that are less damaging but still important since they constitute 25% of the total number.

    3. The Proposed methodology



      We have developed a methodology enabling the construction of an objective network respecting the CREAM model and Bloom’s theory, while minimising the potential errors.

      The proposed methodology, based on a course-driven approach, is as follows:

      1- Identify the major, most general objectives and mark them as Entry points.

      2- Refine the objectives by finding their components (relations of type is-composed-of). At this step we have a set of trees, refined at the desired granularity.

      3- While comparing the different composition trees together, identify the objectives that are duplicated and merge them. This has the effect of connecting some trees together.

      4- Identify prerequisite relations between existing objectives (relations of type is-prerequisite-to). Special care should be taken in avoiding loops in the relations, relations contradicting Bloom's theory and, more subtly, prerequisite relations short-circuiting composition relations.

      5- Add additional objectives representing special prerequisites that are not part of a composition sub-tree.

      6- At any step of the process, the Entry points should remain as such, and no further Entry points should emerge.

      By forcing the use of a top-down approach, we keep the developer from doing mistakes such as losing track of the main objectives of the course. The global organisation of the network is assured at the end of step 2. It is also easier to see the degree of refinement needed for each of the objectives, when they are in separate hierarchies. Merging subtrees and adding prerequisite links are critical operations that will be successful only, if the initial hierarchy classifies clearly what's in the developer's mind.

      This process can be transformed into an incremental one as long as the developer is well supervised, and one ensures that the restrictions are respected anytime. This methodology will be integrated in the existing tools.

    5. Using the proposed methodology to build a curriculum
In order to validate the methodology described in 6.2., we built a small curriculum for teaching basic skills in racquetball. First, in order to have a starting point, we needed to identify the entry points. We decided that: rules, techniques and match would be the three main themes in our tutoring. Here is a short description of each theme: The curriculum built in this section is illustrated in Appendix A. After naming the entry points, we need to expand them. For instance, the techniques objective is expanded as racquet basics and body position. When each objective is refined to the desired level, it is time to merge the ones that are redundant; in this curriculum, no merging were necessary since duplicate objectives were of a different level in Bloom's hierarchy.

Prerequisite relations were then added, with the priority given to the same-name objectives which are of different levels (such as points/AC and points/CO). Other prerequisite relations were added. One may point out that more relations should be added, but we believe that this curriculum already shows enough connectivity. Finally, objectives from lower levels of Bloom's taxonomy were added as prerequisites at points where we wanted to be sure to cover all the domain.

The reader can verify that the three rules discussed in section 5 are respected in this curriculum.

  1. Conclusion
The organisation of the material to be taught (the curriculum) is a key element in the building of an ITS. Since it occupies the heart of the system, any lack in curriculum quality will decrease dramatically the quality of the whole system. As we have shown, building a curriculum is an error-prone process. We believe that, even if we cannot eliminate all errors because of the complexity structure, it is still possible to control some aspects of the process.

In this paper, we have developed a generic evaluation framework based on three axes of intention: data structure definition, pedagogical model, and teaching goals. Using a sample curriculum based on the CREAM-O model and developed by high-school teachers, we have evaluated its conformity to the data structure definition and to the pedagogical model, leaving the teaching goals' evaluation to the teachers themselves.

Using three simple rules we developed, we tracked errors representing more than 40% of the original curriculum content (in terms of relations). This helped us in defining a methodology for curriculum development with the CREAM-O model; these guidelines will be integrated to the existing tools. To test this new methodology, a sample curriculum was built and commented.

We hope that this work will help researchers in the ITS community in defining generic evaluation models for curriculum quality evaluation.

Acknowledgements: This works has been supported by the TeleLearning Network of Centres of Excellence.


[Aïmeur & Frasson, 1996] Aïmeur E., Frasson C. Analyzing a new learning strategy according to different knowledge levels. Computers in education, 27(2), pp. 115-127.

[Anderson & al, 1990] Anderson J.R., Boyle C.F., Corbett A., Lewis M. Cognitive modeling and intelligent tutoring. Artificial Intelligence, 42.

[Bloom, 1956] Bloom B.J. Taxonomy of educational objectives : the classification of educational goals. New York : Longmans, Green.

[Gagné, 1984] Gagné, R. M. The conditions of Learning, 4th edn. Les éditions HRW Ltée. Montréal.

[Gaines, 1990] Gaines, B. Knowledge Acquisition Systems. Knowledge Engineering, vol 1, Fundamentals, pp 52-102, Edited by Hojjat Adeli, McGraw-Hill.

[Koedinger & Anderson, 1993] Koedinger K.R., Anderson J.R. Effective use of intelligent software in high school math classrooms. In Proceedings of the World Conference on Artificial Intelligence in Education, (pp. 241-248). Charlottesville, VA: Association for the Advancement of Computer in Education.

[Lalonde & Robert, 1997] Lalonde M., Robert C. Construction de cours et outils de développement. TeleLearning 6.2.1. Internal report, Université de Montréal, DIRO.

[Mark & Greer, 1991] Mark M.A., Greer J.E. The VCR Tutor: evaluating instructional effectiveness. In Proceedings of the Thirteenth Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Lawrence Erlbaum Associates.

[McArthur & al, 1993] McArthur D., Lewis M.W., Bishay M. The role of artificial intelligence in education: current progress and future prospects. RAND, Santa Monica, CA, DRU-472-NSF.

[McCalla, 1990] McCalla G. The search for adaptability, flexibility, and individualization: approaches to curriculum in intelligent tutoring systems. In Adaptative learning environments: foundations and frontiers, Berlin: Springer-Verlag.

[Millet & Gouardères, 1996] Approche qualitative de la ré-ingénierie d'un système tuteur intelligent à partir d'une méthodologie d'évaluation. In Proceedings of the third international conference, ITS'96, Montréal.

[Murray, 1996] Murray T. Special purpose ontologies and the representation of pedagogical knowledge. In Proceedings of the International Conference for the Learning Sciences (ICLS-96), Evanston, IL, 1996. AACE: Charlottesville, VA.

[Newell & Simon, 1972] Newell, A., Simon, H.A. Human Problem Solving. Prentice Hall, Inc., Englewood Cliffs, NJ.

[Nkambou, 1996] Nkambou, R. Modélisation des connaissances de la matière dans un système tutoriel intelligent: modèles, outils et applications. Doctoral thesis. Université de Montréal.

[Nkambou et al, 1996] Nkambou R., Frasson M.-C., Frasson C. Generating courses in an Intelligent tutoring systems. In Proceedings of the 9th International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, pp 261-266, New York, NY. Gordon and Breach Science.

[Obermeier, 1989] Obermeier, K.K. Natural language processing technologies in artificial intelligence : the science and industry perspective. Halsted Press ed. Ellis Horwood, Chichester, England.

[Pressman, 1997] Pressman, R.S. Software Engineering A Practitioner's Approach, 4th Edition, McGraw Hill.

APPENDIX A: A curriculum for teaching racquetball.

[return to Telelearning home page]