Research and Evaluation

  Overview 

These materials support a discussion on the topic of the role research and evaluation play in the management of open and distance learning programmes. All educational programmes need to be informed by research. The emphasis here is on the particular ways in which research and evaluation can inform the work of managers of open and distance learning programmes.

Source materials for this topic

Calder, J. Programme evaluation and quality. London: Kogan Page, 1994.

Evans, T. (ed.) Research in distance education. Geelong: Deakin University Press, 1990.

Perraton, H. Theories, generalization, and practice in distance education. In Open Learning, 2:3, November 1987.

Schuemer, R. (ed.) Evaluation concepts and practice in selected distance education institutions. Hagen: ziff, 1991.

Thorpe, M. Evaluating open and distance learning. 2nd ed. Harlow: Longman, 1993.

 Underlying issues in research 

Commissioning and interpreting research

If we are to understand fully the outcomes of someone else’s research project and make sound management decisions based on them, we need to know

·      why a particular topic was chosen;

·      who carried out the research;

·      what research methods were employed; and

·      how the results were reported.

When carrying out or commissioning research, we need to be aware of the choices we are making.

Research can be carried out in open and distance learning on a great variety of topics, for a number of different purposes, and at a number of levels. For example at the ‘macro’ level, one can evaluate an entire system; at the ‘micro’ level, one can evaluate a particular course offered within that system.

Discussion: What research is carried out in the participants’ programmes? To what purposes? What references are made to research in the case studies that are part of this kit?

 

Research for System evaluation

The following kinds of research measures are typical of evaluation at the systems level:

·    basic measures of activity: collection of basic management information on such issues as number of enrolled learners, number of courses on offer, cost of programme.

·    measures of efficiency: how many learners successfully complete their courses, what workload they attempt, what level of learner throughput is, how cost-effective their programmes are compared with alternative forms of provision.

·    outcomes: learner performance (exam results, standardized tests), use of materials by other institutions.

·    programme aims: for example, have aims such as increasing access and equity been met by composition of learner body? is rural-urban distribution appropriate?

·    policy evaluation: policy evaluation can be formative (e.g., market research), monitoring (e.g., surveys on impact of costs on learners), evaluation of policy changes, experiments or pilot studies.

·    organisational evaluation: scrutinising the financial management and general organisation and methods, including monitoring tutors’ marking patterns and turnaround time for assignments, evaluating course team approach to course preparation.

Research for course evaluation

The following kinds of measures are typically involved in evaluation at the level of a course:

·    formative evaluation: draft materials can be circulated for comment from within and outside the organisation, or tried out on prospective learners (developmental testing).

·    summative evaluation: intended to provide information about a course or materials in use; gathering feedback from learners on extent of utilisation, overall view of the teaching, general style of presentation, specific content issues.

·    cross-sectional studies: study of a particular innovation or component used in a number of courses, aimed at drawing out generalisations from the use of a particular aspect of the teaching or to establish the effectiveness of a particular strategy or teaching medium.

 Management and research 

A model

The following activities, as we have seen, are typical of the tasks involved in the management function:

·      planning;

·      decision making;

·      leadership;

·      implementation; and

·      evaluation.

These activities require information if the best possible decisions and choices are to be made at each stage. An example of a basic planning, research and evaluation model is outlined on the following page. ‘Research’ in this diagram includes 'evaluation' as one particular form of research that feeds into the overall management process.

Applying research in practice

What is research for?

Discussion: You might want to ask your participants this question before launching into this discussion.

Research can contribute to improving open and distance learning practice in a number of ways, including:

·     research for planning and accountability;

·     research for good teaching and learning;

Basic planning, research and evaluation model

 

·     impact evaluation; and

·     understanding the world of open and distance learning.

For running open and distance learning organisations or units, research could be seen as falling into two main categories:

·     research for the design and development of the system, including

the design of the structure of the institution; and

exploring market needs, including research for the strategic plan;

·     research for day-to-day activities, involving formative evaluation of areas such as

correspondence tuition;

face-to-face tuition;

course materials;

learner progress; and

facilities;

and gathering baseline information such as

number of enrolments;

demographic data;

completed assignments;

examination attendance;

pass rates; and

number of dropouts.

Who are the researchers?

Discussion: You might want to ask your participants this question, in terms of their own programmes.

There are arguments for having evaluation done by researchers external to the institution because they will be more objective. There have also been arguments against using outside evaluators:

·     they often have no experience of open and distance learning;

·     they do not appreciate learner-centred approaches;

·     they are unacquainted with the systems in place;

·     they are unable to formulate appropriate questions; and

·     they are not necessarily objective.

In many cases, however, institutions have no choice about external evaluation because of external funding or political and legislative decisions.

When is research to be undertaken?

Discussion: Again, a useful question to ask your participants before going on to present what follows.

There may be periods where organisations go into a steady state where little development and change occurs, which would include research.

The concern with research and evaluation and with innovative responses to environmental pressures may ebb and flow over a period of time rather than merely decline. 

 Quality assessment

Quality and the manager

An examination of quality issues is useful in clarifying questions of educational policy that arise in the manager’s regular work. At the same time, a concern for quality is more likely to bring ideological questions into the open than to reduce decisions to the merely technical.

Examples:

Raising the quality and quantity of tutorials could be expected to raise the quality of an open institution’s work in terms of process and performance. Because high-quality and high-quantity tutorials cost considerably more, however, they might or might not raise efficiency if this were measured in terms of cost per graduate. They might also have a negative affect on adequacy, since to cover costs the institution would have to raise its fees, thus making access more problematic and defeating the purpose of the institution.

Academic advisors may argue that the division between English language and English literature is artificial and anti-educational, and that a course which combines the two is more academically sound. But the performance many learners seek is not the widening of their understanding but a pass on the English language examination. This presents managers with a dilemma: should they stick with the academically sound, or go for a different definition of adequacy (i.e., meeting the demand for an examination pass), performance (passing in English language and never mind the broader aims of the English teachers) and process (back to the language exercises divorced from literature).

Measuring effort, performance, and efficiency

Research into various aspects of organisational performance can yield valuable information on the quality of that performance and pinpoint areas that need improvement. These aspects include effort, performance, efficiency, adequacy, and process.

Effort

Measuring effort can be quite important for the manager, especially where relevant data is available over long periods. But measuring effort is of value in assessing open and distance learning mainly as a source of data for measures of efficiency.

Performance

Two measures of internal efficiency have been widely used to examine performance:

·    a measure of learning, often using examination pass rates as an indicator; and

·    successful completion rate, which again may be of most value as giving data by which to examine efficiency.

Efficiency

Is distance teaching a cost-effective way of teaching, compared with conventional methods? Hilary Perraton has generated from existing research two generalizations by way of an answer (November 1987):

·    if in a distance-teaching system the costs of face-to-face support rise to the level of those in conventional education, then the costs of distance teaching cannot compare favourably with those of the conventional system; and

·    a favourable economic outcome for any one distance-teaching course is a function of three factors — the number of learners, the amount of face-to-face study and the sophistication of the media used.

What this emphasises is the need for three systems of analysis and the relation between them. As a guide to administrative planning it needs

·    advice from the teaching system about the sophistication of the teaching media needed for a particular audience and subject, and about the role of face-to-face learning; and

·    information from the assessment system about trade-offs such as that between the numbers to be reached and the amount of face-to-face learning permitted.

Adequacy

Is an educational programme adequate in relation to the educational needs it is addressing? The answers will depend on the political stance of the evaluator and the purpose for which the programme was designed.

Process

Is the process of education at a distance comparable with that of conventional education? Two problems here are that:

·    not much literature is available: the issue is seldom directly addressed; and

·    points of comparison are few: should open and distance learning be compared with the best of conventional education or with the average?

It is reasonable to compare the reality of conventional and open and distance learning if we assume that a learner might have a hypothetical choice between well run programmes taught in the two ways.

We also need to ask whether there are conditions necessary for the development of the capacity for dialogue which are absent from a programme of open and distance learning, or exceptionally difficult for it. Do the limited opportunities for debate for open and distance learning learners and the reliance that their courses unavoidable place on text necessarily disadvantage the learners?

An example of evaluating instruction

As we have seen in Topic 6, in distance education instruction tends to be carried by materials, using a variety of media, that have been designed to structure and facilitate learning as students work through them. A number of aspects of the process of designing, developing and delivering this instruction lend themselves to evaluation, the results of which can be very useful to managers in improving it quality.

The instructional design process

The following aspects of the process that is used to design learning materials make an appropriate focus for Quality Assurance evaluation:

·    the planning process by which the materials were produced;

·    the proposed aims, objectives and content of the materials being designed;

·    the proposed teaching strategy; and

·    the appropriateness and effectiveness of the media chosen for implementing the strategy.

Preliminary evaluation

All of these aspects might be examined before the students ever begin studying the materials. It can be valuable to have an outside ‘expert’ look over your materials before you make them available to students, paying attention to aspects such as academic credibility and likely effectiveness.

Academic credibility

You might want to ask some expert or experts in the subject matter questions such as the following about your open and distance learning materials.

Subject Matter Checklist for Learning Materials

 

q       Are the aims and objectives sufficiently explicit?

q       Do the aims seem relevant to the needs of the target audiences?

q       Do the objectives support the aims?

q       Are there any additional aims and objectives we should include?

q       Is the content up-to-date?

q       Is the content accurate?

q       Are the content and presentation culturally appropriate?

q       Are there any important omissions?

q       Do there seem to be any faults of emphasis?

q       Are the assertions made adequately supported by evidence?

q       Do the materials avoid oversimplification or overgeneralization?

q       Are they true to the nature of the subject/discipline?

q       Are they balanced, and at pains to present opposing points of view where appropriate?

q       Are the media that have been selected being exploited appropriately and to their full potential?

Likely effectiveness

Here are some questions that can be asked regarding how educationally effective the materials are likely to be.

Checklist for Educational Effectiveness in Learning Materials

q     Does the structure seem sensible and coherent, with introductions of previews, and summaries or reviews used where appropriate, and means available for allowing students with different needs to use the lesson in different ways?

q     Are adequate steps taken to motivate the students and make clear to them what they are to do with the material and to get out of it?

q     Are the materials pitched at the right level of difficulty and matched to assumed prerequisite skills and understandings of students?

q     Is the tone that of a rigorous but friendly tutor, lively and interesting?

q     Is the language plain and straightforward?

q     Are analogies, examples, case studies and illustrations used where appropriate to develop understanding?

q     Are questions, exercises, and activities properly integrated into the materials to encourage students in self-assessment and practice of relevant skills?

q     Are print and electronic media effectively integrated?

q     Is the form of presentation conducive to effective learning?

q     Are students given sufficient information and practice of a kind likely to help them achieve the objectives?

q    Is the relationship between assessment items and aims and objectives clear?

q     Are assessment items clear in what they demand of students?

q     Are assessment items likely to result in answers that can be marked with reasonable consensus of agreement among different markers?

q     Is the likely student workload reasonable for the topic?

 

Discussion: A useful exercise at this point is to have sample course materials available for participants to assess against these checklists for appropriateness to their own contexts.

Developmental testing

Developmental testing involves trying out materials with students in the hope of developing or improving those materials for the benefit of other or future students.

Methods of developmental testing include:

·    tutorial tryouts: trying the materials out on one student or a small group of students; and

·    field trials: using larger numbers of students (20-30) in circumstances as similar as possible to those in which your eventual students will work.

Continuous monitoring

Once the learning materials are in delivery, you will want to ‘keep an eye on things’ to see what problem areas need addressing, good things that are emerging and should be enhanced, and prepare for end-of-course evaluation.

Mechanisms available for this kind of formative evaluation include:

·    a course log book: used to record the main things you notice in the running of the course and the main in-course corrections you have used;

·    casual evaluation: appraising what is happening in day-to-day situations and responding to it; and

·    deliberate evaluation: actively seeking specific kinds of information, through discussions and interviews, questionnaires.

Summative evaluation

When the course is completed, a summative evaluation of its effectiveness addresses questions such as in the following checklist.

Checklist for the Summative Evaluation of the Effectiveness of Learning Materials

q       Did the course attract enough students?

q       Were they sufficiently qualified?

q       Did enough of them last the course?

q       Was the standard high enough?

q       Was the course cost-effective?

q       Were the students satisfied?

q       Were other stakeholders satisfied?

q       What needs to be changed?

Typical instruments and sources for obtaining this information include:

·      questionnaires: for students, for tutors, for others involved in delivery; and

·      interviews: with selected students, with tutors, with others involved in delivery.

The results of these evaluations can then be fed back into the design process as a basis for improvements to both the process and the results.

 Practice exercise 

Evaluating effort, performance, and efficiency

Instructions: Divide participants into three working groups. Assign each group the task of finding indicators one might use in assessing the following criteria of quality in an open and distance learning programme: effort, performance, and efficiency (e.g., exam results, enrolments). To complicate matters further, ask each group to consider these criteria in terms of formal education programmes (those which lead to credentials) and non-formal education programmes (those which do not lead to credentials). Allow a half hour for small group discussion and then have each group share their findings with the group as a whole.

Timeframe: Approximately one hour.

Materials: Flipchart paper and pens.