Topic 6:

Quality, Research and Evaluation

 

Overview

• Source materials for this topic

Quality, relevance, and effectiveness in open and distance learning

• Quality assurance
• Relevance
• Effectiveness

Research and evaluation

• Purposes of research
• Applying research in practice
• Research for quality assessment

Practice exercise

• Effort, performance, and efficiency

 

 Contents 

These materials support a discussion on the topic of how open and distance learning providers assure quality, relevance, and effectiveness in distance learning programmes, and the role research and evaluation plays in assuring quality.

The first subsection deals with issues of quality, relevance, and effectiveness. The second subsection deals with issues of research and evaluation.

Calder, J. Programme evaluation and quality. London: Kogan Page, 1994.

Evans, T. (ed.). Research in distance education. Geelong: Deakin University Press, 1990.

Perraton, H. Theories, generalisation, and practice in distance education. In Open Learning, 2:3, November 1987.

Robinson, Bernadette. Assuring quality in open and distance learning. In F. Lockwood (ed.), Materials production in open and distance learning, pp. 185–94. London: Paul Chapman, 1994.

Schuemer, R. (ed.). Evaluation concepts and practice in selected distance education institutions. Hagen: ziff, 1991.

Tait, A. (ed.). Quality assurance in open and distance learning: European and international perspectives. Cambridge: The Open University, 1993.

Thorpe, M. Evaluating open and distance learning. 2nd ed. Harlow: Longman, 1993.

 Contents 

Discussion: Begin this discussion by distributing pieces of paper, one to each participant, and asking participants to write down — in letters large enough for the group to see — their definition of ‘quality’ in an educational setting. When participants have finished, ask one of them to collect the sheets and pin them up in front of the group. Draw out the features common to them all, point out the differences, and ask whether there are features to be added.

The features mentioned might include the following:

• chosen standards or criteria;
• the relative nature of quality;
• services as well as products;
• perceptions as well as measured 
• outcomes; and relevance.

Additional points to be made might include the following:

• Everyone agrees on the desirability of quality.
• There is less agreement, however, on what quality is.
• This is because ‘quality’ does not exist in isolation from its context of use.

• Also, judgments differ according to whose views are being sought;

• for example: there is an amalgam of different meanings under the label ‘quality’; different stakeholders have different perspectives on quality; and

  different functional areas within a single organisation have different

  views

Priorities will vary according to

• who is making the assessment; and

• for what purposes the assessment is being made.

Discussion: Ask participants to provide examples of these points about quality from their own experience.

There are important differences between ‘quality assurance’ and ‘quality control’.

• Quality is a characteristic of the products and services an organisation offers.
• Quality assurance is a process directed toward achieving that characteristic. It is the set of activities that an organisation undertakes to ensure that standards are specified and reached consistently for a product or service.
• Quality control operates retrospectively, ‘inspecting out’ or discarding faulty products that fail to conform to a predetermined standard.
• Quality control and quality assurance, together with the assessment of quality systems — the monitoring, evaluation, and audit of procedures — are overlapping functions in regulating how an organisation or venture works.

All of these have a role in quality management approaches, the best known of which is total quality management.In summary:

• quality assurance involves pro-active measures taken to avoid faults;
• quality control involves re-active measures taken to remove faults; and
• quality assurance plus quality control plus continuous monitoring and evaluation equal total quality management.

Discussion: Do any of these terms or distinctions cause participants difficulties? Examples are always useful in clarifying such terms. Ask participants to provide examples from their own experience as you preview the list.

While ‘quality assurance’ may be a recently applied term in the educational context, there is nothing new about educational organisations’ undertaking systematic review and inspection of products and services to ensure their quality.

Discussion: Take this opportunity to solicit examples from participants of the ways in which the processes of review and inspection have been used in their contexts to ensure quality of educational products and services. In addition, almost all the case studies that are included with this training kit contain examples of processes aimed at improving quality.

More recent use of and emphasis on the label, ‘quality assurance’, can be attributed to factors such as the following:

• governments’ interest in return on public investment in education relative to other areas of expenditure;
• the assertion that education and training is essential to economic recovery, growth, and competitiveness;
• the assertion that the institutions responsible for education in the recent past have failed in their mission to meet demand because of ivory tower or anti-business attitudes; and
• insistence that education costs should be reduced and educational organisations made more accountable.

Discussion: Does this list of external factors fit with your participants’ experience? Do they have other factors to add?

Discussion: Begin this discussion by asking participants for their thoughts on whether this industrial approach to quality assurance is an appropriate one for education. Where does the approach fit? Conversely, what problems arise?

Is this industrially-based approach to quality assurance appropriate to educational institutions? Some of the terminology characteristic of the literature on quality assurance may be troublesome or inappropriate when applied to education. Given the nature of the ‘business’ in which educational institutions are engaged, debate centres on the terminology characteristic of total quality management in particular.

The term fitness of purpose can usefully force us to ask questions about our ends, for example, about the nature of our audience or the style of our teaching.

Purposes in an educational institution are varied, and in some cases conflict. For example, our job as educators is to facilitate our students’ learning. At the same time, however, we are expected to enforce certain educational standards of performance, which our learners may fail to meet. No business faces such a conflict.

Oversimplified notions drawn from the business sector and uncritically applied in educational contexts ignore the sometimes contradictory demands of various stakeholders, including

• students;
• academic and professional interest groups;
• research funders and practitioners;
• governments;
• employers;
• society at large; and
• future generations.

Discussion: Ask participants for examples from their own experience of ways in which the interests of these various stakeholders can contradict each other.

The aims of the educational process are to bring about changes in learners’

• knowledge;
• skills; and
• attitudes.

Upon successful completion of the process set out by the educational organisation, the learner may be awarded a credential of some kind.

These outcomes — changes in knowledge, skills, and attitudes and awards of credentials — may be called ‘products’ but they are considerably more complex than are the products of a manufacturing process.

In quality assurance, all actors within and outside an organisation are customers, providing a service to others.

Unlike businesses, in higher education institutions we have to fail ‘customers’ (learners) from time to time, acting in accordance with other stakeholders such as professional bodies, academic peers, and prospective employers.

Thus there remain elements in the relationship that the student has with any formal educational programme that are not based on the purchase of a service.

The ‘services’ provided by educational organisations are as varied and complex as their ‘products’. Services in support of learning include:

• provision of information to prospective applicants;
• pre-enrolment counselling and advising;
• screening of applicants;
• enrolment and registration;
• teaching;
• supporting learning by means of tutoring, counselling, advising, materials provision, libraries, and learning technologies;
• assessing learners’ performance; and
• post-course advice and counselling.

Of these services, screening of applicants and assessing their performance in particular set off education provision from other kinds of services.

To implement these quality assurance procedures, it is helpful to ask the following kinds of questions. (These questions are based on workshop materials developed for the International Extension College by Bernadette Robinson. They later appeared in her article, ‘Assuring quality in open and distance learning’, in F. Lockwood, ed., Materials production in open and distance learning (1994).)

Discussion: Do your participants agree with these points? What would they add, by way of agreement or disagreement?

 Contents 

Questions about quality and relevance are closely linked. To be relevant, open and distance learning courses and programmes need

• to meet both the national and local community needs for which they are intended;
• to match their content, design, and choice of media and technology to the intended learners and their contexts.

This indicates that relevance operates on several levels. These include

• policy;
• programme or course;
• materials and their mode of delivery; and
• learners.

Here is a checklist for determining the extent to which open and distance learning programmes, including both courseware and services, are relevant at these various levels.

3.1 Checklist for the Relevance of an Open and Distance Learning Programme

• Are the objectives relevant to identified need, that is national, regional, social, or educational?
• Is open and distance learning the appropriate means of fulfilling these objectives?
• Is the policy realistic in terms of resources, agreed priorities, culture, and context?
• Can the open and distance learning programme be implemented through the country’s existing infrastructure?

• Does the programme match the policy objectives?
• Is it designed to meet the target audience’s need?
• Is it appropriate for the learners and the communities they live in?

• Are the materials appropriately designed for the policy and programme objectives?
• Are they designed to match the learning levels of the targeted group?
• Are they sound in content and related to the cultural content?

• Is the choice of media appropriate for the learning task?
• Are they accessible to the learners?
• Is it appropriate and sustainable technology for the content?
• Does it match policy intentions and statements?

• Does the programme meet the needs and aspirations of the learners?
• Is the means of delivery and support appropriate?
• Do the learners have the skills necessary to make use of the learning materials?
• Does the course relate to the individual’s life and work?
• Are there enough learners to make the programme work?

A number of problem areas relate to the relevance of open and distance learning programmes.

Open and distance learning courses need to function with large numbers if they are to achieve economies of scale. They also become cost-effective when their production is centralised and limited numbers of experts are used for large numbers of learners. By contrast, relevance needs to be addressed at more local levels, where smaller numbers of learners share common concerns, contexts, and even languages.

Problems generating and disseminating knowledge vary from country to country, and include:

• a lack of focus of professional activity or research base within the country itself instead of outside it;
• lack of an easy means to disseminate knowledge within the country;
• an unfavourable economics of publishing because of smaller numbers of potential purchasers;
• diversity of languages, often within one country;
• reproduction of out-of-date curricula as course materials by lecturers who have difficulty keeping up-to-date; and
• an exacerbation of these problems in open and distance learning, which can provide irrelevant materials more widely and effectively than conventional education.

Needs change, as do technical and professional levels and cultural contexts. Rapidly changing needs can present problems for distance teaching programmes because of their large initial course production costs and their commitment to using existing stocks of materials, which may no longer be relevant to local or national needs.

 Contents 

The point of trying to make open and distance learning relevant to learners is to help make it effective, that is, able to achieve the objectives set for the project or programme.

How are we to judge the effectiveness of open and distance learning? Here are some useful indicators of effectiveness in an open and distance learning programme:

• the throughput of students (important in terms of effectiveness and cost-effectiveness);
• the acceptability of the graduates to employers or other educational institutions;
• its status in the eyes of the community;
• the quality of its materials and services;
• the extent to which the distance learning provision brings economic benefits to a country (for example, in initial training; retraining and upgrading; reaching rural areas; and developing basic technical and vocational skills); and
• student reactions to their learning experiences (since these can encourage or deter other potential learners).

Acceptability and credibility are of particular importance to distance educators, whose provision is often seen as ‘second-rate’. How do we know when our programmes have become credible? Here are some possible indicators:

• acceptance of distance-learning graduates by conventional universities for postgraduate work;
• acceptance of distance qualifications by employers;
• transferability of students between the two kinds of institutions so that work already done in one is given equal credit in the other;
• recognition of teacher qualifications through open and distance learning for increments in salary similar to those for conventionally trained teachers;
• entry to membership of professional bodies or to further professional training on the basis of the open and distance learning courses or qualifications;
• perception of value attributed by the public and by other professionals and academics; and
• use of distance learning materials by conventional institutions because they are the best available.

 Contents 

If we are to understand fully the outcomes of someone else’s research project and make sound management decisions based on them, we need to know

• why a particular topic was chosen;
• who carried out the research;
• what research methods were employed; and
• how the results were reported.

When conducting or commissioning research, we need to be aware of the choices we are making.Research can be conducted in open and distance learning on a great variety of topics and for a number of different purposes. A possible classification scheme follows.

Discussion: What research is carried out in the participants’ programmes? For what purposes?

The various types of research for system evaluation include the following.

• Basic measures of activity: collection of basic management information on such issues as the number of enrolled students, the number of courses offered, and the cost of the programme.
• Measures of efficiency: how many students successfully complete their courses, what workload they attempt, what level of student throughput is, and how cost-effective their programmes are compared with alternative forms of provision.
• Outcomes: student performance (determined using exam results and standardised tests), and the use of materials by other institutions.
• Programme aims: for example, have aims such as increasing access and equity been met by the composition of the student body? Is its rural–urban distribution appropriate?
• Policy evaluation: policy evaluation can be formative (for example, market research), monitoring (for example, surveys on impact of costs on students), an evaluation of policy changes, or experiments or pilot studies.
• Organisational evaluation: scrutinising the financial management and general organisation and methods, including monitoring tutors’ marking patterns and turnaround time for assignments, and evaluating the course team’s approach to course preparation.

The various types of research for course evaluation include the following.

• Formative evaluation: draft materials can be circulated for comment from within and outside the organisation, or tested on prospective students (developmental testing).
• Summative evaluation: intended to provide information about a course or materials in use; gathering feedback from students on the extent of their utilisation and their overall view of the teaching, general style of presentation, and specific content issues.
• Cross-sectional studies: study of a particular innovation or component used in a number of courses, aimed at drawing out generalisations from the use of a particular aspect of the teaching or to establish the effectiveness of a particular strategy or teaching medium.

 Contents 

Discussion: You might want to ask your participants the question, ‘What is research for?’

Research can contribute to improving open and distance learning practice in a number of ways, including:

• research for planning and accountability;
• research for good teaching and learning;
• impact evaluation; and
• understanding the world of open and distance learning.

When administering open and distance learning organisations or units, research could be seen as falling into two main categories:

• research for the design and development of the system, including

• the design of the structure of the institution; and  exploring market needs, including research for the strategic plan

• research for day-to-day activities, involving two areas, 

• formative evaluation and baseline information.    

• correspondence tuition;
• face-to-face tuition;
• course materials;
• learner progress; and
• facilities.

• number of enrolments;
• demographic data;
• completed assignments;
• examination attendance;
• pass rates; and
• number of dropouts.

Discussion: You might want to ask your participants the question, ‘Who are the researchers?’ in relation to their own programmes.

There are arguments for having evaluation done by researchers external to the institution because they will be more objective. There have also been arguments against using outside evaluators:

• they often have no experience of open and distance learning;
• they do not appreciate learner-centred approaches;
• they are unacquainted with the systems in place;
• they are unable to formulate appropriate questions; and
• they are not necessarily objective.

In many cases, however, institutions have no choice about external evaluation because of external funding or political and legislative decisions.

Discussion: Again, a useful question to ask your participants is ‘When is research to be undertaken?’.

There may be periods when organisations go into a steady state during which little development and change occurs, including research.The concern with research and evaluation and with innovative responses to environmental pressures may ebb and flow over a period of time rather than merely decline.

Measuring effort can be quite important for the manager, especially when relevant data is available over long periods. But measuring effort is of value in assessing open and distance learning mainly as a source of data for measures of efficiency.

Two measures of internal efficiency have been widely used to examine performance:

• a measure of learning, often using examination pass rates as an indicator; and
• successful completions rate, which again may be of most value as giving data by which to examine efficiency.

Is distance teaching a cost-effective way of teaching, compared with conventional methods? Hilary Perraton has generated from existing research two generalisations by way of an answer (1987):

• if in a distance-teaching system the costs of face-to-face support rise to the level of those in conventional education, then the costs of distance teaching cannot compare favourably with those of the conventional system; and
• a favourable economic outcome for any one distance-learning course is a function of three factors: the number of students, the amount of face-to-face study; and the sophistication of the media used.

What this list emphasises is the need for three systems of analysis and relation among them. As a guide to administrative planning, the following research is needed:

• advice from the teaching system about the sophistication of the teaching media needed for a particular audience and subject, and about the role of face-to-face learning; and
• information from the assessment system about trade-offs such as those between the numbers to be reached and the amount of face-to-face learning permitted.

Is an educational programme adequate in relation to the educational needs it is addressing? The answers will depend on the political stance of the evaluator and the purpose for which the programme was designed.

Is the process of education at a distance comparable with that of conventional education? Two problems here are that:

• not much literature exists because the issue is seldom directly addressed; and
• points of comparison –are debatable — should open and distance learning be compared with the best of conventional education or with the average?

It is reasonable to compare the reality of conventional and open and distance learning if we assume that a student might have a hypothetical choice between well run average programmes that are taught in both ways.

We also need to ask whether conditions necessary for the development of the capacity for dialogue are absent from a programme of open and distance learning, or exceptionally difficult for it. Do the limited opportunities for debate for open and distance learners and the reliance their courses unavoidably place on text necessarily disadvantage learners?

 Contents 

Instructions: Divide participants into three working groups. Assign each group the task of finding indicators to assess the following criteria of quality in an open and distance learning programme: effort, performance, and efficiency. Examples of indicators might be the number of enrolments, learners’ examination results, or the number of courses produced per year. To complicate matters further, depending on the interests of your participants, you might ask each group to consider these criteria in relation to formal education programmes (those leading to credentials) and non-formal education programmes (those not leading to credentials).

Timeframe: Allow half an hour for small group discussion and then have each group share their findings with the group as a whole.

Materials required: Flip chart paper or overhead transparencies and marker pens.