Performance Indicators

 Overview 

These materials support a discussion on the topic of how managers assure quality, relevance, and effectiveness in open and distance learning programmes, and what indicators of quality performance can be used to assess programme performance in those areas.

 

Source materials for this topic

Davis, D. The real world of performance indicators: a review of their use in selected Commonwealth countries. London: Commonwealth Higher Education Management Services, 1996.

Guri-Rosenblit, S. Quality assurance procedures at the Open University of Israel. In A. Taid (ed.), Quality Assurance in Higher Education: Selected Case Studies. Vancouver: The Commonwealth of Learning, 1997.

Robinson, B. Assuring quality in open and distance learning. In F. Lockwood (ed.), Materials Production in Open and Distance Learning. London: Paul Chapman Publishing, 1994.

Tait, A. (ed.) Quality assurance in open and distance learning: European and international perspectives. Cambridge: Open University, 1993.

 Quality, relevance, and effectiveness in open and distance learning 

Quality

Terminology

Discussion: You might wish to begin this discussion by asking participants for their definitions of quality, posting these definitions where all can read them, and then drawing out the features common to all the definitions.

A dictionary definition of quality is ‘the degree of grade of excellence possessed by a thing; peculiar excellence or superiority; capacity, ability or skill’ (Shorter Oxford English Dictionary).

Educators have more recently looked to fields such as industry, business, and management for helpful definitions. A typical example: Quality refers to a product’s fitness for purposed according to a set of required standards, concentrates on the quality of the product — courses and materials — and is useful for that purpose.

It also ties quality to ‘fitness for purpose’ according to chosen standards or criteria, which points to the relative, not absolute, nature of quality and judgements about it.

But it is probably too narrow to apply to the whole process of open and distance learning. More recent approaches focus on total quality, which includes services as well as products, an important inclusion for open and distance learning.

Judging quality

Despite the importance of the service component, quality in open and distance learning is most often viewed in terms of the materials produced. This is understandable for three reasons:

·    The course materials are tangible and publicly visible, open to scrutiny by all.

·    The materials are the key or sometimes the only means of engaging with the learners in open and distance learning.

·    It is easier to see the quality or lack of it in a product than in less tangible aspects such as teamwork or management.

Aspects of quality

Nevertheless, quality of open and distance learning provision does include far more than materials alone. The following framework outlines four aspects of quality in open and distance learning.

Four Aspects of Quality in Open and Distance Learning

The products or outputs	·    the courses and materials (printed texts, audio, video, and so on)
	·    number of graduates or successful completers
	·    examination pass rates of achievement of intended competencies or practical skills
	·    equivalent results in public examinations
The processes	·    learning and teaching processes such as tutoring, assessing written work, and providing learner feedback, monitoring field workers and tutors, training group leaders
	·    advising learners and keeping track of them
	·    record keeping
	·    co-ordinating groups of external writers
Production and delivery systems	·    course production
	·    print production
	·    scheduling and progress chasing
	·    warehousing and stock control
	·    dispatching materials to learners
	·    transmission of radio programmes
As a general philosophy or ethos	·    policy statements
	·    attitudes of staff
	·    management and training of staff
	·    motto or slogan (for example, ‘Only the best’)
	·    images and messages presented to the public (for example, publicity leaflets, brochures, press reports)

Relevance

Questions about quality and relevance are closely linked. To be relevant, open and distance learning courses and programmes need

·      to meet both the national and local community needs they are intended for; and

·      to match their content, design, and choice of media and technology to the intended learners and their contexts.

This indicates that relevance operates on several levels, including:

·      policy;

·      programme or course;

·      materials and their mode of delivery; and

·      learners.

Here is a checklist for determining the extent to which open and distance learning programmes, including both courseware and services, are relevant at these various levels.

Checklist for Relevance of Open and Distance Learning Programmes

Policy

q     Are the objectives relevant to identified need (national, regional, social, educational)?

q     Is open and distance learning the appropriate means of fulfilling these objectives?

q     Is the policy realistic in terms of resources, agreed priorities, culture and context?

q     Can the open and distance learning programme be implemented through the country’s existing infrastructure?

Programme or course

q     Does the programme match the policy objectives?

q     Is it designed to meet the target audience’s need?

q     Is it appropriate for the learners and the communities they live in?

Materials

Content

q     Are the materials appropriately designed for the policy and programme objectives?

q     Are they designed to match the learning levels of the targeted group?

q     Are they sound in content and related to the cultural content?

Means of delivery (media and technology)

q     Is the choice of media appropriate for the learning task?

q     Are they accessible to the learners?

q     Is it appropriate and sustainable technology for the content?

q     Does it match policy intentions and statements?

Learners: group and individual

q     Are the needs and aspirations of the learners met by the programme?

q     Is the means of delivery and support appropriate?

q     Do the learners have the skills necessary to make use of the materials?

q     Does the course relate to the individual’s life and work?

q     Are there enough learners to make the programme work?

Problems of Relevance in Open and Distance Learning Programmes

In terms of relevance in open and distance learning, there are a number of problem areas.

Achieving a balance between economies of scale and meeting local needs

Open and distance learning courses need to function with large numbers if they are to achieve economies of scale. They also become cost-effective when their production is centralised and limited numbers of experts are used to each large numbers of learners. By contrast, relevance needs to be addressed at more local levels, where smaller numbers of learners share common concerns, contexts and even languages.

Difficulties in the generation and dissemination of knowledge

Problems in the generation and dissemination of knowledge vary from country to country, and include:

·   a lack of focus of professional activity or research base within the country itself instead of outside it;

·   lack of an easy means to disseminate knowledge within the country;

·   an unfavourable economics of publishing because of smaller numbers of potential purchasers;

·   diversity of languages, often within one country;

·   reproduction of out-of-date curricula as course materials by lecturers who have difficulty keeping up-to-date; and

·   an exacerbation of these problems in open and distance learning, which can provide irrelevant materials more widely and effectively than conventional education.

The constantly changing nature of what is relevant

Needs change, as do technical and professional levels and cultural contexts. Rapidly changing needs can present problems for distance teaching programmes because of their large initial course production costs and their commitment to using existing stocks of materials, which may no longer be relevant to local or national needs.

Effectiveness

The point of trying to make open and distance learning relevant to the learners is to help make it effective, that is, able to achieve the objectives set for the project or programme.

How are we to judge the effectiveness of open and distance learning? Here are some useful indicators of effectiveness in an open and distance learning programme:

·    the throughput of learners (important in terms of effectiveness and cost-effectiveness);

·    the acceptability of the graduates or successful completers to employers or other educational institutions;

·    its status in the eyes of the community;

·    the quality of its materials and services;

·    the extent to which the distance learning provision brings economic benefits to a country (for example, in initial training; retraining and upgrading; reaching rural areas; developing basic technical and vocational skills); and

·    learner reactions to their learning experiences (since these can encourage or deter other potential learners).

Acceptability and credibility are of particular importance to distance educators, whose provision is often seen as ‘second-rate’. How do we know when our programmes have become credible? Here are some possible indicators:

·    acceptance of distance taught graduates by conventional universities for postgraduate work;

·    acceptance of distance qualifications by employers;

·    transferability of learners between the two kinds of institutions so that work already done in one is given equal credit in the other;

·    recognition of teacher qualifications through open and distance learning for increments in salary similar to those for conventionally trained teachers;

·    entry permitted to membership of professional bodies or to further professional training on the basis of the open and distance learning course or qualification;

·    perception of value attributed by the public and by other professionals and academics; and

·    use of the distance learning materials by other conventional institutions because they are the best available.

 Some quality assurance tools

Here are five tools that can be useful in identifying and solving problems of quality, relevance, and effectiveness. They were developed by Bernadette Robinson for a workshop delivered by the International Extension College, and later published (Robinson 1994).

Tracking the source using the five whys

The first reason given for a failure of quality may not get to the heart of a problem, yet typically this is where most questioning stops at this point. The reasons need to be tracked back to source. One simple technique is to ask ‘why?’ five times (more if necessary) in answer to each response. This gets to a deeper level of understanding and analysis.

Example: The institution has a system of ‘second marking’ for all assignments submitted by students enrolled in its M.A. course. This means that part-time tutors mark an assignment and then send it to central office for second marking by a member of the full-time teaching staff before it is returned to the student. In one instance a full-time faculty member reduced the grade the tutor/first marker gave to an assignment by only one percentage point, without consulting the tutor. This upset the tutor, making her feel demeaned and incompetent.

Why did the faculty member do this?

Because she wanted to make it appear that she was more than a ‘rubber stamp’.

Why?

Because she came from a university system that does not routinely involve second marking, and did not understand the role of second marker.

Why?

Because this role had never been discussed with her or thoroughly explained.

Why?

Because the director of the programme took it for granted that all universities operated this way and that hence all faculty members would understand the role of second marker and the procedures they were to go through (for example, the obligation to discuss any differences of opinion with the first marker).

Why?

Because assuming something to be the case is easier than checking it out when one is exceedingly busy with this and other duties, including a great deal of travelling.

And so on.

Flow diagrams

Flow diagrams are useful for mapping processes so they can be understood. Flow diagrams provide a way of tracking and displaying how quality is shaped through several stages or phases. As Robinson (1994:190) points out, flow diagrams are useful for:

·     systematically recording steps, decisions, and activities required in a sequence;

·     providing a clear diagrammatic representation of a process as a way of sharing information about what happens;

·     identifying critical points or bottlenecks;

·     displaying the consequences of planned change;

·     standardising practice;

·     training; and

·     tracking and diagnosing the sources of failure.

The following flow diagram presents the process of course approval at the Open University of Israel (Guri-Rosenblit 1997:32).

A flow diagram: Course approval procedures at the Open University of Israel

Pareto analysis

An economist called Pareto is quoted by Robinson (1994:191) as suggesting that 80 percent of problems arise from 20 percent of causes. If you track the causes that create most failures of quality, you can concentrate your efforts on those areas that pay the most dividends. To apply this, you need to:

·     identify the problems;

·     try to quantify them; and

·     use this information to make a simple bar chart which ranks the categories.

Example: An educational organisation tracks the nature of complaints received on a student ‘hot line’ from learners enrolled on their courses, categorises them, and records their frequency. The results of this tracking over the first month of classes are set out below. The fact that the top five complaints all relate to the bookstore operation gives managers a major clue as to where to focus their efforts in order to improve the quality of service to students.

A Pareto Analysis Bar Chart

 

Top Five Complaints	Frequency 50 100 150 200 250 300 350
Set texts not available at start of course.
Not enough set texts available for number of students in class.
Hours of bookstore inconvenient for students who work part-time.
Far too many set texts prescribed at too high a cost.
No used-book sales scheme in place.

Fishbone diagrams

A ‘fishbone’ diagram maps in a structured way:

·     the causes of a problem, both major and minor;

·     the effects; and

·     the factors involved.

It is useful in helping to diagnose and analyse problems with colleagues, especially when there is more than one cause. To use this you need to:

·   identify the problem in a few words and put this as the head of the fishbone;

·   identify the main issues or causes (four is a manageable number);

·   under each of these four, explore the component factors or sub-causes (using a ‘brainstorming’ technique or the ‘five whys’);

·   discuss the completed diagram, trying to distinguish between symptoms and causes; and

·   agree on the one main cause and highlight it so that plans for remedies are focused on it.

 Here is a sample fishbone diagram. The problem under discussion is put into the box at centre right.

A Fishbone Diagram

3.5 Milestones and barriers

This technique charts the learner’s progress through the institution or programme to identify key milestones and barriers. Once identified, the processes that converge on these milestones and barriers are mapped 

and examined in detail. They are then reviewed and evaluated. This highlights critical points in systems, from the learner’s point of view. The same can be done for other key players, for example,

·     corporate clients;

·     tutors; and

·     support staff.

The points of hand-over or junction between stages or players are often critical barriers or milestones. Following is an example.

A Milestone Map

 

 

 Performance indicators

Terminology

Distance educators increasingly are being required by their funding agencies, governments in particular, to develop sets of indicators by which institutional performance can be measured.

This exercise is closely tied to the process of quality assurance. Two important distinctions must be made between these processes, however:

·    quality assurance is primarily focussed on quality, whereas performance indicators tend to focus on quantitative measures; and

·    quality assurance tends to be primarily an internal activity, whereas performance indicators tend to be externally driven and mandated.

The term performance indicator is rather loosely and variously used. There is general agreement, however, that performance indicators, although based on the same data as management information, are clearly evaluative measures, clearly related to institutional or sector goals.

Here is a working definition:

Performance indicators provide a measurement for assessing the quantitative performance of a system.

Discussion: Again, you may wish to begin this discussion by drawing out participants’ own definitions of what constitutes a ‘performance indicator’ and using those as a basis for what follows.

Audiences and purposes

There are many audiences for performance indicators, each with particular purposes and needs. Some of these audiences and their purposes are set out in the following table.

Discussion: You might wish to first draw out participants’ ideas of who constitutes the audiences for performance indicators and the purpose these indicators serve, in the context of their own programmes.

Performance Indicators of Various Audiences

Audience	Purpose
Units and institutions	·      Internal management
	·      Comparison with other units and institutions
	·      Marketing, image building
	·      Evaluation of teaching and research activities of individuals and departments
Government	·      Accountability
	·      Policy and planning
	·      Allocation of resources
	·      Funding
	·      Value of investment in research
	·      Human resources planning
Public	·      Accountability
Learners	·      Institution choice
Industry	·      Research funding
	·      Graduate employment
Research councils	·      Selective distribution of research funds

As you can see, the purpose of quality assurance exercises and the development of performance indicators overlap a great deal. But, again, the performance indicator exercise is most often performed in response to pressure from external agencies. Performance indicators become the object of controversy when they are used as a ranking device to allocate esteem and resources differentially.

Categories

A number of different approaches can be taken to the development of performance indicators that provide a basis for their categorisation. One common approach is based on a production model. Another is based on internal and external measures.

Discussion: What categories do your participants use now?

Measurement of input, process, and output

·      Input: resources used (for example, financial and physical facilities, learners and staff);

·      Process: management of teaching, research, and services; and

·      Output: products of teaching, research, and services.

Internal, external, and operational indicators

·      Internal: market share of undergraduate applications, and graduation rates;

·      External: first destination of graduates, publications, and citations; and

·      Operational: unit costs, staff-to-learner ratio.

Key programme indicators

The following table lists some key programme indicators that are used in these approaches.

Key Programme Indicators

Indicator	Purpose
Learner indicators	·      Population
	·      Entry qualifications
	·      Progression and completion rates
	·      Destination
	·      Learner satisfaction
Staff indicators	·     Qualifications (for example, staff with Ph.D.)
	·      Gender balance
	·      Age ratios
	·      Academic-to-support staff ratios
	·      Value of investment in research
	·      Manpower planning
Resources and finance statistics indicators	·      Operating funds
	·     Research funding
	·     Other income
	·     Staff-to-learner ratio
	·     Expenditures on academic centres, central administration,     library
	·     Different ratios of income-to-expenditure
	·     Other selected financial ratios
Research indicators	·      Number of research students
	·      Research funding
	public sector research funding
	industry research funding
	total research income per academic staff member
	research expenditure per academic staff member
	ratios of research expenditure and income
	·      Publications
	number of journal articles
	number of books
	other publications and conference papers
	·     Patents and licences
	income earned
Estate management and physical resources indicators	·      Space utilisation
	·      Performance in maintenance, improvement, and capital expenditure

Characteristics of ‘good’ indicators

There is general agreement on the criteria for good indicators. They should be:

·      relevant;

·      can be updated;

·      based on reliable figures;

·      understandable; and

·      valid, that is, measure what is supposed to be measured.

Indicators should be clearly related to the defined functions, objectives, and mission of the institution. Where there are common indicators across a system, however, different values might be put on low or high figures by different institutions, reflecting different objectives.

Example: High learner-to-staff ratios are desirable in open and distance learning, since one of the advantages of open and distance learning provision is its ability to make a relatively small number of teachers and other experts available to a wide population. Conventional institutions, however, may well pride themselves on low learner-to-staff ratios, as an indicator of small class sizes and hence a personalised and individualised approach to learners.

Concerns about performance indicators

There are also many concerns about performance indicators. Among these are:

·    the costs of providing additional data when the data required cannot be extracted from existing information;

·    an emphasis on one particular aspect of performance (e.g., financial performance) at the expense of others;

·    the inappropriateness of trying to rank institutions on the basis of performance indicators when institutions have different objectives;

·    the tendency to use performance indicators in isolation rather than in conjunction with measures using professional judgement such as peer review;

·    the fear that diversity among higher education institutions might be lost as institutions seek to maximise performance on the same set of indicators;

·    the imposition of indicators may be used as an instrument of control by government;

·    the limited value of indicators in measuring ‘quality’ including the quality of teaching and learner outcomes; and

·    the fear that the use of performance indicators will stress efficiency and economy rather than quality.

Discussion: What concerns do your participants have about the use of performance indicators in their programmes? What indicators of performance seem to be important to the institutions profiled in the case studies?

 Practice exercise

A working example

Instructions: In order to bring the rather abstract nature of the material in this unit down to a working level, ask each of your participants for an example of an actual problem which they are encountering in their programmes. Ask for as concrete a problem as possible, such as chronically late dispatch of materials, or slow turnaround time on assignment marking.

One good way to elicit these problems is to have each participant write down their problem on a piece of paper; collect the papers, read out the problems, and group them in some way.

On the basis of this grouping, divide your participants into small groups. Give each group a problem to focus on. Ask each group to work together to produce answers to the following questions:

·    How have you become aware of this problem?

·    Track the problem back to its source. One way of doing this is to use the ‘Five Whys’: keep asking ‘Why?’ until you get to the root problem. For example:

Q: Why are materials being dispatched to students after the course has supposed to begin?

A: Because the course writers are not producing them on time.

Q: Why are the course writers not producing them on time?

A: Because they are working for us only part-time and do not take course writing as seriously as their full-time jobs.

Q: Why do they not take course writing as seriously as their full-time jobs?

A: Because course writing does not pay very well and does not earn them any prestige.

Q: Why does it not pay very well, and not earn them any prestige?

A: It doesn’t pay well because we don’t have the money to pay them what they think the work is worth, and is not prestigious because courses do not count as publications?

Q: How much would you have to pay them for them to take course writing seriously? And why do courses not count as publications?

And so on...

·    Once you have identified the root problem, suggest ways in which you could solve it.

·    Identify the resources you would require — for example, money, people, expertise — in order to put your solutions into action, and where you could get these resources.

Have each group report their results to the larger group, and discuss.

Timeframe: The small groups will need about three-quarters of an hour for their discussions. Therefore you can expect the overall exercise to take an hour and a half to complete.

Materials: Flipchart paper or overhead transparencies and pens for group reports.