A report
for the National Council
of the Canadian Evaluation Societyon the experience
of other organizations
PROFESSIONAL CERTIFICATION
|
|
|
© by Bud Long and Natalie Kishchuk, CES, October 1997
|
|
CES main page / Page d'accueil de la SCÉ
|
1. INTRODUCTION
1.1 Background and purpose
1.2 Previous CES actions regarding certification
1.3 Certification versus accreditation/certification
2. METHOD
3. FINDINGS
3.1 Cutting to the chase: a shortcut to the bottom line
3.2 Certification model and related definitions3.2.1 Levels of professional designation
3.2.2 Changing the level of designation
3.2.3 The legal basis for the designation and the organization's structure
3.2.4 Firm membership in an organization
3.2.5 Knowledge and experience requirements and tradeoffs between them
3.2.6 Knowledge requirements: approved courses and degree programs3.3 The intended impacts of certification
3.4 The activities required for certification3.4.1 Inherent activities
3.4.1.1 Developing and maintaining standards: the body of knowledge
3.4.1.2 Developing and maintaining standards: continuing education and recertification
3.4.1.3 Developing and maintaining standards: the education and experience requirement
3.4.1.4 Developing and maintaining standards: ethical behaviour
3.4.1.5 Applying the standards: testing the knowledge of candidates
3.4.1.6 Applying the standards: verifying the experience claimed by candidates
3.4.1.7 Applying the standards: responding to the complaints, appeals and legal action of unsuccessful candidates for certification
3.4.1.8 Applying the standards: monitoring certified members
3.4.1.9 Applying the standards: disciplining certified members
3.4.1.10 Applying the standards: dealing with clients or employers who are dissatisfied with the professional performance of certified members
3.4.1.11 Applying the standards: warning off people who pretend to be certified3.4.2 Activities to assist the candidate to become certified
3.4.2.1 Cross-referencing to recommended textbooks
3.4.2.2 Making it easier to take training and write exams
3.4.2.3 The accreditation of courses and programs
3.4.2.4 The development of courses
3.4.2.5 Grandparenting3.4.3 Activities to enhance the benefits of certification
3.4.3.1 Publicity targeted at potential clients and employers
3.4.3.2 Identification for clients and employers of certified members with the required specialties in a given location3.5 Costs of the certification activities
3.6 So has certification really been worth it?
4. IMPLICATIONS FOR THE CANADIAN EVALUATION SOCIETY
4.1 The level of the designation and the consequent structure of the CES
4.2 Benefits of certification
4.3 Activities required for certification4.3.1 Level of effort required
4.3.2 Inherent activities: developing the body of knowledge
4.3.3 Inherent activities: developing and updating the education and experience requirement
4.3.4 Inherent activities: developing a code of ethics
4.3.5 Inherent activities: testing a candidate's knowledge
4.3.6 Inherent activities: other issues4.4 Activities to assist the candidate to become certified
4.5 Activities to enhance the benefits of certification
4.6 Costs
4.7 Further information needed
As a body representing program evaluators across Canada and promoting the program evaluation function in Canadian institutions, the Canadian Evaluation Society (CES) is concerned with the sustainability, growth and strengthening of the profession. In recent years, this concern has led the Society to consider issues related to increasing professionalization, through means such as professional development programs, development and adoption of practice standards and ethical guidelines, and certification of members.
This latter issue, developing a form of certification for members, would be a major step for the CES. Above and beyond the substantial investment required to develop and maintain certification, it would have important implications at several levels: for evaluation clients, for professional evaluators who may or may not be CES members, and for the CES's vision of the profession and its own role.
The current study has been carried out with the approval of the National Council of the CES and under the broad direction of its Professional Development Committee. The purpose of the study is to inform the deliberations of the Council regarding the adoption of professional certification. It serves this purpose by describing the experience of several other organizations (Organization is used here as a generic term for words like college, institute, society, association, council, forum, etc. In the organizations that we examined, we found that institute was the most common term for the organization that governs the certification process. The term college is generally used for the organizations that perform this function for some self-governing professions like physicians and psychologists -- organizations that we did not examine in any depth; only one organization in our sample used it, the Canadian College of Health Service Executives.) with certification, and by setting out the lessons that the CES might draw from it. It deals with organizations that decided to adopt certification and -- sometimes with reservations -- to maintain it, as well as with those that have considered it but have not yet adopted it or rejected it. We know of no cases where an organization has had certification, but dispensed with it.
Underlying the purpose of this report is a central question: Is certification a goal for the CES? In our view, the priority to be assigned to the wide range of possible CES activities should be determined in light of the answer to this question. If certification is a goal, then there is an obvious, high priority need to define the standards for the profession, above all the minimum body of knowledge and experience that certified evaluators must have. This will require a strong and proactive commitment, systematic planning and the allocation of considerable financial and volunteer resources. If it is not a goal, then the development of these standards ceases to be so urgent (even though they remain desirable in themselves), and the CES can pursue its goals of enhancing the sustainability and strength of program evaluation through its other professional development tools. In our view, it's putting the cart before the horse to argue, as some appear to do, that the CES should save the answer to this critical question until after all the conditions for certification have been met. (This is however what the Canadian Comprehensive Auditing Foundation seems to be doing. See the section entitled Developing and maintaining standards: the body of knowledge. )
The answer to this question will naturally depend on the Society's judgement about whether the pros of certification more than offset the cons. It is hoped that the experience of other organizations, described herein, can contribute to this judgement. (return to Table of contents)
1.2 Previous CES Actions Regarding Certification
To our knowledge, the CES has addressed the topic of certification on only one previous occasion. This was in the study conducted by the Standards Development Committee, reported to the May 1990 annual conference. The report ( Standards for Program Evaluation in Canada: A Discussion Paper (Draft Version to be presented at annual CES Conference, May, 1990), Canadian ) recommended the development of a "set of standards synthesizing values common to Canadian evaluators", the voluntary acceptance of the standards by members as "principles of good evaluation practice", and a "concomitant complementary continuing education program". It was at this time that the CES committed to a strong professional development program available to members in all regions of the country, including the development of the CES ethical guidelines and the Essential Skills Series.
The Standards Committee report did not try to deal systematically with the pros and cons of certification. The Committee thought that only a "sizeable minority of CES members" was interested in "professionalization through accreditation and licensure" ( Nor did the Standards Committee report ever use the word certification. It did however speak of the possibility of licensing, which is generally used to allow only certified people to practice a given profession. See the discussion under the following section of this paper entitled "A certification model and related definitions".). The consultations conducted by the Committee at that time suggested that CES members were generally not well informed about the issue. Although a few were aware of and concerned about efforts by "competing" professions (auditors, marketing researchers, and management consultants), most had not yet to come to firm conclusions about their own positions and had little knowledge of similar issues in other organizations.
Since the 1990 report, various efforts have been made through the Professional Development Committee to pursue the issues of standards development and, indirectly, certification. Important among these has been CES participation on the (American) Joint Committee on Standards for Educational Evaluation, which has informed us about the complexities of standards development. (return to Table of contents)
1.3 Certification versus Accreditation/Certification
The focus of this report is on certification, not certification/accreditation (AC). Certification means the confirmation by an organization of professionals that a person has the knowledge and experience that it judges necessary to work in that profession.
Accreditation means the confirmation by a professional organization that a certain educational course or program, generally one given by a university or college, provides a specific portion of the knowledge required for certification. The organization may, but does not necessarily, go further than this and confirm that the regular exam given at the end of the course will serve as a substitute for an exam held on the same topic by the organization itself.
At the outset of this project, the CES National Council and the Professional Development Committee (PDC) generally used the term "certification/accreditation" as if the two concepts were in some sense equal. The creation by the American Evaluation Association of two parallel senior committees to look at the two topics tended to support this perspective. However, this study has confirmed our earlier opinion that the central concept is certification, and that the accreditation of courses or programs should be considered only as an optional, albeit useful, element in an organization's total certification process. (return to Table of contents)
This report was prepared in accordance with the approach proposed to and agreed by the Professional Development Committee (PDC) during the winter of 1996-97. This involved the development of a questionnaire (see Appendix A) and the gathering of the data from several professional organizations by means of document reviews and interviews. The interviews were with one or more past or present officials from each organization (see Appendix B).
Although the questionnaire was used as the starting point for data collection, not all questions were answered for all organizations. Given informants' time constraints, judgements had to be made during the interviews about which questions were really essential and which could be passed over.
We divided the data gathering tasks by organization, and then collaborated in the preparation of the report. (return to Table of contents)
3.1 Cutting to the Chase: a Shortcut to the Bottom Line
You may wish to take a shortcut to the bottom line, that is, the lessons that the CES could draw from the experience of the other organizations. If so, we suggest that you first examine: the certification model and its definitions, presented next; Appendix B; and the list, given at the start of section 3.4, of the activities that are essential, or at least desirable, for certification. Then turn to 4.0: IMPLICATIONS FOR THE CES. (return to Table of contents)
3.2 Certification Model and Related Definitions
The components of certification are summed up in the following chart:
3.2.1 Levels of professional designation
There are four broad "levels" of professional designation.
A. Exclusive Designation
This is the term we are using for what is clearly the highest level of designation -- that typical of the longstanding self-governing professions, such as lawyers, chartered accountants, architects, professional engineers and medical doctors. For this form of designation, a provincial statute typically:
b) provides that (i) only a "registered" person may purport to be a member of the profession or may practice it, and that anyone who contravenes (i) is guilty of an offence, and
c) authorizes a specific professional body to: i) decide who is so registered, ii) inspect the practice of such people, iii) discipline such people for any breaches of professional ethics or for professional incompetence.
The statute might make certain exceptions to the above. For example, the BC Psychologists Act states that "...nothing in this Act prevents a person...from teaching...as a psychologist [at a university]...or...from [being employed in certain public agencies] where qualifications in psychology are a condition of employment". Moreover, such people may use the title Psychologist.
An interesting variation is found in the Canadian Securities Institute, which provides training and proficiency testing and oversees a supervised work experience requirement for individuals who are licensed by the provincial securities commissions. (That is, the Institute itself does not certify members -- it essentially conducts proficiency testing on behalf of the securities commissions. A licensed status is required to conduct transactions through the stock markets, but many more people take the training and exams than actually end up being licensed with the commissions.) In this (highly regulated) industry, a government agency is essentially moving a relatively new profession towards a self-governing status by requiring the professional designation.
We assume -- but have not confirmed -- that, in contrast to the simpler case of a "protected designation" (see next sub-section), the bill proposing the provincial statute for an "exclusive designation" would normally be introduced into the legislative assembly by the provincial cabinet. (return to Table of contents)
B. Protected Designations
In this case, a specific provincial statute (or an order in council pursuant to a general statute like the BC Society Act -- see below):
b) provides that (i) only a person who is a full member of a particular professional organization may use that title, and (ii) that anyone who contravenes (i) is guilty of an offence.
However, in contrast to the statutes providing for "exclusive" designations, the statute in this case does not state that only a registered person may practice the particular profession. Indeed, in some cases, the statute makes it explicit that there is no such restriction. For example, the Ontario Act referred to in Footnote 5 states that it "does not affect or interfere with the right of any person who is not a...member...of the Institute to describe himself or herself as a planner or professional planner or to practice as a planner or a professional planner or to work in the field of planning". In the absence of the restriction, there is of course no need for the statute to define planning, and in fact the Ontario statute does not do so.
At least in BC, a specific provincial statute that protects the designation must be introduced into the legislative assembly as a private member's bill. This means in turn that the organization must persuade a private member to undertake this task. (One likes to think that, for the CES, the MLA who chairs the Public Accounts Committee might do so.)
However, some provinces may have a general statute that gives the Lieutenant-Governor -- i.e., the provincial cabinet -- the power to pass an order-in-council which protects title with the same legal force that a specific statute does. As noted above, the BC Society Act is apparently an example of such a general statute. (We have been assuming that the equivalent of "protected title" in Quebec is the titre reserve provided by the Office des professions. However, as we go to press, we realize that titre reserve may in fact go beyond protected title and encompass at least some degree of exclusivity. This is something to be clarified. )(return to Table of contents)
C. Certification Marks ("Semi-protected Designations")
Certification marks are awarded by the federal government under the federal Trademarks Act. They are used to identify wares or services that meet a defined standard. They are owned by an organization which then licenses their use to people who meet that standard. Example are the "certified health executive (CHE)" designation of the Canadian College of Health Service Executives and the "Certified Financial Planner (CFP)" designation of the Financial Planners Standards Council of Canada.
Certification marks are like "protected title" with one important difference. It's not a criminal offence for an unlicensed person to use a certification mark. Rather, the organization to which the mark has been awarded must seek damages in a civil court. As with the protected title, there is of course no definition of a profession, nor any restriction on the right of unlicensed persons to call themselves, say, "health executives" or to be employed in that capacity. (return to Table of contents)
D. Non-protected Designation
This is our tag for designations that organizations award to their members, but without any specific legal basis preventing its use by non-members. We saw two examples of this. Although the Toronto Chapter of the Institute of Internal Auditors is incorporated in Ontario (The Institute of Internal Auditors is an international organization without any national or provincial bodies (at least not in Canada). ), this does not give it protected title; nor does it have protection under the federal Trademark Act.
A second example comprises certain provincial affiliates of the Canadian Institute of [land use] Planning. There is apparently no general protection under the federal Trademarks Act and only five of the provinces have protected title under provincial statutes. The others (the four Atlantic provinces , for which there is one affiliate, and Manitoba) are without any specific protection ( It's unclear what the situation of the Territories is. The BC affiliate extends to the Yukon, and the Alberta affiliate to the NWT. However, it's not clear what the territorial statutes provide. ).
Nonetheless, the two institutes believe that, while they do not have protected title, they could still bring a civil action against anyone wrongly stating that he/she had been certified by them. However, they would not have any basis for suing a person who was merely calling him/herself a "certified internal auditor"or "certified planner".
We share the view of many evaluators that, as a matter of principle, program organizations are responsible for self-evaluation,and that in turn the evaluation profession is responsible for supporting this principle. The question is whether certification would be consistent with this view. The risk of inconsistency would lie most obviously with the "exclusive designation", where the scope of evaluation would be defined and only certified people would be able to practice it. But how then to draw the line between what constitutes professional evaluation, and the work that is simply part of the role of a responsible manager? To illustrate, consider on which side of the line each of the following issues should lie: the definition of desired outcomes; selection of related indicators; selection of the data collection methods; the related question of the setting the budget for the evaluation study; the selection of people to plan and to conduct the study; and so on. Under an exclusive designation, managers would have to be certified to make the decision on any of the above issues that were considered to be within the scope of professional evaluation. There may be ways of dealing with this problem, but this would have to be confirmed before any move were made to adopt exclusive designation as a CES goal. (return to Table of contents)
3.2.2 Changing the level of designation
Organizations can of course change the "level" of their designation. The Certified Management Consultant (CMC) designation of the Institute of Certified Management Consultants of Canada was originally protected only by certification marks. It is now protected by legislation in most if not all provinces. Further, the Financial Planners Standards Council of Canada hopes to convince the provincial securities commissions to provide certified financial planners with the exclusive right to sell securities to individuals.
Not all thoughts of "upgrading" have led to much. A few years ago, the Canadian College of Health Services Executives tried without success to have the provincial Ministries of Health give people with the Certified Health Executive (CHE) designation the exclusive right to occupy certain hospital positions. And in the 1960s, the Canadian Institute of Planners considered seeking the exclusive right to practice [land use] planning, but did not pursue it in light of the expected opposition of related professions like civil engineers, architects and landscape architects.
An organization which considered the upgrading of its essentially open membership to adoption of certification and professional designation is the Association des économistes québecois. A proposal to have the title "économiste" reserved under the provincial statutes governing the professions was rejected by the membership in a consultation process. (return to Table of contents)
3.2.3 The legal basis for the designation and the organization's structure
The legal basis of the designation has some bearing on the structure of the organization. If the legal basis is certification marks under the federal Trademarks Act, then the substantive organization can be at the national level, with provincial or local chapters. The Canadian College of Health Service Executives and the Financial Planners Standards Council of Canada are examples of this.
If, on the other hand, the legal basis is provided by a provincial statute or order-in-council (see above the sections on the "exclusive" and "protected" designations), then the legally substantive organization must be at the provincial level. However, in each of the cases we examined, the provincial organizations share a national umbrella organization to provide common services. Perhaps the major common service is the development of the knowledge and experience standards for certification. An example of this type of structure is the Institute of Certified Management Consultants of Canada and its provincial affiliates.
However, there are variations even within this general structure. The provincial organizations of the Institute of Certified Management Consultants of Canada have had their own offices to administer the certification exams, to accredit courses, to maintain the provincial membership lists, and at least in one case to identify the certified practitioners for potential clients. (A similar relationship is found between the national and provincial chapters of the Economic Developers Association.) However, for the sake of greater cost effectiveness, the Institute of Certified Management Consultants of Canada has recently decided to eliminate the provincial offices, other than that in Quebec, and to move all the work to the national office. This office will be headed by a Chief Executive Officer, responsible for implementing Board policy. The provincial institutes will be formally kept in place to the extent needed to meet the requirements of the provincial statutes that created them and protect their title. They will, however, receive all their staff support from the national organization. The national organization admits that having to deal with several provincial administrations and memberships at long range may not be ideal, but expects this difficulty to be outweighed by the advantages of centralization. Under this arrangement strong local chapters will be encouraged in the larger cities. There is some recognition that, at 1200 members, the Toronto chapter might be large enough to justify a part-time staffer. (return to Table of contents)
3.2.4 Firm membership in an organization
The Institute of Certified Management Consultants of Canada has firm memberships (We did not systematically ask the organizations about firm membership. The concept was mentioned in the documents of only the Institute of Certified Management Consultants of Canada, so it seems unlikely that the other organizations provide for them. ). To be accepted, the firm must "serve clients...on managerial and related operating and technical problems in one or more of the main areas of managerial activity", and must do so with "demonstrated competence". It must have "recognized standing" in its field; its professional staff must be "recognized by the Institute as experienced and competent"; and it must have "a record of competence, ethical conduct, high standards independence and stability". In addition, at least 30% of the firm's "practice leaders and professional staff" must either be certified or be participating in the certification process. The firm must also make commitments to raise this number to 50% within three years, and to maintain it thereafter at that level. The firm membership is thus, in part, an extension of the membership of its staff. (return to Table of contents)
3.2.5 Knowledge and experience requirements and the tradeoffs between them
The most basic components of professional certification are the units of knowledge and experience that individuals must possess in order to become and remain certified. Nearly all of the certifying organizations that we examined have standards for both knowledge and experience. However, within the knowledge requirements, some treat as substitutes for each other the taking of approved courses and the passing of exams set by the organization (Institute of Certified Management Consultants of Canada). At the same time, however, other organizations require both the taking of approved courses and the passing of exams, whether or not there was an exam at the end of the course (Financial Planners Standards Council of Canada; Economic Developers Association).
Some organizations also allow trade-offs between Knowledge and Experience. For example, the more relevant the academic background of the candidate the less is the amount of experience required (Canadian Institute of [land use] Planning). (return to Table of contents)
3.2.6 Knowledge requirements: approved courses and degree programs
There are basically of two types of approved courses: first, those developed by the organization itself to cover exactly the knowledge required by its professional standards; and second, those provided by other institutions and accredited by the organization as meeting a specified portion of the knowledge standards.
Different levels of program accreditation can be assigned, based on the programs' degree of relevance to the profession. For example, the Canadian Institute of [land use] Planning and its provincial affiliates assign a program that is specifically in land use planning a higher standing than a degree in, say, geography. However, a degree in geography is considered to be at least "related" to a degree in planning and has a higher standing than an "unrelated" degree, say, in economics. A graduate degree can be given a higher level of accreditation (i.e., a larger number of credits toward the quantity required for certification) than an undergraduate degree (Economic Developers Association).
The certification process can involve submission of transcripts to ascertain whether the required number of accredited credits or points has been obtained (e.g., Economic Developers Association). (return to Table of contents)
3.3 The Intended Impacts of Certification
We can begin to turn now to the central question of whether the intended impacts of certification outweigh its costs. The organizations' documents and the interviews mentioned the following impacts (it is unclear which were "intended" before certification and which have since emerged as "unintended but desirable"):
We suggest that these impacts are usefully restated as follows:
b) more income for practitioners,
c) better quality of practice
d) more public respect for the profession.
Impact (a) was mentioned by only the Canadian College of Health Services Executives. Quite apart from whether or not it is achievable (see below), it seems a questionable benefit, for it means that people being certified will effectively be forced to subsidize other members of the organization.
As for Impact (b), the presumption is that it will be easier to sell the services of a profession:
Note that the issue here may be not merely to increase demand from where it is now, but to sustain it in the face of competition from similar professions, especially ones that already have certification, or which are moving toward it (The certification of physicians is justified by the presumed ignorance of the individual consumer of medical services. However, this justification applies to only one of the examined organizations, the Financial Planners Standards Council of Canada. The others all serve institutions, which would generally be better able to choose suitable practitioners than the typical consumer of medical services . However, institutional consumers might still find it very helpful to be able to restrict their search for suitable practitioners to those who have had to meet a strong minimum standard of knowledge and experience. Whether consumers of evaluation services would actually do so is something that needs examination. This is discussed further in the last major section of this paper.).
The difficulty with Impact (b) is that it may fall very unevenly on practitioners. Raising the status of the profession by the adoption of certification will almost certainly also raise the status of certified practitioners relative to uncertified practitioners. Thus, for the uncertified, any increase in income owing to the general increase in demand will be offset at least in part, and perhaps more than completely, by the decline in income from a shift of demand towards the certified. Thus, for those who are not certified, Impact (b) will be problematic.
The better quality of practice (Impact c) would likely be a step on the way to the greater respect for the profession (Impact d). And both would likely be steps on the way to the achievement of greater income (Impact (b) (Although the Association des économistes québécois ultimately decided not to proceed with certification, it nonetheless made one of the clearer links between better quality of practice on the one hand and greater public respect and income on the other. The Association expected that one of the benefits of certification would be an improvement in the quality of the work carried out under the name of the profession. The Association wanted to avoid the loss of credibility that resulted from unqualified individuals calling themselves economists and doing incompetent work. ). However, they could potentially be desired as ends in themselves regardless of the effect they have on practitioners' incomes. Again, the greater respect will be primarily for those practitioners who are certified, perhaps at the expense of those who remain uncertified.
So what does the experience of the examined organizations tell us about the actual benefits of certification? Unfortunately, none had conducted an evaluation to answer this question. However, some of the informants did have relevant opinions to express.
Impact (a) -- more revenue for the organization: The single organization with
this goal has had certification in place now for almost 20 years. It feels that it has generally lost more money than it has gained.
Impact (b)-- more income for practitioners: Most remarks clearly concerned the relative income advantage that the certified would have over the uncertified. The Institute of Internal Auditors informant said that, although there is "variability" in client attitudes, "more and more" companies are asking that their internal auditors have the CIA designation, or at least be enrolled in the Institute of Internal Auditors certification process; recent examples of this are Bell Tel and two of the major banks. Similarly, one of the informants from the BC affiliate of the Canadian Institute of [land use] Planning said that, at least in BC, about nine out of 10 job advertisements call for membership or "eligibility" for membership. In contrast, we understand that, at least in the last year or so, few if any job advertisements have specified the need for, or even the desirability of, the Certified Health Executive designation of the Canadian College of Health Service Executives. Indeed, some members of that organization are apparently now asking whether certification has been any help to them in their careers. The experience of the Institute of Certified Management Consultants of
Canada seems to be rather in the middle, with the informant saying that the CMC designation was only one of several factors determining the degree of a management consultant's success.
Impact (c) -- better quality of practice: While this seemed to be a general goal behind many moves toward certification and some respondents felt that the overall quality of practice had improved, this was difficult to separate from the training and other professional development activities which are inherent in certification (i.e., if there is professional development without certification, could the same improvements be expected?) Although the Association des économistes québécois ultimately decided not to proceed with certification, it nonetheless expected that one of the benefits of certification would be an improvement in the quality of the work carried out under the name of the profession. The Association wanted to avoid the loss of credibility that resulted from unqualified individuals calling themselves economists and doing incompetent work.
Impact (d) -- more public respect for the profession: No comments bore directly on certification's achievement of this goal. Certainly increased respect for the certified practitioners can be inferred from the increased demand for them in the job market. However, perhaps not everyone would consider this as an indicator of respect for the profession as a whole. (return to Table of contents)
3.4 The Activities Required for Certification
Not all the activities associated with certification are "required" to the same extent. Some are inherent in certification. Others are not inherent, but assist the candidate to become certified (but not by making the requirements easier). Still others are not inherent, but enhance the benefits of certification (although not necessarily to the extent needed to cover the cost).
The activities are given in summary form immediately below to provide an overall view. They are then each discussed in more detail.
INHERENT ACTIVITIES:
a) developing and maintaining standards for:
b) applying the standards to members
ACTIVITIES TO ASSIST THE CANDIDATE TO BECOME CERTIFIED:
ACTIVITIES TO ENHANCE THE BENEFITS OF CERTIFICATION:
Consider each of these activities in more detail.
3.4.1.1 Developing standards: the body of knowledge
The body of knowledge is what defines a profession. It's what the members of the profession have in common with each other and what distinguishes them from the members of other professions. The absence of a body of knowledge implies the absence of a profession in anything but name.
As the model near the beginning of this paper shows, the body of knowledge is the basis for certification. Its acquisition is what is supposedly assured in the certification process. Even the required experience is essentially a means of assuring that the candidate has the practical knowledge needed to apply the "theoretical" body of knowledge.
The organizations we examined came in rather different ways to their body of knowledge. At one extreme is the Institute of Internal Auditors, which with
its 35,000 members in 215 chapters world-wide can afford an international headquarters of about 100 staff (located in Florida). It is this office, supplemented as needed by curriculum consultants, that provides the support needed by the Institute's committees in the development and maintenance of the body of knowledge (The Institute of Internal Auditors has no national or provincial structure in Canada, only local chapters. Although the Toronto Chapter's 800 members apparently make it the sixth largest chapter of the Institute, it is still only one of many voices in the process for developing the body of knowledge.).
The present body of knowledge covers:
At the other extreme are the 3500 member Institute of Certified Management Consultants of Canada and the 500 member Economic Developers Association. Each developed and maintains its body of knowledge entirely by its own efforts. They used committees of volunteers for this purpose, although this has not been easy to do given the limited time that volunteers can usually devote to such matters. The periodic review and upgrading of knowledge and experience standards has been a lot easier than their initial development, but still quite time-consuming.
The Institute of Certified Management Consultants of Canada has been focussing recently on the review and definition of its "core competencies", and a new document setting out the approved body of knowledge is to be completed and available in the early fall of 1997 (Sylvia Cook of the Institute of Certified Management Consultants of Canada said a copy of the Common Body of Knowledge would be available "for a fee". The Institute of Certified Management Consultants of Canada phone number is 800-268-1148. She also recommended another document that the relevant committee apparently found useful. This is entitled Professional profiles management consultants -- a body of expertise, skills and attributes. It is published by Acme at phone 212-697-9693 and fax 212-949-6571. The cost is US$15.). For this purpose, the Institute has been relying on local committees, each one of which has assumed responsibility for a particular issue; it scorns attempts to form "great Canadian committees" except at the review and consolidation stage. At least in the past -- and we understand no basic changes are to be made -- the body of knowledge, as reflected in the exams, has comprised the following subjects:
Rather between these extremes are two organizations that obtained the right to use the body of knowledge of their American counterparts and then "Canadianized" it. This was true of the Canadian College of Health Services Executives and the Financial Planners Standards Council of Canada.
The Canadian College of Health Service Executives purchased the knowledge standards from the American College, although the price was not known by either of the informants. The College then called upon its senior members to develop "competency questions" for use in exams. It was largely in the production of these questions that the American standards were effectively modified for use in Canada. Apparently there was some help in this work received from a New York consulting firm and the American College itself. By these means, the Canadian College has acquired a rather detailed table setting out the "professional competencies" required for each of:
The Financial Planners Standards Council of Canada purchased the right to use the knowledge standards of the American Certified Financial Planners Board of Standards. The American Board had developed the standards by conducting three surveys (presumably directed at its members, leaders in the profession, academic organizations, etc. but this was not clear from the interviews). The adaptation for Canadian use was done by two experts who had taught and written in the field. A "curriculum review" is now underway, being carried out by focus groups composed of (largely volunteer) members of the Council's nine parent associations. We did not obtain a copy of the present body of knowledge, but presumably it comprises the same fields in which post-certification training must fall in order to meet the continuing education requirement. These fields are:
Not only must an organization create the body of knowledge, but it must also keep it up to date. This was a problem for the Canadian College of Health Service Executives. This organization originally concentrated on the management of acute treatment hospitals. Its body on knowledge did not therefore adequately reflect the increasing importance in the health system of chronic care institutions, of home care and of prevention rather than treatment. However, this was corrected by a major review in 1994 -- apparently the first -- which comprised a collaborative effort by over 100 members, curriculum development consultants, employers, academics and government officials.
With the exception of one organization, the body of knowledge is an absolute concept. It comprises what certified people should in principle know (even though, in practice, perfect marks are not required to pass the exams based on the body of knowledge).
A given knowledge element is either a component of the body or it is not. The interesting and important exception is the Economic Developers Association, which
has adopted an "inclusionary" approach to the development of its body of knowledge. This is now based on a member survey, conducted in 1996, which identified all the possible types of work conducted by designated economic developers. The Association requires the mastery of a basic general body of knowledge and, in addition, the mastery of at least one specialty area. A member does not have to know all specialties. The advantage of this arrangement is that certification would thus become a matter of degree, not an either/or thing. There would then be more room for compromise: in a dispute over a given knowledge element, the choice would no longer be just between including or excluding it from the body of knowledge; the choice would be among including it in the basic body of knowledge, excluding it from all bodies of knowledge or including it in the body of knowledge of one or more of the specialties. Certainly, Association members are convinced that this approach not only is workable but also strengthens their profession.
We have seen what the certifying organizations have done for the formulation of a body of knowledge. It's also instructive to consider the efforts being made by those organizations that do not certify their members.
Since last fall, the Canadian Comprehensive Auditing Foundation has been working on "proficiency requirements", primarily by means of several focus groups established for the purpose. The results of these focus groups have been consolidated and have been sent to the participants for review and elaboration. The overall report is to be submitted to the Research Committee this fall. (The Director of Research, David Moynaugh, said the CES could have a copy once it's approved by the Committee).
The basic question addressed by the focus groups was: "What judgements do comprehensive auditors have to make, and what knowledge, skills and experience are needed to make the judgements?" Judgements must be made about the strategic issues of where to audit and of the scope of the audit. But they must also be made about operational issues like when to bring in and how to deal with/oversee specialists.
The knowledge requirements that emerged from this review covered:
Skills are what's needed for the application of knowledge.
Experience was apparently the most difficult thing for the Foundation's focus groups to define and agree on. If we understood correctly, the issue is about the way in which the intensity of the experience with any one area of knowledge can be recognized by adjustments in the required length of time of the experience.
These proficiency requirements are expected to be used for the following purposes in the management of comprehensive auditing shops: in the allocation of personnel
(e.g., given the types of judgement the position requires, is it occupied by the right person?), in recruitment, in succession planning, and in professional development programs (Mr. Moynaugh (Canadian Comprehensive Auditing Foundation) mentioned that a recent Public Service Commission document, which sets out a "holistic competence profile" (knowledge, skills and aptitudes), might be useful to us. This reflects, at least in part, the thinking of one Lyle Spencer, a guru in competence models, who makes a basic distinction between "surface" ( skills and knowledge) and "core" (values) competencies. Spencer argues that agencies should hire the core and train the surface. The PSC had the document in draft last year and has been revising it. Apparently one Ken Grant of the PSC is in charge of this project. Mr. Moynaugh also said that the American Institute of Certified Public Accountants has a Web Site that might be informative. We should check the work of the Special Committee on Assurance Services (700 pages on the Web). The equivalent body, the CICA (Canadian Institute of Chartered Accountants), is also looking at this same area.).
Another non-certifying organization is the Planning Forum. This is an international body of about 4000 members (1200 in Canada, of which 800 are in Toronto), with its headquarters in Chicago. It has been shifting its emphasis from mere "planning and budgeting to "strategic management and strategic leadership". The Planning Forum has not yet started to seriously discuss what knowledge is required for the effective performance of this latter function.
The Association des economistes quebecois, which considered but decided against certification, relies on the active participation of its members in its professional development activities (conferences and workshops) to communicate and foster the acquisition and maintenance of skills and knowledge. (return to Table of contents)
3.4.1.2 Maintaining standards: Continuing education and recertification
New ideas and concerns must be reflected in the body of knowledge on which new candidates for certification are tested. But they should also be reflected in the requirements for continuing education that are applied to people who have already been certified.
In fact, we found quite a variation in the types of requirements for continuing education. The Institute of Certified Management Consultants of Canada is working on a new standard that is to be available in the fall of 1997; it is unclear what has been in place to date, but in any event, it seems to have been optional, for it has apparently never been implemented in British Columbia. The Institute of Internal Auditors requires 110 hours of continuing education in each three year period. It can take the form of formal education, published writing, oral presentations, or participation in specified professional development activities. The Canadian Institute of [land use] Planning affiliates hold professional development events, but attendance is not mandatory.
Since February 1996, the Canadian College of Health Services Executives has a Maintenance of Certification Program, which calls for 20 hours of activities per year. Those hours can comprise such activities as continuing professional development with the College or its chapters, attendance at conferences or workshops, written publications, presentations, teaching, volunteer service in policy development in the health field, attendance at board meetings, and mentoring in a planned fashion. Consideration was being given to making attendance at the annual conference mandatory, as it apparently is in the United States, but the idea was abandoned because of health program cutbacks.
The Financial Planners Standards Council of Canada is to require 30 hours per year in areas to be specified by Council, such as insurance or estate planning, investments, ethics, or client counselling. The Economic Developers Association requires recertification every three years or the designation is withdrawn. Recertification requires the accumulation of five certification points, which can be gained by attending courses or conferences or by acting as a facilitator or moderator in an event relevant to the profession. The Canadian Securities Institute requires that, once having passed its exam and undergone a supervised work period of six months (under the supervision of a certified supervisor), a candidate must take an additional 180 hours of training in the first 30 months after certification. Members' employers may in fact require even more extensive post-certification activity, as the knowledge requirements evolve very quickly in this field.
Most of the organizations seem to have been made little systematic attempt to tie the continuing education requirement to changes in the body of knowledge. However, a few have done so. For example, a body composed of representatives of industry, stock market and securities commission representatives advises the Canadian Securities Institute on modifications needed in the training curriculum because of changes in products or legislation. In the case of the Economic Developers Association, a committee composed of university and industry representatives makes recommendations for the updating of the Association's two main five-day training sessions and three-day seminars, which are offered exclusively by the University of Waterloo. (return to Table of contents)
3.4.1.3 Developing and updating standards: the education and experience requirement
Education and experience have been lumped together here, because the organizations often allow trade-offs between them. Generally the requirement is for a bachelor degree and a minimum of two years of experience in the field of the particular profession (Presumably the ideal experience would be in the application of the required body of knowledge. However, that would imply that the candidate should meet the knowledge requirement before the experience in applying that knowledge should start to be counted. In fact, at least one organization in our sample accepts experience gained before the passing of the knowledge examination. ). However, there are a number of exceptions to this.
The Institute of Certified Management Consultants of Canada requires a degree or five years of experience that is deemed by the organization to be the equivalent. It also requires 1200 hours per year for three years of management consulting experience, which must include five projects "signed-off" by a sponsor. It appears, however, that there is some potential, decided on a case-by-case basis, for trade-offs between the three year period on the one hand and, on the other, the quality of the experience and education. Some papers from the Ontario Institute say that experience as an "internal consultant" will not be accepted unless the "environment" is one which "promotes"..."independence and objectivity".
The Institute of Internal Auditors requires a degree or the CGA or CMA accounting designations. It also requires two years of experience, but will accept a master's degree or one year of related experience (mainly in law or accounting) for one of those two years. The Canadian Institute of [land use] Planning requires a person with a "recognized planning" degree to have a minimum of two years experience in professional planning. A person with a "related" degree such as geography will require four years experience, and with an "unrelated" degree six years.
Until 1994, at the time of a major review of its requirements, the Canadian College of Health Services Executives required no degree at all. Since then, however, that organization has required a masters degree or the equivalent. It also requires two years of consecutive experience. The Financial Planners Standards Council of Canada requires two years of experience related to financial planning, but within five years (before or after) of the time of the exam. It does not require a degree.
The Economic Developers Association has a complex point system which recognizes not just the degree, but the relevance of the courses making up the degree to the profession of economic development. Candidates holding a bachelor's degree must have earned a certain number of points by taking courses judged by the Association to be relevant; those with a masters need fewer points. Official transcripts are required. Members must also have worked in the profession for three years (but this is assured by the structure of the required training courses -- see below). Those without a degree may still qualify by writing a major paper (50 pages) on a relevant topic.
The Association des économistes québecois, which has no form of certification, requires that members hold a bachelor's degree in economics.
We have not examined the criteria used by the organizations for the establishment of the education and experience requirements. The fact that a bachelor degree and two years of experience is common suggests a certain amount of imitation.
Note that it is only the Canadian Institute of [land use] Planning that sets what is in effect an experience requirement for the maintenance of certification: continued employment as a [land use] planner. (return to Table of contents)
3.4.1.4 Developing and maintaining standards: ethical behaviour
All organizations have codes of ethics that candidates for certification must accept in order to become, and to follow in order to remain, members in good standing. When adherence to a code of ethics is a condition for certification, that code must be precise enough to make clear to members and to the organization that passes judgement on them just what types of behaviour are and are not acceptable. (return to Table of contents)
3.4.1.5 Applying the standards: testing the knowledge of candidates
This is another area of great variation across the organizations. It is in effect an "operationalization" of the body of knowledge. It entails issues that will be all too familiar to the educators among us (And our apologies to them if our summary here is less than perfect. ), some of which are:
The Institute of Certified Management Consultants of Canada exams are all written. As already noted, they comprise:
The exams are designed and marked at the national level and administered at the provincial level. No appeals are accepted. The whole process requires the heavy involvement of volunteers.
The Institute of Internal Auditors has a written 210 minute exam on each of the four topics that form the body of knowledge. The exams contain both multiple-choice and essay questions. The four exams are held on two consecutive days on two occasions each year at "major cities around the world". The Institute's large international headquarters handles the development of the exam and their conduct (likely with some help from the local chapter, although we did not verify this).
The Canadian Institute of [land use] Planning's approach to exams is perhaps the most complex of any. For those with a "recognized planning degree", only an oral exam is required. For others, i.e., those with either a related degree (e.g., geography) or an unrelated degree, there is both the oral exam and a written exam. The oral exam is to last 30 -- 60 minutes, and the written exam three hours. They are both conducted and marked by the provincial affiliates. The written exam, which comprises three questions that have apparently remained unchanged for years, is distributed at least a month in advance to the candidates. For the oral exam, 24 sample questions are specified in the membership manual to serve as a guide for the discussion, but are not to be distributed to the candidate. The interviewers are supposed to add questions based on a review of the candidates' background as reflected in his/her file.
In the opinion of an Institute informant who had been an examiner for some years, the three written questions are very effective in distinguishing between suitable and unsuitable candidates. Only rarely is there a disagreement between the two initial markers about who should pass and who should fail (a third, "National Examiner" from another affiliate resolves such disagreements, where they persist). Presumably answers to the questions are kept in a secure location or destroyed in order to keep them out of the hands of future candidates (although we did not verify this). The oral interview is conducted by at least two and preferably three examiners.
The Canadian College of Health Services Executives requires the passing of one four hour exam of 48 multi-choice questions that test the candidate's knowledge of the "professional competencies" referred to earlier, as well as the preparation of two reports of about 2500 words each on independent study projects (only one project is required of a person with a masters degree with a "health care focus" earned within the last five years). Until recently, the exams were held only at certain central points, but they can now be done anywhere that a candidate can arrange for supervision.
Before writing the exam, candidates must attend one of the College's leadership assessments. These half-day sessions are held at two or three points in Canada each year, followed by an optional half-day seminar in writing skills.
The College's rather modest objective of changing only 12 (i.e., 25%) of the multi-choice questions every year has been a major challenge for them. The practice has been to seek them from the 21 local chapters. But after the 1994 review, it was "like pulling teeth" to get them from those bodies. And last year in the nation-wide canvass, only four or five suggestions were made, all from one chapter, and not all of them were acceptable. After the questions have been used in the exam, the administrator with the aid of some of the exams markers identifies those questions that are too easy or too hard, and deletes them from further use.
Candidates for the Certified Economic Developer designation of the Economic Developers Association must pass a 4.5 hour examination with both oral and written sections. They do this after completing two five day training sessions (one per year in consecutive years) and at least six three-day seminars. Many choices are provided among the written questions, so that candidates can demonstrate their knowledge in their particular economic development specialty. Highly detailed scoring protocols have been developed for all of the open-ended exam questions, and the marking of exams is handled by the Association's permanent staff. A special committee reviews borderline cases.
The Financial Planners Standards Council of Canada requires candidates to complete the six course "modules" that are now provided by its parent bodies. Candidates may do the modules either by correspondence or at the 18 community colleges which offer them. The modules, based on the Council's curriculum guidelines, can apparently be completed in 18 -- 30 months of part-time study. As an alternative, candidates can take the programs -- accredited by the Council as covering the same material -- that are offered by Wilfred Laurier University and the British Columbia Institute of Technology (BCIT). Other post-secondary programs are expected to be accredited in the near future.
Candidates must not only take these courses or programs, and any exams that they require, but also the Council's subsequent Professional Proficiency Examination, which "test[s] the ability to apply [financial planning] knowledge to practical situations". The exam is developed by the volunteer Examination Sub-Committee of the Council's Education Committee, with the support of the organization's permanent staff, and follows the Council's curriculum guideline. The exams are to be held twice a year, presumably in different parts of Canada (although we did not check on this). The Council held its first exam in June, 1997 for about 300 candidates; it expects to have 1500 candidates per exam within the next year or so.
The Canadian Securities Institute has, in addition to its 150 hour training program, a mandatory 50 hour program on professional ethics with a separate exam.
Note from the preceding paragraphs that many of the organizations require not only a demonstration by exam of an understanding of the body of knowledge, but also the taking of specified training. We did not pursue the rationale for making this training mandatory. Certainly, the courses are a financial burden to the candidates, who might be able, if they had the option, to acquire the same knowledge by less costly means, for example, from properly referenced texts. Making the training mandatory is rather like an addition to the education requirement, rather than simply a test of the candidates' knowledge. (return to Table of contents)
3.4.1.6 Applying the standards: verifying the experience claimed by candidates
A few examples will demonstrate the types of requirements that are imposed by professional organizations.
The Institute of Certified Management Consultants of Canada requires each candidate to have two sponsors, preferably certified members of the organization. They must attest to the accuracy of the information in the candidate's application, including that regarding previous employment, and must "sign-off" the summaries of five consulting assignments. An elaborate process is in place to ascertain whether the experience of internal management consultants has been in an environment that promotes independence and objectivity. Among other things, this includes criteria for such an environment, acceptable sources of evidence and assessment methods.
The Canadian College of Health Services Executives asks for a list of previous employment, and for three references, at least one of which is to be a current or previous supervisor, and at least two of which are to be certified members of the College. The Canadian Institute of [land use] Planning requires candidates to keep a "log book" for all the experience claimed. Each "item of experience" must be described in some detail, and signed by the candidate's supervisor and by a certified member. The Economic Developers Association asks for three references from individuals who are knowledgeable about the applicant's duties in the field of economic development. (return to Table of contents)
3.4.1.7 Applying the standards: responding to the complaints, appeals and legal action of unsuccessful candidates for certification
The informants said that there are occasional complaints and formal appeals from unsuccessful candidates, but none knew of cases where legal action had been taken. Nonetheless, the organization bylaws must provide machinery in the form of committees and procedures for the review of unsuccessful candidatures. Further, the provincial laws that protect a designation generally provide for an eventual appeal to the courts. This is true, for example, of the provincial legislation protecting the Registered Professional Planner designation of the Ontario component of the Canadian Institute of [land use] Planning. (return to Table of contents)
3.4.1.8 Applying the standards: monitoring certified members
Such monitoring is largely to ascertain whether certified members meet the requirements for continuing education and ethical behaviour. Those organizations we know about each ask their certified members for an annual report on their continuing education activities; usually the organizations do this in conjunction with the membership renewal. It appears that breaches of the codes of ethics are addressed on the basis of specific complaints, presumably from dissatisfied employers or clients.
Monitoring would also have to identify both non-payment of annual dues and failures to meet any requirement to continue to work in the field. Monitoring the former would happen automatically as part of the collection of annual dues. The latter could be easily enforced by requiring an annual report on continued employment confirmed by clients or employers.
In any event, monitoring means that the organization needs to know whom to monitor. This implies in turn that the membership lists must be current, reflecting new members, members transferring between provinces, and discontinued members. (return to Table of contents)
3.4.1.9 Applying the standards: disciplining certified members
Disciplinary action can have several levels of severity. At the one extreme is permanent withdrawal of the designation and expulsion from the organization. At the other extreme would be, say, a written reprimand or warning from the chapter president. In between are such penalties as the temporary loss of the designation, or the publication of a written reprimand.
Machinery and rules are necessary for disciplinary action. Committees must be established to make the disciplinary decisions, and they must be given rules and procedures to follow. Among other things, the rules must establish where the burden of proof lies, whether the criminal law principle of "beyond all reasonable doubt" or the civil law principle of the "balance of probabilities" is to apply, and what detail and form the judgement is to have.
The rules also have to state when appeals are permitted and how they are to be handled. Where there are provincial organizations, the initial disciplinary action will often be taken by a provincial committee, with appeals to the national committee. Provincial legislation that protects title to the designation may set out the procedures for internal appeals and for appeals at some point to the courts. (Again, the Ontario legislation protecting the Registered Professional Planner designation does so).
This all sounds like heavy stuff, and indeed, putting the machinery and rules in place likely requires a fair effort. However, the good news is that none of the examined organizations have had to use the discipline machinery and rules very often. Even where there are breaches, most are sufficiently minor to be corrected with a letter of warning, sometimes from the organization's lawyer, without any further disciplinary action by the organization. The Institute of Certified Management Consultants of Canada said that it was very rare for the extreme penalty, the permanent withdrawal of the designation, to be imposed. (return to Table of contents)
3.4.1.10 Applying the standards: dealing with clients or employers who are dissatisfied with the professional performance of certified members
None of the informants knew of any examples of legal action directed at the organization by dissatisfied clients or employers. Indeed, they didn't even mention complaints from clients or employers. Nonetheless, at least some of the organizations (the Canadian Institute of [land use] Planning is an example) have insurance that protects the staff and directors against legal action. (return to Table of contents)
3.4.1.11 Applying the standards: warning off people who pretend to be certified
In the earlier discussion of professional designations, the reader will have noticed a certain vagueness reflected in words like "evidently", "appear" or "seem". This is because the informants themselves were often rather vague about the nature and implications of the legal basis for their designations. The reason for this appears to be that policing the use of the designation is only a minor task. There have been only a few cases of misuse of the designations, and nearly all have been corrected by a phone call or letter to the offending person, who often seems to be acting in ignorance (A senior staff member of an organization with only the "semi-protected" designation under the federal Trademarks Act was completely unaware of the possibility of a protected title under provincial legislation. Clearly the means of protecting title had not been a subject of very intense discussion in that organization, which rather confirms the conclusion that title protection is not a major practical issue.). (return to Table of contents)
3.4.2 Activities to assist the candidate to become certified
Once an organization had developed its body of knowledge and the related exams, it could in principle leave it entirely to the candidate to find means of acquiring that knowledge and passing those exams. The Institute of Internal Auditors does this and the Canadian College of Health Services Executives comes very close to doing it (As noted earlier, the Canadian College of Health Services Executives has a one-half day Leadership Skills Workshop, which candidates for the Certified Health Executive designation must take before writing the examination. An optional half-day workshop in writing skills is offered during the same day.). However, some organizations take additional steps to assist candidates. These vary in difficulty from the easy to the onerous. (return to Table of contents)
3.4.2.1 Cross-referencing to recommended textbooks
The simplest method is to annotate the statement of the body of knowledge with cross-references to the appropriate sections of various textbooks, preferably to ones that are competent, economical and readily available.
The Institute of Certified Management Consultants of Canada has done this for the "functional" portions of its body of knowledge, using texts that are sold by Coles or Advantage Books. We did not enquire whether this organization has arranged any price discounts with these suppliers. In any event, the texts are not exactly inexpensive -- averaging about $65, plus the costs of shipping from Toronto. Nor are they always in print, but where they are not, the Institute will provide photocopies of the text. (return to Table of contents)
3.4.2.2 Making it easier to take training and write exams
The mandatory five day and three day courses of the Economic Developers Association, which are offered exclusively by the University of Waterloo, are delivered in major locations across the country. One of the five day courses is always held in conjunction with the annual conference. As the principal resource person, the UW program director travels to these locations to deliver much of the program himself. However, his sessions are supplemented by local experts, including certified members of the Association. Travel discount packages are arranged by the Association for those candidates who are not located in the selected major locations. The Association has considered offering a distance education program, but has found the cost prohibitive.
The oral exam of the Canadian Institute of [land use] Planning is often conducted on a telephone conference call with the candidate and the two or three examiners. The Canadian College of Health Services Executives allows the multiple-choice exam to be written wherever the candidate can arrange for supervision. It is also considering how that exam might be conducted in remote locations by electronic methods. (return to Table of contents)
3.4.2.3 The accreditation of courses and programs
Another form of assistance to the candidate is the accreditation of courses and programs. Accreditation is carried out in what might be termed proactive and passive ways. A typical proactive approach is the periodic visits of an "education committee", from either the national or provincial organization, to each of the institutions to confirm that the organization's standards continue to be met. This proactive approach generally applies only to a program.
A variation of this proactive approach was undertaken by the Economic Developers Association when it conducted a survey of all universities in Canada to create a list of courses relevant to members' work. (A previous survey had established what work members were actually doing.)
The passive approach to accreditation is generally used for individual courses. Here a candidate will identify a course that appears to cover a particular portion of the knowledge standards. The candidate then requests the organization's agreement to accept the course as a substitute for taking the organization's exam on that portion. This approach is taken by the Institute of Certified Management Consultants of Canada for the several "functional" portions of its knowledge standards. By this means, the Institute has built up a "bank" of approved courses. Although this approach requires the organization to review the content of the courses, at the same time it reduces the organization's exam administration burden. We did not verify this, but presumably it periodically reviews the syllabus of each course to ensure that it has not shifted significantly over the years. Although being able to take accredited courses is a useful option for candidates, it would be rather more costly to them than a combination of home study and the writing of the organization's exams.
A variation on accreditation would be to arrange with the post-secondary institutions giving accredited courses to allow candidates to write the exams without taking the courses. This would avoid forcing candidates to take courses covering material they already know. It would also reduce the number of exams that an organization would have to conduct itself. It would be especially useful for examining candidates in locations distant from major centres. However, we have no examples of this arrangement. (return to Table of contents)
3.4.2.4. The development of courses
A fourth form of assistance to the candidate is the development of courses that are tailored to various portions of the body of knowledge. The organization might do this by itself or in collaboration with a post-secondary institution. Examples of the former are the half-day leadership assessment workshop and the associated writing skills workshop of the Canadian College of Health Service Executives, the one-day workshops given by the Institute of Certified Management Consultants of Canada before each of its Introductory and Comprehensive exams, and the 150 hour training program offered by the Canadian Securities Institute.
Examples of courses developed by an organization in collaboration with post-secondary institutions are those of the parent bodies of the Financial Planners Standards Council of Canada and the five day training and three-day seminar programs that the Economic Developers Association developed jointly with the University of Waterloo.
Most of these courses are mandatory (The single obvious exception is the Introductory Workshop of the Institute of Certified Management Consultants of Canada. A less obvious example is provided by the fact that candidates need take only six of the three day seminars that the Economic Developers Association offers through the University of Waterloo.). As noted earlier, they are therefore effectively part of the education requirement for certification. They are not simply a means of helping the candidates to pass the related certification exam. (return to Table of contents)
By this term we mean the automatic award of the designation to members who are in place at the time of the adoption of certification. The Institute of Certified Management Consultants of Canada allowed its members to use the designation for 12 months without writing the exams. The approach of the Canadian Institute of [land use] Planning is lost from the readily available corporate memory, as is that of the Canadian Securities Institute and the Economic Developers Association (although informants of the latter two were certain that some form of grandparenting had existed). The Canadian College of Health Service Executives created, and still has, a category of associate membership for those who have not acquired the CHE designation. The Financial Planners Standards Council of Canada, and probably some of its parent organizations, were created for the express purpose of providing certification, so the question of grandparenting did not arise for it. As we go to our electronic "press", it is still unclear what was done by the Institute of Internal Auditors, which had existed from 1942 to 1972 without certification. (return to Table of contents)
3.4.3 Activities to enhance the benefits of certification
We have found two examples of activities that are not an inherent part of the certification process, but help to achieve the desired impacts of increased income and respect for the profession, at least for certified members.
3.4.3.1 Publicity targeted at potential clients and employers.
This publicity is to convince clients and employers of the importance of the designation. Three of the organizations mentioned the need for it without prompting.
The head of the Institute of Internal Auditors Toronto chapter said that he always asks to dinner each new certified member and the most senior supervisor available to ensure that the latter person has a full appreciation of the significance of the designation.
The Institute of Certified Management Consultants of Canada believes the level of awareness among clients and employers for its CMC designation could still be raised substantially (although it's clearly at a significant level already -- see next section). Past attempts to develop publicity programs have been side-tracked by the concentration of the organization's limited financial and volunteer resources on the certification process itself. The intention had been for the national institute to develop informational material and for the provincial institutes to distribute it. Presumably the national office will assume responsibility for the total publicity function under the new centralized support structure of this Institute.
The Financial Planners Standards Council of Canada said that the lack of public awareness for the CFP designation is reflected in the fact that only 10% of (personal)
financial planners have so far obtained it. The organization is apparently planning a publicity program to overcome this.
The acquisition of an exclusive right to practice in a particular field would of course be a powerful substitute for publicity. We have already seen the interest that the Canadian College of Health Services Executives, Canadian Institute of [land use] Planning and Financial Planners Standards Council of Canada have had in "upgrading" their designations to this level. However, regardless of any other consideration, a strong publicity campaign would be necessary to gain the provincial governments' agreement to pass the legislation needed for that purpose. (return to Table of contents)
3.4.3.2 Identification for clients and employers of certified members with the required specialties in a given location
The BC Institute of the Institute of Certified Management Consultants of Canada
said that one of its major services is the identification for potential clients and employers of appropriate certified members. Other organizations might also do this, although the question was not regularly posed in the interviews. In any event, the provision of this service also requires that the membership lists be current, reflecting new members, members transferring between provinces, and discontinued members.
Note in passing that this activity and the preceding publicity activity have a rather symbiotic relationship. The publicity would advertise, among other things, the member identification service. And reference to the identification "hits" could be included in subsequent publicity. (return to Table of contents)
3.5 Costs of the Certification Activities
A quick examination of Appendix B will show that, in our sample of certifying organizations, there is a crude inverse correlation between the number of members on the one hand and the annual membership and certification fees on the other. This correlation is unsurprising since certification encompasses a number of fixed costs, and the more members they can be "spread" over, the less the cost per person.
Further, the average annual fee of certifying organizations ($255) is higher than that of the four non-certifying organizations ($125), including the CES.
These figures do not take account of the volunteer contribution, or of possible earnings from seminars and conferences. However, these resources also likely vary in proportion -- directly this time -- to the size of the membership, so they are not likely to offset the fixed cost that a small organization must ask its relatively few members to cover. (return to Table of contents)
3.6 So Has Certification Really Been Worth It?
None of the organizations we examined, or indeed were able to find, had adopted certification but then had become sufficiently disappointed to dispense with it. However, this fact is not a satisfactory guide for the CES. The difficulty is that, for those organizations that had adopted certification, the initial investment in dollars, volunteer time and hassle had already been made and was thus a "sunk cost". But this investment has not yet been made by the CES; for the CES, it is not a sunk cost. Thus, the relevant guide for the CES would be an answer from the certifying organizations to the following question: "If you had to start all over again, and yet you still had the wisdom of hindsight, would you proceed or not?"
We did not ask the informants to speculate in this fashion. However, four of the organizations -- the Institute of Certified Management Consultants of Canada, Canadian College of Health Services Executives, Financial Planners Standards Council of Canada, and Economic Developers Association -- have made extensive reviews of their certification processes. In the end, all decided to retain certification, even though some also decided addtional expenditures were needed to support it.
An informant from the Institute of Certified Management Consultants of Canada said that certificatIon had been a recurring issue almost from the beginning and emphasized that any organization contemplating certification should think long and hard about whether the benefits will be worth the costs. A large commitment of volunteer time and money was needed to run the certification process (including, in the case of the Institute, the eight exams in two languages) and the supplementary activities (including the cost of publicity directed at clients and employers). The recent review of certification led to the conclusion that on balance certification was still desirable, but that it had to be managed more economically -- thus the decision to centralize the support facilities in the national office. The CMC designation should be "retained and embraced", but primarily as a "marker" of professionals who strive for excellence. Moreover, we were told that the benefit was by no means an assurance of financial rewards; there were management consultants without the CMC designation who were doing quite nicely, and others with it who were not doing especially well (What is not clear is whether, other things equal, people with the CMC designation do better than those without, and if so, by how much. A study would have been necessary to address this question, but apparently none has been carried out. During the interview, we gathered, without obtaining confirmation, that the comment by the informant was based only on impressions. Also mentioned was the fact that, although it was the large firms that were originally the driving force behind certification and the creation of the Institute, now only about 30% of the CMCs work for those firms. So if certification had ever been seen by the big firms as a means of dominating the profession, it has not obviously been successful.).
As already noted, the Canadian College of Health Service Executives had been disappointed in its expectation that certification would be a money-maker for it as an organization. Moreover, the CHE designation seems to carry little weight in recruitment decisions. Further, the task of keeping the knowledge base up-to-date and of renewing the exam questions had been demanding. Still, although the certification requirements and process had been reviewed and modified in 1994, in the final analysis certification itself was retained.
The Financial Planners Standards Council of Canada is a new organization. On the other hand, it is the offspring of several organizations, at least some of which had a form of certification. These organizations decided it was essential to keep certification, but thought it could be more efficiently administered on a consolidated basis, and thus created the Council.
The Economic Developers Association felt quite strongly that certification had contributed to the viability of profession. Although progress would likely have been made toward this in the absence of the certification program, the enormous amount of effort required has been worth it.
The other certifying organizations seem to have little doubts about their choice. However, the Institute of Internal Auditors is a very large organization able to afford a strong permanent staff. The Canadian Institute of [land use] Planning has had certification in some form for so many decades that its members would likely be unable to imagine anything else. In any event, this Institute has apparently not had any agonizing reappraisals, and seems satisfied with the recognition that employers give to the certified planner designation.
So far, we have been speaking about the attitude of the certifying organizations to the worth of certification. What do the non-certifying organizations have to say about it?
Many of the members of the Canadian Comprehensive Auditing Foundation are interested in moving in toward certification. but the majority believes that there are a lot of steps to go through before this can be done. Certainly, the existing project on proficiency requirements does not encompass the issue (A "strong minority" of existing Foundation members already have one or other of the accounting designations. Among the majority are a number of non-accounting "specialists". The focus groups working on the "proficiency requirements" considered whether the accounting skills required to obtain an accounting designation should be among those requirements. The members of these groups split largely along the lines of who had, and who did not have, such a designation. The majority prevailed in rejecting the idea. It was apparently suggested during the proficiency requirements project that, even if the present research work were not to lead to a comprehensive auditor designation, it might have some impact on the certification requirements of the three accounting associations. The Management Accountants organization is in fact now updating its concept of proficiency and considering consequent changes to its certification requirements. It's unclear however whether, in this project, this organization is taking account of the Foundation's work on proficiency requirements.
). The view seems to be that at least some of the steps that need to be taken for certification are justified regardless of whether or not the organization decides to certify.
As already noted, the Planning Forum has been shifting its emphasis from mere "planning and budgeting to "strategic management and strategic leadership". This organization has not yet started to consider what knowledge is required for the effective performance of such a function.
The Association des économistes québécois was the only organization in our
sample to explicitly consider and reject certification. In this case, the discussion of certification was promoted by several Board members. An open debate was conducted through several mechanisms, including a "for" and "against" exchange in the Association's newsletter, focus group-type meetings in several regions, and a member poll. The membership clearly rejected certification on the grounds it would not be required in most of the work settings occupied by members, and would thus present no particular advantages. However, our informant said that the existing Board members are sufficiently interested in the idea that they will likely bring it forward again in two or three years. (return to Table of contents)
4 IMPLICATIONS FOR THE CANADIAN EVALUATION SOCIETY
So what are the implications for the CES? We need to consider this largely under
the same headings used to discuss the approaches and experience of the other organizations.
4.1 The Level of the Designation and the Consequent Structure of the CES
We have seen there are broadly four types of designation, each with a particular level of "protection" and with a particular set of consequences for the structure of the organization. We believe the "semi-protected" title provided by certification marks under the federal Trademarks Act would likely be quite suitable for the CES.
This level of protection is relatively easy to acquire because it involves only one government. In contrast, fully protected title would require the passing of legislation -- likely by means of a private members' bill -- in most provinces. Moreover, unlike fully protected title, it would not require the creation of provincial organizations to serve as the legally substantive bodies within the CES (To use a crude analogy from poli-sci 100, the CES would remain essentially a unitary state, which delegates certain functions to local governments. It would not have to become a confederation of states that delegate certain functions to a central government. The new structure of the Institute of Certified Management Consultants of Canada doesn't fit easily into this analogy: it's rather like a confederation with a single public service operated by the central government -- an uncommon arrangement to our knowledge.). Yet, if needed, it would provide a specific basis for legal action against an impostor. This would be a civil, not criminal, action, but this appears sufficient since impostors have not been a serious problem for any of the organizations, including those which have had this type of protection. As suggested earlier, even without certification marks, a civil action would likely be successful against those who wrongly asserted they were certified members of the CES. However, without the certification marks, there would be nothing to prevent them from using the CES designation (e.g., CPE -- Certified Program Evaluator).
In not carving out an exclusive field of practice, the "semi-protected" title would avoid the difficulties of possible conflicts with other organizations (see 3.2.2) and of possible inconsistency with the responsibility of program agencies for self-evaluation (see 3.2.1). (return to Table of contents)
Recall the four Intended Impacts that we distilled from those mentioned by the organizations we examined:
(a) more revenue for the organization,
(b) more income for practitioners,
(c) better quality practice,
(d) more public respect for the profession.
In its deliberations, the CES might well define other possible benefits that certification would yield. However, this distillation and some commentary might serve as a useful starting point.
For reasons already discussed, Impact (a) seems unrealistic and ethically questionable. Impacts (b), (c) and (d) would apply primarily to those practitioners who become certified. They would not apply -- or worse, perhaps apply "negatively" -- to others. If the present membership of the CES is composed of significant numbers of each type of member, certification would likely be quite divisive. The requirements proposed for certification would thus likely be very heavily scrutinized.
One option here would be to put the matter to a vote (after perhaps a consultation process modelled on that used by the Association des économistes québécois or parts thereof). If the majority were in favour and still large enough to make it financially feasible, then the CES would presumably proceed with certification. If the majority were against it, then the CES would not proceed.
Regardless of the size of a favourable majority, it's hard to predict what impact certification would have on the size of CES membership. Some present members who are opposed to certification might leave never to return. However, one hears of people who now practice evaluation but are not CES members, and the desirability of becoming certified might change their minds. The net effect on CES membership would of course be a function of both factors (It would be conceivable -- how likely is another thing -- that a pro-certification minority could create a new, splinter organization for the introduction of certification. Again, it would have to be large enough to make the start-up costs manageable, and would be more viable if it included opinion leaders in the profession. In any event, if this were to happen, then even if they were in the majority, the opponents would find themselves in the same invidious position they would have been in had the CES itself proceeded. However reluctantly, many would eventually feel it necessary to try to join the new organization.).
Impacts (b), (c) and (d) will be achieved only if clients and employers are impressed with the importance of certification and the related professional designation. But the more they are impressed, the more they will make invidious comparisons between the certified and the uncertified, and the more therefore will the certified gain and the uncertified lose. We fear that certification that doesn't lead to such comparisons is unlikely to serve any purpose at all.
We have seen that the experience of the examined organizations is inconclusive about the attitude of employers and clients to certification. The Institute of Internal Auditors and Canadian Institute of [land use] Planning report positive results, although this is after many years of existence and, at least in the case of the Institute of Internal Auditors, some effort to publicize the importance of the designation. The Institute of
Certified Management Consultants of Canada is more guarded, although its recent review ended with a renewed "embrace" of certification. The Canadian College of Health Service Executives is doubtful that its designation carries much weight with employers or clients.
For these organizations, certification does not seem to have been a very divisive force. However, their experience may not be very relevant to the CES. With one exception, they all appear to have existed for only a few years before certification was introduced, so that members would not have become used to the idea of a non-certifying organization. Indeed, our impression is that the Institute of Certified Management Consultants of Canada, the Canadian College of Health Services Executives and the Canadian Institute of [land use] Planning were created with at least a vaguely defined goal of certification. For example, the creation of the Institute of Certified Management Consultants of Canada in its original incarnation was accompanied by a call for "professional standards of competence and ethics". The Financial Planning Standards Council of Canada (and probably many of its parent organizations) were established for the very purpose of certification.
It's true that the Institute of Internal Auditors had been in place for 30 years before introducing certification (in 1972). However, its Board of Governors acted in response to a broad-based desire within the membership for professional recognition. We didn't explore this very far in the interview, but it seems that many of the internal auditors had neither much formal education nor other designations, and were competing with many people who already had one of the three accounting designations. Certification was therefore attractive to them as a means of levelling the playing field. In contrast, CES members' average level of education is quite high, and in general they are not yet competing against the certified members of another profession. (return to Table of contents)
4.3 Activities Required for Certification
4.3.1 Level of effort required
The activities required for certification will require a great deal of volunteer time. Much of it will be in the form of participation on committees, working groups, task forces and so on. Recall that, in creating such groups for its recent review, the Institute of Certified Management Consultants of Canada did not try to create "great Canadian committees", each with representation from all parts of the country. Rather it created local groups whose members could meet easily and economically, and assigned to each group one of the sets of problems that the organization had to deal with. Only the review and final approval of the several committees' recommendations were done by a national group. This Institute suggested that, were the CES to proceed with certification, it adopt a similar approach. This seems to us like a good suggestion. (return to Table of contents)
4.3.2 Inherent activities: developing the body of knowledge
In the casual discussions we've had with CES members about certification, perhaps the most recurring concern has been the difficulty of developing and updating (For example, the arrival of fifth-generation evaluation, if it appeared to have something more than a snappy name, would require some updating.) a body of knowledge in what is invariably called a "multi-disciplinary" profession. This concern is certainly reflected in the literature (See for example Worthen B.R., "Is evaluation a mature profession" New Directions for Program Evaluation; The preparation of professional evaluators: issues, perspectives, and programs Altschuld and Engle eds. Jossey-Bass, summer 1994, p.9. ). Based on our review of other organizations' experience, we now think that it has been exaggerated.
First, it's not at all clear that evaluation is a "multi-disciplinary" profession. Certainly, one finds practitioners that come from several academic disciplines, but evaluation does not require the full range of knowledge found within them. Rather it requires primarily a sub-discipline -- social science research -- that not only is well-defined and highly developed, but is composed of elements that are shared by the academic disciplines from which the large majority of evaluation practitioners come. Indeed, the membership of the BC Chapter (with 90 members out of 100 reporting) is almost 90% from the social sciences. The remaining 10% are from business and commerce and even they get exposed to statistics and survey research methods. In the ways that count -- i.e., in terms of the content that evaluators would likely think important to include in a certification examination -- academic backgrounds could not be much more homogeneous. Certainly, they are less varied than those of the Institute of Certified Management Consultants of Canada, with its significant complement of science and engineering graduates.
Also very desirable would likely be some knowledge of the constitution, structure, culture and operations of each of the three levels of government and not-for-profit organizations. So also would be the principles of project management. But these too are all well-developed fields of knowledge, and indeed are widely available in courses offered by post-secondary institutions across the country (more on that later) (Caron reaches rather similar conclusions in his paper of a few years ago on this subject. See especially pages 69-75, Caron,D.J. "Knowledge required to perform the duties of an evaluator", Canadian Journal of Program Evaluation, April/May 1993.). It should not be difficult to decide on the elements of these portions of the body of knowledge.
Probably there should also be instruction on the various "philosophies" of evaluation, such as those reflected in the writings of the likes of Scriven, Campbell, Stake, Patton, Lincoln & Guba, Cronbach, Rossi, the Office of the Comptroller General and so on. Although those developing the body of knowledge might well differ in their evaluation ideology, they need reach no agreement on this; rather, they need reach agreement on only the writers to be included, and that shouldn't prove too challenging.
Certainly, this body of knowledge appears to be at least as easy to specify as that of the Institute of Certified Management Consultants of Canada or the Canadian College of Health Services Executives. And certainly this body of knowledge could be at least as easily related as that of the Institute of Certified Management Consultants of Canada to specific existing courses at a large number of post-secondary institutions and to sections of a great variety of text books.
In short, we are skeptical about the contention that evaluation is "multi-disciplinary" in any real sense of the word (And even if it were, it would be still be just as desirable, if more difficult, to define what the profession professes to practice.). We expect that a committee of members with advanced training, primarily in the social sciences, could easily define the types of knowledge that evaluators should ideally have. The elements of the Essential Skills Series would be a good starting point for this.
More difficult would be the decision on the required depth of each type of knowledge. Here it's a question of what one needs in order to do a given task him/herself, as opposed to doing it with the help of an expert. And (to draw on the thinking of the Canadian Comprehensive Auditing Foundation), this then raises the question of what one needs to know in order to "judge" when to call in a given type of expert, to find the right expert, and to manage him/her most effectively.
Also tricky is deciding what would constitute "agreement" over the contents of the body of knowledge. In general, the larger the number of elements in the body of knowledge the smaller the proportion of members that would accept it. There would likely be many elements that would receive near unanimous approval among those members who were ready to accept some form of certification. The easy option would be to adopt only those elements as the body of knowledge. But there would be a much larger number of elements on which a small majority would agree, and the issue would then be whether it should impose its views on the large minority. Beyond that there would be any number of bodies of knowledge that only minorities of various sizes would support. For both the small majority or any minority the question would be whether the number of members that it constitutes would be large enough to provide the volunteer and dollar resources needed for certification.
Very relevant in this regard is the "inclusionary" approach of the Economic Developers Association, discussed in 3.4.1.1. Recall that, by requiring the mastery of a basic body of knowledge and the body of knowledge of only one of several possible specialties, the Association has made the required knowledge a matter of degree. This arrangement means in turn that there is more room for compromise in the definition of the body of knowledge. It would thus significantly take the edge off, but not eliminate, the difficulties described in the preceding paragraph. (return to Table of contents)
4.3.3 Inherent activities: developing and updating the education and experience requirement
Most of the organizations required a bachelor degree and a minimum of two years relevant experience. The Institute of Certified Management Consultants of Canada, which represents the professional group that is closest to evaluators, requires three years and proof of five consulting assignments. On the other hand, it will substitute for the bachelor degree an additional five years of experience that it deems to be the equivalent. The Canadian Institute of [land use] Planning requires four years experience if the degree is only "related to" land use planning, and six years for a degree that is "unrelated to" planning.
In the CES, all but 14 of the 822 members that reported on their education have at least a bachelor degree, so requiring one as a matter of course would not likely be very contentious. Still, we wonder whether there's a need for it if a candidate could pass exams that rigorously test his/her understanding of a substantial and challenging body of knowledge. True, a bachelor degree is routinely used by employers as a screening device to keep the number of applicants manageable, but it's unclear that this is a problem for the CES. If the degree is simply a proxy for greater maturity, the CES could follow the practice of the Institute of Certified Management Consultants of Canada and substitute a few years work experience for the degree.
Actual practice in applying the body of knowledge is clearly important. Thus a minimum of two years or three years experience hardly seem excessive. If the CES wanted to limit its membership to those who have demonstrated a commitment to evaluation, an experience requirement of this magnitude would provide a reasonably valid screen ( If the CES were to contemplate a more senior designation -- something we have not systematically investigated or discussed earlier -- then additional experience should perhaps be a requirement for it. Of the organizations we examined, only the Canadian College of Health Services Executives has a second level designation, this in its Fellow of the College of Health Service Executives. To obtain this designation, one must already have the basic CHE designation, have five years of experience (instead of two), undertake a "work project" (three case studies, thesis or a "special project"), undergo an oral review, and agree to make future "contributions to the College".). (return to Table of contents)
4.3.4 Inherent activities: developing a code of ethics
The CES already has a code of ethics. Presumably the CES would follow the practice of the other organizations and make adherence to the code a condition for the maintenance of certification. We think, however, that the code would then need to be reviewed to ensure that it is sufficiently clear and unambiguous for this purpose. Mechanisms for dealing with breaches in ethics would have to be developed.
In the event of certification, the code might need to be amended in substance. For example, the CES might wish to emphasize in the code not only that the techniques and attitudes of evaluation would remain available for use by managers and administrators in all components of a public or private enterprise, but also that it was the responsibility of evaluators to promote their widespread and general use. Indeed, the skills needed to meet such a requirement of the code of ethics might need to be added to the body of knowledge. (return to Table of contents)
4.3.5 Inherent activities: testing a candidate's knowledge
We have already seen the large variation among the organizations in the manner in which they test for knowledge. The most suitable methods of testing should be selected in light of the character of the CES body of knowledge. The many professional educators among the CES membership should be able to advise on these methods. In doing so, they may wish to take account of the practices and experience, described here, of the other organizations, including the steps taken to make the exams more convenient for the candidate. We would only draw attention to the point that testing has been a major burden for both the Canadian College of Health Service Executives and the Institute of Certified Management Consultants of Canada, which have relied heavily on multiple choice questions and on their development by volunteers. The Institute of Internal Auditors also uses multiple choice questions, but it is large enough to afford professional staff to prepare the exams. (return to Table of contents)
4.3.6 Inherent activities: other issues
These other inherent activities are: verifying the experience claimed by candidates; responding to the complaints, appeals and legal action of unsuccessful candidates; monitoring certified members with regard to continuing education and ethical behaviour; dealing with clients and employers who have been dissatisfied with the performance of certified members; and warning off impostors, if necessary by legal action.
These have not been major, or even significant, tasks for the examined organizations, and there's no reason to believe they would be for the CES. However, the processes for dealing with them must be defined in the bylaws so that they can be quickly put into operation when necessary. The practice and experience of the examined organizations provide suggestions on the form the processes might take. (return to Table of contents)
4.4 Activities to Assist the Candidate to Become Certified
Very helpful to candidates and probably not too onerous for the CES would be the facilitation of exam writing (Canadian College of Health Services Executives), and the accreditation of post-secondary courses or programs as providing a specific portion of the body of knowledge (Institute of Certified Management Consultants of Canada). So would be the cross-referencing of the body of knowledge to recommended texts (as done by Institute of Certified Management Consultants of Canada). Very likely a special price discount could be worked out for any texts from Sage Publications that the CES used for this purpose.
More problematic is the development, perhaps in collaboration with post-secondary
institutions, of new courses or programs tailored to teach specific portions of the body of knowledge, including professional ethics. For much of the body of knowledge, this may not be necessary, since most of that knowledge would likely be covered by rather common post-secondary courses. However, except in large centres, there would probably not be enough demand for courses in the philosophy of evaluation or professional ethics, regardless of whether it were offered by the CES itself or by a collaborating institution. Thus, for many candidates, these might be subjects they would have to acquire from a text, preferably cross-referenced, or from special documents, and then be examined by the CES. (return to Table of contents)
4.5 Activities to Enhance the Benefits of Certification
The two activities -- rather symbiotic, as already noted -- are publicity targeted at potential employers and clients, and the identification for employers and clients of certified members with required specialties. How much to do of either is a judgement about the consequent costs and benefits. These tasks should be approached gradually, and the results evaluated. (return to Table of contents)
The costs that would be critical to the membership are the annual fee and the cost of becoming certified. The Institute of Certified Management Consultants of Canada and the Canadian College of Health Service Executives are the organizations most comparable in size, structure and type to a certifying CES. Respectively, the annual fees are $390 and $365 and the certification fees $900 and $800. The certification costs for the Institute of Certified Management Consultants of Canada are significantly more if the candidate substitutes an accredited course for each of the six functional exams. Any travel costs are also additional.
Annual fees are of course a deduction on taxable income, so that the net cost to the candidates would be reduced by their marginal tax rates. Thus, with the middle marginal tax rate, a fee of, say, $400 would be reduced to $260. This would be about $22 per month, or about $16 more than now being paid.
We did not check on this with the tax authorities, but presumably the certification fees would also be a deduction from taxable income, while the tuition fees for required courses would be a non-refundable tax credit (It's less obvious how Revenue Canada would treat a fee for a course that was a mandatory component of certification. Would it be part of the certification costs or a tuition fee? We leave it to others to find out.).
The consideration of costs raises the question of acquiring the economies of large scale operations. An organization can do this by becoming part of a larger organization or, while retaining a distinct identity, by sharing goods and services with another organization. Both the Institute of Internal Auditors and Financial Planners Standards Council of Canada fees are quite low. The Institute of Internal Auditors is a very large international organization, of which the Canadian chapters together form only a small part. The Financial Standards Council not only is the offspring of several smaller Canadian organizations but also has arranged, for a fee, to use the body of knowledge of its much larger American cousin. However, the Canadian College of Health Services Executives, whose fees are not low, also acquired the body of knowledge of its American counterpart, but had to make considerable modifications to the document for its use in Canada; to our knowledge, this organization has not continued to share certification services.
The CES should at least keep in mind the potential for international collaboration.
Its American counterpart does not yet certify, but it seemed for a while to be moving actively towards that goal, and may yet achieve it. If the American Association were ready in time, the CES might purchase its body of knowledge and the exams that are based upon it. As an interesting but problematic alternative, the CES might follow what was, at least in principle, Canada's strategy of promoting multilateral international arrangements in order to minimize its dependance on bilateral arrangements with the American elephant. The CES role in the 1995 international evaluation conference would facilitate and provide credibility to any demarches that it made for this purpose towards evaluation organizations in other countries or regions.
There appear to be no related organizations within Canada that might collaborate with the CES in the manner that the several parent organizations of the Financial Planning Standards Council did with each other. The Canadian Comprehensive Auditing Foundation seems to be fairly close in subject matter -- anyone who doubts this should consider the potential scope of the triple E audit -- but it is apparently uninterested in pursuing certification. Still, the Foundation recently prepared a document, to be available shortly, setting out its "proficiency requirements". This suggests that it is in fact progressing in that direction, perhaps even moving ahead of the CES. The Foundation document might be a useful contribution to the development of a CES counterpart document, regardless of the intentions of either organization about certification. It might also provide a concrete and up-to-date indication of the potential for collaboration in a move towards certification.
The final point on costs is that the CES now has a surplus of $100,000, built up over the last several few years, mainly from a few remunerative conferences. Were certification adopted as a goal, then these funds might be used in part to make some of the investments necessary to achieve it. (return to Table of contents)
4.7 Further Information Needed
The attitude of clients and employers is the major piece of information that is missing for a rational decision on certification. It is they who will increase the demand for evaluation in general and it is they who will decided how much preference they would give to someone with the designation. Clearly a survey is needed. Although its type and scale are beyond the scope of this paper, presumably it would be targeted to governments at all three levels, including in particular the central agencies that oversee the evaluation function, and to not-for-profit organizations.
Another missing piece is the provincial practices and laws regarding the protection of title. We have been working only with a few examples from BC, Ontario and Québec. As a result, we do not know just how much variation there is, although our impression is that, as often happens, there is a lot of inter-provincial copying. However, the substantial Quebec legislation regarding professional associations might be quite different. The need to do a more extensive examination of provincial legislation would be much reduced, if not eliminated, by a decision to use only certification marks under the federal Trademarks Act.
Finally, there is the question of the attitude of the CES membership. At the very least, before being asked to express an opinion, members should have the benefit of a balanced statement about what certification would imply. This in turn would mean that the Council would first have to decide, in approximate terms, the particular form certification would best take, and discuss its advantages and disadvantages in comparison with the status quo. It remains to be seen whether there would be enough agreement among its members for the Council as a whole to take a position on the best course of action. (return to Table of contents)
April 23, 1997
QUESTIONNAIRE FOR ACCREDITATION/CERTIFICATION (AC) STUDY
(NOTE THAT THE NUMBERING SYSTEM IS RATHER IMPROVISED. FOR EXAMPLE, QUESTION 99 (A) -- THERE ARE NOT REALLY AS MANY QUESTIONS AS THIS -- WOULD NOT NECESSARILY BE AN ELABORATION OF QUESTION 99. RATHER, IT IS LIKELY WE SIMPLY WANTED TO INSERT A QUESTION BETWEEN 99 AND 100, AND WANTED TO AVOID RENUMBERING ALL THE SUBSEQUENT QUESTIONS.)
QUESTIONS FOR ALL ORGANIZATIONS
THE ORGANIZATION
1) What sort of work does a member of your organization perform? Is there available a formal statement describing or defining this work?
2) How many members do you have?
3) What are the annual fees required of your members?
3a) What are each of the other main sources of revenue and how large are they?
4) How many people are there on the governing body of the organization?
5) How often does it normally meet?
6) How much is the organization's annual budget and what is its allocation among your main activities? Would it be possible to have a copy of the budget document?
7) What are the purposes of your organization (as opposed to the type of work performed by your members)? Is there a statement setting them out?
7a) Could I obtain a copy of the current legal document(s) (e.g., letter patent, or legislative act possibly with regulations pursuant to it) that establishes your organization?
QUESTIONS FOR ORGANIZATIONS THAT HAVE ADOPTED SOME FORM OF AC
PRESENT AC ARRANGEMENTS
8) Please describe the nature of your certification/accreditation process and requirements. (We would of course have to explain how we are using those terms and find out what terms they use and exactly how they define them.)
8a ) What broadly must a certified member know and what type and amount of experience must he/she have? Are there specialties for which further certification and/or experience is necessary? Is there a written statement summarizing the required knowledge and experience, both at the general and (if applicable) specialized levels?
8b) What criteria must accredited courses meet?
8c) What process do you have for the review (continuing or periodic) of your certification/accreditation requirements (both for the original award of AC and for its maintenance over time)?
8d) In practice, are many changes made to these requirements as a result of such reviews?
9) What is the cost to the individual of becoming certified, and to educational institutions of having courses accredited?
10a) What legal powers do you have over the practice of your profession? For example, can you prevent anyone other than a certified member of your organization from calling him/herself a ------------------? Or can you prevent an institution from saying that a particular non-accredited course is relevant to achieving your designation?
10b) Is it a federal or provincial statute that gives you this power, and what is its name? Would it be possible to get a copy? [IN PRACTICE, THIS REPEATS ON 7(a)]
10c) How in practice do you enforce your legal authority? What are the problems and costs associated with this?
10d) When introduced, were the certification criteria applicable to all members? If not, who was exempted, at least for a period? In particular, were existing members exempted? If they were exempted only for a certain time, how long was it?
10e) Has anyone who has failed to gain certification tried to sue the organization? If so, what was the outcome?
11) When did your organization seriously begin to consider AC? When did it formally introduce it?
12) What are the legal consequences to the organization of a professional mistake being made by a certified member? What measures, such as the purchase of liability assurance, does your organization take to deal with such consequences, and what are their costs?
12a) Once a person is certified, what criteria if any must he/she meet to retain the certification? In particular, is there a minimum amount of continuing education that must be taken in a specified time period, and if so, what are the criteria and process for deciding what education is acceptable for this purpose? Or is there a minimum amount of work (in the particular field) that must be carried out in each specified time period, and if so, what are the criteria and process for deciding what constitutes such work?
[NOTE: THE QUESTIONS ABOUT "PROCESS" HERE ARE NOT THE SAME AS THAT IN 8c. FIRST, 8c DEALS WITH BOTH THE ORIGINAL REWARD AS WELL AS MAINTENANCE. SECOND, THE PROCESS REFERRED TO IN 8c IS THAT FOR ESTABLISHING THE REQUIREMENTS FOR CERTIFICATION. THE PROCESS REFERRED TO HERE IS THAT NEEDED FOR DECIDING ON A PERSON BY PERSON BASIS WHETHER OR NOT THOSE REQUIREMENTS ARE MET BY THE CONTINUING EDUCATION AND/OR EXPERIENCE HE/SHE HAS BEEN ACQUIRING. OF COURSE, THE MORE SPECIFIC THE REQUIREMENTS THE LESS ONEROUS IS THE TASK OF DECIDING WHETHER OR NOT THEY HAVE BEEN MET.]
HISTORY
13) Who in the organization took the initiative to consider AC? Was there any particular spur for this initiative?
13a) What structure and process did you establish for the consideration of AC?
14) How much support was there for this from the membership at large? How did you seek the views of members?
15) Did the membership need a lot of convincing that AC should at least be seriously considered? If so, how did the organization do this?
16) What process was there for taking the decision to proceed with AC?
17) Once the decision was taken, how did you organize for its implementation? In particular, how did you go about reaching agreement on the knowledge and experience members would need to have to be certified, or on the content courses would need to have to be accredited?
18) What goals, objectives, or purposes did your organization have which it thought AC would help achieve?
18a) What academic background in terms of level and discipline did your membership have at the time AC was under consideration?
18b) TO BE ASKED ONLY IF IT APPEARS THAT ACADEMIC BACKGROUND WAS QUITE DIVERSE IN LEVEL AND/OR DISCIPLINE: What difficulties if any did this diversity create for reaching agreement on AC? How were these difficulties addressed?
ASSESSMENT
19) To what extent do you think those goals etc. have been achieved? What evidence do you have for this?
20) To what extent do you think those goals etc. would have been achieved without AC, and what is the evidence for this?
21) Have there ever been estimates made of the cost, in terms of volunteer time and out of pocket expenditures, for (a) establishing AC and (b) maintaining it annually? If so, what are those estimates? How much of the total cost has been reimbursed by the fees charged to individuals for certification and to institutions for course accreditation?
22) What improvements could have been made to the structures and processes for considering AC and for taking the decision to adopt it?
23) In what ways do you think the structure and process for implementing the decision to adopt AC could have been improved?
24) Have you made, or are you considered making, any changes to the particular form of AM that you originally adopted? If so, what are the changes and why are you thinking of making them?
25) Is the membership satisfied with your form of AC? If not, why not?
25 a) Has there been a significant change in academic background, in level or discipline, since the introduction of AC?
QUESTIONS FOR ORGANIZATIONS THAT DO NOT HAVE ANY FORM OF AC
HISTORY
26) What consideration, if any, has been given by your organization to the adoption of AC ?
27) Who took the initiative? Was there a particular spur for this initiative?
28) What organizational structure and process did you create for this consideration?
29) What process was there for obtaining the views of members? What were those views?
30) Was there an explicit decision either not to proceed, or did the issue simply never come to a head? If there was an explicit decision, who took it, the governing body or the membership at large? (For example, did the governing body recommend proceeding but the general membership rejected the recommendation?)
30a) What were the particular forms of AC that were being considered?
31) What goals, objectives, purposes did the proponents think AC would achieve?
32) What were the major reasons for not proceeding? For example, was the predominant view that the goals etc. weren't worth the financial cost and the hassle, or that AC couldn't achieve them anyway, or that they could be achieved more easily by other means (if applicable, what were those other means)?
33) What were the expected costs to the organization and how were they to be borne?
ASSESSMENT
34) What now is the general view about not proceeding?
35) Has there been any thought given to reconsideration of AC? If so, what is the present character and status of this reconsideration?
36) Have the goals of AC been achieved to some extent by other means?
FURTHER QUESTION FOR ALL ORGS
37) Do you have any other comments you would like to make about your organization's experience with AC?
(return to Table of contents)
|
|
Number of members |
Annual fee (for certified members ) $ |
Certification fee ($) |
Designation; level of protection |
|
Association des économistes québécois |
530 |
$110
|
NA |
NA |
|
Canadian Comprehensive Auditing Foundation |
950 |
$45 government $95 other |
NA |
NA |
|
Canadian College of Health Services Executives |
1300 (1800 other) |
$365 |
800 |
Certified Health Executive (CHE): Certification marks |
|
CES |
1340 |
$75 |
NA |
NA |
|
Canadian Institute of [land use] Planning |
4500 |
225 |
120 |
Name of designation varies: Protected in four provinces; non-protected in others |
|
Canadian Securities Institute |
30,000 in profession (?) |
Not the certifying body |
NA |
NA |
|
Economic Developers Association |
500 |
$350 principal $200 discounted (small budget organization) $150 multiple from same employer |
$100 for exam; about $2200 for courses and seminars over three years |
Certified Economic Developer: Certification marks |
|
Financial Planners Standards Council of Canada |
3750 |
125 |
300 |
Certified Financial Planner (CFP): Certification marks |
|
Institute of Certified Management Consultants of Canada |
3500 (?) |
390 |
900 |
Certified Management Consultant (CMC): Protected |
|
Institute of Internal Auditors |
35,000 or 55,000 (?) (780 in Toronto) |
160 |
60 |
Certified Internal Auditor (CIA): Non-protected |
|
Planning Forum |
6000 (1200 Canadian) |
240 |
NA |
NA |
CES main page
/ Page d'accueil de la SCÉ