When the rubber hits the road: How do we know our graduates are competent?

Psychology is entering an era of increasing accountability in student training. To meet this expectation, a paradigm shift in the teaching of postgraduate psychology is required. But what constitutes competence in our profession/graduates? A shift to a ‘competency model' is needed (Lichtenberg, et al., 2007; Voudouris, 2009, 2010) that assists in both assessing students' competency as well as enhancing their employability. However, this will require new methods for both graduate assessment and the management of assessment outcomes (Lichtenberg, et al., 2007; Leigh et al., 2007).

The focus on psychology students attaining skill competency, rather than simply checking that specified domains of education and training are undertaken, is reflected in the recent Australian Psychology Accreditation Council (APAC) standards which specify competencies that all postgraduate psychology courses must now demonstrate that their graduates meet. These represent core competencies required for all psychologists, but have not thus far been extended to specific competencies for specialist psychology training. This article discusses the specific competencies for organisational psychologists and briefly presents a job analysis approach to assessing student competency for the Master of Industrial and Organisational Psychology program at Deakin University (Deakin). Results so far are promising.

Determining the competencies

The competency approach requires identification of competencies for various professional programs. Bartram (2004) defines competencies as "sets of behaviours that are instrumental in the delivery of desired results or outcomes" (Bartram, 2004, p.246). Competencies therefore capture the repertoire of procedures, skills and abilities, attitudes, beliefs and values, dispositions and personal characteristics, self perceptions, and motivations that enable a range of work demands to be met effectively (Rubin 2007).

At Deakin, the competencies required for graduate organisational psychologists were determined through job analysis and competency modelling. Key stakeholders such as employers, industry bodies, the Australian Psychological Society and its College of Organisational Psychologists were consulted. Interviews were conducted with academics involved in both lecturing, supervising placements and managing the course, and recent course graduates/practising organisational psychologists.

Interviews also utilised the SHL universal competency cards (SHL Group Ltd, 2004) to identify and standardise the behaviours important for newly graduated organisational psychologists. Each card contained a list of behaviours relevant to a specific competency and interviewees were required to rate each card on a scale ranging from critical or essential to not relevant. Stakeholders were also required to provide workplace examples relating to the ratings and frequency of the behaviours.

Interviews were thematically analysed to ensure that no critical behaviours were overlooked in the final specification of required professional competencies. A final list of competencies were determined, with a number of behaviour dimensions identified for each competency. The identified competencies are listed in Table 1.

Competency assessment

The multidimensional nature of competencies requires a multi-faceted assessment approach (Leigh et al., 2007). Displayed behaviour of competence can be demonstrated through the utilisation of an Assessment Centre (AC) approach. This is designed to assess a person's competence utilising multiple criteria, multiple methods and multiple observers. Various tasks and tests are carefully selected so as to provide the best measure for the relevant competencies.

Assessment of behaviour and performance is therefore more direct, relying on instruments designed to measure ‘samples' of behaviour rather than the signs of underlying traits. The AC process has the advantage of deriving observable behaviour of the competencies and yielding higher criterion-related validity than other selection instruments (Turnage & Muchinsky, 1984; Robertson & Iles, 1988; Howard, 1997).

The Deakin program's approach

The AC approach was used in the organisational psychology program at Deakin to assess competency, albeit as a development tool. This was chosen to be conducted early in the course, allowing formulation of individualised learning objectives. Student learning could then be tracked over a two-year period and targeted feedback provided. On completion of the two-year Masters program, students' achievements were reviewed in relation to their attainment of the identified competencies.

Evaluation through Assessment Centre methodology

The Post Graduate-Development Assessment Centre (PG-DAC) was established to assess the identified competencies. Tests included an in-tray exercise, meeting and presentation role play, written report and leaderless group discussion. Students also completed an online personality assessment. Assessors included the placement coordinators, recent graduates, and an industry partner - PeopleMeasures - that provided industry expertise in an honorary capacity to assist with conducting the PG-DAC.

Comprehensive feedback was provided to each student and targeted placement development plans were formulated. This plan included the identification of strengths and development areas, along with individualised placement learning objectives. As a result, placement supervisors were equipped with the defined needs of the student, which assisted in the selection of the appropriate placement activities to enable the student to meet their objectives.

Results thus far

The performance evaluations collected to date (n=16) were measured by behaviourally-based rating scales completed by both students and their organisational supervisors. A longitudinal design was utilised with numerous evaluation points from work placement stakeholders. Despite a small sample size, predictive validity from the PG-DAC was positive and in line with most other research in this area. The ratings illustrated the positive impact of this methodology, where learning objectives are clear and articulated and tracked at an individual level.

Conclusion

The development of the PG-DAC enabled program coordinators to measure, develop and assess student competency in an effort to inform education processes and ensure graduate success. Early evaluation of this approach has been positive (Keele et al., 2010), but requires further and more detailed assessment. The application of the methodology appears to provide a much needed strategy for not only assessing professional competencies of specialty areas of psychology practice, but also providing a standardised way forward for universities to prepare, plan and continue to develop professional competencies in the context of work placements.

The principal author can be contacted at kathryn.vontreuer@deakin.edu.au.

References

Bartram, D. (2004). Assessment in Organisations. Applied Psychology: An International Review, 53(2), 237-259.

Bartram, D. (2005). The great eight competencies: A criterion-centric approach to validation. Journal of Applied Psychology, 90(6), 1185-1203.

Keele, S., Sturre, V, von Treuer, K.M. & Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students - a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45-64.

Leigh, I.R., Bebeau, M.J., Rubin, N.J., Smith, I.L., Lichtenberg, J.W., Portnoy, S., & Kaslow, N.J. (2007). Competency Assessment models. Professional Psychology: Research and Practice, 38(5), 463-473.

Lichtenberg, J.W., Bebeau, M.J., Smith, I.L., Portnoy, S.M., Leigh, I.W., Rubin, N.J., & Kaslow, N.J. (2007). Challenges to the assessment of competence and competencies. Professional Psychology: Research and Practice, 38(5), 474-478.

Rubin, N.J., Leigh, I.W. Nelson, P.D., Smith, I.L., Bebeau, M., Lichtenberg, J.W. Portnoy, S., Kaslow, N.J. (2007). The competency movement within psychology: An historical perspective. Professional Psychology: Research and Practice, 38(5), 452-462.

Voudouris, N. (2009, December). On defining competencies in the training of Australian psychologists. InPsych, 31(6), 32-33.

Voudouris, N. (2010, December). Towards assessment of professional competence in Australian psychology. InPsych, 32(6), 24-26.