Summary
Aims and objectives
The aim of the HACC Service Standards Consumer Appraisal Data Development Project was to further the development of a tool and method for collecting client appraisals of agency performance against the HACC National Service Standards. This project follows on from and is a complement to an earlier project that developed the Instrument to measure agency compliance with the HACC National Service Standards (Jenkins et al 1998).
The first phase of this work involved investigating and reporting on work that has occurred in the area. This literature review is reported separately and has the following reference: Cooper, D and Jenkins, A. 1999. Obtaining consumer feedback from clients of home based care services. Canberra: AIHW. (Welfare Division working paper no. 21).
The second phase of this work involved field-testing. This was done with four principal objectives in mind:
- to refine the survey tool developed by the HACC Standards Working Group;
- to test the usefulness of survey methods in providing information about the quality of the service provided by an agency;
- to test the capacity of this tool to stand alone as an accurate indicator of agency service quality as described by the HACC National Service Standards and to analyse the extent to which consumer appraisals provide a useful means of validating and informing HACC agency appraisals; and
- to examine the viability of survey methods according to such criteria as cost, timeliness, practicality, acceptability to clients, and usefulness to service providers.
Overall findings
The Consumer Survey Instrument (CSI) can be used as a means of gathering consumer feedback in relation to service quality. For the majority of consumers, this tool reliably measures consumers’ opinions of service quality as they experience it.
The CSI was found to be valid as a measure of service quality experienced by consumers in so far as it was clear, appropriate and relevant to their experiences and opinions.
Specific revisions to the CSI have been put forward to improve this reliability.
All three survey methods (mail survey, telephone interview, and focus groups) examined in this study have the capacity to provide valid and reliable information in relation to the HACC National Service Standards. Telephone interviews were found to be the most effective means of collecting consumer feedback about service quality; mail surveys were found to be the most efficient; and focus group were least efficient and of questionable effectiveness.
It cannot be concluded that this information can be used on its own as a measure of service quality because it remains to be shown that the CSI is sufficiently valid that it will accurately reflect the true level of service quality delivered to consumers. To properly undertake this validity test it is necessary to compare consumer appraisals with appraisals collected by another means; in this case, with Instrument assessment scores. These scores are not available at the time of writing.
No one method can be recommended for use in all agencies because particular characteristics of agencies and consumers may predispose toward or against the use of a particular method. For example, telephone interviews are not recommended for consumers of non-English speaking background.
The views of Aboriginal consumers can be successfully accessed through telephone interviews, face-to-face interviews and focus groups. The success of any of these methods is dependent on the broader context in which they are undertaken. They should reflect a genuine acknowledgment of the value of Indigenous elders’ views and be a part of the on-going consultation with and community involvement of elders in their welfare and health services.
None of the methods tested were observed to be sufficiently effective in eliciting critical comment from consumers of non-English speaking background. A more innovative approach is required for these groups.
The methods tested did not yield an exact representation of consumers from the HACC target population as measured by national statistics but telephone interviews and mail surveys adequately represented these consumers. Focus groups systematically under represented older and more dependent clients.
Consumer feedback should not be summarised by adding across items owing to difficulties in adequately accounting for missing data, valid “don’t know” responses, “other” responses and skewed data.
While each of the survey methods examined encountered difficulties overcoming the problems of high reported satisfaction that are typical of feedback obtained from this consumer group, half of the items were not affected by this response set. Thus the Instrument was shown to be capable of measuring feedback in such as way that it could distinguish different levels of service quality.
In terms of time elapsed (timeliness), when the administrative and process requirements of telephone interviews and focus groups are taken into account, mail surveys, telephone interviews, and focus groups weigh up equally.
Focus groups were the most time costly means of measuring consumer feedback.
Mail surveys posed the least administrative burden on service providers.
Focus groups were very well received as a means of seeking feedback from consumers, however telephone interviews and mail surveys offer a more acceptable alternative means of involvement for many, particularly for more highly dependent clients and carers.
All things being equal, telephone interviews are the most useful means of gathering consumer feedback.
Recommendations for further work
Having established that the CSI provides a reliable measure of consumer appraisal of service quality in both its content and method of application, it remains to be shown that the CSI is sufficiently valid that it will accurately reflect the true level of service quality delivered to consumers. To properly undertake this validity test it is necessary to compare consumer appraisals with appraisals collected by another means; in this case, with Instrument assessment scores when they become available.
Product of the project
As a result of the extensive field testing and detailed qualitative and quantitative analyses described in this report a set of revised Consumer Survey Instruments has been developed for use in assisting the assessment of HACC funded agencies against the National Service Standards.