Consumer outcomes in mental health care Print this page

This section reports confidence intervals in the data tables and visualisations. A confidence interval is a range of values that quantifies the statistical uncertainty in estimates that result from natural or random variation. For example, in the number of services provided and the number of persons using services over time. There are also non-random sources of uncertainty, such as incomplete reporting, that are not captured by confidence intervals.

Generally, confidence intervals describe how different an estimate could have been if the underlying conditions stayed the same but random fluctuations had led to a different set of data. Accordingly, it is recommended that confidence intervals are reported alongside a number estimate.

Confidence intervals are calculated with a stated probability (commonly 95%); this means we can be 95% confident that the confidence interval includes the true value if the assumptions made in the construction of the confidence interval hold. Larger numbers of observations yield more precise estimates with narrower confidence intervals. Confidence intervals can be used to perform tests of statistical significance. If the 95% confidence intervals do not overlap – that is, they do not include the same values in the range – the difference can be said to be statistically significant (note that differences can be significant in a subset of cases where the ranges do overlap).

In this section, 95% confidence intervals are shown in most figures and all tables.

Further information about confidence intervals, including calculation methods, statistical assumptions behind the calculation and sources of variability can be found in the data source section.

During 2019–20 for people aged 11–17, Involuntary status was recorded for over 1 in 6 discharge collection occasions in acute inpatient care (591 or 17.3%) and about 1 in 50 (273 or 1.9%) for ambulatory care (Figure NOCC.2).

For people aged 18–64, Involuntary status was recorded for over 2 in 5 discharge collection occasions in acute inpatient settings (17,525 or 40.9%) and over 1 in 8 (9,490 or 12.9%) for ambulatory settings.

For people aged 65 years and older, Involuntary status was recorded for nearly 2 in 5 discharge collection occasions in acute inpatient settings (1,493 or 37.9%) and over 1 in 11 (1,210 or 9.4%) in ambulatory settings.

Clinician-rated measures were completed at much higher proportions than consumer-rated measures during 2019–20. This is the case for each type of collection occasion (admission, discharge and review) and across all age bands for both acute inpatient and ambulatory settings.

For example, clinician-rated measures were collected at admission for between 89.7% and 94.5% of expected collection occasions while consumer-rated measures were collected between 19.1% and 63.5% of the time (numbers depend on age band and service setting). Data are available in Tables NOCC.5, NOCC.6, and NOCC.7.

The same clinician-rated measures have been adopted by all states and territories implementing the NOCC. There is more variability for the consumer-rated measures, with states and territories adopting different measures.

For these reasons, the clinician-rated suite of Health of the Nation Outcome Scales (HoNOS) is used in this report to provide a national picture of the mental health related problems faced by consumers at admission to a mental health service. This makes use of the most consistent and comparable data across states and territories.

More information about consumer- and clinician-rated measures is in the data source section.

During 2019–20, the mental health-related problems most frequently affecting consumers aged 11–17 years were Emotional (89.0% of collection occasions in acute inpatient care, 89.2% in ambulatory care), Family (71.3% acute inpatient, 70.7% ambulatory), and Peers (58.9% acute inpatient, 59.4% ambulatory).

Problems with Self-injury, Substance misuse and Hallucinations more frequently affected consumers in acute inpatient care (68.3%, 27.2% and 25.5% respectively) than ambulatory care (43.7%, 16.6% and 15.2% respectively). Somatic problems more frequently affected consumers in ambulatory care (31.6%) than acute inpatient care (15.4%).

During 2019–20, the mental health-related problems most frequently affecting consumers aged 18–64 years were Depressed mood (59.7% of collection occasions in acute inpatient care, 53.7% in ambulatory care), Other mental and behavioural problems (59.3% acute inpatient, 61.0% ambulatory), and Relationships (49.2% acute inpatient, 44.8% ambulatory). Substance misuse was rated a clinically significant problem for 49.8% of collection occasions in acute inpatient care and 32.2% of occasions in ambulatory care. The presence of clinically significant problems in Other mental and behavioural problems indicate comorbid problems for the consumer.

Problems with Hallucinations, Behaviour, Substance misuse and Self-injury more frequently affected consumers in acute inpatient care (45.1%, 39.6%, 49.8% and 36.3% respectively) than in ambulatory care (21.3%, 21.4%, 32.2% and 20.0% respectively).

During 2019–20, the mental health-related problems most frequently affecting consumers aged 65 years and older were Other mental and behavioural problems indicating comorbid problems (69.6% of collection occasions in acute inpatient, 53.0% ambulatory), Physical illness or disability problems (54.8% acute inpatient, 57.2% ambulatory) and Depressed mood (63.5% acute inpatient care, 48.4% in ambulatory care).

Problems with Hallucinations, Self-injury and Behaviour more frequently affected consumers in acute inpatient care (45.4%, 28.2% and 45.2% respectively) than in ambulatory care (21.9%, 8.8% and 25.9% respectively).

During 2019–20, for consumers aged 18–64, Mental disorder not otherwise specified was the most frequently recorded diagnosis at discharge across both settings (17.7% acute inpatient, 16.0% ambulatory), and Schizophrenia was the second (16.1% acute inpatient, 12.6% ambulatory). In both settings, Depressive episode was also among the most frequently recorded principal diagnoses (9.5% acute inpatient, 9.9% ambulatory) (Figure NOCC.4).

Other frequently recorded diagnoses were Reaction to severe stress and adjustment disorders (11.7%) in ambulatory settings and Mental and behavioural disorders due to other psychoactive substance use (8.7%) in acute inpatient settings.

During 2019–20, for consumers aged 11–17, Other anxiety disorders was among the most frequently recorded mental health-related principal diagnoses at discharge (8.4% of collection occasions in acute inpatient, and 17.5% in ambulatory), as were Reaction to severe stress and adjustment disorders (16.8% acute inpatient, 14.3% ambulatory), Mental disorder not otherwise specified (15.6% acute inpatient, 12.3% ambulatory) and Depressive episode (15.6% acute inpatient, 9.7% ambulatory) (Figure NOCC.4).

In 2019–20, for consumers aged 65 years and older the most frequently recorded mental health-related principal diagnoses at discharge were Depressive episode (22.3% acute inpatient, 17.4% ambulatory), Dementia (13.3% acute inpatient, 12.4% ambulatory), Mental disorder not otherwise specified (10.7% acute inpatient, 11.5% ambulatory), and Schizophrenia (10.0% acute inpatient, 7.7% ambulatory).

Bipolar affective disorders was also among the five most frequently recorded diagnoses at discharge in acute inpatient settings (10.9%), as were Recurrent depressive disorders in ambulatory settings (8.4%).

The most frequent consumer groups are:

• Completed acute inpatient (96.2% of acute inpatient episodes) – episodes that started and finished within the reporting year with a duration longer than 3 days. Consumers are often very unwell at admission, but their symptoms can often be treated quite effectively and reasonably quickly. Many of the consumers in this group will be discharged to ambulatory services once their symptoms have begun to improve (Table NOCC.13).
• Completed ambulatory (36.5% of ambulatory episodes) – episodes that started and finished within the reporting year with a duration longer than 14 days. Consumers may be seen in ambulatory services only or in ambulatory services following an episode of acute inpatient care. The severity of symptoms at the beginning of an episode is generally better than for consumers in acute inpatient care.
• Ongoing ambulatory (48.2% of ambulatory episodes) – episodes of care that were still open at the end of the reporting year. Consumers may be affected by illnesses that are persistent or episodic in nature. Goals of care for these consumers may be to reduce symptoms, improve functioning or maintain their current state of wellness and prevent deterioration. Thus for some consumers, no significant change may be a positive result that provides a basis for longer-term treatment.

For both clinician and consumer-rated measures during 2019–20, most episodes for consumers aged 18–64 showed Improvement in completed acute inpatient care (73.4% clinician-rated and 65.3% consumer-rated) and completed ambulatory care (51.6% clinician-rated and 51.0% consumer-rated) (Figure NOCC.6).

No change was the biggest single category for consumers in this age band in ongoing ambulatory care on both clinician-rated (57.2%) and consumer-rated (68.1%) measures.

On clinician-rated measures, consumers aged 18–64 showed Deterioration in 17.5% of ongoing ambulatory episodes, 6.8% of completed ambulatory episodes and 4.5% of completed acute inpatient episodes.

On consumer-rated measures, consumers in this age band showed Deterioration in 10.5% of ongoing ambulatory episodes, 3.6% of completed ambulatory episodes and 3.7% of completed acute inpatient episodes.

During 2019–20, on clinician-rated measures most episodes for consumers aged 11–17 showed Improvement in completed acute inpatient care (57.9% of episodes) and completed ambulatory care (55.0%). For consumers in this age band the biggest single category for those in ongoing ambulatory care was no change (46.7%) (Figure NOCC.6).

Consumers showed Deterioration on clinician-rated measures in 15.2% of ongoing ambulatory episodes, 9.5% of completed acute inpatient episodes and 5.5% of completed ambulatory episodes.

During 2019–20, on clinician-rated measures most episodes for consumers aged 65 and older in completed acute inpatient care showed Improvement (72.6% of episodes). No change was the biggest category in this age band for consumers in completed ambulatory care (48.5%) and ongoing ambulatory care (59.2%) (Figure NOCC.6).

On clinician-rated measures, consumers showed Deterioration in 14.8% of ongoing ambulatory episodes, 5.6% of completed ambulatory episodes and 5.2% of completed acute inpatient episodes.

On consumer-rated measures, most episodes for consumers aged 65 and older in completed acute inpatient care showed Improvement (59.9% of episodes). For ongoing ambulatory care and completed ambulatory care most episodes (68.6% and 63.5% of episodes, respectively) showed No change.

On consumer-rated measures, consumers showed Deterioration in 9.6% of ongoing ambulatory episodes, 1.3% completed ambulatory episodes and 5.3% completed acute inpatient episodes.

The National Outcomes and Casemix Collection (NOCC) is a nationally agreed data collection for the routine collection and reporting of consumer outcomes using clinical measures. Under the National Mental Health Policy 1992, Australian governments committed to national monitoring of the effectiveness of public mental health services. The primary objective of the NOCC was to establish the routine use of outcome measures in all publicly funded or managed mental health services, where such measures contribute both to improved practice and service management (Burgess et al. 2015). The NOCC has been progressively implemented in state and territory public sector specialised mental health services from 2001 with all jurisdictions reporting by June 2005.

The NOCC captures information about consumers’ health and wellbeing during their mental health care using standardised clinical measures, which is used to report on outcomes.

The NOCC also gathers ‘casemix’ information, which is information about the mix of people who are receiving mental health services according to their clinical status and the nature of the care they are receiving. The casemix information collected in the NOCC supports the introduction of the first version of the Australian Mental Health Care Classification (AMHCC) (Independent Hospital Pricing Authority 2018).

Public reporting of the NOCC

National indicator set: Change in consumers’ clinical outcomes

The Key Performance Indicators for Australian Public Mental Health Services (KPIs) were developed for the purpose of improving public mental health services. KPI 01 Change in mental health consumers’ clinical outcomes draws on data from the NOCC for reporting. These indicators contribute to measuring the performance and progress of mental health services in Australia.

The original KPI set was released in 2005, with the aim to measure and improve the performance of public mental health services. The indicators have been revised over time through the former National Mental Health Performance Subcommittee (NMHPSC) of the former Mental Health Information Strategy Standing Committee (MHISSC), to drive and incorporate data improvements, jurisdictional implementation and results of nation-wide projects (NMHPSC 2013).

Change in mental health consumers’ clinical outcomes was added to the national KPI set in 2011 (NMHPSC 2013).

The Key Performance Indicators are published on Mental health services in Australia (MHSA). Due to differences in methodology there is variation in the NOCC data that are reported for the KPIs, this section, and other reporting tools.

Other reporting tools

Other NOCC public reporting products focus on the clinical utility of the collection, through the publication of ‘normative’ reference data for the clinical measures that assist clinicians and other users to better understand the outcomes and variability in the population under care.

A Web Decision Support Tool allows users to compare an individual consumer’s scores at a single point in time, or change in scores over time, against normative data from ‘like’ consumers around Australia. In addition, scores on clinician- and consumer-rated measures can be displayed side-by-side which facilitates engagement with the consumer/family around different perspectives on mental health status.

A Reports Portal allows users to create tailored reports that provide different statistical summaries of the NOCC data, for example the change in scores on various measures across the course of given episodes. More granular reports can be created by selecting from a range of variables, for example age, measure (including item level), service setting, collection occasion, collection reason, jurisdiction, diagnosis, sex, legal status.

In the Web Decision Support Tool and the Reports Portal, NOCC data are reported at national and state/territory levels.

NOCC protocol

The collection of the standard clinical measures is guided by an underlying conceptual model and national protocol. The clinical measures included in the NOCC protocol are at Box NOCC.1. The measures are specific to the consumers’ age group (Child or adolescent aged less than 18 years; Adult aged 18–64 years; and Older person aged 65 years and over) and may be used for the purpose of measuring outcomes or describing casemix.

● Measure is used for the specified purpose of measuring outcomes or describing casemix.

○ Not an outcomes measure but is important for the interpretation of outcome data.

a) These measures are completed by the consumer. The specific measure used varies across states and territories – K10+ (New South Wales, Northern Territory, South Australia and Western Australia), BASIS-32 (Australian Capital Territory, Tasmania and Victoria), and MHI-38 (Queensland).

b) The NOCC includes three versions of the SDQ: SDQ-PC (parent report measure for children aged 4–10 years); SDQ-PY (parent report measure for youth aged 11–17 years); and SDQ-YR (self-report measure for youth aged 11–17 years).
 

Clinical measures are completed at key Collection occasions during the consumer’s episode of mental health care (at admission, review and discharge). The measures are specific to service setting (inpatient, residential and ambulatory) and the consumer’s age group. Outcome measures that are collected on at least two occasions allow assessment of change in health status.

Casemix measures are used to describe the mix of people who are receiving mental health services, grouped according to their clinical status and the pattern of services they are receiving. Casemix measures need only be collected at the single most appropriate point for describing and classifying each episode.

Other data elements in the NOCC provide context for interpreting the information gathered using the clinical measures. These include defining attributes of collection occasions, for example mental health provider entity identifier, person identifier, age group, mental health service setting, reason for collection, collection occasion date and person-level socio-demographic characteristics.

More information can be found on the Australian Mental Health Outcomes and Classification Network (AMHOCN) website:

• Protocol: National Outcomes and Casemix Collection: Technical specification of State and Territory reporting requirements, version: 2.03
• Measures: National Outcomes and Casemix Collection: Overview of Clinician-Rated and Consumer Self-Report Measures, version 2.1.

Scope and coverage

NOCC data are reported annually, based on financial year. An individual consumer’s measures are not linked across years. The NOCC does enable an individual consumer’s clinical status and functioning to be described at different points of treatment within a single year. However, many consumers, due to the nature of their mental illness, receive care for longer periods and often across multiple settings and organisations. The approach used to report outcomes from the NOCC separates consumers’ care into segments – for example, inpatient versus ambulatory care – within a single year, rather than tracking outcomes across treatment settings and time.

NOCC coverage estimates are reported as an indicator of the extent to which the NOCC protocol has been successfully implemented. They are derived by comparing the number of consumers with clinical ratings in the NOCC in a reporting period (the numerator) to the overall number of consumers reported as receiving clinical care from state and territory public mental health services in the same reporting period (the denominator).

The numerator is derived from the NOCC. For the purposes of coverage estimates, counts of consumers included in the NOCC are unique at the jurisdictional level within the reporting period. The denominator is sourced from aggregated data supplied by jurisdictions to the AIHW for the purposes of calculating MHS Key Performance Indicator 9 (KPI 9) New client index. General and specific caveats affecting the quality of these data are provided in the Key Performance Indicators for Australian Public Mental Health Services tables (200kB XLSX).

The consumer’s age at each collection occasion may not align with the NOCC protocol in terms of the age group specific services received and the measures completed. This may occur, for example, when consumers aged less than 18 years receive ‘adult services’, or consumers aged 18–64 years receive ‘older persons’ services, etc. Over the 6-year period covered by this section, approximately 2.1% of collection occasions did not align. For the purposes of this section, this small percentage of collection occasions has been excluded, resulting in a small underestimate of coverage.

Statistical units

• The statistical counting unit used to describe coverage of the NOCC is the consumer, a unique individual within a jurisdiction.
• The statistical counting unit used to describe NOCC volume is a collection occasion, a meaningful point during a period of contact between a consumer and a mental health service organisation within the reporting period.
• The statistical counting unit used to describe outcomes is an episode of mental health care, the period of contact between a consumer in a single setting within a mental health service organisation bounded by the ‘first’ and ‘last’ collection occasions within the reporting period.

More information about the statistical counting units and the conceptual basis for the reporting of the NOCC can be found in the Reporting Framework for the National Outcomes and Casemix Collection.

Data quality information

Data validation

Data are supplied annually by all states and territories and are validated to ensure the data conform to the NOCC protocol under the NOCC Technical Specifications ‘business rules’. Jurisdictional representatives respond to any issues before the data are accepted as the most reliable current data collection. This process may highlight issues with historical data. In such cases, historical data may be resupplied to ensure data are consistent. Only data that form valid sequences of collection occasions within non-overlapping episodes of mental health care are used for public reporting. Further information about the NOCC data set business rules can be found on the AMHOCN website.

Data quality over time

Data should be consistent across most jurisdictions and across years within most jurisdictions, with the following exceptions.

The Australian Capital Territory transitioned to a new information system in 2016–17. This has impacted the integrity of the unique counts of consumers that were supplied for the purposes of calculating NOCC coverage.

The Australian Capital Territory does not reliably report principal type of admitted patient care program provided by specialised inpatient mental health services. In this section, all Australian Capital Territory inpatient services are considered ‘acute care’.

New South Wales transitioned to new information systems in the 2015–16 and 2016–17 periods. This occurred along different timelines region by region. The change had an impact on the ability of staff to record data as they were trained and adjusted to the new systems.

The NOCC Technical Specifications were updated with effect from 2015–16 to include new data elements for country of birth, Aboriginal and Torres Strait Islander status, and consumers’ area of usual residence. The latter, reported as Statistical Area Level (SA2) from the Australian Statistical Geography Standard, is used to derive socio-economic status and remoteness measures. Partial reporting of these measures (4 jurisdictions) commenced in 2015–16; full reporting commenced in 2016–17. These measures are reported only for years in which there was full reporting.

The concepts episode type and consumer group are based on information about service setting, collection occasion and the reason for collection. Reason for collection is a NOCC data element comprising domains that describe key events triggering admission (e.g., new referral), review (e.g., 3-month review) or discharge (e.g., no further care) collection occasions. The Australian Capital Territory did not reliably report the reason for collection data element with respect to ambulatory discharges before 2018–19. This means that estimates for ‘Completed’ ambulatory episode types and the ‘Completed ambulatory’ consumer group are not available prior to 2018–19.

Clinical measures

The collection of routine outcome measures in everyday clinical practice may be challenging and gaps in collection can subsequently occur. It is important to understand these challenges as they can impact on the volume of data that is available for reporting and introduce systematic biases. An important challenge in the design of the NOCC protocol has been to minimise the burden of collection.

There are limited exceptions to the protocol for defined instances when the collection of measures is not required – for example, if it is not appropriate to offer the measure where the consumer’s current clinical state is of sufficient severity that it could not be reliably completed and/or would cause additional distress; or if an episode of mental health care is too brief to allow meaningful opportunity to show change at the time of discharge; or when the care of the consumer is transferred from the ambulatory service to an inpatient or residential service of the same Mental Health Service Organisation. In these instances, the collection occasion is not considered to be in-scope for reporting.

Other challenges reflect the reality of everyday clinical practice – for example, when the consumer is not available to be offered the measure at the intended collection occasion, such as at discharge. In these instances, the collection occasion is considered to be in-scope for reporting.

Clinician-rated measures

HoNOSCA (Health of the Nation Outcome Scales for Children and Adolescents):

HoNOS/HoNOS 65+ (Health of the Nation Outcome Scales for working age adults and older adults):

Consumer-rated measures

SDQ-YR (The Strengths and Difficulties Questionnaire Youth Report):

Adults and Older Persons

Kessler Psychological Distress Scale (K10)/K10 Plus (K10+):

Behaviour and Symptom Identification Scale (BASIS-32):

Mental Health Inventory (MHI-38):

Outcomes methodology

In order to measure outcomes:

• the same measure must be collected, and
• on two collection occasions that form a logical sequence, and
• within an episode of mental health care, and
• within a single setting.

When these conditions are met, these are called matched pairs.

The NOCC protocol defines specific instances when the clinical measures are not in-scope for collection; these instances are a complex mix of factors that vary across age bands, settings, types of collection occasions, and the specific measure. This can impact the number of collection occasions that are available for describing the clinical characteristics of consumers at different points in their engagement with services, and for describing the outcomes of clinical care.

Not all collection occasions are eligible to form matched pairs. Specifically, discharge ratings on the clinician- and consumer/carer-rated measures are not required for brief episodes of ambulatory care (14 days or less) or brief acute inpatient care (3 days or less) because this brief period does not provide a meaningful opportunity to measure change. In addition, discharge ratings on the clinician- and consumer/carer-rated measures are not required when the consumer is transferred to an inpatient or residential setting within the same organisation, because the measures will be collected upon admission to the new setting.

Episodes of mental health care for an individual consumer are derived from the sequence of collection occasions recorded within the annual reporting period (refer to the Reporting Framework for the National Outcomes and Casemix Collection for further information). Episodes can only be derived if there is a valid sequence of collection occasions (e.g., an admission collection occasion followed by a discharge collection occasion). Episodes cannot be derived if there is an invalid sequence of collection occasions (e.g., a review collection occasion followed by an admission collection occasion) or where there is only a single collection occasion reported for a consumer in the reporting period.

Not all episodes are in-scope for reporting on outcomes based on the consumer-rated measure, specifically because some consumers are not offered the measure.

Outcomes classification

Public reporting of the outcomes from the NOCC is based on an effect size methodology. Specifically, mental health outcomes – that is, the difference or change between scores at the start and end of an episode of mental health care – were classified using an effect size metric as significant improvement, no significant change or significant deterioration. The advantage of this method is that change values derived from the different consumer-rated measures are converted into standardised units so that they can be combined for national reporting. For each measure, a medium effect size threshold of half a standard deviation (Cohen 1988) of the score was calculated from all admission collection occasions, separately for acute inpatient and ambulatory settings and for each outcome measure.

The effect size thresholds for each measure reported in this section are shown in Box NOCC.2. For example, for episodes in which consumer outcomes were based on the HoNOS family of measures, this corresponded to an absolute threshold of change score of 3 in both acute inpatient and ambulatory settings. Outcomes were then classified as significant improvement if the change score was 4 or more, no significant change if the change score was between -3 and 3, and significant deterioration if the change score was -4 or less.

Box NOCC.2. Medium effect size thresholds by measure

SD = standard deviation; Max.=maximum; Min.=minimum.

a) Represents all admission collection occasion scores in these settings.

b) The intervals of change for the MHI-38 have opposite signs to the other measures because, unlike the other measures, higher scores indicate better mental health. Therefore, a negative change score indicates improvement and a positive change score indicates deterioration.

Use of confidence intervals

This section makes use of confidence intervals to reflect some of the statistical variability (‘uncertainty’) in estimates derived from the NOCC. It is acknowledged that there are different views on the appropriateness of using inferential statistics, such as confidence intervals, for population parameters (Redelings et al. 2012), noting that the NOCC is intended to comprise the complete population receiving care from public sector specialised mental health services. We adopted the approach used by Public Health England, which recommends that a confidence interval should be presented alongside a point estimate whenever an inference is being made from a set of observations to the underlying process or ‘risk’ that generated them (Eayres 2008; Redelings et al. 2012). In this section, confidence intervals are shown in most figures and are included in the NOCC tables.

A confidence interval is a range of values that is used to quantify the random variability or fluctuations that can occur naturally, for example in the numbers of services used and of persons using services over time. Generally, confidence intervals describe how different an estimate could have been if the underlying conditions stayed the same but random variability had led to a different set of data (Eayres 2008).

A confidence interval does not quantify all variability inherent in a statistic. In the NOCC, a key source of variability is incomplete reporting. This can occur when a clinician does not collect a measure on a particular collection occasion as prescribed by the NOCC protocol, or when a consumer is not available to be offered a consumer-rated measure on a given collection occasion, for example at discharge where the consumer is ‘lost to follow-up’ and ‘administratively’ discharged. Other sources of non-random variability include systematic differences between jurisdictions in their implementation of the national protocol. For example, although the national protocol was updated in 2017–18 to capture the data element Phase of care, Victoria had not commenced its collection by 2019–20. Systems are in place to encourage standardised data collection, and to check for patterns of non-random variability (refer to Data validation, above), however, some non-random variability is likely to remain (Kreisfeld and Harrison 2020).

The width of the confidence interval is determined by 3 factors. The first factor relates to the extent of variability in the phenomenon being measured. In this section, almost all estimates derived from the NOCC are proportions, calculated by dividing the numerator by the denominator. The underlying distribution of a proportion is assumed to follow a binomial distribution, and the corresponding variability is taken into account in the calculation of the confidence intervals. Following the approach recommended by Public Health England (Public Health England 2018), the Wilson Score method was used to calculate the confidence intervals (Newcombe & Altman 2000; Wilson 1927). This method has the advantage of generating an interval when the numerator, and therefore the proportion, is zero. Because the binomial distribution is non-normal, the resulting confidence intervals are asymmetrical. That is, the size of the margin of error between the lower 95% value and the estimate will not necessarily be equal to the size of the margin of error between the upper 95% value and the estimate.

The second factor is the ‘level of confidence’, the desired probability that the interval includes the true value. In reporting of public health measures, a 95% level of probability is commonly used, and means that we can be 95% confident that the true value lies within the interval. Confidence intervals can be used to test for statistical differences between estimates. If the 95% confidence intervals for two reported estimates do not overlap, then there is 95% confidence that the difference between them is statistically significant. This is considered a conservative method; it is not always the case that overlapping confidence intervals do not indicate a statistically significant difference (Public Health England 2018). More exact methods are available but have not been used in this section.

The third factor is the population size from which the estimate is derived. Larger population sizes yield more precise estimates with narrower confidence intervals. In this section, estimates are provided for groups that vary widely in clinical population size. For example, there is at least a 30-fold variation in the number of people receiving clinical care from specialised mental health services in the Northern Territory compared to New South Wales. Similarly, there is wide variation in the size of some population subgroups, for example between the number of Indigenous Australians receiving care compared to non-Indigenous Australians, between the number of people living in very remote locations compared to major cities, and between the number of people receiving care in residential services compared to ambulatory services. In the absence of information about the precision of the estimate, small differences between groups or small fluctuations for a group over time could be incorrectly interpreted as meaningful (AIHW: Kreisfeld and Harrison 2020; Redelings et al. 2012).

AIHW: Kreisfeld R and Harrison J 2020. Indigenous injury deaths: 2011–12 to 2015–16. Injury research and statistics series no. 130. Cat. no. INJCAT 210. Canberra: AIHW.

Burgess P, Pirkis J and Coombs T 2015. Routine outcome measurement in Australia. International Review of Psychiatry 27(4):264–75.

Burgess P, Trauer T, Coombs T, McKay R and Pirkis J 2009. What does 'clinical significance' mean in the context of the Health of the Nation Outcome Scales? Australasian Psychiatry 17(2):141–8.

Burns A, Beevor A, Lelliott P, Wing J, Blakey A, Orrell M, Mulinga J and Hadden S. 1999. Health of the Nation Outcome Scales for elderly people (HoNOS 65+). British Journal of Psychiatry 174:424–7.

CIHI (Canadian Institute for Health Information) 2021. Outcomes. Viewed 18 May 2021.

Cohen J 1988. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, New Jersey: Lawrence Erlbaum Associates Inc.

Eayres D 2008. Commonly Used Public Health Statistics and their Confidence Intervals. Viewed 10 February 2021.

Eisen SV, Dickey B and Sederer LI 2000. A self-report symptom and problem rating scale to increase inpatients' involvement in treatment. Psychiatric Services 51(3):349–53.

Eisen SV, Dill DL and Grob MC 1994. Reliability and validity of a brief patient-report instrument for psychiatric outcome evaluation. Hospital and Community Psychiatry 45(3):242–7.

Goodman R 1997. The Strengths and Difficulties Questionnaire: a research note. Journal of Child Psychology and Psychiatry 38(5):581–6.

Gowers SG, Harrington RC, Whitton A, Lelliott P, Beevor A, Wing JK, Jezzard R 1999. Brief scale for measuring the outcomes of emotional and behavioural disorders in children: Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA). British Journal of Psychiatry 174:413-6

Independent Hospital Pricing Authority 2018. Australian Mental Health Care Classification v1.0 User Manual. Sydney: IHPA.

Kessler R, Andrews G, Colpe L, Hiripi E, Mroczek D, Normand S, Walters E and Zaslavsky A 2002. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine 32(6):959–76.

NMHPSC (National Mental Health Performance Subcommittee) (2013) Key Performance Indicators for Australian Public Mental Health Services, 3^rd edn, NMHPSC, Australian Health Ministers Advisory Council’s Mental Health Drug and Alcohol Principal Committee (MHDAPC).

Newcombe RG and Altman DG 2000. Proportions and their differences. In: Altman DG, Machin D, Bryant T, Gardner M (edn) Statistics with confidence. 2nd edition. London: BMJ Books, 46–8.

Public Health England 2018. Technical Guide: Confidence Intervals. Version: 25 May 2018. Viewed 10 February 2021.

Redelings M, Sorvillo F, Smith L and Greenland S 2012. Why confidence intervals should be used in reporting studies of complete populations. The Open Public Health Journal 5:52–4.

Veit C and Ware JJ 1983. The structure of psychological distress and well-being in general populations. Journal of Consulting and Clinical Psychology 51(5):730–42.

Wilson EB 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158):209–12.

Wing J, Curtis R and Beevor A 1994. ‘Health of the Nation’: measuring mental health outcomes. Psychiatric Bulletin 18:690-1.

Wing JK, Beevor AS, Curtis RH, Park SB, Hadden S and Burns A 1998. Health of the Nation Outcome Scales (HoNOS). Research and development. British Journal of Psychiatry 172:11-8.