National Key Performance Indicators for Aboriginal and Torres Strait Islander Primary Health Care: results to June 2018

The nKPIs, like performance indicator systems generally, are useful but imperfect measures of system characteristics that are agreed to be important. To maximise their usefulness, data users need to understand where and how nKPI data may depart from the reality that the indicators are trying to measure. The following issues should be considered when interpreting the data presented:

General considerations

• Information on data quality can be found on the AIHW’s Metadata Online Registry (METeOR).
• The number of organisations that provided valid data are different for different indicators. This means that the analysis of results for each indicator may not be based on the same number of organisations.
• Due to the small number of nKPI reporting organisations, data for Tasmania are combined with that for Victoria and data for the Australian Capital Territory are combined with that for New South Wales.
• There might be double-counting of the same client at multiple organisations due to a high level of mobility among Indigenous Australians. Nationally, the extent of this is unknown and difficult to quantify.
• Where an organisation has a small denominator—that is, fewer than 20 Indigenous regular clients—small changes in the numerator can have a large impact on the overall proportion for that organisation.
• For the most recent data, June 2018, the proportion of organisations with a denominator of fewer than 20 Indigenous regular clients exceeded 10% of all contributing organisations for 12 of the 24 indicators. One indicator measure, PI15: Immunised against influenza—clients with COPD, had 78% of organisations with a denominator of fewer than 20 Indigenous regular clients. This was higher than for all other measures.
• Due to the way in which data have been extracted for organisations funded by the Northern Territory Government since December 2015 (the exclusion of measurements or tests conducted outside an individual organisation), results might be underestimated for:
  • PI03: MBS health assessment—aged 0–4
  • PI05: Glycated haemoglobin (HbA1c) result recorded
  • PI07: General Practitioner Management Plan
  • PI08: Team Care Arrangement
  • PI09: Smoking status recorded
  • PI14: Immunised against influenza—aged 50 and over
  • PI15: Immunised against influenza—clients with type 2 diabetes or COPD
  • PI16: Alcohol consumption status recorded
  • PI22: Cervical screening
  • PI23: Blood pressure result recorded—clients with type 2 diabetes.
• Starting with the June 2017 collection, changes were made to the electronic data extraction method for most organisations. AIHW analysis identified time-series anomalies that indicated that the change in extraction method constitutes a break in series. This means that the June 2017 collection represents a new baseline for the collection moving forward, as data from earlier collections are not comparable with data from June 2017 onwards.
• Until June 2016, the nKPI data were extracted from health organisations’ clinical software systems, largely by a single tool, PenCAT. This tool was compatible with Medical Director, Best Practice, PractiX, Communicare and a version of Medinet. Some organisations also submitted manually. Organisations using MMEx transmitted data directly.
• For the June 2017 nKPI data collection, the Department of Health introduced a new direct load reporting process, which allowed Communicare, Medical Director and PCIS clinical software systems to generate nKPI data within their clinical system and transmit it directly. Best Practice organisations were provided with the Telstra Health tool, Elicio, to extract, transform and send their data. Some Best Practice organisations used a Structured Query Language script developed by the Improvement Foundation in conjunction with the manual submission form. Some changes were made to the MMEx extraction process as a result of the Data Validation Project.

Regular clients

A regular client is defined as a person who has attended the primary health care organisation at least 3 times in the previous 2 years. Starting from the June 2018 collection, the definition of a regular client excludes deceased patients. All of the indicators, except the two birthweight indicators, use the regular client definition. The following points should be noted when interpreting results:

• The concept of a visit varies within CISs, and might be captured differently for services, clinical items, diagnoses, clinical procedures, episodes of care, client contact, and other variables used in CIS data (DMA 2017). Vendors have been addressing these issues but any subsequent residual impact on results has not been quantified.
• During the June 2018 collection, updates to the clinical software Medical Director Insights and the Communicare Launchbar were released. These updates meant that deceased patients were excluded from the regular client count. However, 10 organisations submitted nKPI data using an older version of MD Insights and 22 organisations submitted nKPI data using an older version of the Communicare Launchbar. Deceased clients may have been included in data for these organisations. At the national level, this may lead to an overrepresentation of the number of regular clients and a lower proportion of people receiving appropriate health care.
• There are various scenarios where a client would or would not be considered a regular client, which should be considered when interpreting the data. These might include the following:
  • Some clients might attend an organisation 3 times in 2 years, but have another primary health care organisation as their primary place of care. At the organisational level, this provides an invalid measure of the extent to which a person is receiving appropriate care from the provider they visit 3 or more times, but which is not their main provider. Examples include where a patient declines a particular service, having recently received it at their usual health organisation, or a clinician being able to see results due to a linked CIS or shared electronic health records. At the national level, this will lead to double-counting of that person and underestimate the national proportion of people who are receiving appropriate health care.
  • Some clients might be transient and stay in a community only temporarily. Organisations with a large proportion of transient clients who are counted as regular clients might appear to have poorer results than other organisations, as they have less capacity to follow up on patients, including those with chronic diseases. These organisations might also choose to allow a client’s usual primary health care organisation to provide some MBS item services, including health checks, General Practitioner Management Plans (GPMPs), and Team Care Arrangements (TCAs). This would underestimate the national proportion of people who are receiving appropriate health care. Analysis of data in 2016 found that the nKPI definition leads to a higher count of regular clients for many indicators compared with a definition that restricts the denominator to the usual clients of a health care organisation. The impact of this was that out of a possible 24 measures, 21 measures had better results when the definition was restricted (see AIHW 2017).
  • Clients might access different health care organisations in the same general location, and might not use the same organisation consistently. They might use various organisations for different purposes; for example, favouring one when they want increased privacy and another because it bulk bills (Bailie et al. 2013). This behaviour might be more common in regions with more health care options, and less frequent in areas where local health care options are more limited (for example, Very remote areas). This could result in variations in the make-up of regular clients between regions.
• Organisations operating out of regional centres in a given remoteness area (for example, Alice Springs, which is classified as a Remote area) might have higher levels of regular clients who are not their usual clients than other Remote organisations. This is because they might be regional centres used in transit, and because they provide a wider array of health care options.

Maternal and child health indicators

• Babies’ records (rather than mothers’ records) are the specified source of data for indicators on birthweight recorded and results. But data from organisations using MMEx source this information from the mothers' records (DMA 2017). The impact of this on results has not been quantified. The standard nKPI Indigenous regular client definition does not apply to these indicators—the baby is considered a client and counted in the nKPIs even if they attended only once, and their parents are not regular clients of the organisation. This might lead to the inclusion of babies who visited the organisation purely for acute care, and whose carers might not have been able to confirm birthweight.
• Multiple births should not be included in birthweight results, as babies born as part of multiple births are more likely to have a lower birthweight. But Medical Director and Communicare do not exclude multiple births, as this information is not captured in the baby’s record. Nor do Medical Director and Communicare exclude babies with ‘unknown gestational age’ from the low birthweight indicator (DMA 2017). Although this finding was not expected to significantly affect the nKPIs, it is possible it might inflate the proportion of low birthweight babies recorded in the data.
• Babies’ birthweight and antenatal visits data may be underestimated, as results for Northern Territory Government organisations were provided by the Northern Territory Government Midwifery Group Practice, but not entered as having occurred at the client’s usual health centre. This was rectified for some Northern Territory Government organisations in December 2017 but may affect some data included in this report.
• Antenatal visits data for organisations using Communicare and Medical Director may have been affected by data extraction issues related to the recording of the categories 'No visit recorded’ and ‘Timing of visit not recorded’. The issue was identified in data for June 2017, December 2017 and June 2018. Further information is provided in Chapter 2 of the AIHW (2018) report National Key Performance Indicators for Aboriginal and Torres Strait Islander primary health care: results for 2017.  
• MBS items are not claimed by all organisations, either because they do not have a general practitioner (GP) present, they are not eligible to claim them, or clients are seen by other clinicians. As a result, the indicators based on MBS items might not reflect all related health care activities carried out in an organisation. These indicators include MBS health assessment (item 715) for children aged 0–4. In the case of child health checks, children may receive comprehensive health checks provided within a model of care that does not suit or allow for the check to be claimed as an MBS item. MBS health checks are counted in Communicare at a point in the process before its submission. Only claims explicitly discarded after a rejection are subsequently excluded (DMA 2017). The impact of this has not been quantified.
• GP availability might be limited in some areas, and have an impact on the results reported by organisations. For example, limited GP availability might affect an organisation being able to claim MBS items (child and adult health checks, GPMPs, and TCAs).
• Shared care arrangements between hospitals and primary health care organisations, between primary health care organisations, or between primary health care organisations and other providers of similar care are not consistently supported by automatic data sharing. This could lead to lower rates of data recording for some indicators, such as birthweight results and antenatal care. Similarly, it will be difficult for organisations to obtain information on their regular clients who may choose to receive cervical screening elsewhere.
• Smoking status categories are not yet fully agreed. For example, there is not yet universally accepted guidance on how long a person needs to have quit smoking to be considered an ex-smoker rather than a smoker. An increased number of types of ex‑smokers might improve data quality, and lead to more frequent updating of clients’ records.
• Smoking status of women who gave birth in the previous 12 months records smoking status during pregnancy retrospectively, and the information is updated only when women’s smoking status category is changed. Therefore, this indicator is a proxy for smoking during pregnancy.
• Data extraction for Northern Territory Government organisations excludes measurements or tests conducted outside an individual organisation since December 2015, so results might be an underestimate for PI03: MBS health assessment—aged 0–4. This may also occur for other organisations, though the extent of this is not clear.
• Child immunisation data for the nKPI collection indicates that primary health care records are capturing far fewer cases of fully immunised Indigenous children than Australian Immunisation Register (AIR) data. nKPI data may therefore be an underestimate.
• Small denominators can cause fluctuations in data over time, therefore results should be interpreted with caution. For maternal and child health indicators, 9%–54% of organisations contributing to these indicators in June 2018 had denominators of fewer than 20 clients.

Preventative health indicators

• MBS items are not claimed by all organisations, either because they do not have a general practitioner (GP) present, they are not eligible to claim them, or clients are seen by other clinicians. As a result, the indicators based on MBS items might not reflect all related health-care activities carried out in an organisation. These indicators include MBS health assessment (item 715) for adults aged 25 and over. In addition, the business rule in Communicare is that an item 715 claim is counted at a point in the process before its submission. Only claims explicitly discarded after a rejection are subsequently excluded (DMA 2017). The impact of this has not been quantified.
• Influenza vaccination does not include clients who are offered a vaccination, but refuse. Also, organisations might not have records of immunisations that occurred at other places, such as workplaces.
• GP availability might be limited in some areas, and have an impact on the results reported by organisations. For example, limited GP availability might affect an organisation’s ability to claim MBS items (child and adult health checks, GPMPs, and TCAs).
• Shared care arrangements between hospitals and primary health organisations, between primary care organisations, or between primary health care organisations and other providers of similar care are not consistently supported by automatic data sharing. This could lead to lower rates of data recording for some indicators. It may be difficult for organisations to obtain information on their regular clients who may choose to receive cervical screening elsewhere.
• Smoking status categories are not yet fully agreed upon. For example, there is not yet universally accepted guidance on how long a person needs to have quit smoking to be considered an ex-smoker rather than a smoker. An increased number of types of ex‑smokers may enhance data quality, and lead to more frequent updating of clients’ records.
• Time-stamped records normally ensure that a record or activity is fairly recent. A number of indicators (smoking status recorded, smoking status result, and alcohol consumption) are based on the most recent record for the client (that is, treated as having been updated in the previous 2 years), regardless of how old that record is. As a result, the indicator might not reflect the current smoking or alcohol consumption status of the Indigenous regular client population, unless the data have been collected recently for all or most clients.
• Differential BMI testing might occur in some organisations where BMI might be more likely to be measured in clients who look underweight, overweight, or obese. This would result in the proportion of overweight or obese Indigenous regular clients being higher than it actually is.
• Recording of alcohol consumption status (PI16) is not restricted to a particular test or format for this indicator. Organisations can use tests such as AUDIT or AUDIT-C, or simply record whether or not the client consumes alcohol. However, for the indicator on AUDIT‑C results (PI17), only AUDIT-C results are included. This means that, for some organisations, test results in PI17 are a subset of the tests reporting in PI16.
• Risk factors to enable a CVD risk assessment require information on diabetes status. For the June 2017 collection, MMEx restricted the count of clients with all the necessary risk factor information (that is, the numerator) to clients with a type 2 diabetes diagnosis, leading to an under-count for this indicator. MMEx results for June 2017 are excluded from results presented.
• Absolute cardiovascular risk assessments can be calculated using the NVDPA or the CARPA method. As the CARPA method applies an extra 5% loading for Indigenous Australians, nKPI data should have the 5% loading removed to make the data comparable with NVDPA data. As the PCIS system is unable to deduct the 5% because the data are captured as categorical scores (low, medium, high), organisations using PCIS (predominantly the Northern Territory Government) are not included in the results presented. Additionally, data do not capture clients without known CVD whose risk factors mean they are categorised as ‘high risk’ and therefore do not require a risk assessment.
• Data extraction for Northern Territory Government organisations excludes measurements or tests conducted outside an individual organisation. This means results might be underestimated for PI09: Smoking status recorded, PI16: Alcohol consumption status recorded, and PI22: Cervical screening. This may also occur for other organisations, though the extent of this is not clear.
• Cervical screenings are conducted for female regular clients who are Indigenous, aged 20–74. In June 2018, this indicator was revised to align with the new National Cervical Screening Program (NCSP) where the previous Pap test is replaced by a HPV test from 1 December 2017. The key changes were that:
  • data are to be collected on clients who had either a Papanicolaou smear (Pap test) conducted prior to 1 December 2017 or a human papillomavirus (HPV test) conducted from 1 December 2017
  • the HPV test can be based on a sample collected by a health practitioner or on a self-collected sample
  • the age range for this nKPI indicator has been revised to 20–74 for a transitional period. This is to accommodate the former reporting age range (20–69) and the new age range (25–74).
  An update for Medical Director Insights that incorporated the indicator changes was released during the July 2018 submission period. Where organisations were identified as using an older version of this CIS, and therefore submitting data that did not align with the PI22 changes, data were excluded from the results. As a result, the number of cervical screenings may be underrepresented in the results presented. In addition, some data mapping issues related to the pathology codes used were identified for organisations using MMEX. This issues was addressed in August 2018 and some organisations were able to amend data values. The impact of this issue has not been quantified.
• Small denominators can cause fluctuations in the data over time, therefore results should be interpreted with caution. For preventative health indicators, 0%–32% of organisations contributing to these indicators in June 2018 had denominators of fewer than 20 clients.

Chronic disease management indicators

• MBS items are not claimed by all organisations, either because they do not have a general practitioner (GP) present, they are not eligible to claim them, or clients are seen by other clinicians. As a result, the indicators based on MBS items might not reflect all related health-care activities carried out in an organisation. These indicators include GPMPs and TCAs for clients with type 2 diabetes.
• Influenza vaccination does not include clients who are offered a vaccination, but refuse. Also, organisations might not have records of immunisations that occurred at other places, such as workplaces.
• Pathology results held at an organisation might not reflect all pathology tests that have occurred for its Indigenous regular clients. Organisations without systems in place might not have recorded the information, or results might not have been picked up accurately.
• GP availability might be limited in some areas, and have an impact on the results reported by organisations. For example, limited GP availability might affect an organisation’s ability to claim MBS items (child and adult health checks, GPMPs, and TCAs).
• Access to allied health providers might be limited in some areas, in which case TCAs might not be practical. This is often the case in remote regions.
• Clinical definitions for type 2 diabetes, CVD and COPD vary across CISs, as different coding schemes are used. Medical Director uses doctor command language (DOCLE) codes, Communicare uses International Classification of Primary Care 2nd edition (ICPC2), and MMEx uses Systematized Nomenclature of Medicine (SNOMED). This leads to some variation in the patients who will be picked up by different CISs (DMA 2017).
• Data extraction for Northern Territory Government organisations excludes measurements or tests conducted outside an individual organisation since December 2015. This means results might be underestimated for PI05: HbA1c result recorded, PI07: GPMP, PI08: TCA, PI14: Immunised against influenza—aged 50 and over, PI15: Immunised against influenza—clients with type 2 diabetes or COPD, PI23: Blood pressure result recorded—clients with type 2 diabetes. This may also occur for other organisations, though the extent of this is not clear.
• Kidney function test recorded and result (type 2 diabetes and CVD) data were excluded from June 2018 national reporting for organisations using Medical Director Insights v1.5. Kidney function test recorded (type 2 diabetes) has had ongoing data quality issues since June 2017. Because of this, results for this indicator are not presented for June 2017. In December 2017, results from organisations using Best Practice and Medical Director were excluded.
• Small denominators can cause fluctuations in the data over time, therefore results should be interpreted with caution. For chronic disease management indicators 8%–78% of organisations contributing to these indicators in June 2018 had denominators of fewer than 20 Indigenous regular clients.

References

AIHW (Australian Institute of Health and Welfare) 2017. National Key Performance Indicators for Aboriginal and Torres Strait Islander primary health care: results from June 2016. National key performance indicators for Aboriginal and Torres Strait Islander primary health care series no 4. Cat. no. IHW 177. Canberra: AIHW.

AIHW (2018). National Key Performance Indicators for Aboriginal and Torres Strait Islander primary health care: results for 2017. National key performance indicators for Aboriginal and Torres Strait Islander primary health care series no. 5. Cat. no. IHW 200. Canberra: AIHW.

Ballie RS, Griffin J, Kelaher M, McNeair T, Percival N, Laycock A et al. 2013. Sentinel sites evaluation: final report. Report prepared by Menzies School of Health Research for the Department of Health. Canberra: DoH.

DMA (Doll Martin Associates) 2017. National Key Performance Indicators for Aboriginal and Torres Strait Islander primary health care: Data Validation Project report. Canberra: DoH.