ABDS 2018 quality index

In light of the assessments of measuring uncertainty described previously, the Expert Advisory Group for the ABDS 2011 concluded that this was beyond the scope and resources of the project. However, they supported the need for clearly defined indicators to accompany each set of estimates (DALY, YLL, YLD and attributable burden) to provide users with guidance on the quality of the data underpinning the estimate, and to inform interpretation. Such indicators should inform users not only of the type of data used to derive the estimate, but also its coverage and any transformations required to produce inputs suitable to the YLL, YLD, DALY and risk factor attribution estimation process.

To help users understand the potential sources of uncertainty associated with the estimates, the 2-dimensional index developed for the ABDS 2011 was used for the ABDS 2018 burden estimates. This index was derived based on:

  • the relevance of the underlying epidemiological data
  • the methods used to transform that data into a form required by this analysis.

These dimensions are explained in greater detail in the following section.

The index was designed to help users understand the reliability and limitations of the estimates, especially which patterns and differences were likely to be genuine, and which could be influenced by uncertainties in the data or methods that made them less reliable. The higher the index the more relevant and accurate the estimate was.

To be useful in assessing the impact of different data sources and transformation methods, the final index also took into account the contribution of the underlying data to the overall estimate. For example, a particular data source might have contributed a large proportion of the overall YLD for a single disease, while another might have only contributed a small proportion.

Based on the processes required to produce the various estimates for burden of disease, and the experience of the ABDS project team in collating and analysing data for this purpose, the following key assumptions and core dimensions were developed to provide users with a succinct and coherent assessment of the quality of the estimates.

Key assumptions

To create the index, all standard inputs, methods and assumptions underpinning the estimates were referred to the Australian Burden of Disease Expert Advisory Group and/or disease and risk factor experts for review. Assumptions on which this framework was based include: 

  • for YLL:
    • the reference life table (defined by the GBD 2010 and 2013) was appropriate for use in the Australian context
  • for YLD:
    • the conceptual models mapping sequelae to health states that form the basis of estimates were appropriate as per expert review
    • the health states and disability weights (defined by the GBD 2013) were appropriate to:
      • the conditions being estimated
      • the national and Indigenous populations.
    • the assigned average durations of health loss for sequelae that last for less than 1 year were an accurate reflection of the time spent in a particular health state. Duration has a direct impact on the point prevalence of each sequela (for these sequelae, prevalence = incidence x duration). Durations used in the ABDS were based on accepted clinical research or judgment, and were supplied or reviewed by the expert panels as part of the model.       
  • for risk factors:
    • the risk–outcome pairs, minimum exposure levels and effect sizes (used in the risk factor analysis) defined by the GBD 2019 and other studies were appropriate for:
      • the particular risk factor
      • the Australian context.

Index dimensions

Dimension I: Relevance of the underlying epidemiological data

This dimension refers to the data used to generate the estimate, and includes concepts of data quality, currency and coverage, and suitability to the model being used. These were drawn together into a single score of 5 to 1, as outlined in Table 5.1. The higher the score the more relevant, current and complete the data.

Data source

All input data to the ABDS were required to meet quality guidelines endorsed by the study’s Expert Advisory Group and Indigenous Reference Group to ensure that the highest quality data available were included in the study (see Additional material in Overarching methods and choices for ABDS 2018). However, there was still a wide variability of data reliability. This approach facilitated comparison between data sourced from:

  • disease registers, administrative data, large national surveys, meta-analyses, modelled estimates and single epidemiological studies
  • Australian compared with international sources.

Generally, higher scores were given to Australia-wide unit record or survey data, and lower scores to small surveys and epidemiological studies or international data of limited generalisability.

Data currency and coverage

Data currency refers to how close in time the data were to the reference year. The ABDS 2018 aimed to source data as close to the reference year as possible. While this was possible for most key data sources, it was not possible for all data sources. Data for conditions that are known to be stable over short periods of time were considered current if referring to within 2 years of the reference date (for example, cancer incidence data). Data for conditions that varied from year to year, such as some infectious diseases, were considered current if specific to the reference year.

Data coverage refers to the proportion of the population covered by the data. For example, national versus sub-national, or all age groups versus particular age groups. Generally, the wider the coverage, the higher the score.

Data specificity

Data specificity refers to the suitability of the data to the condition and measure being analysed. Specificity depended very much on the relationship between the condition and the data source. For example:

  • hospitals data for conditions with a high hospitalisation rate (such as appendicitis, amputation) scored higher than conditions with a medium or low hospitalisation rate (such as soft tissue injuries) when hospitalisations were used to estimate prevalence

  • for survey data, clinically diagnosed conditions scored higher than self-reported conditions.

Table 5.1: ABDS quality index, Dimension I – Data relevance scores

Score

Criteria

5

Current data from one of the following: fully enumerated disease register (such as a cancer register) or administrative data, unlinked hospitalisation data for condition with a high likelihood of hospitalisation or national Australian survey (such as the AHS) of either (a) diagnostically confirmed conditions/sequelae or (b) established high correlation between self-report and clinical diagnosis specific to the population with no major variability due to small numbers.

No severity distribution needed, or high-quality empirical data on this distribution were available.

4

Same as ‘5’ BUT not fully enumerated with either known gaps in coverage or not diagnostically confirmed or within 2 years of the reference date or there was some variability due to small numbers (for example, a particular age group) or had high RSEs or severity not available.

It was also used for estimates with components that scored between 5 and 3.

3

Same as ‘4’ BUT with medium specificity of the data source to the condition/sequela being estimated. For example:

  • for survey data, there was known medium correlation between what was collected (for example, measurement, self-report and clinical diagnosis) and the condition
  • for hospitals data, condition had a medium likelihood of hospitalisation (that is, condition only results in hospitalisation in severe or certain cases).

Also, data were from a single, large area (more than 1 state/territory) Australian study of very good quality or from a systemic meta-analysis that could be generalised or from a review of Australian studies with medium currency.

It was also used for estimates with components that scored between 4 and 2.

2

Data were from one of the following: small Australian studies of good quality, small international area study with good sampling that could be generalised to the Australian population, a systematic and meta-analysis that could be generalised, a review of Australian and/or international (for example, other high-income countries) studies. Additionally, the data source was specific to the condition/sequela being estimated and either the data were collected less than 5 years previously for a disease or condition that had a known trend of changing over time or data were collected more than 5 years previously for a disease or condition that had a known trend of not changing over time.

It was also used for estimates with components that scored between 3 and 1.

1

Data were from one of the following: a small Australian study and refers to data more than 5 years from the reference year for a disease or condition that has a known or unknown trend of changing over time, a small number of overseas research studies of questionable generalisability to the Australian context or a secondary data source for indirect prevalence estimates.

Dimension II: Methods of data transformation

This dimension refers to the methods used to transform the data to generate the estimate. It included processes used to fill data gaps, such as:

  • projecting data from 1 year to the reference year to overcome issues of currency
  • applying age and sex distributions or rate ratios from a secondary data source to overcome data gaps
  • applying adjustment factors to overcome issues of data specificity
  • smoothing or combining data to overcome variability in the source data due to sampling or small numbers.

As for Dimension I, these were also drawn together into a single score of 5 to 1, as outlined in Table 5.2.

Table 5.2: ABDS quality index, Dimension II – Data transformation scores

Score

Criteria

5

Data were directly applied to the model and minimal or no extra modelling was required.

Severity distribution (if required) was obtained directly from the data..

4

Rates were projected to the reference year, taking into account changes in underlying trend, and applied to reference population/broad sex or age distributions were converted to 5-year age groups using trend analyses/pooled data from multiple years or sources with comparable definitions/ratios of related and primary data (for example, incidence-to-separations ratio from 1 state) applied to primary data (for example, applied to national separations data). Severity distribution (if required) was obtained from an Australian study.

It was also used for estimates with components that scored between 5 and 3.

3

One of the following transformations was used: rates from another year were applied to the same population for the reference year not accounting for any change in the underlying trend, rates from another population were applied to the reference population for the reference year where there was evidence or expert advice supporting no difference in the underlying prevalence between populations/age or sex distribution from alternative (but relevant) data source applied to the base data, pooled data from multiple sources with differing definitions after standardisation, applied New Zealand Burden of Disease prevalence rates or severity distributions based on linked data, severity distribution obtained from international studies similar to Australia (such as other high-income countries or GBD high-income severity distribution)/ratios of related and similarly defined secondary data (for example, incidence-to-separations ratio) applied to primary data (for example, prevalence).

It was also used for estimates with components that scored between 4 and 2.

2

One of the following transformations was used: other epidemiological measures were modelled to produce the estimates; indirect modelling methods were used, including indirect modelling of prevalence from other measures, such as incidence, mortality, and so on; GBD global severity distribution was used.

It was also used for estimates with components that scored between 3 and 1.

1

Transformations were done using one of the following: inference of distributions from other slightly related data sources; based on expert advice only; indirect modelling methods where the data source had an inconsistent definition of the condition, had a low coverage factor or data were not within 5 years of the reference year; or the severity distribution from another disease or condition was applied as a proxy.

Deriving the ABDS quality index

The ABDS quality index operated at the disease or risk factor level, and was applied to the YLL, YLD and attributable burden for the 2018 national estimates. The quality of DALY estimates is the weighted average of the YLL and YLD estimate.

The index was built from the lowest level of estimate using the 2 dimensions outlined previously, weighted for the contribution to the overall disease-level estimate, as follows:

  • for YLL, it was applied at the disease level
  • for YLD, it was applied at the sequelae level, weighted by the contribution to the overall YLD, and summed to produce an index at the disease level
  • for risk factors, it was applied at the measure of exposure level (for example, second‑hand smoking), then summed to produce an index at the risk factor level (for example, tobacco use).

The index for each dimension is derived and reported separately for YLD and risk factors (see tables below) to help interpret results.

Scoring

Each dimension was scored from 5 to 1. Although these are linear units, it should not be assumed that each score is proportionally equal. This was dealt with by scaling, as follows:

Each score was weighted by the proportion it contributed to the estimate in question. As the maximum score for a disease was 500 (that is, score of 5 contributing to 100% of the estimate) and the minimum 100 (a score of 1 contributing 100%), this was divided by 5 to give an overall score in the range 20–100.

This overall score was then divided into an index (A–E) for Dimension I/Dimension II, as follows:

A.    90 or more (highly relevant/accurate – estimate was derived from comprehensive and highly relevant data/little or no data transformation was required)

B.    75 to less than 90 (relevant/accurate)

C.    45 to less than 75 (moderately relevant/accurate – estimate was derived from reasonably comprehensive and relevant data/moderate transformations required, taking into account known trends in the underlying data, such as over time or age-distributions)

D.   30 to less than 45 (somewhat relevant/accurate)

E.    Less than 30 (questionable relevance/accuracy – use with caution, as estimate was derived from less comprehensive or relevant data/moderate transformations required with trends unknown or unaccounted for).

Sub-national, 2015, 2011 and 2003 estimates

The data and methods used for 2018 estimates underpinned the sub-national, 2015, 2011 and 2003 estimates, so the quality of these estimates must be considered together with the broad sub-national, 2015, 2011 and 2003 methods described in Overarching methods and choices for ABDS 2018, and the specific details described in Disease specific methods and Risk factor specific methods.

Derived ratings

Fatal estimates

Using the ABDS quality index, the mortality data were considered to be comprehensive and relevant with little or no transformation required other than the redistribution of a small proportion of deaths that were not considered appropriate for burden of disease analyses (see Years of life lost (YLL)). Therefore, all fatal burden estimates are highly indicative of the YLL due to these diseases. One exception to this is the fatal injury burden by nature of injury, as injury-related deaths are classified by the external cause – subsequent mapping was needed to estimate the fatal burden by nature.

Non-fatal estimates

The table below lists the quality index for YLD assigned to each disease, and a concise summary of any data issues. Each rating must be interpreted carefully together with the statement accompanying the index and the disease specific methods described in Disease specific methods. Care is needed when using estimates that have a rating of D or E, which are considered to be somewhat relevant/accurate or of questionable dependability, respectively.

Attributable burden estimates

The quality index ratings for risk factor estimates, and a summary of key data issues and gaps are listed in the table below. For each risk factor, it was only possible to rate the quality of the data used to estimate the direct PAFs or the exposure data used to calculate the PAFs. Many other inputs (such as relative risks) were included in these calculations, but it was not feasible in the scope of this project to determine the quality of these inputs.

For risk factors with multiple measures of exposure such as tobacco use, the quality measures have been summarised to reflect the measures with the most attributable burden. Each rating should be interpreted together with the statement accompanying the index and the risk factor-specific methods described in Risk factor specific methods.

This interactive data visualisation reports on the quality information regarding the non-fatal burden estimates of each disease and injury for the national population and for the Aboriginal and Torres Strait Islander population. The specific disease or injury can be selected by the user. There are 2 sections – the first section displays the quality information of the estimates for the national Australian population, the second section displays the quality information of the estimates for the Aboriginal and Torres Strait Islander population. For each disease and injury, there are two scores – one for data and one for methods. Each score is a whole number out of 5. There is also a description of the data and methods used to obtain the non-fatal burden estimate.

Visualisation not available for printing

This interactive data visualisation reports on the quality information regarding the risk factor exposure data estimates for the national population and for the Aboriginal and Torres Strait Islander population. The specific risk factor can be selected by the user. There are 2 sections – the first section displays the quality information of the risk factor estimates for the national Australian population, the second section displays the quality information of the risk factor estimates for the Aboriginal and Torres Strait Islander population. For each risk factor, there are two scores – one for data and one for methods. Each score is a whole number out of 5. There is also a description of the data and methods used to obtain risk factor exposure data.

Visualisation not available for printing
Previous page
Next page