- Open Access
Do adults in contact with Australia's public sector mental health services get better?
Australia and New Zealand Health Policy volume 3, Article number: 9 (2006)
This paper describes the outcomes of episodes of care for adults in public sector mental health services across Australia, with a view to informing the debate on service quality. Health of the Nation Outcome Scales (HoNOS) change scores and effect sizes were calculated for 14,659 acute inpatient episodes and 23,692 community episodes. The results showed that people in contact with public sector mental health services generally do get better, although the magnitude of improvement depends on the setting and episode type. This confirmatory finding is particularly positive, given current community concerns about the quality and effectiveness of mental health services.
In Australia, as in other countries, there has been increasing concern about how well mental health services serve the community. Australia's National Mental Health Strategy has provided a strong policy platform for mental health service reform over the last decade or more , emphasising co-ordination between acute inpatient care and community services such that people can move between them according to their level of need. However, there is a view that this has not translated into optimal service delivery 'on the ground'. Some view this as a problem with the implementation of the Strategy , whereas others argue that the implementation is moving in the right direction, but that change may not have occurred extensively or quickly enough .
Various sources of evidence have been cited in the debate over the nature and quality of Australia's mental health services, and their capacity to meet consumers' needs. Key among these are several reports on the direct experiences of consumers and carers, which draw on the findings of large-scale consultations [4–6]. Also crucial are a series of National Mental Health Reports which have monitored the progress of the National Mental Health Strategy by providing data on changing patterns of mental health expenditure and service provision . In addition, there have been two evaluations of the plans that operationalise National Mental Health Strategy, each of which relied on a combination of quantitative data (taken primarily from the National Mental Health Reports), and qualitative data (derived from consultations with key informants and commentary from international experts) [8, 9].
What has been missing to date is any systematic evidence regarding whether people in receipt of mental health care improve. Consumer-level outcome data that can inform this question are necessary, and are now being collected by public sector inpatient and community mental health services in all Australian states/territories, and being routinely submitted by each state/territory to the Australian Government. Australia is an international leader in this regard – the United States, the United Kingdom and several continental European countries have also begun to routinely collect outcome data, but have not done so on such a comprehensive scale [10–12].
Collectively, the Australian outcomes data are known as the Mental Health National Outcomes and Casemix Collection (MH-NOCC). For adults, two clinician-rated outcome measures are used in each state/territory, along with one of three consumer-rated measures (see Table 1 for details). The Australian Mental Health Outcomes and Classification Network (AMHOCN) has been charged with the task of aggregating, analysing and reporting on these data . The current paper describes the change in problem severity for adults in public sector acute inpatient and community mental health services across Australia, with a view to informing the debate on the quality of these services. The paper draws on a comprehensive report which is available on the MH-NOCC website, or from the authors on request [14, 15].
Under the MH-NOCC protocol, individual providers are responsible for administering the clinician-rated outcome measures (and offering consumers the consumer-rated measures) during given 'episodes of care', at particular 'collection occasions'. An episode is defined as '... a more or less continuous period of contact between a consumer and a mental health service organisation that occurs within the one mental health service setting [e.g., inpatient or community]' . Collection occasions occur at set points and for different reasons: admission (new referral; transfer from other setting; other), at review (91 days; other), and at discharge (no further care; change of setting; death; other).
MH-NOCC data were available for the period July 2000 to April 2005. Data were provided at the collection occasion level, and then 'cleaned' and aggregated to 'episodes' by the AMHOCN study team. All episodes comprised more than one collection occasion, but it should be noted that not all episodes were bounded by admission and discharge collection occasions. The first collection occasion for those in ongoing treatment at the point of implementation of the national collection was a review, rather than an admission; and the last collection occasion for those in ongoing treatment when the MH-NOCC data were submitted was a review, rather than a discharge. In other words, certain valid sequences bounded episodes, creating the following episode sub-types: admission to review; admission to discharge; review to review; review to discharge.
The current paper focuses on one outcome measure only, the clinician-rated Health of the Nation Outcome Scales (HoNOS), developed by Wing and colleagues in the United Kingdom . The HoNOS is a general measure of problem severity, and comprises 12 items that collectively cover the sorts of problems that may be experienced by people with a mental illness – namely, (1) Overactive, aggressive, disruptive or agitated behaviour; (2) Non-accidental self-injury; (3) Problem drinking or drug taking; (4) Cognitive problems; (5) Physical illness or disability problems; (6) Problems associated with hallucinations and delusions; (7) Problems with depressed mood; (8) Other mental and behavioural problems; (9) Problems with relationships; (10) Problems with activities of daily living; (11) Problems with living conditions; and (12) Problems with occupation and activities . Psychometric studies have generally shown the HoNOS to have adequate to good validity, reliability, sensitivity to change and utility .
Each HoNOS item is rated from 0 (no problem) to 4 (very severe problem), resulting in individual item scores, subscale scores and a total score. HoNOS measures were included in the current analysis if at least 10 of the 12 items had valid clinical ratings. Missing items were treated as contributing 0 to the relevant subscale score, and to the total score.
HoNOS change scores and effect sizes were calculated for valid episode of care. The change score was defined as the mean difference between the 'start' and 'end' scores: a positive change score indicated a reduction in clinical severity (i.e., improvement) and a negative change score indicated an increase in clinical severity (i.e., deterioration). The effect size quantified the difference in change scores within episodes (i.e., was calculated from a repeated measures design perspective) , with an effect size of 0.00 indicating no change, an effect size of 0.20 considered 'small' and indicating negligible clinical importance, an effect size of 0.50 regarded as 'medium', indicating moderate clinical importance, and an effect size of 0.80 viewed as 'large', indicating critical clinical importance .
In total, data were available from 101,820 collection occasions from acute inpatient settings, and 183,071 from community settings. In the former setting, these collection occasions aggregated to 21,911 valid episodes, 14,659 (66.9%) of which had sufficiently complete HoNOS ratings at the first and last collection occasions. In the latter, they aggregated to 36,803 valid episodes, 23,692 (64.4%) with eligible HoNOS ratings.
Table 2 shows the HoNOS change scores for acute inpatient episodes, by episode sub-type. For episodes bounded by admission and discharge, which constitute the majority of acute inpatient episodes, the change score and effect size statistic provide evidence of improvement of 'critical clinical importance' (mean change score = 7.3; effect size = 1.00, 95%CI = 0.87–1.12). For episodes bounded by admission and review, the change score and effect size statistic indicate more modest, but still 'moderately clinically important' improvement (mean change score = 3.1; effect size = 0.43, 95%CI = 0.14–0.72). The same is true for episodes bounded by review and discharge (mean change score = 3.3; effect size = 0.48, 95%CI = 0.10–0.86). For episodes bounded by two reviews, the change score and effect size statistic indicate virtually no shift (mean change score = 0.4; effect size = 0.07, 95%CI = 0.05–0.10).
Table 3 shows the HoNOS change scores for community episodes, by episode sub-type. For episodes bounded by admission and discharge, the change score and effect size statistic point to improvement of 'moderate clinical importance' (mean change score = 3.4; effect size = 0.52, 95%CI = 0.45–0.60). For episodes bounded by admission and review, the change score and effect size statistic indicate lesser improvement of 'small to moderate clinical importance' (mean change score = 1.8; effect size = 0.30, 95%CI = 0.25–0.35). This is also the case for episodes bounded by review and discharge (mean change score = 1.6; effect size = 0.28, 95%CI = 0.23–0.34). For episodes bounded by two reviews, the change score and effect size statistics indicate negligible improvement (mean change score = 0.7; effect size = 0.13, 95%CI = 0.12–0.15).
Tables 2 and 3 both provide a further breakdown of the change scores within each of the four episode sub-types described above, presenting results at the level of reason for collection occasion within each. There is some variability within each episode sub-type. So, for example, for community episodes bounded by admission and discharge, the improvement is universally greater in episodes in which the reason for discharge is no further care or 'other' than in episodes where the discharge is to a changed setting or death (where in fact there is sometimes deterioration).
Also of note in Tables 2 and 3 are the mean start and end scores for each episode sub-type, from which the change scores are derived. They indicate that, as would be expected, the typical level of severity of symptoms is higher for people admitted to acute inpatient episodes than those admitted to community episodes. There is less difference between severity levels at discharge from the two settings. Taken together, these findings indicate that there is greater scope for improvement in the acute inpatient setting, but that the overall outcome may be approximately the same.
The findings suggest that, on the whole, people in contact with Australia's public sector mental health services do get better. The magnitude of improvement is greater in acute inpatient episodes than in community episodes. This makes sense, given that people who are admitted to the former setting would typically experience greater problem severity than those admitted to the latter, and would therefore have greater room for improvement. In both acute inpatient and community settings, the extent of improvement is greater in episodes that begin with admission and end with discharge, than it is in episodes that are punctuated at one end or the other by a review. Again, this finding has good face validity, in the sense that improvement would be expected to occur on a gradient. Within settings, the patterns of improvement are also intuitively sensible.
Some caution should be exercised in interpreting the above findings, since they cannot be interpreted as causal. To demonstrate that mental health services were making people better would require a different study design, and would require evidence that those who were not in contact with mental health services did not show similar degrees of improvement.
In addition, the study had several limitations related to data volume and quality. Firstly, although all Australian states/territories are now submitting MH-NOCC data, the progress with which individual jurisdictions have implemented routine outcome measurement has not been uniform, so the majority (almost three quarters) of collection occasions were recorded in 2003–04 and 2004–05. Secondly, there was no way of knowing the level of compliance with administering the HoNOS, since no denominator data were available on the total number of consumers under care. Thirdly, some collection occasions were 'dropped' at the point of converting them to episodes (because they did not constitute valid sequences) and some episodes were not included in the analysis (because they did not include the requisite valid HoNOS ratings), which may have introduced systematic biases into the results if particular types of consumers were more likely to be the focus of these collection occasions or episodes. Fourthly, the reliability and validity of the HoNOS ratings are arguably open to question, although it should be noted that clinicians across the country have been trained in the instrument's use, and there are no financial incentives for 'gaming' .
These limitations aside, the findings address an evidence gap in a way that can inform the debate on how well Australia's public sector mental health services are serving the community. Until now, no comprehensive data have been available on consumer outcomes; the current study provides such data, and suggests that, in general, these outcomes are positive. This is particularly impressive, given that these services primarily provide treatment for people with psychotic and affective disorders , which can be chronic and disabling. Of course, the aggregate nature of the data means that the results are reported as averages, so there will be some consumers who do not experience improvements and may even deteriorate during contact with mental health services. Equally, the data do not address the question of unmet need – and in particular the availability of services in times of crisis – which is of concern to many [4–6]. In other words, the data should not be over-interpreted since they represent only one piece of the jigsaw, but they do suggest that, in the main, people in contact with Australia's public sector mental health services get better.
The study demonstrates the value of routine outcome measurement, in terms of informing questions of service quality and effectiveness. Routine outcome measurement was introduced in Australia under the National Mental Health Strategy, in recognition of the fact that the quality and effectiveness of services can only be achieved through the development of sound information to support service planning and delivery. In mental health circles, Australia is considered a world leader in this regard . Other areas of health could learn from the experiences of the mental health sector in implementing routine outcome measurement and using the data to address key policy and planning questions, both within Australia and overseas .
The current study provides a global picture of adult consumers' improvement in public sector acute inpatient and community mental health services, as measured by the HoNOS. The challenge for mental health services is now to use this information to inform service and program development to support high quality care. There is also a need for further research endeavours, and future work by the AMHOCN study team will pursue more detailed analysis of the above data (e.g., stratifying by factors such as age, sex and diagnostic group, and controlling for factors such as the initial HoNOS profile), consider additional consumer groups (children and adolescents; older people) and additional settings (community residential services), and explore outcomes assessed by different instruments, both clinician-rated and consumer-rated.
To conclude, the current study suggests that adults in contact with public sector acute inpatient and community mental health services experience a reduction in problem severity during the course of a typical episode of care. While one would hope that this would be the case, it has not been demonstrated in any systematic way before. This confirmatory finding is particularly positive, given current community concerns about the quality and effectiveness of mental health services.
Australian Health Ministers: National Mental Health Policy. 1992, Canberra, Australian Government Publishing Service
Hickie IB, Groom GL, McGorry PD, Davenport TA, Luscombe GM: Australian mental health reform: Time for real outcomes. Medical Journal of Australia. 2005, 182: 401-406.
Whiteford HA, Buckingham WJ: Ten years of mental health service reform in Australia: Are we getting it right?. Medical Journal of Australia. 2005, 182: 396-400.
Mental Health Council of Australia: Out of Hospital: Out of Mind. 2003, Canberra, Mental Health Council of Australia
Mental Health Council of Australia: Not For Service: Experiences of Injustice and Despair in Mental Health Care in Australia. 2005, Canberra, Mental Health Council of Australia
SANE Australia: SANE Mental Health Report, 2002-03. 2003, Melbourne, SANE Australia
Department of Health and Ageing: National Mental Health Report 2005: Ninth Report - Summary of Ten Years of Reform in Australia's Mental Health Services under the National Mental Health Strategy 1993-2003. 2005, Canberra, Commonwealth of Australia
Steering Committee for the National Mental Health Plan (2003-2008): Evaluation of the Second National Mental Health Plan. 2003, Canberra, Commonwealth Department of Health and Ageing
National Mental Health Strategy Evaluation Steering Committee for the Australian Health Ministers' Advisory Council: Evaluation of the National Mental Health Strategy: Final Report. 1997, Canberra, Commonwealth Department of Health and Family Services
Brower LA: The Ohio Mental Health Consumer Outcomes System: reflections on a major policy initiative in the US. Clinical Psychology and Psychotherapy. 2003, 10: 400-406. 10.1002/cpp.386.
Ruggeri M: Feasibility, usefulness, limitations and perspectives of routine outcome assessment: the South Verona Outcome Project. Epidemiologia e Psichiatria Sociale. 2002, 11: 177-185.
Slade M: The use of patient-level outcomes to inform treatment. Epidemiologia e Psichiatria Sociale. 2002, 11: 20-27.
Pirkis J, Burgess P, Coombs T, Clarke A, Jones-Ellis D, Dickson R: Routine measurement of outcomes in Australian public sector mental health services. Australia and New Zealand Health Policy. 2005, 2: 8-10.1186/1743-8462-2-8.
Australian Mental Health Outcomes and Classification Network: National Outcomes and Casemix Collection Standard Reports, 1st Edition, Version 1.1. 2005, Brisbane, Australian Mental Health Outcomes and Classification Network
Department of Health and Ageing: Mental Health National Outcomes and Casemix Collection: Technical specification of State and Territory reporting requirements for the outcomes and casemix components of 'Agreed Data', Version 1.50. 2003, Canberra, Commonwealth Department of Health and Ageing
Wing JK, Beevor AS, Curtis RH, Park SB, Hadden S, Burns A: Health of the Nation Outcome Scales (HoNOS). Research and development. British Journal of Psychiatry. 1998, 172: 11-18.
Pirkis J, Burgess P, Kirk P, Dodson S, Coombs T, Williams M: A review of the psychometric properties of the Health of the Nation Outcome Scales (HoNOS) family of measures. Health and Quality of Life Outcomes. 2005, 3: 76-10.1186/1477-7525-3-76.
Morris SB: Distribution of the standardized mean change effect size for meta-analysis on repeated measures. British Journal of Mathematical and Statistical Psychology. 2000, 53: 17-29. 10.1348/000711000159150.
Cohen J: Statistical Power Analysis for Behavioural Sciences. 1987, Hillsdale, N.J., Erlbaum
Australian Institute of Health and Welfare: Mental Health Services in Australia, 2001-02. 2004, Canberra, Australian Institute of Health and Welfare
Slade M: Routine outcome assessment in mental health services. Psychological Medicine. 2002, 32: 1339-1343. 10.1017/S0033291701004974.
Eagar K: When performance measurement isn't enough: Towards national benchmarking for Australia's mental health services: ; Canberra. 2003.
This work was funded by the Health Priorities and Suicide Prevention Branch of the Australian Government's Department of Health and Ageing. The authors would like to thank Adam Clarke, David Jones-Ellis and Brad Bowman of Strategic Data Pty Ltd for their assistance in receipt, organisation and management of the data. They would also like to thank Allen Morris-Yates and Bill Buckingham for their assistance in checking and interpreting the data, and Harvey Whiteford for providing comments on an earlier draft of the paper.
Declaration of competing interests
The author(s) declare that they have no competing interests.
PB, JP and TC conceptualised the idea for the paper, PB undertook the analyses, and PB, JP and TC all prepared the manuscript. All authors read and approved the final manuscript.
About this article
Cite this article
Burgess, P., Pirkis, J. & Coombs, T. Do adults in contact with Australia's public sector mental health services get better?. Aust N Z Health Policy 3, 9 (2006). https://doi.org/10.1186/1743-8462-3-9
- Mental Health Service
- Change Score
- Community Mental Health Service
- Effect Size Statistic
- Routine Outcome Measurement