States/Territories' progress regarding data collection systems and infrastructure
Ultimately, States/Territories are aiming to have streamlined data collection systems that allow the outcome data collected via the NOCC dataset to be linked to the admitted and non-admitted activity data in the NMDS. This will allow outcome data to be 'attached' to given inpatient and community episodes of care. This has advantages in terms of allowing outcomes for consumers to be 'tracked' across episodes, and is necessary for progressing casemix development work that requires outcome data and resource use data to be combined within episodes.
All States/Territories have developed data collection systems, or are in the final stages of doing so. For some, this has involved 'starting from scratch'; for others it has required modifications to existing systems. For example, the systems used in Queensland to capture admitted and non-admitted NMDS information did not have the functionality to incorporate outcome measures, so an additional system was developed to do so. By contrast, in the Australian Capital Territory, the system used by all community teams to collect non-admitted NMDS data, was modified to collect outcome data and extended to inpatient services, where it runs alongside a separate patient administration system for the collection of inpatient activity data.
States/Territories differ in terms of the number of systems that are currently involved in the collection of routine outcome data. The simplest scenario is one where outcome measurement functionality has been added to an existing system for recording activity in community mental health settings, and has been extended into inpatient settings (as with the system in the Australian Capital Territory, described above). This also occurs in Victoria, Tasmania and the Northern Territory. Other States rely on as many as four statewide systems to collect NOCC and NMDS information, sometimes with further degrees of complexity between areas or metropolitan/country settings.
Linking NOCC and admitted and non-admitted NMDS datasets is impeded in most States/Territories by the lack of a unique identifier. Typically, linkage is only possible for parts of the data (usually NOCC and non-admitted NMDS data) and/or by conducting quite complex record linkage tasks. The exception is the Northern Territory, which has a client master index that allocates each consumer a unique identifier that allows him/her to be 'tracked' across episodes, across services, and over time. Other States/Territories are working towards improvements, but have some way to go. Western Australia's data collection system has a unique identifier that will allow episodes of care to be attributed to the same individual, regardless of location or time, but its 'roll-out' is not yet completed. Queensland and New South Wales have plans to reconcile their unique identifier systems via specific projects. This will mean that States will assign a unique identifier to a given individual that he or she will 'carry' across all health services, including mental health services, but this will not occur in the near future.
States/Territories have differing levels of infrastructure to support the NOCC and NMDS collections. Human resources vary, with some States/Territories having a number of personnel deployed to train and support clinicians and managers, and others relying on one or two core individuals. So, for example, Queensland has Zonal Outcomes Co-ordinators and Mental Health Information Support Officers providing 'on the ground' support, whereas Tasmania has a small, centrally-located team performing the same function. Physical resources also vary, with some States/Territories having sophisticated online data entry systems (e.g., the Australian Capital Territory), others relying on batch entry of paper-based forms (e.g., Tasmania), and still others using a combination of the two (e.g., New South Wales and South Australia).
States/Territories' progress regarding training and retraining of staff
All States/Territories have implemented comprehensive training programs and have trained substantial proportions of their mental health workforces in routine outcome measurement. According to stakeholders, well over 7,000 clinicians and managers across Australia have received direct training, and possibly half as many again have received training under a train-the-trainer model. This figure is consistent with that of 10,000 reported by the Department of Health and Ageing, which is estimated to represent approximately 60% the public sector mental health workforce [12].
The direct training approach is seen as having the benefit of consistency, while the train-the-trainer approach is seen as fostering capacity building and being less labour intensive and cheaper. Some States/Territories have considered accrediting trainers, so that the advantages of both approaches can be combined. Managers are also more commonly being recruited as trainers, as part of a move to secure their commitment in leading the change process. South Australia has been innovative here, building capacity by training staff as trainers through the Certificate 4 in Workplace Training and Assessment, and investing in training in content knowledge around outcome measures in the NOCC collection. In this way, South Australia has addressed some of the difficulties inherent in more standard train-the-trainer approaches.
Many States/Territories are now beginning to consider issues of ongoing training and support. High levels of staff turnover in some States/Territories mean that there are new staff who have not been trained, and lags between training and implementation in some jurisdictions have resulted in skills being lost. In addition, many States/Territories are recognising the need for a second wave of training that goes beyond how to use the outcome measures and focuses more on how to interpret the results of specific measures (at individual and aggregate levels).
Some States/Territories have implemented ongoing training strategies. Western Australia, for example, has begun refresher training. Tasmania has implemented a second round of training, focusing on the outcome measures that were not covered in the original training (i.e., the LSP-16 and the BASIS-32). Queensland has established an ongoing training program that emphasises sustainability, clinical utility and building capacity, and involves its Zonal Outcomes Co-ordinators modelling for clinicians how outcome data can be used in clinical management. Most other jurisdictions have plans in place to implement a second wave of training that focuses on the clinical and management utility of outcome measurement.
Novel, clinician-focused approaches, such as the use of vignettes and interactive case studies in Victoria and Western Australia, have underpinned the initial and ongoing training in many States/Territories. Training has also typically involved the development of resources (e.g., guides and glossaries for specific measures, consumer/carer brochures), many of which are located on individual State/Territory websites.
States/Territories' progress regarding the implementation of routine outcome measurement
States/Territories are now implementing routine outcome measurement, albeit with very variable degrees of progress. By May 2004, Victoria had provided data for 2000–01, 2001–02 and 2002–03; New South Wales for 2001–02 and 2002–03; Tasmania for 2001–02; and Western Australia, Queensland and the Northern Territory for 2002–03 (partial year only in the latter two). For South Australia and the Australian Capital Territory, 2003–04 data will constitute the first report. Within States/Territories, there is considerable patchiness in terms of coverage, compliance and completeness. There is variability by setting (with community services generally having higher coverage than inpatient services) and by outcome measure (with clinician-rated measures being completed to a greater extent than consumer-rated measures). Strong leadership at all levels has been associated with high levels of overall performance in terms of implementation.
Beyond initial training and rollout, some States/Territories have considered how to sustain and build upon current efforts with regard to routine outcome measurement. There is recognition by these (and other) States/Territories that unless routine outcome measurement becomes embedded in the process of clinical care, it will not be seen as a priority by clinicians and managers. So, for example, in New South Wales outcome measurement has been embedded in a standard protocol, which involves triage, assessment, review and discharge documentation. Specifically, a suite of clinical modules has been developed that not only includes an outcomes module but also includes the incorporation of outcome measures into the process of care. For example, the collaborative care planning module encourages collaboration between the clinician and consumer, and prompts review of the clinician-rated HoNOS and the consumer-rated K-10. The process of embedding outcome measurement within the clinical process of care is enhanced by providing clinical interpretations of given scores on particular measures. All New South Wales Area Mental Health Services will have the same modules, produced as standard medical record stationery for use within clinical files.
States/Territories' progress regarding analysis and reporting of data
Some States/Territories have also begun to consider how best to provide feedback to staff. There is recognition that without appropriate and timely feedback in the form of relevant reports that shed light on clinical and management issues, the current momentum will falter and data quality and comprehensiveness will be jeopardised. Feedback in the form of reports is required at a variety of levels. Some States/Territories have developed individual-level reports that allow clinicians to profile an individual consumer's scores on a range of outcome measures, either at a single point in time or over time. For example, in the Australian Capital Territory, the data capture system produces an electronic management plan, similar to the New South Wales module described above, which incorporates areas that the clinician and consumer might want to address, given the consumer's profile on the outcome measures. Similarly, in Western Australia, HoNOS scores of greater than 2 on Items 1 (Overactive, aggressive, disruptive or agitated behaviour) and 2 (Non-accidental self injury) trigger a risk assessment, and an alert is registered on the system.
Other States/Territories are generating aggregate-level reports about compliance. For instance, Western Australia generates Statewide compliance reports that are distributed to mental health services every six weeks, and the Office of Mental Health works with services that are experiencing difficulties with compliance to review the systems in place for monitoring the NOCC collection.
A few States/Territories have started producing some rudimentary, aggregate-level reports that provide information about groups of consumers under the care of a given clinician, team or service. Tasmania, for example, has produced monthly reports for its Southern Region, which include aggregate-level data on average HoNOS scores at admission, review and discharge. Some States/Territories have begun to consider how best to provide these reports to areas and services. New South Wales, for example, has conducted a project involving workshops in all area health services, using their own data to demonstrate the clinical and management utility of the information. A similar process has been undertaken in Queensland.
A range of factors has hampered efforts at analysis and reporting to date. These include resource issues (e.g., lack of personnel and technological constraints), data quality, a lack of clarity about which reports will have greatest clinical and management utility, and the absence of relevant normative and/or benchmarking data.