The Australian experiment: the use of evidence based medicine for the reimbursement of surgical and diagnostic procedures (1998–2004)

Abstract

Background

In 1998 a formal process using the criteria of safety, effectiveness and cost-effectiveness (evidence based medicine) on the introduction and use of new medical procedures was implemented in Australia. As part of this process an expert panel, the Medical Services Advisory Committee (MSAC) was set up. This paper examines the effectiveness of this process based on the original criteria, that is, evidence based medicine.

Method

The data for this analysis was sourced primarily from that made available in the public domain. The MSAC web site provided Minutes from MSAC meetings; Annual Reports; Assessment and Review reports; Progress status; and Archived material.

Results

The total number of applications submitted to the MSAC has been relatively low averaging approximately only fourteen per year. Additionally, the source of applications has quickly shifted to the medical devices, equipment and diagnostic industry as being the major source of applications. An overall average time for the processing of an application is eighteen months. Negative recommendations were in most cases based on insufficient clinical evidence rather than clinical evidence that clearly demonstrated a lack of clinical effectiveness. It was rare for a recommendation, either positive or negative, to be based on cost-effectiveness.

Conclusion

New medical procedures are often the result of a process of experimentation rather than formally conducted research. Affordability and the question of who should pay for the generation, collection and analysis of the clinical evidence is perhaps the most difficult to answer. This is especially the case where the new procedure is the result of a process of experimentation with an old procedure. A cost-effective way needs to be found to collect acceptable levels of evidence proving the clinical effectiveness of these new procedures, otherwise the formal processes of evaluation such as that used by the Australian MSAC since 1998 will continue to run the risk of committing Type II errors, that is, denying access to medical procedures that are beneficial and efficient.

Introduction

Arising out of the uncertainty in decision making in any health care sector decision makers are faced with the dilemma of determining which has the greater risk, making available medical procedures that are ineffective or even harmful (Type I error) or, denying access to medical procedures that are beneficial and efficient (Type II error). Owing to the long shadow of thalidomide, there may be an over-emphasis by decision makers on the avoidance of a Type I error. Additionally, the growing availability of new technology and the resultant cost blow-outs may also have biased decision makers against making a Type I error. The combined effect may result in an unacceptable level of denying access to medical procedures that are beneficial and efficient (Type II errors). This potential dilemma can be shown diagrammatically as in Table 1.

Table 1 Dilemma associated with Type I and Type II Errors

In April 1998 the then Federal Minister for Health and Family Services announced that Australia had become the first nation in the world to formally adopt evidence-based medicine (EBM) as a key feature of its health system with all new medical procedures being independently evaluated by an expert panel before being admitted to the Medicare Benefits Schedule.

"This new vetting procedure will make quality a central feature of the health system by ensuring that only medical procedures and new technologies which were safe, cost-effective and of real benefit to patients were funded through Medicare. We will no longer find ourselves in the untenable position of new procedures being used in Australia simply on the basis of anecdotal evidence or because of aggressive marketing. The introduction of evidence based medicine and the committee means that the gap between research knowledge and clinical practice will narrow, and patients will benefit earlier from the most advanced procedures drawing on the best scientific and medical evidence."[1]

In Australia, for a medical procedure to attract funding covering the fee for the medical practitioner paid in the case of privately insured patients (43% of the population [2]), the procedure must have a Medicare Benefits Schedule (MBS) Item Number. Although this Item Number only directly determines the basic scheduled fee payable to the medical practitioner, indirectly it determines the payment of all other costs associated with the procedure (theatre, bed-days, equipment and prosthesis) since these payments are contingent on the procedure having this MBS Item Number.

Evaluation of evidence accompanying applications from the medical profession for the listing of new medical services on the Australian Medicare Benefits Schedule (MBS) is not new. The assessment of evidence has always been an integral part of the listing process of medical technologies and services on the MBS via a mix of specialist consultative and advisory bodies. The creation of the MSAC was seen as a way of formalising and strengthening this process, especially in terms of the cost-effectiveness criterion.

The guidelines for this new system of applications to the Medical Services Advisory Committee (MSAC) were loosely based on the Australian Guidelines for submissions to the Pharmaceutical Benefits Advisory Committee (PBAC) for the evaluation of all new pharmaceutical seeking listing on the Australian Pharmaceutical Benefits Schedule (PBS) that was made mandatory in Australia in January 1993 [3]. A listing on this schedule results in the pharmaceutical being subsidised by the Australian Government.

Although there are many similarities between the process used for pharmaceuticals and the new process for medical procedures, there are a number of important differences.

Submissions to the PBAC for PBS listing of a pharmaceutical include the evaluation of all the evidence by the company making the submission (usually the manufacturer or distributor). This evidence is reviewed by the Pharmaceutical Benefits Branch (PBB) and the process is totally confidential until the outcome of the submission is made public (a one page summary referred to as the Public Summary Document). In contrast, applications to the MSAC are evaluated by contractors employed by the Medicare Benefits Branch (MBB). Additionally, this process has a reasonably high level of transparency with publication on a Web site of the receipt of the application, its progress, and a detailed written report of the outcome.

Approximately one hundred submissions are received by the Pharmaceutical Benefits Branch (PBB) each year. On average, approximately nineteen applications or references are received by the Medicare Benefits Branch (MBB) for the listing of procedures.

The safety and efficacy of a pharmaceutical must already have been established by the Australian Therapeutic Good Administration (TGA) leaving only the effectiveness and cost-effectiveness to be established in the submission to the PBAC. An application to the MSAC evaluates safety, efficacy, effectiveness and cost-effectiveness of the procedure all in the one evaluation.

The time between the lodgement of the submission and the listing of the pharmaceutical on the PBS, assuming a positive outcome, is nine months [4]. A period of twenty four months between the lodgement of an application to the MSAC and the listing of the procedure on the MBS is considered to be standard.

Pharmaceuticals listed on the PBS are subsidised by the Australian Government for the whole population regardless of private health insurance status. The listing of a procedure on the MBS result in subsidy only to those covered by private health insurance, approximately 43% of the population as at December 2004 [5].

The establishment of the Medical Services Advisory Committee (MSAC) by the Australian Federal Government had three key objectives [1].

• Only medical procedures and new technologies which were safe, cost-effective and of real benefit to patients would be funded through Medicare.

• There would be a more rigorous assessment by the MSAC to ensure that the medical procedure and new technology was safe, cost effective and of real benefit to the patient.

• The gap between research knowledge and clinical practice would narrow, and patients would benefit earlier from the most advanced procedures drawing on the best scientific and medical evidence.

After more than six years of operation, it may now be timely to review how effective the MSAC has been, both in terms of screening the introduction of new procedures as well as a possible hurdle to the introduction of new technology supplied by the medical devices and equipment industry. This review of the achievement of the original objective of the MSAC is based on:

1. Usage Rate of the MSAC process and source of applications

2. Time taken for processing MSAC applications and references

3. Recommendations and outcomes of MSAC applications and references

Method

Data sources

The data for this analysis was sourced primarily from that made available in the public domain. The MSAC wed site provided minutes from MSAC meetings, annual reports, assessment and review reports, progress status and archived material.

In addition to this primary source, especially in the case where contradiction arose, verbal clarification was sought from various officers in the Health Technology Section and the applicants of individual applications.

Calculation of duration of assessment

In Tables 2a and 2b, the calculation of the duration of assessment for applications was the time between acknowledgment of receipt of the application at a MSAC meeting and the date of sign-off by the Minister for Health and Ageing. Applications submitted before December 2004 that had not been signed off by the Minister have a duration shown as a number of months with a plus sign representing the duration between acknowledgment of receipt at a meeting of the MSAC and December 2004. These estimates are considered to be conservative since the application could have been received up to three months before being acknowledged at a meeting. For example, Application 1042 was acknowledged at the May MSAC meeting but was actually received 1^st March 2001.

Table 2a Application to the MSAC and Duration of Assessment

Table 2b References to the MSAC and Duration of Assessment

Results

Usage rate of the MSAC process

The MSAC system cannot be said to be fulfilling its criteria if the system is not utilised. Between April 1998 and the end of December 2004 ninety-one applications and thirty-three references had been made to the Medicare Benefits Branch, Department of Health and Ageing of the Australian Government for evaluation by the MSAC. In addition to assessments initiated by applications (applications), MSAC's terms of reference allow the Committee to undertake reviews (references) as referred by the Minister or the Department of Health and Ageing. The procedures covered by these application and references are shown in Tables 2a and 2b.

Although Tables 2a and 2b show a wide diversity of applications and references for new procedures requiring subsidy via the establishment of new MBS Item Numbers, the total number is low averaging only approximately nineteen per year. Compare this to the number of existing MBS Item Numbers. The November 2004 edition of the Medicare Benefits Schedule runs for 696 pages with 190 pages of listings for therapeutic procedures (with up to twenty Item Numbers per page), another 56 pages for diagnostic procedures and diagnostic imaging, and 25 pages of pathology services.

Medical practitioners as a source of MSAC applications and 'Item Drift'

When the MSAC was established in 1998 it was assumed that the majority of applications would originate from the professional medical associations that represent the medical practitioners since the Medicare Benefits Schedule (MBS) is primarily a schedule of fees for the payment of the medical practitioner. A summary of the sources of MSAC application and references, divided into financial Years (1^st July to 30^th June), is shown in Table 3. In the June 2003 to July 2004 Financial Year, only one of the fifteen applications came from the medical profession. The remainder were sponsored by industry.

Table 3 Applications & References – Duration of Assessment

Possibly, some early experiences with this new system resulted in alerting the medical practitioners that an MSAC application is a time consuming and risky process. An MSAC application results in attention being focused on the usage of the existing MBS Item Number to cover the new procedure. If the MSAC application is unsuccessful, the wording of the existing Item Number will more than likely be modified specifically to exclude the new procedure.

A further possible explanation for this low usage rate of the MSAC process by medical practitioners is a practice that may best be referred to as 'Item Drift'. This is the practice of an established procedure evolving into a new procedure over time with the medical practitioner's fee being claimed under the existing MBS Item Number. This practice is somewhat facilitated by the often broad and vague wording of the descriptors of the existing Item Numbers. A clear example of this is the low number of laparoscopic procedures listed on the MBS despite laparoscopic surgery being common practice. The additional time and skills required by the medical practitioner to perform a new procedure can be covered by simply increasing the gap payable by the patient (total fee charged minus amount covered by the MBS Item Number).

Industry as a source of MSAC applications

Despite the original expectation that the majority of MSAC applications would originate from professional medical organisations, the source of applications has quickly moved to the medical devices and equipment industry as being the major, and virtually only, source of applications. In 2004 all but one MSAC application came from the medical devices and equipment industry, raising the question as to why this industry is taking such a leading role in a process primarily designed to facilitate fees for medical practitioners.

The answer lies with the nature of the new procedures. A close examination reveals that without exception all new MSAC applications cover procedures that include the use of new technology, that is, capital equipment, consumables, disposables, prostheses or medical devices. The payment for capital equipment, consumables and disposables is determined by a process referred to as Theatre Banding that determines the payment of theatre costs. The 'banding' of a procedure is directly linked to the MBS Item Number used by the medical practitioner. If the medical practitioner uses an existing MBS Item Number the new capital equipment, consumables or disposables is not covered. Similarly, the payment for prostheses or medical devices is covered by a listing on The Prostheses List. One of the criteria for a listing on this schedule is a relevant MBS Item Number. It can be difficult to explain how an MBS Item Number that has been in existence for over a decade can cover a procedure that includes prostheses new on the market in 2004. Up until 2004, the requirement for the procedure used to implant the prosthesis or medical device to be covered by a relevant MBS Item Number in order for the prosthesis or medical device to qualify for reimbursement, has rarely been enforced. However, with the rapidly growing volume and costs of prostheses and medical devices, private health insurers are now lobbying for this criterion to be enforced.

Time taken for processing MSAC applications and references

According to the original press statement announcing the introduction of the MSAC, "patients will benefit earlier from the most advanced procedures drawing on the best scientific and medical evidence"[6]. The details of the processing times for each application or reference is shown in Tables 1a and 1b and summarised in Table 3. The overall average processing time was approximately 18 months for applications and 22 months for references.

The steps involved in processing an application can be briefly outlined as follows:

• Pre-lodgement meeting between Applicant and Health Technology Section.

• Application lodged with the Health Technology Section.

• Written information to Applicant that application has been received and deemed eligible.

• Project Officer from the Health Technology Section allocated.

• Chairperson of Advisory Panel selected from the MSAC at next scheduled meeting.

• Formation of Advisory Panel – letters sent out to relevant 'Craft Groups' for nominations.

• Evaluators appointed by Health Technology Section and requested to draft a protocol.

• First meeting of the Advisory Panel and Evaluators – refinement of draft protocol.

• Draft protocol sent out to Applicant for comments (14 days).

• Comments on draft protocol reviewed by Advisory Panel Chairperson and other members of panel if necessary.

• Evaluators evaluate application (three months).

• Evaluators draft Assessment Report presented to the Advisory Panel.

• Advisory Panel review draft Assessment Report – Evaluators incorporate changes.

• Reviewed draft Assessment Report sent out to Applicant for comments (one month).

• Response to the Applicant's comments to the Advisory Panel by Evaluators.

• Assessment Report (complete with recommendation by the Advisory Panel) and Applicant's comments presented to the next MSAC meeting.

• Recommendation of the MSAC sent to the Minister for Health and Ageing.

• Recommendation ratified by the Minister for Health and Ageing.

Note that it is only the Applicant and the Evaluators that have any time limits imposed.

The role of the Advisory Panel is to assist in the assessment of each application and provide expert input into the assessment process as well as ensuring that the Evaluator's assessment is clinically relevant. Although the Advisory panel is central to the process, it is also a major cause of delay owing to the time taken to form this panel. The formation and organisation of the first meeting of this panel can take in excess of six months. During this time the only progress made by the application is the briefing of the Evaluators and the resulting draft protocol from the Evaluators. For example, an application received in December did not have the first meeting of the Advisory Panel until the following August.

The processing times detailed in Tables 2a and 2b do not take into account the time between the lodgement of the application and the next MSAC meeting, a period of up to three months. However, far more importantly, once a decision has been ratified by the Minister, the application has to be processed by another committee, most often the Medicare Benefits Consultative Committee (MBCC) for the wording of the MBS descriptor and the determination of the fee for the service. This process is not commenced until the Minister has approved the MSAC recommendation. As a consequence a period of two years between the date of the lodgement of an application and the actual listing of the new procedure on the MBS as a claimable Item Number is not uncommon. This is clearly illustrated in Table 5 using a selection of the applications lodged for the first time in the second half of 2004.

Table 4 Sources of Applications and References by Financial Year

Table 5 Processing times for Applications

Recommendations and outcomes of applications and references to the MSAC

Table 6 summarises the known and ratified (by the Minister for Health and Ageing) outcomes of all MSAC applications and references submitted up to the end of December 2004. It is difficult to quote a percentage of positive recommendations since these are divided into positive, partial positive and interim recommendations.

Table 6 Outcomes of Applications and References

A positive recommendation is basically a recommendation for an MBS Item Number covering the total indication applied for by the sponsor of the original application. A partial positive recommendation is a recommendation for an Item Number covering only part of the original indication applied for by the sponsor. An interim recommendation is temporary funding and is approved when the evidence is inconclusive but suggests that the procedure could be at least as safe, more effective, and more cost-effective than the existing comparable procedure. In these circumstances, the MSAC usually recommends interim funding for a period of three years to enable data collection and further evaluation of the procedure. These applications require a reapplication at the end of the three years based on the additional evidence collected during that time in order to maintain funding. For example, applications 1029 and 1031 that received interim funding in 2002 were re-submitted as applications 1089 and 1092. Application 1089 has been granted an extension of interim funding in order to allow time for the collection of data.

Tables 7a, 7b, 7c and 7d give a summary of the published findings from completed applications under the headings of safety, effectiveness, cost-effectiveness, comment and type. A simple qualitative analysis of these tables brings up some interesting points.

Table 7a Applications with Positive Recommendations

Table 7b Applications with Partial Positive Recommendations

Table 7c Applications with Interim Recommendations*

Table 7d Applications with Negative Recommendations

As would be expected, any procedure that had a serious safety concern was not recommended. Applications that related to a procedure likely to be carried out on a small number of patients were more likely to be given a positive recommendation. Although often applications with positive or partial positive recommendations were based on 'solid' clinical evidence of effectiveness, this was not always the case, especially in the case of interim funding. Importantly, negative recommendations were in most cases based on insufficient clinical evidence rather than clinical evidence that clearly demonstrated a lack of clinical effectiveness. It was rare for a recommendation, either positive or negative, to be based on cost-effectiveness since less than 10% of the literature search carried out by the evaluators resulted in finding any acceptable papers covering this criterion. Although, logically, this was to be expected for applications with negative recommendations due to insufficient clinical evidence, what was unexpected was that it appeared to be equally true for those applications with positive recommendations. Diagnostic procedures have a much higher total positive, partial positive or interim recommendation rate compared to surgical or therapeutic procedures. Therapeutical procedures were far more likely to be ineligible compared to surgical or diagnostic procedures.

The number of negative recommendations based on insufficient clinical evidence raises the important issue of what level of clinical evidence is sufficient. With pharmaceuticals the gold standard is the Phase Three, double blind, randomised, head-to-head clinical trial with statistically significant outcomes. In 1995, the National Health and Medical Research Council (NHMRC) prescribed a schedule of levels of evidence by which the efficacy of treatments could be assessed [7]. The 1999 revised version of these levels of evidence is shown as Table 8.

Table 8 NHMRC Levels of Evidence

However, the generation of evidence for a medical procedure has to take into account a number of factors that do not apply to a pharmaceutical. These include:

A new medical procedure often results from a process of experimentation and variation of an existing established procedure. Small incremental changes, over time, can result in a new procedure substantially different from the original procedure. However, it is unclear at what point the procedure became a 'new' or different procedure.

Clearly, a pharmaceutical company has a financial incentive in investing in running a clinical trial on their product. However, in the case of a procedure the identification of who should carry out the clinical trial is far less clear. Individual medical practitioners or their association do not have the financial resources or the incentive.

In the case of a procedure that involves an implantable prosthesis or an item of capital equipment, the manufacturer or distributor may have an incentive to carry out the clinical trial. However, even if a medical device company does go to the expense of running a clinical trial resulting in a successful application to the MSAC, the resulting new MBS Item Number cannot be restricted to the use of that company's product. Any similar product from a competitor can be used in the procedure. This reduces the possible competitive advantage and thus incentive to carry out the clinical trial.

The best timing of a clinical trial for a procedure can be difficult to determine. If the trial is carried out too early the outcome may not be optimal due to the lack of the medical practitioner's experience or practice in performing the new procedure. Conversely, a delay in running the clinical trial results in a reduction in the 'window of time' on the return of the financial investment in the clinical trial and the new technology.

In the case of a procedure that involves new technology, there is the additional timing problem created by the on-going development and refinement of the prosthesis or capital equipment. Unlike a pharmaceutical that enters the market as a finished product, prosthesis and capital equipment continue to evolve once in the market based on feedback from the medical practitioners and patients, resulting in newer and better versions. A clinical trial based on the first version will often generate less than optimal results. However, a delay in carrying out the clinical trial and the resulting delay in funding reduces the financial viability of the product.

Clinical trials with statistically significant outcomes are expensive to conduct. Unlike pharmaceuticals, where the potential market is often measured in tens of millions of dollars per annum, the market for prostheses and capital equipment is far smaller. This limits the affordability of clinical trial covering procedures.

Clearly, there are a number of challenges involved in the generation of high level clinical evidence for medical procedures. At the same time, a public funding body such as the Department of Health and Ageing of the Australian Government cannot risk giving approval for a medical procedure that has not been proven to be safe and effective.

Worldwide, there has been a rush of guidelines for the evaluation of medical procedures with the more recent efforts acknowledging the differences between pharmaceuticals, surgical procedures and diagnostic tests. Unfortunately, the requirements of these guidelines are based more on what should be, rather than what is, since there would appear to have been less of a rush to generate the evidence required by these guidelines. As a consequence, the majority of negative recommendations from the MSAC continue to be based on insufficient evidence.

Discussion and implications for health policy

Proposed timetable for the processing of applications to the MSAC

Presently there is no set timetable for the processing of applications to the MSAC. Despite the fact that one of the most crucial stages, the evaluation of the evidence, is given a set time of three months, unexplained delays in other stages of the process have resulted in applications taking in excess of two years to generate a new MBS Item Number. Table 9 shows a possible timetable for the processing of applications covering the major steps in the process. A set timetable would add a degree of certainty to the process for the Applicant and would eliminate the huge variation in the current processing time.

Table 9 Possible timetable for the application process

Appointment of Advisory Panels and the role of the Australian Medical Association (AMA)

The clinical and economic benefits of a positive MSAC recommendation can be dramatically reduced if the recommendation is not arrived at in a timely fashion. The MSAC process presently averages two years between the acceptance of the application and the listing of the new Item Numbers on the MBS. A major cause of this lengthy delay is the amount of time spent forming the Advisory Panel and agreeing on the dates of the first and subsequent meetings of this panel.

Currently, when a new application to the MSAC is received by the Medicare Benefits Branch, one of the first steps in the process is to write to the relevant medical association (Craft Group) and ask for nominations for positions on the Advisory Panel. This part of the process is open-ended in terms of timing of the response from the Craft Group and can take several months.

An example of the make up of expertise of the membership of an Advisory Panel is that for Application 1089, brachytherapy for the treatment of prostate cancer. This panel included a nuclear medicine specialist (Chair and MSAC member), two urologists, two radiation oncologists and, a consumer representative.

Discussions with the Australian Medical Association (AMA) and Craft Groups aimed at formulating guidelines, including timing, for the nomination process could facilitate a more rapid appointment of Advisory Panel members and allow the first Advisory Panel meeting to be convened within six weeks of the lodgement of an application.

Interim funding

Currently, where the evidence is inconclusive but suggests that the procedure could be at least as safe and possibly more effective and cost-effective as the existing comparable procedure, the MSAC may recommend interim funding for three years subject to the condition that additional data be collected to allow further and longer term evaluation of the procedure.

In its submission to the productivity commission inquiry into impact of advances in medical technology on health care expenditure in Australia the Faculty of Radiation Oncology (RANZCR) stated that "the implementation of new technology into Australia is affected by perceptions of efficacy, international experience, and MSAC approval for funding under the MBS. The introduction of new technology is not necessarily consistent with MSAC requirements in that the lag time to generate high-level evidence of benefit is often not consistent with clinical imperatives to introduce new technology. Furthermore, a chicken and egg situation can develop where funding for equipment is needed to generate evidence of effectiveness."[12]

RANZCR went on to say that 'the nature of equipment-based developments and the process of obtaining sufficient data for approval is a complex one, and there has been at least one instance in the Faculty's field (brachytherapy for prostate cancer) in which MSAC provided an interim approval. There have been challenges in collecting data to gain ongoing approval of MBS funding, especially when this is undertaken in an arm's-length manner from the equipment manufacturers, along the lines of clinical trial data collection protocols."

If interim funding in an expanded form, subject to the on-going collection of data, is to be a solution to the 'insufficient evidence' problem, there are clearly a number of issues that need to be resolved. Foremost of these issues are the funding of the cost of the data collection, the duration of the data collection and, who will be responsible for the collection of the data. A major drawback of an interim funding system is the problems associated with the discontinuation of funding should the data collected show the procedure not to be cost-effective. Owing to the fairly recent nature of the implementation of the MSAC process only one application that was initially granted interim funding has been reassessed. This application was brachytherapy for the treatment of prostate cancer (Application 1089) and it has been granted an extension of interim funding in order to allow more time for data collection.

A rare example of a government funded audit of a procedure granted interim funding by the MSAC is the audit of endoluminal repair of abdominal aortic aneurysms (Application 1006). The 1999 MSAC report on this procedure found that although it appeared to be effective in the short-term, there was insufficient evidence relating to the long-term safety and effectiveness. The Australian Audit of Endoluminal Graft (ELG) Repair of Abdominal Aortic Aneurysms (AAA) was established in 1999 to evaluate the mid- to long-term safety and efficacy of the procedure in the Australian setting. Results from the audit will help to inform funding decisions made by the Australian Government [14].

As at May 2005, this audit based on procedures carried out between November 1999 and May 2001 is on-going with a cost already measured in the hundreds of thousands of dollars.

Unlike pharmaceuticals, the financial beneficiary from the reimbursement of a new medical procedure can be difficult to identify. This raises the question of who should pay for the production, collection and analysis of the evidence. Table 4 shows that for the four year since July 2000, 86% of applications received by the MSAC have come from industry. This is due to the fact that these new medical procedures involve the use of new technology manufactured or distributed by medical device, medical equipment or diagnostic companies and an MBS listing is crucial to the commercial viability of these products.

Clearly, industry needs to be informed, and thus aware of the minimal level of evidence required by the MSAC process. If industry is to take a more active financial role in the generation of evidence it may also be justified in seeking a greater role in the process. At this stage industry has no representative on either the MSAC or any of the Advisory Panels. With new procedures that result from the development of substantial capital equipment rather than new variations of existing procedures (experimentation), for example robotically assisted minimally invasive surgery, the collection of affordable and acceptable clinical data should be possible.

Ongoing and large scale evaluation of the cost effectiveness and the incorporation of economic considerations into guidelines

In their 2001 paper, Eccles et al [15] explored the methods of incorporating cost issues within clinical guidelines and concluded that unlike other areas of guideline development, there is little practical or theoretical experience to direct the incorporation of cost issues within clinical guidelines.

In 2005 Richardson [16] concluded that:

"The scale of present evaluation activities is inadequate. In an industry absorbing 9 percent of the GDP – the country's largest industry – there should be ongoing and large scale evaluation and re-evaluation of the cost effectiveness of the services provided. Evaluation should be based upon a comparison with the full spectrum of substitute services. A failure to do this almost certainly ensures that there will be widespread and significant allocative inefficiency in the level and mix of services."

Despite the fact that the MSAC was the first of its kind in the world, the National Institute for Clinical Excellence (NICE) has overtaken it in terms of expertise due to a far greater allocation of resources. Presently the Health Technology Section that manages the evaluation process is understaffed and as a consequence struggling to cope with the workload. There is an urgent need for an injection of staff qualified to deal with not only the day to day administration of the evaluation process but also the development of methodology aimed at generating acceptable clinical evidence upon which to base cost-effectiveness evaluations. Methodology developed for these new procedures could also be applied to existing procedures that have raised some concern. A major policy issue that has become clear as a result of the evaluation of new procedures is that many of the existing 'gold standards' used as comparators are anything but gold and are based on little or no real evidence.

Conclusion

New medical procedures are often the result of a process of experimentation rather than formally conducted research. The medical procedures being evaluated by the MSAC are, in the majority of cases, procedures that have been carried out elsewhere in the word for some time. Ironically this includes diagnostic tests using capital equipment developed in Australian itself. This raises the question as to exactly why there is such a low rate of positive recommendations from the MSAC. The answer may lie with what is considered to be an acceptable level of evidence and the fact that we have yet to develop a financially viable formal process for the generation of this level of evidence in Australia or elsewhere in the world.

The key characteristics of this clinical evidence are affordability, timeliness, systematic collection, unbiased and, not anecdotal.

When possible, interim funding of a new procedure can ensure the systematic collection of evidence while allowing the use of the new procedure. The answer to the timeliness question lies in a combination of a greater use of interim funding as well as the elimination of the unproductive time usage that exists in the current MSAC process.

Affordability and the question of who should pay for the generation, collection and analysis of the clinical evidence is perhaps the most difficult to answer especially in the case where the new procedure is the result of a process of experimentation with an old procedure. In cases where new technology is used, industry may be able to play a role here. However, this must be dependant on industry having a greater role in the MSAC decision making process.

Up to this point in time the use of economics and cost-effect analysis has not been feasible due to a lack of acceptable levels of clinical evidence. Currently, only the cost side of the equation is possible in the vast majority of new medical procedures with this often working against the new procedure since virtually all new procedures involve the use of expensive new technology.

Unless a cost-effective way is found to collect acceptable levels of evidence proving the clinical effectiveness of these new procedures, formal processes of evaluation such as that used by the Australian MSAC since 1998 will continue to run a very high risk of committing Type II errors, that is, denying access to medical procedures that are beneficial and efficient rather than risk a Type I error, approval of a non-beneficial procedure.