Document MJR1eyRJ1arYKaLpEgQQDrd49

HSE Health & Safety Executive Health and Safety Executive Occupational Medicine and Hygiene Laboratory MDHS 71 Methods for the Determination of Hazardous Substances March 1991 Analytical quality in workplace air monitoring INTRODUCTION 1 Under the Health and Safety at Work etc Act 1974 there is a general requirement for employers to protect the health of their workforce, but with the introduction of the Control of Substances Hazardous to Health (COSHH) Regulations the duties of employers are stated more clearly and in more detail. Exposure of employees to hazardous substances must either be prevented, or where this is not reasonably practicable, adequately controlled. For most toxic substances the major route by which exposure occurs is inhalation and the demonstration of adequate control may well require the measurement of airborne concentrations. The results of such measurements are used to make decisions which often have significant implications both for the health of employees and the financial burden on employers. It is therefore vital that the measurements should have sufficient integrity. 3 The Health and Safety Executive (HSE) has for some years provided practical guidance on several aspects of the sampling and analysis matrix. Thus Guidance Note EH42 has covered sampling strategy, and the Methods for the Determination of Hazardous Substances (MDHS) series published by the HSE Committee on Analytical Requirements has detailed sampling and analytical methods. Apart from the use of properly validated methods, the other important requirements for maintaining analytical quality are internal quality control and external quality assessment. 4 This MDHS is intended to provide guidance to laboratories on the practical application of internal quality control procedures within the laboratory, although it also includes some information on external quality assessment. It is not intended to be exhaustive nor to set down detailed requirements for the operation of quality control schemes in particular laboratories. 2 The factors which contribute to the quality of the final measurement, which is usually a worker's personal exposure, are illustrated below. Sampling Analysis DEFINITIONS 5 The terminology used in this document is essentially the same as that developed by Whitehead and Woodford.1 Internal Quality Control (IQC) is the set of procedures undertaken by the laboratory staff for continuous monitoring of operations and results in order to decide whether the results are reliable enough to be released; IQC primarily monitors the day-to-day consistency of results on quality control samples. I Interpretation/ decision making External Quality Assessment (EQA) refers to a system for objectively checking laboratory results by means of an external agency. It includes comparison of a laboratory's results at intervals with those of other laboratories, the main object being the establishment of inter:laboratory comparability. EQA is designed to assess the accuracy of a laboratory's results. The internationally agreed term `proficiency testing' is slowly gaining acceptance in the UK; it means the same as external quality assessment and the two terms are interchangeable. Quality assurance programme/system refers to the sum total of a laboratory's activities aimed at achieving the 1 required standard of analysis. IQC and EQA are very important components of a quality assurance programme but it must also include staff training, administrative procedures, management structure etc. ANALYTICAL QUALITY REQUIREMENTS 6 The level of analytical quality required for effective occupational hygiene monitoring must be considered before procedures for establishing a quality assurance programme are discussed. Statistical quality control procedures can determine what is currently achievable in terms of intra- and inter-laboratory precision and bias and may throw some light on the relative accuracy of different methods, but they do not determine what are desirable levels of accuracy and precision. 7 The existence of legislative requirements to take corrective action when exposure limits are exceeded has a bearing on analytical quality requirements. Laboratories with a negative analytical bias may be putting workers' health at risk; those with a positive bias may be placing an additional financial burden on an industrial company and so putting it at a competitive disadvantage. 8 The Comite Europeen de Normalisation (CEN) Working Group draft proposal on performance criteria2 has suggested limits to the overall uncertainty in occupational hygiene analyses. This overall uncertainty would include bias and imprecision in the combined sampling and analytical methods. Appendix A contains a detailed discussion of the relationship between the imprecisions in sampling and in analysis and how they affect the overall imprecision. When the imprecision of the analytical method falls below half that of the sampling method, its contribution to the overall imprecision becomes very small. For example, if the coefficient of variation (CV) of a sampling method were 10%, an analytical method with a coefficient of variation of 5% would be adequate since at that level the contribution of the analytical method to the overall imprecision would be small in comparison to the contribution of the sampling method. 9 For many analytes, measured at the occupational exposure limit, methods with a between-batch imprecision of about 5% and a bias of no more than 5% may be suitable. Poorer precision and accuracy may be acceptable when, measuring air concentrations at one tenth or ten times the occupational exposure limit than when measuring close to the occupational exposure limit. GUIDANCE ON ESTABLISHING A QUALITY ASSURANCE PROGRAMME Laboratory management 10 The senior management of the company to which a laboratory belongs must be committed to the achievement and maintenance of a high standard of quality in all aspects of the laboratory's work. 11 The term `quality control' (QC) is used normally in the sense of the monitoring and control of the precision and accuracy of laboratory measurement, and this paper is concerned mainly with this use of the term. However, the statistical control of analytical quality is only one aspect of laboratory quality control. A broader definition would include the monitoring and control of all errors arising within the laboratory between the receipt of the sample and the despatch of the report. 12 In some respects even this definition is too narrow if quality is to be seen in the wider terms of the laboratory's role in ensuring the usefulness of a result. Thus, for example, the quality of communication between the laboratory and the occupational hygienist is a very important aspect of quality control which should not be overlooked. Good communication would help to ensure the production of timely results, a clearly presented report, or even that an appropriate analysis was undertaken, all of which are essential if results are to be understood and appropriate action taken. A report which arrived too late, or one which was presented badly and was thus misinterpreted, would negate all the effort put into the analysis. 13 Analytical quality does not depend solely on the skill of the analyst and on statistical techniques for monitoring the accuracy and precision of analytical methods. An important decision in analytical quality control is the initial selection of the analytical method to be used, which must be robust, and whose performance characteristics (bias, precision, detection limit etc) must have been determined and documented. The method selected must be appropriate for the level of qualification and skill of the analyst who is to use it. 14 The laboratory manager has a central role to play in choosing appropriate analytical methods and is also responsible for several equally important influences on analytical quality.3 These include the establishment and auditing of administrative procedures for sample identification, for the reporting of results and for the maintenance of records; the management of the purchase and maintenance of equipment and the purchase of reagents and other consumable supplies; and the training and deployment of staff. 15 A Quality Manager should be appointed who is responsible for establishing and monitoring the laboratory's quality assurance programme. Responsibility for day-to-day quality control matters may be devolved to the analyst, but the role of the Quality Manager should be to conduct a more thorough analysis of the quality control data, looking for trends, and trying to associate changes in results with specific events. 16 The Quality Manager must be aware that a quality assurance programme should be managed with some sensitivity. It is very easy for staff to feel that their performance is being judged. But a quality assurance scheme should not simply assess the performance of individual members of staff. It is the job of the laboratory manager to ensure that staff are properly trained and that they are not required to perform analyses for which 2 their level of knowledge, training and skill is inadequate. For many methods unknown factors may influence results.4 A quality assurance programme will help to identify those parts of the method which are operator dependent so that steps can be taken to eliminate this dependency. If a method gives different results with different analysts, the Quality Manager should be aware that the fault may lie with the method and not the analysts. 17 The role of management in achieving analytical quality control will not be considered further here, but HSE encourages laboratories to obtain NAMAS accreditation which would demonstrate that the laboratory's management and administrative procedures were sufficient to ensure that its work was conducted with integrity. Internal quality control (IQC) 18 The aim of a quality control scheme is to check every stage of an analysis. One way to check that an analytical procedure is functioning correctly is to take a matrix-matched sample of known value through the entire analytical procedure in the same way as the unknown samples. Any untoward losses or contamination experienced by the unknown samples during the analytical process are likely to be experienced by the quality control sample also and the analyst may be alerted to the fact that all was not well. 19 It is recommended that for analyses which are determined frequently, a batch of matrix-matched quality control material should be prepared. If it is not possible or practicable to do this, alternative approaches can be found in Appendix B. A suitable quality control material might be a filter spiked with a metal salt, or a charcoal tube to which a known quantity of analyte has been added. The quality control material must be stored under appropriate conditions which ensure maximum stability of the samples. One or more samples from this set of quality control material is then analysed with each analytical batch and the results compared with a target value. Determine the standard deviation (see paragraph B7 for details) of the method prior to implementing internal quality control. 20 Plot the value obtained for the quality control sample on a chart. It is much easier to detect changes in this way than by looking at lists of figures recorded in a notebook. The simplest form of chart is the Shewhart chart which is described in more detail in Appendix B. improvement in the precision of the results obtained, so that the old warning limits may no longer be appropriate. 23 A more sophisticated chart, the cumulative sum or cusum chart, can be a very informative way to represent data. The principle is that a target value, for example the mean value of the quality control material, is subtracted from each observed value and the cumulative sum of the deviations from the target is plotted against the serial number of the observation. Cusum charts are very useful for retrospective analysis of data to identify trends in performance associated with any changes in analytical methodology. More details of the cusum technique are given in Appendix B. 24 Appendix C describes a practical example of the role of Shewhart and cusum charts in internal quality control. 25 Be prepared to take action if the results of the quality control sample indicate that action is needed. Further analysis should not be conducted until the analytical procedure is brought under control. If the whole sample is destroyed during the analysis, have a clear policy for the reporting of results from analytical batches where IQC results are suspect. 26 Keep a log-book for each method for documenting any changes to the method, eg servicing of instrumentation, new standards, new reagents, change of analyst. In this way it may be possible to associate changes in the bias or precision of the method with specific events. 27 If stock standard solutions are used, do not change them and the quality control material at the same time. A new batch of quality control material should be run in parallel with the old to accumulate data on its mean value and consistency. 28 In summary, the quality control system adopted should be simple to operate. The use of a Shewhart chart with a simple set of rules would be suitable for dayto-day control. People may find cusums more difficult to work with and interpret.6 Cusum charts can prove more effective in detecting long-term changes and are perhaps better used by supervisory staff to conduct retrospective assessments of quality control. Commercial computer programs are available which will plot charts and flag results which are out of control. These can remove much of the tedium of maintaining IQC charts. 21 A Shewhart chart with simple rules, ie warning at 2SD, action at 3SD, is probably sufficient for a laboratory starting up an IQC scheme. A more complex set of rules can be employed once staff are familiar with IQC procedures. 22 Check the standard deviation of the method at intervals and revise the warning and action limits if necessary. The data used to calculate the initial standard deviation may not have been representative. The application of an IQC scheme may have resulted in an External quality assessment (EQA) 29 The primary purpose of an external quality assessment scheme is to provide an objective measure of the analytical performance achieved by a laboratory. That performance could be assessed against a defined standard of performance or by comparison with the performance of a group of laboratories. 30 One means of EQA is the regular analysis of Certified Reference Materials (CRMs). The availability of 3 CRMs makes it possible, in principle, to merge the functions of IQC and EQA into one, but in general it is not recommended that this should be done. There are usually good practical reasons for keeping the two functions separate, not least the cost of CRMs. More importantly, however, CRMs may become contaminated or damaged through daily use as quality control samples. Instead, routine IQC should be based on in house samples with periodic analysis of CRMs to provide an assessment of accuracy. results obtained would also be distributed about a mean value. In other words, the observed result for the concentration of analyte in any given sample contains two sources of variability, which can be expressed as the standard deviation of the sampling method (Ss) and the standard deviation of the analytical method (Sa). The variation of the observed result expressed in terms of a standard deviation (S0) will be greater than the sampling or analytical variation singly and will be equal to 31 Information on CRMs which have been developed for occupational hygiene analyses can be obtained from: The Office of Reference Materials Laboratory of the Government Chemist Queens Road Teddington Middlesex TW11 OLY Tel: 081-943-7565 Fax:081-943-2767 32 A better method of EQA is participation in an external quality assessment scheme. While the regular analysis of CRMs is useful, a laboratory will gain much more if it participates in an external quality assessment scheme. The Quality Manager will obtain information, over an extended period of time, of how the laboratory performs in relation to others. A laboratory experiencing problems with an analytical method may be able to receive help from the organisers of the scheme. Details of some external quality assessment schemes for occupational hygiene analyses can be found in Appendix E. 33 If no external quality assessment scheme provides for analyses which your laboratory performs frequently, approach the organisers of an existing scheme and ask for your analytes to be added to the scheme. If they are unable to do this, consider exchanging samples with other laboratories or even setting up an external quality assessment scheme to deal with your special requirements. 34 At present there are few EQA schemes for workplace air analyses, but new schemes dealing with particular analyses may be established in the future. In order to help those who wish to set up new EQA schemes and to help potential users of EQA schemes to evaluate them, the purposes of external quality assessment schemes and the characteristics they should have are discussed more fully in Appendix D. VSs2 + Sa2 A2 Table 1 demonstrates the position when Ss is 100 arbitrary units and Sa varies from 10 to 1000; S0 varies from 100.5 to 1005 units. A3 The more imprecise the analytical method, the more it contributes to the overall error. For most purposes it would be considered acceptable if analytical variation contributed 10% or less to the overall imprecision of the measurement. Therefore the analytical variation should be less than about 45% of the sampling variation. If sampling error is high in relation to analytical error, trying to improve the analytical error is not important. Table 1 sa 10 20 30 40 50 60 70 80 90 100 200 500 1000 So 100.5 102.0 104.4 107.7 111.8 116.6 122.1 128.1 134.5 141.4 223.6 519.6 1005.0 Ss= 100 units; SQ is the observed standard deviation for a number of samples collected from the same atmosphere; Sa is the standard deviation of the analytical procedure. APPENDIX B APPENDIX A Internal quality control The relationship between sampling, analytical and overall imprecision A1 Assume that the concentration of an analyte in air is constant and that, if a large number of samples of the air are collected, the quantity of analyte in the samples will be distributed normally about a mean value. If any one sample were to be analysed repeatedly, the individual B1 The purpose of this appendix is to provide background information for the laboratory manager who wishes to set up an internal quality control scheme. B2 There are some difficulties in devising a suitable quality control scheme for occupational hygiene analyses. It would be desirable to have a supply of samples, all with the same known concentration of 4 analyte, in a matrix identical to that of the unknown samples, to be taken through the whole of the analytical procedure concurrently with the unknown samples. B3 In practice, this ideal may be impracticable. If, for example, the laboratory measured lead on filters, the quality control sample could be a filter spiked with lead. But a large batch of filters would normally be prepared by spiking them with a solution of a readily soluble lead salt, while the real samples might contain a less readily soluble compound such as lead oxide, and may also contain other contaminating, and potentially interfering, material. Thus the matrices of the real and quality control samples would be different and the two types of sample might behave differently. B4 If a particular analysis is performed only infrequently, it may be inappropriate to use a matrixmatched quality control sample because of the cost of preparing and storing the sample. Again, if the analyte were unstable, the use of a quality control sample would not be appropriate. Where it was not possible or realistic to provide a matrix-matched quality control sample, suitable substitutes might be a solution of the analyte prepared independently from the calibration standards or a small quantity of known weight of the pure analyte which would then be taken through the complete analytical procedure. The percentage recovery of the analyte could then be recorded for QC purposes. The analyst would need to record some other aspect of the analysis to help make a judgement about its reliability. The absorbance of a standard, the slope of the standard curve or some aspect of the instrument, such as lamp current, may be appropriate. Quality control charts B5 The quality control chart is an essential part of the decision making process. A detailed description of quality control chart methodology is given in BS 5700.5 Two common procedures are discussed here. The Shewhart chart B6 This was first devised by Shewhart6 for use in manufacturing. On the x-axis is plotted the date of the analysis or analytical batch number and on the y-axis the determined value for the control material. Lines parallel to the x-axis are drawn from the y-axis at values denoting warning and action limits. B7 Determine at least 20 values for the quality control reading, each from a different analytical batch. It is assumed that the distribution of these values is Gaussian. Calculate the mean value and the standard deviation (SD) of the readings. The value for the quality control sample is plotted on the y-axis and the date or analytical batch number is plotted on the x-axis. Lines parallel to the x-axis are drawn at the mean, mean +2SD and mean +3SD values. The lines at the mean +2SD are regarded as warning limits, ie if the result falls between mean +2SD and mean +3SD or mean -2SD and mean 3SD it is taken as a warning that something may be wrong. One would expect by chance that one result in twenty would fall outside the warning limits. The lines at mean +3SD and mean -3SD are regarded as action limits, ie the results from that analytical batch cannot be reported without a senior member of staff being told and appropriate remedial action taken. B8 The Shewhart chart is easy to use and understand. It is easy to determine whether or not a result is outside the permitted limits and it is easy to understand how these limits are calculated. Cusum techniques B9 The cumulative sum or cusum chart,7'8 which is a plot of the cumulative sum of the deviations from a target value, provides another approach to plotting quality control data. There are two factors which must be selected with care in setting up a cusum chart; these are the choice of the target value and the scale of the chart. B10 It is easier for the eye to detect a change in slope away from the horizontal than it is to detect a change in an already sloping line. If the target value is the best estimate of the mean value obtained for the quality control reading, a method which is in control will produce a line roughly parallel to the x-axis. B11 The scale of the chart is also important. If the vertical scale is too great the swings due to normal variation of the method become too wide for a change in the angle of the slope to be detected easily. If the vertical scale is too small, it becomes difficult to detect any change at all. The accepted convention for deciding the appropriate scale is first to decide the scale on the horizontal axis, say 5 mm, 1 cm or 2 cm per observation, depending on the use of the chart (smaller scale for desk use, larger for wall charts). This distance is then taken to represent as close to 2 standard deviations on the vertical scale as is practicable. B12 With Shewhart charts (B6ff.), the deviation from the target value of the result obtained for the quality control material is represented by the height of the point plotted on the y-axis. On the cusum chart, deviation from the target value is represented by the slope of the line. When the average value corresponds to the target value, the line of the cusum is roughly parallel to the x-axis. When the average is greater than the target value, the line of the cusum slopes upwards; when the average is less than the target value, the cusum slopes downwards. The greater the discrepancy, the steeper the slope. B13 The cusum chart is sensitive to small changes in bias, but can be more difficult to use and interpret. It is more difficult to make a qualitative assessment of a change in the slope of a cusum than it is to determine on a Shewhart chart whether a data point is acceptable or not. B14 Quantitative interpretation of a cusum can be made using a V-mask, a template which can be laid on the chart. The mask can be designed to provide a particular probability of detecting a systematic change in 5 the analytical process, but nevertheless the significance of the change may be more difficult to interpret. A detailed description of the use of V-masks can be found in BS 5073.8 APPENDIX C An example of the application of quality control techniques Quality control rules B15 It would be convenient if a batch of analyses which was in control never produced an IQC result outside the warning or action limit and conversely, every batch which was out of control would produce an IQC result outside the warning or action limit. The statistical nature of IQC charting means that this ideal cannot be achieved. Thus a quality control scheme must have rules which set out the action to be taken when the values for the quality control material fall outside specified limits. However, several apparently arbitrary rules exist: eg the analyses should be repeated if the quality control value falls outside a specified limit; limits of 2 standard deviations and +3 standard deviations are common. In the former case 1 in 20 batches which are in control will be falsely rejected; in the latter about 1 in 300 will be falsely rejected, but this latter rule is not sensitive to small changes in bias, and analyses which are out of control may be accepted. B16 If one in twenty batches, which is nevertheless in control, is reanalysed, this represents an increase in workload of more than 5%, once the time and effort spent investigating a non-existent fault is taken into account. B17 The setting of action limits involves dealing with probabilities and balancing the probability of false rejection against the probability of accepting analyses which are out of control. If one sets the limit of acceptability wide there is a danger of letting through analyses which are out of control. Conversely, if limits are set so tightly that the probability of detecting out of control analyses is very high, there is a danger of increasing the false rejection rate. In addition, interpretation of quality control results becomes more difficult as more IQC samples are included in each batch. A rule or set of rules is needed which will combine a low probability of rejection of good analyses with an acceptable probability of rejection of analyses which are out of control. Cl The examples in this appendix demonstrate the application of the various quality control techniques discussed previously. The analytical method for which data were collected was the determination of lead on filters by atomic absorption spectrophotometry. The data were obtained from a laboratory which agreed to analyse matrix-matched samples as a data gathering exercise. The values obtained were recorded, but no attempt was made to plot them or use them for quality control purposes. It would not have been appropriate to impose an untried quality control scheme on the laboratory. C2 The laboratory operated the following calibration and internal quality control scheme. At monthly intervals the instrument was calibrated to provide a multi-point standard curve and the slope of this curve was stored by the instrument. The instrument was calibrated daily using a single standard and a blank solution. An aqueous lead solution (with a different source from those standards used to prepare the standard curve, and referred to as the routine quality control solution) was analysed as a quality control check and its value plotted on a Shewhart chart. C3 In addition to the procedures described in C2 the laboratory was asked, for a trial period, to analyse with each analytical batch a filter spiked with a solution of a lead salt, so that there was nominally 80 pg of lead on the filter. This filter is referred to as the trial quality control filter. The laboratory was also asked to record the absorbance of the calibration standard. C4 The different quality control techniques and their merits and shortcomings are discussed below. The routine quality control solution The Shewhart chart C5 In Figure 1, the values obtained for the routine quality control solution are shown plotted on a Shewhart chart. Most of the time the results fall within the warning limits, indicating that the method is in control. B18 Westgard et al. have proposed a multi-rule quality control scheme,9 based on a Shewhart quality control chart. The rules are simple and the scheme is suitable for the smaller laboratory using up to 4 quality control samples per analytical run. B19 Quality control rules reduce the opportunities for the application of subjective judgement in deciding whether a batch is in control or not. Their application should result in an improvement in the quality of the analytical work, for in the absence of rules, there is always a temptation to find reasons for accepting a batch which is out of control. The cusum chart C6 When the values obtained for the routine quality control solution are plotted as a cusum (Figure 2) some changes in slope can be seen. Some of these are associated with distinct events such as a change of analyst or the use of new standards, demonstrating that a cusum chart can detect changes in bias which are not immediately obvious from the Shewhart chart. The changes in slope are not consistent; for example, a change in analyst does not always result in a change in the slope of the cusum. In this case the cusum chart indicates that the procedures used in standard preparation may need to be examined. 6 5-4- Batch number Fig 1 Shewhart chart of values obtained for the routine quality control solution. Fig 2 Cusum plot of values obtained for the routine quality control solution. 7 Cumulative sum Absorbance of calibration standard Batch number Fig 3 Shewhart chart of values obtained for trial quality control filter. The trial quality control filter C7 The data obtained for the trial quality control filter were examined retrospectively. The mean and standard deviation were determined as described in para B7. Two samples were analysed in each batch and their mean values plotted on a Shewhart chart as shown in Figure 3. (There is no relationship between the batch numbers in Figures 1 and 2 and those in Figure 3, the data being collected at different times.) On some days more than one batch was analysed. It is obvious, even without a cusum plot, that there was a significant change at batch number 11 which continued until batch number 24. These results must be kept in perspective. The coefficient of variation of the method when it is well controlled is about 1.2%. Over the whole of the period for which data were collected, the coefficient of variation is still less than 6%. C8 Examination of the method's log book showed that the change in the value for lead on the filter coincided with a change of analyst and returned to its original value when a new lamp was installed in the atomic absorption spectrophotometer. The values obtained for the absorbance of the calibration standard were plotted on the Shewhart chart (Figure 3) and showed an inverse relationship between the absorbance and the value obtained for the trial quality control filter. This relationship implied that the change in bias was related to the performance of the instrument and not to some other part of the analytical procedure. C9 A thorough investigation revealed the cause of the variation in results. The lamp in the spectrophotometer was old. The monthly calibration had been carried out by one analyst who subsequently obtained consistent results for the lead on the filters. While the same analyst set up the instrument the results were not affected, but a different analyst, with a different method of `optimising' the instrument, obtained a different relationship between concentration and response which deviated from linearity at the higher concentrations. He therefore obtained different results for the trial quality control filter. The concentration of the routine quality control solution was such that it always lay on the linear part of the curve. The problem was solved when a new lamp was installed and the instrument recalibrated because the new calibration curve was linear. CIO As a result of the investigation, the laboratory has made a number of changes to its procedures. A stricter protocol has been written for calibrating and optimising the spectrophotometer. Individual analysts will not be allowed to set up the instrument in their own way. The calibration curve is now checked regularly for linearity. An IQC sample whose value lies at the top end of the calibration curve is run with each analytical batch. C11 Several lessons can be drawn from this example, not least that it is easy to believe that all is well. The laboratory's own QC scheme had not indicated that anything was wrong. Modern automated equipment can produce results with very good precision, but it is easy to 8 accept these results uncritically and to assume that all is in order. It is essential to keep comprehensive records so that it is easier to determine what has caused the loss of analytical quality control. Standard laboratory protocols for sample analysis must be devised and adhered to. An IQC sample which is treated in the same way as unknown samples should, if at all possible, be included in each analytical batch. It should be chosen carefully so that changes in its value will accurately reflect changes in the precision and bias of the method. A cusum chart can show changes that are not immediately obvious on Shewhart charts. APPENDIX D Setting a target value D6 There are several ways of setting a target value for the external quality assessment material, all of which have their strengths and weaknesses. Perhaps the value most commonly used is the group mean or consensus value, ie the mean of all results after exclusion of outliers. It is not necessarily the case that the consensus value is the same as the true concentration of the analyte in the quality assurance sample. The group mean may be biased if, for example, a significant number of participants use an inaccurate method; again, if there are only a small number of participating laboratories, a few poor performers may have a disproportionate effect on the group mean. Nevertheless, the group mean seems to work well in practice. External quality assessment D1 This appendix gives information on the objectives of an external quality assessment scheme. It gives the reader guidance on the points to look for in an existing external quality assessment scheme. Existing schemes may not meet the needs of certain groups of analysts; this appendix may help those who wish to set up their own informal schemes. The purpose of an EQA scheme D2 The primary purpose of an external quality assessment scheme is to provide an objective measure of the analytical performance achieved by a laboratory. In addition, external quality assessment schemes can serve several other purposes.10 They can be used to determine the `state of the art' of the measurement of a particular analyte. They can supplement internal quality control procedures. They can be used to obtain consensus values for samples, and they can have an educational effect in improving laboratory performance. They can give information on the relative performances of different analytical methods, and on the influence on laboratory performance of factors such as the size of the laboratory and the level of education and training of its staff, the work-load and the frequency with which analyses are performed.11 D3 Experience in clinical chemistry and with the United Kingdom External Quality Assessment Scheme (UKEQAS) for Lead in Blood has shown that it is often possible to identify inaccurate methods and that over a period of time there develops a consensus on which methods are the most accurate. D4 Because of the range of possible objectives for interlaboratory assesssment schemes, the organisers of such a scheme must have a clear primary objective and must design the scheme so that this objective can be met. D5 Details of well established external quality assessment schemes in occupational hygiene analysis are given in Appendix E. D7 The group mean has proved its usefulness as a basis for comparison in UKEQAS for Lead in Blood where the overall performance of the participants has shown a marked improvement over the years.12 D8 The mean value determined by a sub-group of reference laboratories is sometimes used to determine the target value and this approach should overcome the problem of bias caused by poorly performing laboratories. However, if a small number of reference laboratories are used, one which is out of control and obtaining inaccurate results could have an undue influence on the target value. D9 For practical purposes the group mean appears to work well, but if the number of participating laboratories is small, the use of a select group of reference laboratories to set the target value may be a better approach. Samples DIO The samples distributed should represent real samples as closely as possible and should have the required stability. Quality control checks should be performed on a random selection of each set of samples prepared to ensure that their quality is appropriate to the purpose. Frequency of sample distribution Dll Regular sample distribution is essential if any external quality assessment scheme is to be effective in improving laboratory performance. Distribution should be frequent enough to give a regular feed-back on performance to the participants but should not be so frequent that the external quality assessment could be substituted for internal quality control.13 A decision on the frequency with which to distribute samples should also take into account the extra work and cost incurred by the participating laboratories. It is unlikely that a scheme would be useful if the frequency of distribution fell below once every three months. 9 Measurement of performance D12 The usual practice in external quality assessment schemes is to return to each participant information on the target values for each round and some measure of the distribution (such as SD) of results about the target value. However, experience has shown that provision of this information by itself does not bring about an improvement in the overall quality of performance and further processing of the information is necessary. Each participating laboratory must be given some numerical indication of its relative performance, and the approach adopted by the Workplace Analysis Scheme for Proficiency (WASP) is to calculate a Performance Index and a Running Performance Index, and to allocate the laboratory to a Performance Category. Dealing with poor performers D13 It was stated earlier that external quality assessment schemes may have an educational function and this should be borne in mind when dealing with laboratories which consistently perform badly in such schemes. The aim of the organisers of an external quality assessment scheme should not be to punish a laboratory or to expel it from the scheme but to try to bring about an improvement in its performance. It is essential therefore that these schemes are organised by practising scientists who are well qualified to give the necessary professional advice. D14 Participants in the WASP scheme have access to an advisory panel of analysts who are able to offer advice to those laboratories whose performance falls below the acceptable level. Poor performers can be put in touch with good performers using the same equipment, can be given advice on the preparation of calibration standards and internal quality control material and, to help them check accuracy, can be issued with samples of surplus external quality assessment material for which a consensus value has been obtained. charcoal-packed tubes for solvent desorption. Benzene, toluene and m-xylene on Tenax-packed Perkin-Elmer thermal desorption tubes. Additional analytes will become available in the future. Operation of scheme Samples sent out quarterly by a contract laboratory in January, April, July and October each year. Participants receive four samples per round of each type requested plus appropriate blank filters and tubes. Assessment of performance For each analyte and sample a `target' result is calculated as the mean of all laboratories' results after exclusion of outliers. Individual results are then compared with this `target' figure and a numerical performance index is calculated which reflects the deviation from the target value. On the basis of at least four rounds, a laboratory is allocated a performance category for each analyte. Trends in performance over extended time periods can be displayed graphically. Registration and charges Potential participants should contact: Dr H M Jackson Occupational Medicine and Hygiene Laboratory Health and Safety Executive 403 Edgware Road London NW2 6LN UK Tel: 081-450-8911 Fax: 081-452-2961 There is an annual fee for participation to cover preparation and despatch of the samples. The charges vary between different sample types and are subject to periodic review. APPENDIX E External quality assessment schemes in occupational hygiene analysis 2 Title Proficiency Analytical Testing (PAT). There are known to be two general schemes in operation for this type of analysis, as well as a number of schemes for specific analytes. 1 Title Promotor National Institute for Occupational Safety and Health, USA. Workplace Analysis Scheme for Proficiency (WASP). Promotor Health and Safety Executive, UK. Range of analytes Lead, cadmium and chromium on both membrane and glass fibre filters. Benzene, toluene and m-xylene on Range of analytes Lead, cadmium, zinc and chromium on 37mm diameter filters (selection of three per round). Silica (quartz) on 37mm diameter membrane filters. Asbestos fibres on 25mm diameter membrane filters. Organic solvents (selection of three per round from benzene, chloroform, carbon tetrachloride, 1,2-dichloroethane, o-xylene, p-dioxane, trichloroethylene and toluene) on charcoal or silica gel packed tubes. 10 Operation of scheme Samples sent out quarterly in January, April, July and October each year. Participants receive four samples per round of each type requested plus appropriate blank filters and tubes. Assessment of performance For each analyte and sample a `reference' value is calculated as the mean of the results obtained by a set of reference laboratories after elimination of outliers. The standard deviations of the results obtained by the reference laboratories are also calculated. Upper and lower performance limits are then set equal to the reference value 3 standard deviations and results lying outside these limits are judged to be outliers. Laboratories are then assigned proficiency ratings over four rounds based on the number of `acceptable' results (ie within performance limits) obtained. Ratings are given for overall performance on each category type, ie metals, silica, asbestos and solvents. only high fibre density samples but it is planned to introduce low density samples in due course. Assessment of performance A `reference' value is obtained for each sample using automated image analysis systems. The ratio of each reported result to the reference value is calculated giving eight ratios per laboratory per round. On the basis of results over four rounds, each laboratory is categorised according to the following criteria. Category 1 > 75% results with ratios <1.7 >0.7 Category 2 > 75% results with ratios <2.2 >0.55 Category 3 < 75% results with ratios <2.2 >0.55 Categories 1 and 2 are classified as `satisfactory' and category 3 `unsatisfactory'. `Unsatisfactory' laboratories are offered advice and re-training by the National Training Programme which is run in conjunction with the RICE scheme. Registration and charges Registration and charges Potential participants should contact: Potential participants should contact: Mr J Groff PAT Program National Institute for Occupational Safety and Health 4676 Columbia Parkway Cincinnati OHIO 45226 USA Tel: 513 684 4357 Fax: 513 841 4500 Fees are payable for participation in PAT to cover both administration and provision of samples. The charges vary between different sample types and are subject to annual review. 3 Title Regular Interlaboratory Counting Exchange (RICE). Promotor Health and Safety Executive and Institute of Occupational Medicine, UK. Analyte Asbestos fibres on membrane filter. Number count using light microscopy. Institute of Occupational Medicine 8 Roxburgh Place Edinburgh EH8 9SU UK Tel: 031-667-5131 Participation in RICE is restricted to laboratories in the UK and Eire. However, a similar scheme is operated with international participation, and interested parties should contact the above address. There is an annual fee for participation in the RICE scheme. Other schemes for specific analytes which are on a smaller scale and less formal than the above are the AQUA scheme for isocyanates and the WHO/EURO scheme for man-made mineral fibres. The AQUA scheme is presently tied to a single analytical method (MDHS 25, Health and Safety Executive), and interested parties should contact: Mr D Bagon Home Counties Region Field Consultant Group Health and Safety Executive 14 Cardiff Road Luton LU1 1PP Telephone: 0582-34121 Fax: 0582-459775 Information on the WHO/EURO scheme can be obtained from the Institute of Occupational Medicine. Operation of scheme Samples sent out quarterly by the Institute of Occupational Medicine. Participants are divided into groups and eight samples are circulated around each group per round. At present the formal scheme covers REFERENCES 1 Whitehead TP and Woodford FP. External quality assessment of clinical laboratories in the United Kingdom. J. Clin. Pathol., 34 (1981), 947-957. 11 2 Draft CEN Standard for general requirements for the performance of procedures for workplace measurements. 3 Garfield FM. Quality Assurance Principles for Analytical Laboratories. Arlington: Association of Official Analytical Chemists, 1984. ISBN 093558426 9. 4 Georges RJ. Measurement of some sources of variability in colorimetric analysis. Ann. Clin. Biochem., 10 (1973), 75-78. 5 British Standards Institute. BS 5700:1984. Guide to process control using quality control chart methods and cusum techniques. 6 Shewhart WA. Economic Control of Quality of Manufactured Product. London: MacMillan, 1931. 7 Woodward RH and Goldsmith PL. Cumulative Sum Techniques. ICI Monograph No.3. Edinburgh: Oliver & Boyd, 1964. 8 British Standards Institute. BS 5703: parts 1 -4. Guide to Data Analysis and Quality Control Using Cusum Techniques. 9 Westgard JO, Barry PL, Hunt MR and Groth T. A multi rule Shewhart chart for quality control in clinical chemistry. Clin. Chem., 27 (1981), 493-501. 10 Buttner J, Borth R, Boutwell JH et al. IFCC approved recommendation (1983) on quality control in clinical chemistry. Part 5. External quality control. J. din. Chem. Clin. Biochem., 21 (1983), 885-892. 11 Maclagan NC, Kind PRN, Daly JF et al. Factors affecting analytical performance in clinical chemistry laboratories. Report of a working party sponsored by The Nuffield Provincial Hospital Trust. London: 1980. 12 Bullock DG, Smith NJ and Whitehead TP. External quality assessment of assays of lead in blood. Clin. Chem., 32 (1986), 1884-1889. 13 World Health Organisation. External quality assessment of health laboratories. Copenhagen: World Health Organisation, 1981. ISBN 92 890 12021. iRv v. o & Printed and published by the Health and Safety Executive ISBN 0-7176-0366-0 9 7 80717 11603664 3.00 CIO 3/91