QUALIMAMA : an assessment tool for mammography services quality

The objective of this research was to develop and validate a system of indicators to audit the overall quality of mammography services that includes the factors related to the image, dose and environment settings, applying it for the characterization of these services in two cities of southern Bahia-Brazil. It was proposed a new quality indicator system for mammography called QUALIMAMA, based in the main current regulatory documents: the guides of the International Atomic Energy AgencyIAEA (2009 and 2011), the European Guide (2006), the Ordinances MS 453/98, MS 2898/2013 and 306 RDC/ANVISA. The QUALIMAMA was applied to evaluate 5 clinics in the micro region of Ilhéus-Itabuna, pointing precisely insufficiencies in these services, which may be impacting on the incidence rates of breast cancer in the region, and will serve as a basis for the improvement of mammography services in the assessed clinics. We compared the results obtained in the application of the proposed system with the assessments carried out based on the main regulatory documents prevailing in the world. The system showed a high level of integration of indicators, allowing a more complete picture of the quality in its various dimensions.


INTRODUCTION
Mammography is the most widely used radiodiagnosis examination for early detection of breast cancer, which is the most common cancer in women worldwide [1].The most part of mammograms is associated with programs of breast cancer screening, performed in distinct ways around the world [2][3][4].
The mammography exam allows to detect microcalcifications, which are not palpable on clinical examination and self-examination, find out tissue changes in initial stages.However, there are some problems in its realization, such as, difficulty to perform the organized screening program and the bad quality mammograms that may generate false-positive or false-negative results.
The concern about mammography available services is increasing, and several studies [5][6][7][8][9] have been dedicated to the optimization of services in order to obtain the best image quality with the lowest possible radiation dose.This has become more relevant, in the period of transition between technologies (screen-film and digital mammography).
As is pointed out by many authors [10][11][12] one of the most important challenges of medical radiodiagnosis is to strengthen the Quality Assurance Programme in mammography.The quality of mammography service, in principle, must meet the same standards anywhere in the world, but it is possible to notice the lack of a unique system of indicators capable to evaluate, integrally, the different aspects related to the quality.Nowadays coexist different tools for this purpose [13][14][15].
A broad review of the guides of quality assurance programs in mammography was recently reported [15], aiming to support a future harmonization of mammography practices internationally.The authors discuss 14 documents that are currently being applied in mammography services.They highlight differences in required tests for the performance of image acquisition systems, processing and visualization.The authors point out that the reference values and acceptability criteria are diversified among the guides, which can make possible the situation of a mammography service to be classified as conforming in accordance with one specific guide and at the same time being classified as nonconforming using another one.The authors pointed out that among the set of guides reviewed the IAEA Quality Assurance Programme for Digital Mammography [16], is the most updated and focused on digital technology.
In Brazil, radiology services follow the regulatory document MS453/98 [17], which establishes the guidelines and radiological protection and radiodiagnostic in medical and dentistry facilities.One of the considerations for the elaboration of this document was the concern about the risks of ionizing radiation and the need to establish a national radiological protection policy in radiodiagnostic.Specifically for the mammography service, was published by the regulatory document MS531/2012 in March 26, 2012, establishing the National Program of Quality in Mammography (PNQM) and establishes some image requirements and radiographic quality [18], reviewed by the MS2.898/2013 [19].
The present study was motivated by two main issues: the absence of an integrated system for assessing the quality of mammography services that consider factors related to the image, dose and environment; and the absence of information on the quality of these services in the state of Bahia.The objective of this work is to develop and validate a system of indicators to evaluate the mammography services integrating the previous mentioned factors.

Sample selection and data collection.
The universe of the research sample contemplated all clinics that do mammography in the cities of Ilhéus and Itabuna, present or not in the National Register of Health Establishments which agreed to participate.Were found eleven registered units, of which were identified only eight units in effective operation.A sample of five clinics representing 62.5% of the sample universe was chosen.
The data collection strategy was planned in three stages: first an exploratory research was conducted, then it was developed the pilot version of the observation guide and applied in test regime in a single clinic.In the third step, after corrections and enhancements, the guide it was applied throughout the selected sample.
The exploratory and descriptive field research was carried out in order to identify the main difficulties and deficiencies in the functioning of mammography units.A non-participative observation was performed with a focus at: image quality, radiation dose and environmental risk and its implications on quality control in mammography.
The first version of the observation guide was designed and implemented in a clinic.After the pilot application, the guide was reviewed for the definitive implementation in all clinics.The results were used in the system of indicators developed, called QUALIMAMA.The clinics were identified by letters, keeping protected their business names and private data.The present research had quantitative and qualitative character allowing to qualify the mammography services in categories by using numerical data.

Method of the System Development
The system of indicators was developed to assess the quality of mammography services, taking as criteria the image quality, radiation dose reduction and environmental risk.National and international regulatory documents were identified, which were analyzed and integrated to develop an evaluation system as complete as possible.The integration of the indicators was performed considering the addition of all items included in the regulatory documents and in case of overlapping the evaluation questions in several guides, it was chosen the one with more restrictive criteria.The following regulatory documents were used: Radiodiagnóstico Médico e Odontológico.Agencia Nacional de vigilância sanitária (Brasil).

Structure of the System of Indicators
The complete system called QUALIMAMA composed of 231 indicators was divided into four dimensions: Technological basis, Operating protocol, Image quality and Environmental risk, as shown in Table 1.
The technological basis enables to identify the conditions of the used technology.The operation's protocol involves records of the services offered since the acquisition, processing and visualization of images, including both the equipment, the environment and the professionals responsible for handling them.The image quality assesses the techniques used and the results of quality control tests.The environmental risk includes techniques and procedures performed during the use of mammography equipment in order to reduce environmental impact.  When the technology is conventional, the weight of the ionizing radiation and chemical aspects will be 7,5%, respectively.

The base of QUALIMAMA indicators system
All dimension of the system has indicators related to the technologies: Digital Radiology (DR), Computed Radiology (CR) and conventional Film (F) separately.It was not assigned different scores to the type of technology used, because the criteria adopted in building the system was to value the integrated quality of service in the three technologies.
Each aspect was divided into indicators that were rated on a 3 points ordinal scale.The score -1 (least one) was applied when the item is in non-compliance with the minimum requirements of the standards; 0 (zero) when complies and +1 (plus one) when the item complies and is better than what is required.As an example are shown in Table 2 some indicators, detailing the evaluation criteria used in the ternary scale.A complete table of indicators and their evaluation criteria is available in Costa, 2015 [24], and soon also in QUALIMAMA software that will be available on www.uesc.brsite.
Table 1 shows the number of indicators in each aspect and dimension according to the respective used technology.It can be observed that the system has a total of 179 indicators for screen-film technology, 187 for computed radiography and 178 for digital mammography, some of them are common among them.For each dimension (D) was assigned a weight (  ), in percentage, in accordance with the degree of importance of each aspect in the final outcome of the service.The dimension "Operation's Protocol" received a higher weight due to covering the largest number of indicators and includes aspects that interfere in the other dimensions.The weight of each dimension (  ) comprises the sum of the weights from each feature (   ) in the entire system, as can be observed in Table 1.
The value for one aspect (A) considered in the system is obtained by adding the value attributed to each indicator belonging to this aspect.Using equation (1) as follows: where: is the value of the score (-1; 0 or 1) referring to the evaluation of service for a specific indicator (i) , in aspect (j).
is the total number of indicators (i) defined for a aspect (A) To calculate the value of the dimension (D) the equation ( 2) is used, where:   is the value of the aspect (j).   is the weight of the aspect pondered within the total number of aspects (Aj) that compose the system.  is the total number of aspects (j) in the dimension (D).
The Integrated Quality Index (K) of the evaluated mammography service is obtained by adding the value calculated for each dimension considered in the proposed indicators system, according to equation (3), where: is the score value for the dimension m regarded.  is the total number of dimensions considered.
K is the final classification obtained by the system and the value of which may vary in scores from 0 to 100.The Integrated Quality Index K informs, in a single numeric indicator, the overall quality of the mammography service considering the weights of the different aspects in correspondence to its importance.According to the final pondered summation, each service will be classified concerning the quality shown in Table 3.This classification proposed in Table 3 considers the reality of mammography services.It is desirable to obtain 100 points.However, the proposed scale presupposes the continuous improvement of mammography services through the progressive acquisition of superior classification.

Analysis by Dimension.
In order to verify which areas need more attention in its improvement, it is performed the analysis by dimension.As part of QUALIMAMA evaluation system, graphical analysis tool on the radar format was included.To work the indicators in this type of representation, the value of aspects, without weights, are shifted to the scale of positive values, using the equation 4: where: is the value of the aspect (  ) obtained by equation (1) shifted to the positive values scale.
We define the new value of the dimension (D) as the sum of all the values of the aspects    in this dimension, in a scale with a maximum value of 10, according to equation (5). where: á is the maximum value of the sum of points (   )to a certain dimension.
is the total number of aspects (j) in the dimension (D).

Indicators in binary system (A*, D* and K*)
The QUALIMAMA-b is one of the formats in which are expressed the results of the proposed system that uses a binary system for clinical evaluation .It is composed of the same indicators and dimensions of QUALIMAMA, and it is distinguished by the form of assessment.This binary system can be used to assess compliance and non-compliance according to the standards used, applying the ordinal scale (-1 or 0) and without the use of weights in the dimensions.The QUALIMAMA-b format complements the standard format because it allows a direct view of the percentages of non-compliance.The binary system was also applied to regulatory documents IAEA/2011, EC/2006, MS 453/1998 and MS 2898/2013, for comparison of Dimension (D*) and Integrated Quality Index (K*).
The calculation of the aspect is performed as in standard QUALIMAMA, using equation (1).It uses equation 6, to shift the value of the aspect to the positive scale. where: Aj is defined in equation (1).
It is calculated the value of dimensions, using the equation (7). where: nD is the total number of aspects of the dimension D, N j max is the maximum score in the dimension D, i.e.,    = 100 •   It is performed the calculation of Integrated Quality Index K* without considering weights, and with binary scale according to equation (8).
where:   is the total number of dimensions (m) in the applied document.
After calculating the Integrated Quality Index (K*), the clinic is classified according to Table 3.The QUALIMAMA classification system which uses the ternary ordinal scale differs from QUALIMAMA-b that uses binary scale system due to the different meaning of the scale values for each system.For binary system the value K* = 100 reflects a compliance for the full set of indicators, any obtained value below this one, is consequence of requisites that were not fulfilled.In the QUALIMAMA system, this situation of full conformity can be achieved with the value of K =50 and values up to 100 could be consequence of performance above the minimum required.

Method of comparative analysis of the system
A performance comparison of the assessed services in accordance with national and international documents used with the developed system QUALIMAMA was performed.The approach consisted in expressing the items of the regulatory documents on indicators in the binary scale.It was carried out a comparative analysis of results using the methods of multivariate statistics: analysis of Hierarchical Classification (Cluster) and analysis of bivariate correlations with Pearson's coefficient, using the SPSS v18.0 software.

Results of QUALIMAMA's application in the clinics
The result of the K value obtained by the evaluated services is shown in Figure 1.It is visible also a moderate difference between the services located in both cities.In Figure 2 is presented the results in radar graphic form which allows easily view the most deficient areas of services.It is observed that in all services the values of dimension "Image Quality" are below the reference value 5, which indicates that there are non-conformities, sharp situation in the services A and B. In the studied sample, the comparative ranking was dominated by the dimension "Image Quality", because in other dimensions the results remain close to the reference value 5.The quality index (K) of the A and B services departs from the recommendable value even more due to the deficiencies that appear in the Operating Protocol dimension.It indicates the need to improve the management, organization and self-assessment in the units.

Comparative analysis of QUALIMAMA system with the regulatory documents
In order to evaluate the effectiveness of QUALIMAMA system as evaluative tool, this was compared with the national and international regulatory documents.For this, was considered the number of items by dimension and also the results of the evaluation of the services studied.Some elements that differentiate these documents are reflected in the number and type of indicators.
In the figure 1 is shown the Integrated Quality Index (K or K*) obtained by each service in different regulatory documents.The scores observed in services A and B are generally lower than for the other services.This fact, is particularly evident using the EC 2006 guideline in which the aspects, "Quality Control" and "Image Quality" have a predominant weight in this guideline.In figure 1 can be observed that the evaluation of the same clinic, by different regulatory documents, shows a wide variation between them.The QUALIMAMA and the QUALIMAMA b, generally vary less among themselves than the others documents, reflecting the trend of the others evaluations.
The analysis of the services based on the classification of Integrated Quality Index (K* or K) is presented in Table 4.It is observed that the sensibility of the ordinal classification according to IAEA/2009 did not allows to differentiate the general quality of the evaluated clinics.The others regulatory documents allow to obtain results that can distinguish the classification until three values for these same services.According to the result obtained with the QUALIMAMA b and the brazilian regulatory documents, the service E shows a better performance among the analyzed services

Comparative statistical analysis of the documents based on the score obtained by services
The value of K* obtained by QUALIMAMA-b shows an intermediate euclidean distance with the other documents, the most distant evaluation was obtained using the regulatory document MS 2898/13.The intermediate position of the evaluation by QUALIMAMA-b system indicates that it is closer to the evaluation made by the other regulatory documents.It was observed a gap of evaluations obtained by QUALIMAMA and QUALIMAMA b.This fact reflects the effects of the weight and scale in the integrated evaluation.
The bivariate correlations analysis of K or K * values for all services and by all the documents allowed to evaluate the correspondence between the methodologies and their behavior between each other.The results show that all methodologies have positive correlation coefficients and values higher than 0.7, indicating a reasonable degree of correlation between these tools.Thus it is expected that the behavior of the comparative results between the services must be reproduced when evaluated with the different standards.It is interesting to highlight that the QUALIMAMA methodology shows the highest correlation coefficients, that are above 0.9 for all the regulatory documents.

DISCUSSION
It is observed a low quality of mammography services in the region, requiring an urgent implementation of a policy targeted to raise the culture of the quality assurance.It is noticed a difference between service's quality in the first city (services A and B) and the second one (services C, D and E), which obtained better results possibly due a more frequent quality monitoring.However none of them achieved the recommended standard.
It is noted that the Brazilian regulatory document MS 453/98 refers to radiodiagnosis services, not particularly mammography, and highlights the radiological protection and the unit's operating records.The MS 2898/2013 regulates the quality assurance program in mammography, emphasizing the quality of mammographic image.Precisely because it is a more specific standard, the results obtained from it, differs from the others.
With regard to international guides, used by many countries as a reference, the EC / 2006 describes the procedures that involve the screening and diagnosis of breast cancer, in addition to mammography quality control tests.The regulatory document IAEA / 2011 is the most current and is dedicated to digital technologies.The results of evaluations carried out using these documents and QUALIMAMA b show differences particularly for clinic E. This difference is related to the requirements of the documents control and the radiation protection practice, which are not present in the international standards mentioned above.
On the basis of the analysis of correlation coefficient it is possible to state that the evaluation by the QUALIMAMA system incorporates coherently the key elements of the others regulatory documents.

CONCLUSION
In this study it was developed and applied a new quality assessment tool in mammography, QUALIMAMA.The QUALIMAMA indicators system gathers in one single tool all the quality assessment components that appear in several documents that regulate the sector in the country and in the world.
The new QUALIMAMA indicator system offers a quantitative approach at three distinct levels: the first is performed based on the value of the Integral Quality Index K; in the second, the Dimensions are analyzed by the radar chart and the third analyzes the performance of each aspect.These attributes allow to characterize the QUALIMAMA as a practical system, easy to use, holistic, that facilitates the internal audit of the services.This can strengthen and accelerate the quality's consolidation process in DR, CR or screen-film mammography services.
The QUALIMAMA results in the cities of Ilhéus and Itabuna, showed that all the assessed clinics was classified as not recommended for this system.So, will be required a great effort by the managers to meet the requirements of the regulatory documents.The main deficiencies found are related to indicators concerning the image quality, which can have a direct impact on the accuracy of diagnosis and consequently in preventing breast cancer.
The QUALIMAMA indicators system has the necessary structure and appropriate characteristics to be easily modified, including new standards and their updates.In order to facilitate its application, the QUALIMAMA v1.0 software is under development which will make assessments of mammography services with the regulatory documents used in this study.
The proposed methodology can contribute in the implementation of quality assurance programs of mammography services and encourage to the continuous improvement of the total quality program, including internal audits.The improvements in the quality of mammography services should have a positive impact on the indicators of public health related to the incidence of breast cancer.

ACKNOWLEDGMENT
The authors thank the radiology clinics in the cities of Ilhéus and Itabuna for the valuable contribution made in providing all the necessary information for the present research.The authors thank the research funding agency Financiadora de Estudos e Projetos (FINEP) for funding the above work.

Table 1 .
Number of indicators in each dimension of QUALIMAMA system per technology 1

Table 2
Examples of indicators system

Table 3 .
Quality classification of mammography service according to the QUALIMAMA

Table 4 .
Ordinal classification* obtained by the services, from the Integr Quality Index (K*), in each regulatory document 1. International Agency for Research on Cancer.Latest world cancer statistics Global cancer burden rises to 14.1 million new cases in 2012: marked increase in breast cancers must be addressed.2013.Disponível em: <http://www.iarc.fr/en/media-centre/pr/2013/pdfs/pr223_E.pdf>.Acesso em: 22 dez.2013.