ACCURACY ASSESSMENT ANALYSIS OF MAPBIOMAS’ LAND COVER (AND LAND USE) MAPPING


Accuracy analysis is the main way of assessing the quality of mapping performed by Mapbiomas. In addition to telling the overall classification accuracy, the analysis also reveals the accuracy and error rate for each classified class. MapBiomas evaluated global and per-class classification accuracy for each year between 2000 and 2016.

Accuracy estimates were based on the evaluation of a sample of pixels, which we call the reference database. The number of pixels in the reference database was pre-determined by statistical sampling techniques. Each year, each pixel in the reference database was carefully evaluated by three technicians trained in visual interpretation of Lansat images. The evaluation of a pixel in a given year was considered valid only when two or three technicians agreed on the class observed in the pixel. Accuracy assessment is then done using metrics that compare the class mapped with the class evaluated by the technicians in the reference database. We report yearly accuracy estimates with c.a. of 5% error for each class in the mapping.

In each year, the accuracy analysis is done by cross-tabulating the sample frequencies of the mapped and real classes, in the format of Table 1. The frequencies ni,j represent the number of pixels in the sample classified as class i, and evaluated as class j. The line marginal totals, Formula1 0a79614e9b08b886c3e50adc030dfad303a6a07b461b14d2b1453456ff4945dd, represent the number of samples mapped as class i, while the column marginal totals,Formula2 6336a4b841e299449e55016c91a6408c82f8a8fb2b3c814a652df1743fbc0710 , represent the number of samples that were evaluated by the technicians as class j. Table 1 is commonly called the error matrix or confusion matrix.

Table 1: Generic sample error matrix

Tabela1 mapbiomas en b2deaed6c143c48c9dc8977eb72b331d324be3e289c74b3db53a202fdd1c749a

From the results in Table 1, the sample proportions in each cell of the table are estimated by Formula3 7d5fddb94e58ea97c1546e4f60f907ba16f1825966a149fa23dc0142522f1c68 Formula4 8f2e5226c87b8d2c1df56fd741ecc9dc6a1596aada528bc661e8288e81454660. The matrix of values Formula5 2947dbf0589ad8ae2e84012661079f130d322b94ca06cf615a9a7d7f610d8ffe is used to estimate:

  1. User’s Accuracies: These are the estimates of the fractions of pixels, for each classified class, correctly classified. The user’s accuracy is associated with the error of commission, which is the error of assigning a pixel to class i, when it belongs to some other class. The user’s accuracy for class i is estimated by Formula6 32ba01730e7c8021c4338555c742f9d6ea3fd2ae0eee75110ac8c066c12fa534 and the commission error by Formula7 89781df0133f266b373046ae5aa8c0c3822c027e69b4d78860281e3ab9f01327. These metrics are associated with the reliability of each classified class.
  2. Producer’s Accuracies: Are the sample fraction of pixels of each land cover/use class correctly assigned to their classes by the classifiers. The producer's accuracy is associated with the omission error, which occurs when we fail to map a class j pixel correctly. The producer 's accuracy for class j is estimated by Formula8 dfd0c71796b0bd34c90b3a97a03731c6200e8d3b0c40e72cc284487f07d05129 and the omission error by Formula9 d481a689c409c96b1d80c1061073dab92f986201845c6cf8b29c611e99b7757b. These metrics are associated with the sensitivity of the classifier, that is, the ability to correctly distinguish one class from another.
  3. Global Accuracy: It is the estimate of the overall hit rate. The estimate is given by Formula10 n 54dffbcce756d3fa579000e6d6aea99c2faf2cf01031361735f9ba28b097c849, the sum of the main diagonal of the proportions matrix. The complement of the total accuracy, or the total error Formula11 a1620f148d11084951dba9653625e938e0cbad8d5d934713ac62c96b132cddf3 is still decomposed into area (or quantity) disagreement and allocation disagreement1. Area disagreement measures the fraction of the error attributed to the amount of area allocated incorrectly to the classes by the mapping, while the mismatch allocation to the ratio of class-displacement errors.

The matrix also provides estimates of the different types of errors. For example, we show estimates of true class area composition in each mapping class. Thus, in addition to the hit rate of the class mapped as forest, for example, we also estimate the fraction of these areas that could be pasture or other classes of cover and land use, for each year. We understand that this level of transparency informs users and maximizes the potential of use across multiple types of users. For this, we build an application to facilitate the visualization of the accuracy and the errors of the mapping

1- Pontius Jr, R. G., & Millones, M. (2011). Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment. International Journal of Remote Sensing, 32(15), 4407-4429.


About the graphics


Overall Statistics

They show the mean annual total accuracy and the error, decomposed in area and allocation disagreements.

Acuracia img1 en 4f4c495009fb0c1fd51fab9cd8d6bedaa1184d3f3110a3a3ac781dd801b282f7

Graph 1. Annual Total Accuracy Chart:

This graph shows the total accuracy and the total error per year. The total error is decomposed into area and allocation disagreements. Accuracy is plotted at the top and errors at the bottom of the chart.

Acuracia img2 en 0f77d92f6df0271351e4b3ac1afcf4da6e718ed0fa6faeff20fa5dc592a34f35

Graph 2: Matrix of errors

This graph shows the user’s and producer’s accuracy, and the confusions between classes for each year. The first shows the confusions for each mapped class. The second shows the confusions for each real class.

Acuracia img3 en fc0f2b205bb0f97780bfa3d3fe9efdf33c001b5ba7eed16e249a248f1e1f082e
Acuracia img4 en b629521598b76a6e71dbcdcf143d708c2dbe91181296840f5d2d96db3b5fe807

Graph 3: Class History

This graph allows you to inspect the confusions of a particular class over time. The user’s r and producer’s accuracies for each class is displayed along with the confusions in each year.

Acuracia img5 en 4f525e303ae084be9d98633f14d36395643a5e631216910b8617fe177795c7d0
Acuracia img6 en 057dcc3bf5d36aad0136e4a3c2df22d6203e42ff5e0650fc8f41da13d6cf2669