We present counterfactual situation testing (CST), a causal data mining framework for detecting individual discrimination in a dataset of classifier decisions. CST answers the question "what would have been the model outcome had the individual, or complainant, been of a different protected status?"in an actionable and meaningful way. It extends the legally-grounded situation testing of Thanh et al. [62] by operationalizing the notion of fairness given the difference of Kohler-Hausmann [38] using counterfactual reasoning. In standard situation testing we find for each complainant similar protected and non-protected instances in the dataset; construct respectively a control and test group; and compare the groups such that a difference in decision outcomes implies a case of potential individual discrimination. In CST we avoid this idealized comparison by establishing the test group on the complainant's counterfactual generated via the steps of abduction, action, and prediction. The counterfactual reflects how the protected attribute, when changed, affects the other seemingly neutral attributes of the complainant. Under CST we, thus, test for discrimination by comparing similar individuals within each group but dissimilar individuals across both groups for each complainant. Evaluating it on two classification scenarios, CST uncovers a greater number of cases than ST, even when the classifier is counterfactually fair.
Counterfactual Situation Testing: Uncovering Discrimination under Fairness given the Difference
Alvarez, Jose Manuel
;
2023
Abstract
We present counterfactual situation testing (CST), a causal data mining framework for detecting individual discrimination in a dataset of classifier decisions. CST answers the question "what would have been the model outcome had the individual, or complainant, been of a different protected status?"in an actionable and meaningful way. It extends the legally-grounded situation testing of Thanh et al. [62] by operationalizing the notion of fairness given the difference of Kohler-Hausmann [38] using counterfactual reasoning. In standard situation testing we find for each complainant similar protected and non-protected instances in the dataset; construct respectively a control and test group; and compare the groups such that a difference in decision outcomes implies a case of potential individual discrimination. In CST we avoid this idealized comparison by establishing the test group on the complainant's counterfactual generated via the steps of abduction, action, and prediction. The counterfactual reflects how the protected attribute, when changed, affects the other seemingly neutral attributes of the complainant. Under CST we, thus, test for discrimination by comparing similar individuals within each group but dissimilar individuals across both groups for each complainant. Evaluating it on two classification scenarios, CST uncovers a greater number of cases than ST, even when the classifier is counterfactually fair.File | Dimensione | Formato | |
---|---|---|---|
Alvarez and Ruggieri 2023.pdf
accesso aperto
Descrizione: Counterfactual Situation Testing
Tipologia:
Published version
Licenza:
Creative Commons
Dimensione
698.37 kB
Formato
Adobe PDF
|
698.37 kB | Adobe PDF | |
Appendix_for_CounterfactualSituationTesting-17-18.pdf
accesso aperto
Tipologia:
Altro materiale allegato
Licenza:
Creative Commons
Dimensione
453.13 kB
Formato
Adobe PDF
|
453.13 kB | Adobe PDF |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.