Задумчивая крыса

Performance scores for an ML-based predictor f are inflated due to data leakage if f uses illegitimate information to achieve the reported scores, that is, information in the training data that generalizes to the test data but not to the inference-time data.

Biological data often show class imbalance. For example, there may be more healthy samples than disease samples, more non-binding pairs of molecules than binding ones, or more major than minor subvariants of a phenotype. ML requires balanced training datasets or weighted losses. Otherwise, the model may perform well by predicting only the majority class. Unbalanced training datasets may require undersampling the majority or oversampling the underrepresented class(es). An uninformed sampling strategy can create new shortcuts in predicting the label distributions, for example, if the sampling is done uniformly at random, but the major class(es) follow a different distribution. The performance may seem impressive when the model is evaluated on a test set containing the same biases. However, when a minority class sample is seen at inference time, the performance may be poor as the shortcut is not applicable.