Does the Data Processing Inequality Reflect Practice? On the Utility of Low-Level Tasks

Why it matters (2 min)

• The data processing inequality is an information-theoretic principle stating that the information content of a signal cannot be increased by processing the observations.
• In particular, it suggests that there is no benefit in enhancing the signal or encoding it before addressing a classification problem.
• Open receipts to verify and go deeper.

Go deeper (8 min)

Context

The data processing inequality is an information-theoretic principle stating that the information content of a signal cannot be increased by processing the observations. In particular, it suggests that there is no benefit in enhancing the signal or encoding it before addressing a classification problem. This assertion can be proven to be true for the case of the optimal Bayes classifier. However, in practice, it is common to perform "low-level" tasks before "high-level" downstream tasks despite the overwhelming capabilities of modern deep neural networks. In this paper, we aim to understand when and why low-level processing can be beneficial for classification. We present a comprehensive theoretical study of a binary classification setup, where we consider a classifier that is tightly connected to the optimal Bayes classifier and converges to it as the number of training samples increases. We prove that for any finite number of training samples, there exists a pre-classification processing that improves the classification accuracy. We also explore the effect of class separation, training set size, and class balance on the relative gain from this procedure. We support our theory…

For builders

Builder: scan the abstract + experiments; look for code, datasets, and evals.

Verify

Prefer primary announcements, papers, repos, and changelogs over reposts.

Receipts

1. Does the Data Processing Inequality Reflect Practice? On the Utility of Low-Level Tasks (arXiv stat.ML)