Recall error in self-reported income

Most of what is known about household income comes from asking people. That method has been examined by a small body of validation research, which links what respondents said to what an employer or an administrative agency recorded for the same person and period.

Greg Duncan and Daniel Hill carried out one of the early exercises of this kind in the 1980s, using the Panel Study of Income Dynamics Validation Study: survey responses from employees of a single large manufacturing firm were compared with that firm's own payroll records. Reports of annual earnings tracked the records reasonably closely. Reports of hourly wages and of hours worked did considerably worse, and errors in the two components did not cancel out.

Error that leans

John Bound and Alan Krueger extended the approach in 1991 by matching Current Population Survey responses to Social Security earnings records. Their central observation concerned the shape of the error rather than its size: it was negatively correlated with true earnings. Respondents at the lower end tended to report figures above their recorded earnings, and respondents at the upper end tended to report below. Error of that form compresses the measured distribution, which matters for any statistic about spread or inequality, and it violates the assumption of random noise that a great deal of applied work quietly relies on.

Bound, together with Charles Brown and Nancy Mathiowetz, later assembled this literature into a chapter of the Handbook of Econometrics. Two of its recurring themes are relevant to any discussion of earnings from online work. First, accuracy depends on the regularity of the income: a fixed salary from one employer is recalled well, while variable amounts from several sources are recalled poorly. Second, longer reference periods degrade recall, and questions about a full year are answered by reconstruction rather than by memory.

The record is not a perfect ruler

It would be convenient to treat administrative records as ground truth, and the validation literature declines to do so. Arie Kapteyn and Jelmer Ypma, examining linked survey and register data, showed that the act of linking introduces its own error: records are matched to the wrong person, or matched imperfectly, and the resulting mismatch can look exactly like respondent error in the statistics. Estimates of survey inaccuracy that ignore mismatch will overstate how badly respondents perform.

The validation studies also share a structural limitation. They are typically drawn from settings where a linkable record exists — one large employer, a national insurance system, a formal payroll. Work arranged through many short engagements is the case least likely to appear in such a study and, by the same literature's own reasoning, the case where reporting error should be largest.

What this implies for platform figures

Income from online platforms tends to be variable in amount, intermittent in timing, split across more than one source, and reported for a period in the past. Every one of those attributes appears in the validation literature as a condition under which recall degrades. This does not establish that any particular survey figure is wrong, and no study cited here measures online platform income directly. It establishes something narrower and more durable: that self-reported figures in this domain carry a wider band of uncertainty than self-reported figures for salaried employment, and that the direction of the error should not be assumed to be neutral.

Where a study reports earnings without stating whether it asked or observed, over what reference period, and against what record, the number is not yet interpretable — not because it is dishonest, but because a necessary part of it has not been printed.

Return to the register