Validation Using Imperfect Reference Instruments (Comparative or Relative Validation)

Collecting unbiased reference intake data is logistically and economically difficult. Therefore, instead of using unbiased reference measures, studies seeking to assess the [glossary term:] validity of self-report instruments commonly use imperfect references (e.g., 24HR or [glossary term:] food record) that have some systematic [glossary term:] bias, but less than that in the instrument being validated. However, data collected using imperfect [glossary term:] reference instruments contain error, including [glossary term:] intake-related bias, that may be correlated with error in the main instrument. Using them in statistical models under the assumption that they meet the requirements of an unbiased reference may result in inflated estimates of correlation coefficients between the main instrument and true usual intakes and [glossary term:] attenuation factors that are more optimistic than those obtained using [glossary term:] recovery biomarkers. Results of comparative or [glossary term:] relative validation studies must therefore be interpreted with caution (see Interpretation of Validation Studies).

Validation with an imperfect reference instrument is not typically done for 24HRs and food records because there are no other self-report instruments that are thought to be superior (i.e., less biased) and one is not a reasonable reference instrument for the other. The following are a few important design and analysis considerations for conducting validation studies of FFQs or screeners as the main instrument using an imperfect reference instrument:

• As noted above, data from multiple administrations (at least two) of each instrument is needed to assess [glossary term:] within-person random error (also known as within-person variability). Knowledge of the within-person random error is essential for estimating the correlation coefficient between self-reported and true usual intakes.
• Include two administrations of an FFQ or screener before and after at least two administrations of the reference instrument, though this number and order of administrations is not always feasible. Because FFQs and screeners query past intake (e.g., past year), it is preferable that at least one administration of the reference instrument be collected before one administration of an FFQ or screener to allow for reporting of intakes during the same time period.
• Analyses should examine validity of energy-adjusted measures in addition to absolute intakes, as biomarker-based validation studies have shown that [glossary term:] energy adjustment (Learn More about Energy Adjustment) tends to improve attenuation factors and correlation coefficients for FFQs.

[glossary term:] Concentration biomarkers are also sometimes used as [glossary term:] reference instruments in validation studies, although they too are imperfect reference instruments. In general, analyses using concentration biomarkers consist of simple correlations between biomarker-based and self-reported estimates of intake or partial correlations that can be adjusted for [glossary term:] covariates of interest. Because concentration biomarkers do not measure dietary intake directly, and are subject to complex metabolic regulatory controls, such studies are of limited value.

The method of triads also has been used to assess the validity of dietary assessment instruments. In this method, information about intake of a given dietary constituent (e.g., beta-carotene) is modeled using three different measures (e.g., FFQ, 24HR, and concentration biomarker). In applying this model, two assumptions are made: 1) the three measurements are linearly related to true [glossary term:] usual dietary intake; and 2) given [glossary term:] true intake, errors in the three measurements are independent of each other. Under these assumptions, the correlation between any of the three measurements and true intake can be estimated [10]. When the measurements include two measures of [glossary term:] self-reported intake, such as an FFQ and a 24HR, however, the assumption of independence between their errors given true intake is unrealistic in many situations [11]. In addition, when the measurements include self-reported intake and concentration biomarkers, the potential dependency of both on factors, such as body mass index or smoking, may lead to correlations between their errors. In such situations, estimates of correlations between the instruments and true usual intake based on the method of triads will be biased.