The recent literature on evaluating manpower training programs demonstrates that alternative non-experimental estimators of the same program produce an array of estimates of program impact. These findings have led to the call for experiments to be used to perform credible programs evaluation. Missing in all of the recent pessimistic analyses of non-experimental methods is any systematic discussion of how to choose among competing estimators. Two unstated premises underlie the negative assessment of non-experimental evaluation procedures. The first premise is that alternative non-experimental estimation procedure should produce approximately the same program estimate. The other premise is that there is no objective way to choose among alternative non-experimental estimators. The first premise is invalid if there are systematic differences between trainee and comparison group members in observed and unobserved characteristics affecting outcome measures. The truth of the second premise hinges on the content of the available non-experimental data and on the assumptions invoked to justify various non-experimental estimators. This paper explores the value of simple specification tests in selecting an appropriate non-experimental estimator. Previous empirical studies documenting the sensitivity of estimates of programs impacts to non-experimental estimation procedures do not test the fitted models against the available data or else disregard the inference from such tests. A re-analysis of the national support work demonstration data previously analyzed by proponents of social experiments reveals that a simple testing procedure eliminates the range of non-experimental estimators that are at variance with the experimental estimates of the program impact.
Bibliography: Heckman, James J. 1989. "Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower Training." NBER Working Paper 2861 (February), National Bureau of Economic Research, Cambridge, MA.
You should be able to view web pages in your web browser (Internet Explorer, Netscape, etc.)
