Chapter 5: Conclusion
5.1 Coding Errors
To start off with, it should be noted that the database seems to have some problems with coding errors. Tables B.3, B.4, B.6 and B.7 all had unlabeled entries in them in the original database. These entries now appear with the label "Coding Error" in the referenced tables. These errors surfaced because the erroneous codes happened to be ones that weren't used. There are probably other invisible errors where the erroneous codes matched existing codes. This errors-in-variables problem makes relationships more difficult to uncover. So, there were probably some significant relationships in some areas which the statistical tests failed to show.
A more optimistic assessment of these blank entries is that they are correctly coded, but the labels were just omitted inadvertently. This still creates an errors-in-variables problem, although a smaller one, since these entries would be lumped in to categories like "Other" or "Miscellaneous" when most likely they should go in to another category.
5.2 Interpreting Statistical Results
The presence of a statistical relationship doesn't always signify the presence of an underlying relationship which can be used in decision making. To cite a modified version of a caveat that the U.S. Center for Disease Control in Atlanta puts at the end of its reports:
The presence of an association does not of itself indicate a causal relation between a given characteristic and an indicator of expenditures. A cross-sectional survey of this nature cannot provide the type of information needed to establish causal relationships. Information on the magnitude and location of major problems, plus certain associations of moderate or greater degree can, however, provide guidance in selecting areas for further inquiry into causality for planning programs and for evaluating their effectiveness.The original unmodified quote (the same except for references to malnutrition) can be found in Arab Republic of Egypt Nutrition Status Survey, 1978, Atlanta, Georgia: Center for Disease Control, Public Health Service, U.S. Department of Health, Education, and Welfare, December 6, 1978, p. 87. What this quotation means as far as the AFC Membership Survey is that when one observes a positive correlation between, say, household expenditures at the co-op and dissatisfaction with the co-op's quality as we did in Chapter 2, that does not necessarily mean that doing something to decrease satisfaction with the quality of the co-op's food will necessarily improve the co-op's revenues.
The correlation could be due to many things. In this case, it may be that lower-income people are concerned with different sections of the store's stock than higher-income people are. The section they are concerned with, say bulk, may be a very high-quality section of the co-op, while the section higher-income people are concerned with, say meat, may be a very low-quality section of the co-op. Under this interpretation, what the results are saying is that higher-income people are less satisfied with the quality of food they buy than are lower-income people. It doesn't say dissatisfaction is a source of people spending more at the co-op. Decreasing the quality of the co-op's food would probably not, under this interpretation, cause lower-income people to spend more at the co-op. It would more likely lead to a major or minor exodus out of the co-op.
Of course, the result discussed here is rather anomalous, but it points up the importance of a critical analysis of more "reasonable" results which may also be another outcome of the same income effects. The moral is that the correlations must be taken with a grain of salt, as the excerpt above states. Any conclusions made from the correlations presented in this report should be corroborated with other sorts of evidence before being used to guide co-op decision making.
The greatest advantage of this sort of survey methodology is its comprehensivity; that is, it is designed so that all points of view get represented. This can be best seen by comparing it to another sort of survey method: leave a questionnaire out and those who wish to fill it out do so; those who don't, don't. In a survey designed, say, to find out what people's interests as far as social events are, this may not be a bad approach, as those who don't fill out the questionnaire probably just aren't interested in participating in that aspect of the co-op anyway. However, if one wants to find out such things as why people are shopping outside the co-op or leaving the co-op, this latter sort of survey technique is likely to produce very misleading results. The non-participants are exactly the ones who need to be reached for any useful information to be obtained. It is precisely the comprehensivity of the survey analyzed in this report that has allowed the depth of insight into the co-op that probably very few co-ops have.
The disadvantages of this sort of survey methodology are the great amount of time consumed in carrying it out and analyzing the results and the need for extensive computational facilities. Of course, people can get work-hours credit for the time they spend, but the cost to the co-op is still very large. The example provided by this survey and its accompanying reports will hopefully mitigate these costs in future similar surveys in AFC or other co-ops as it provides an example of how to set such a survey up and analyze the results.