the qualities corresponds to the order in size of the average p.e. with but three displacements. A certain amount of this must indeed be ascribed to happy chance, for the differences in the p.e.'s are often infinitesimal, and were there actually perfect correspondence the present methods would be far too coarse to detect it surely. So far as the results go, the qualities that we tend to judge an author by are also those that we tend to grade with the greater accuracy. It is perhaps not unnatural that the traits about which we have the most assurance should also be those that we regard as the most important. The close correspondence of the two may itself be in the nature of an argument for their validity.

The method measures directly an author's possession of a quality with reference to other authors. Indirectly an idea may be obtained of the prominence or absence of a quality relative to the other qualities of his own work. Aside from such errors as would be due to differences in the ranges, etc., he is likely to have more of a quality in which his position is higher than of one in which his position is lower. Thus I, who has a median of 2.1 in Imagination, but one of 6.9 in Wholesomeness, is probably more imaginative than he is wholesome. A table may be constructed in which a plus sign is given to those quality grades which are at the same time both above the author's median of medians and the general median of the grades in that quality, this last always falling somewhere in the neighborhood of 5.5. Minus is assigned to those grades which fall at the same time below the author's median of medians and the general median of the quality, and a zero sign goes to those which fall between the two. Other things being equal, a sign then goes to the qualities that are relatively prominent, a sign to those that are absent, and zero to those which are inconspicuous one way or the other. Such a table contains 35 + signs, 27 signs, and 38 zero signs. The figure, however, has little significance save when it refers to a prominent quality in a low author or a lacking quality in a high one. The following are in order the two highest and the two lowest quality grades received by each author; i. e., the two qualities for which his work is presumably the most and the least distinguished.

III. ON THE VALIDITY OF INDIVIDUAL JUDGMENT AS

MEASURED BY DEPARTURE FROM AN AVERAGE.

If we took a series of graduated weights, and asked a number of persons to serially arrange them in order of their apparent heaviness, we should find, if the differences between the weights were sufficiently small, that no one could save by chance arrange them in correct order, but that there would always be more or less displacement. The person whose arrangement showed the least displacement would approximate closest to the true order, and we should therefore consider him to have the most accurate judgment for weight. Now assuming that the distribution of all the errors made followed that of the probability curve, we should find that the errors compensated and that the average order in which the weights were placed would also be very close to the correct order, closer probably than that of the best individual, though the average number of displacements might be considerable. In estimating the accuracy of our subjects' judgments of weight, it would make little or no difference whether we took as the true order the actual order of heaviness as measured on the scales, or took the average order as the standard. Theoretically, each would give us the same result.

But there are many important qualities, and indeed those most adaptable to measurement by relative position, whose differences we cannot determine in this objective way. The question then arises, are we also here justified in taking the truth of the average order as objective, and measuring the value of a judgment according to its deviation from it? For clearly unless our average approximates to some objective validity, the absolute value of a single judgment is not measured by the amount of its deviation from it. To recur to our weights, suppose we heated and cooled the weights to varying degrees before presenting them to all save one of our subjects, and to him presented them at equal temperatures. The subjects would all feel the colder weights as heavier, and the average order would not be the objectively true one, and the order of the subject perceiving the weights under equal conditions might well be the farthest from the average. Our two groups would give us different results because they were judging from different standards.

It is just this condition that must be guarded against in those measurements where an average order is all that we have to guide us. We have, a priori, no objective measure of the varying standards by which the individuals judge. Still less do we know the relaive values of the standards themselves. In the case of the weights

we know the differing nature of the standards, and can allow for them; but if we did not know them the judgment of the single subject would still be the most useful for us. Practise will overcome many illusory standards of judgment to which normal persons are subject, and I should hardly have the right to assert my judgment of direction to be superior to that of Professor Judd because I was nearer the average than he in amount of subjection to the Zöllner illusion.

In the measurement of mental traits by relative position we have thus two factors that tend to cause individual deviation from the average, namely the absolute inaccuracy of the judgment, the direction of whose errors will be variable, and a differing standard from other members of the group, the direction of whose errors will be constant, at least throughout the individual. We must know the exact nature of the deviations due to these two causes before we can estimate the values of the judgments. We must also know the value of the standards, for it is possible that the opinion of a very accurate judge by one set of standards might be of smaller value than that of a less accurate judge by another. We must show cause why a person who judges literary work by its clearness must have ipso facto a poorer judgment than one who judges it by its imagination.

It is possible that in the estimation of scientific merit, where this method found its first application, there would be more unanimity in the standards of judgment, yet there are some divergences from this cause, since there was an observed tendency for graders to give disproportionately high position to men engaged in the same special work with them and to their own immediate colleagues. The method has here been applied only to the first fifty psychologists, but it gave fairly definite results, and these might be still more definite in others of the sciences. Save for observer A the order is rather variable, and it might be questioned whether a man's estimate of the fifth group should be allowed the same weight with his estimate of the first. This is also a matter subject to a good deal of variation, for the second best judge of the first ten psychologists is the worst of the second, the fifth of the third, the eighth of the fourth, and the sixth of the fifth.

However, where the variations in the standards compensate, as they ought to do in scientific merit, the method is immeasurably more valid than where they not only patently fail to do so but give a false standard, as in literary merit. The conditions are exactly the same as with the varying sizes and temperatures of the weights. Our group of weight-graders constantly gives a small or cold object an undue weight; the group of scientific graders constantly assigns high position to their immediate colleagues and co-workers; the group of literary graders constantly allows a presumably undue weight to

Euphony and Finish. The variation in the accordance of the judges is a little over 2:1, as was the case in Cattell's psychologists; the accordance of the judgments also tends to follow the normal distribution, though there seems to be a slight skew in favor of the more accordant judgments.

It should not be impossible to get a quantitative demonstration of these differing standards. When we have a series of objects graded in respect to a general quality, and then in regard to the main elements of that quality, the relative influence of the elements on the general judgment appears in their degree of correspondence to the general quality. Now while the graders showed a certain unanimity in assigning to various elements of literary merit a certain order of influence, it does not follow that the mature judgment of eminent literary critics would give the same order, or that the graders themselves would give it twenty years hence. Still less does it follow that this standard is the best one for us to abide by, or that it is one which the graders themselves would not be among the first to consciously repudiate. If we had the qualities directly graded in order of value to literary merit, we should hardly expect to find Euphony and Finish first, Clearness and Wholesomeness last. Nor do we.

Such a judgment was obtained from a group of 24 graduates in psychology and education, of about the same intellectual level as those who furnished the literary grades. I see no reason a priori-and there is certainly none evident in the results-why the conscious judgment of this group should not have the same ethical value as that of the literary graders, or why the terms should not have been equally well understood. The group contained a certain proportion of women, about one-third, but this factor did not appear to influence the character of the judgments. The formula by which the qualities were graded was "according to their importance to the fulfilment of the highest function of literature." No definitions of any of the qualities were given, nor does it appear that it would have been advantageous to have given them. This order of importance, with positions and p.e.'s, is shown in the accompanying table (cols. T.C.).

This table, compared with that on p. 22, gives an idea of what we think we judge literary merit by as contrasted with what we actually judge it by. The number of displacements between the two orders is 28 slightly more than we should expect by pure chance. Such correspondence as there is between our naive and conscious. standards is thus slightly in the direction of perversity. It is probably something more than an amusing coincidence that that quality which we are so sure we ought to judge an author by most of all is the one which really plays the least part in our estimate of him, and that the two qualities which ought to have the least share in deter

mining an author's position are those which always show the most remarkable correspondence with it.

The distributions of these grades are unimodal for the most part, and only in Wholesomeness do we find distinct species of high and low grades. It has much the largest p.e. and is the only quality receiving a grade in every place. The species were examined for sex correlations, but none were apparent.

Before the method for the determination of individual standards had been applied, the literary graders had been made aware, through one of the cruder methods, of the general relations of the qualities. It was therefore impossible to obtain from them any order not subject to large constant error. Nevertheless, it seemed worth while to obtain a few records from this group.

Records were obtained from 14 individuals, of whom 12 had taken part in the previous test. The results are given in the last quoted table, cols. E.G. The order and positions here assigned also differ from the objectively determined order by slightly more than the chance number of displacements, but while the number of displacements is almost identical with that of the order given by the other group, there are 11 displacements between the two groups themselves, and in a few cases these discrepancies are outside the limits of the p.e. This may well be due to the constant error mentioned above, and I do not consider that there is sufficient warrant for supposing separate species. An interesting aspect of these results is afforded from the view-point of individual comparisons. The number of displacements that occur between the order of the authors in general merit and their order as assigned in the various qualities by a single individual, gives an idea of that individual's actual standards of judgment. The qualities that vary least from the general merit order are his most important standards. In the grading of the qualities themselves we have the conscious standards by which