Method of Evaluation

We developed a system of evaluation that derived a point total to express the overall evaluation of the specific work within its own category. The point total was a sum of subordinate categories including: (1) the total number of words devoted to Korea, (2) an evaluation of the accuracy of the presentation, (3) ratings for the numbers of visuals (including pictures or drawings, maps, and various kinds of charts and graphs). We also evaluated (4) the bibliography according to its helpfulness to students doing further research. In addition, those works including annotated bibliographies were given twice as many points as those using regular bibliographies.

The points were divided as follows:

Words (total word count) 40 points optimum [quantitative measure]

Accuracy 40 points possible [qualitative]

Portrayal 10 points
Care with facts 10 points
Up-to-date 10 points
Romanization 10 points

Visuals 15 points optimum [quantitative]

Pictures and drawings
Maps
Graphs, charts, and tables

Bibliography 5 points optimum [quantitative]

----------------

100 points possible

In the words, visuals, and bibliography categories, points were assigned to a text based on how it compared with the best text in each category; the best text got the optimum points. In the category of words, for example, the one with the most words would be given the maximum points, 40; the others would be given a percentage of points based on the number of words compared to the optimum. For example, Encyclopedia Britannica devoted 65,000words to Korea, for a maximum of 40 points; Collier's Encyclopedia had 39,000 words, 24 points (60% of the maximum). The highest received 100%of the points while the other received a percentage of the points.

One text may have the highest word count, but have the second highest number of visuals. Visuals were worth up to fifteen points. We rated the total number of visuals, whether they were maps, pictures or drawings, charts or graphs. In this category, for example, Compton's Encyclopedia had the most with 77 visuals (including 14 maps, 40 pictures or drawings, and 23 charts or graphs) and thus received the optimum 15 points. Britannica had 54 visuals (9 maps, 37 pictures, and 8 charts) and received 10.5 points (70% of 15).

Similarly, we rated the bibliography sub-category according to a percentage of five points. Grolier Encyclopedia had 74 references and received the full 5 points; Britannica had 55 and received 3.7 points (74% of 5).

Accuracy was rated qualitatively, and somewhat subjectively, according to the following considerations: overall portrayal of Korea, care with facts, up-to-datedness, and the romanization of Korean words. These were abbreviated on our tables with the letters "P" "C" "U" and "R". Each of these sub-categories was worth 10 points. The criterion of accuracy, with its four parts, was not a percentage of the maximum or optimum, but was a subjective rating wherein all texts had the possibility of earning as much as 40 points.

In the first step of the rating process, we recorded all the errors and then categorized them according to their magnitude, their impact on the overall portrayal of Korea, and, if applicable, their degree of carelessness and out-of-datedness. In addition, we assumed that the careful use of romanization was an indication of precise scholarship. This assumption has been substantiated by our results -- care in romanization is reflected by accuracy in other details. There are multiple anglicized transcriptions possible for Korean, but there is only one standard system. Our approach was to award ten points if the work was completely consistent with the standard system, nine points if there were minor variations, and eight points if they used "newspaper standard romanization" (wherein diacritical marks are deleted). Errors within "newspaper romanization" usage would bring a seven, and grosser errors would yield six or less. Within each category we used the following system for evaluation and determining a rating.

- "Portrayal" meant the overall picture that was presented about Korea. If negative terms were used, if the presentation was out-of-date in some ways even though using recent statistics, if the tone of the presentation was "us versus them" or in anyway ethnocentric, then points were deducted.

- "Care with facts" covered accuracy of factual data and the way such data was presented.

- "Up-to-date" influenced both "portrayal" and "care with facts." If an encyclopedia was published in 1995, but quoted data no more recent than1988, the points were deducted in this area.

- And finally, since there is a standard romanization, publications that honored and used that standard tended to be careful and accurate in other areas of treatment as well. Furthermore, we encourage all publications to adopt the standard. In giving points for romanization, we noted that some publications used what we called "the newspaper standard" of romanization, wherein they did not include the diacritical markings that are part of the academic standard. Such markings include apostrophes on consonants to show aspiration, mostly, and markings on vowels to show differences in pronunciation between two kinds of "o's" and two kinds of "u's". Some publications would use the consonant markers, but not the vowels, and others would mark vowels, but not consonants. One point was deducted for inconsistencies with the consonants, and one point for vowels; the newspaper standard, therefore, would bring a rating of eight. If there were inconsistencies beyond that, the ratings would be seven or lower.

Printed Reference Works