Method of Evaluation
We evaluated CD-ROM's on two basic scales. The first was to compare data on Korea from one CD-ROM to data on Korea from other CD's. The second was to compare treatment of Korea to treatment to other countries on the same CD.
I. Treatment of Korea on one CD compared with the treatment on other CD's
In the first case, we established a point system to facilitate comparison. There were 100 points maximum; 50 for the text and 5 each for ten other items which included the ease of use features of the CD. The 50 points for the text included 20 points for quantity, 20 points for quality, and ten points for accuracy. The ten features that were each awarded up to five points included five features of visual (pictures, movies, maps, statistical tables, and graphics), two audio features (language samples and other sounds, e.g., music samples or sound for a movie), the up-to-date aspect of the data, coverage of North Korea, and what we called "ease of use" -- an evaluation of the internal software design of the CD. Some CD's included all ten features, but none received full points in all ten; CD's tend to specialize in certain areas, but not all. Some are more oriented to pictures and sound; some are nothing but text.
II. Treatment of Korea compared to treatment of other countries within a given CD
The comparison of Korea with other countries was made under the assumption that certain countries are more important than others. Based upon which standards one chooses, one can quantify the relative importance of countries. We chose five factors that we considered important in measuring the standing of a country (population, GNP, per capita GNP, exports and imports), and we ranked the countries with the "best" in the category as one, second as two, etc. We then totaled their rankings for the major countries. The best overall score was a golf score --lower is better. For example, the United States is third in population, first in GNP, second in per capita GNP, first in imports, and first in exports = eight points. Japan was second; ninth in population, third in GNP, eleventh in per capita GNP, fourth in imports, and third in exports = thirty points. France was third, followed by the United Kingdom, Italy, Canada, and Germany. Korea was eighth, followed closely by Spain, Mexico, and the Netherlands. China was first in population, second in GNP, 137th in per capita GNP, and eleventh in both imports and exports = 162 points, or 18th overall.
Korea came out the eighth most important country in the world. Korea was 15th (combined with North Korea) in population, 14th in GNP, 47th in per capita GNP, 13th in imports and 13th in exports. We found it interesting that Korea was not in the top ten in any one category, but in the aggregate it came out number eight. The argument is that population alone does not determine which countries are important, neither does wealth, nor trade, but all of those factors together form an aggregate that is useful for comparative purposes.
Having determined the order of importance of countries for this exercise, we created a template, whereon the countries were all listed in the established order; and we use the template to measure coverage on each CD. By looking at where a given CD provided data (or other media), in comparison to the order in which it should be, we can see where a bias disproportionate to the standard is revealed. If Korea should be eighth in coverage, but ends up twentieth on one CD, or fiftieth on another, it tells us the degree of fairness or bias on that CD.
Our rating system can possibly be criticized. There are not many established standards for critiquing CD-ROM's. The CD's are so different from each other that we were trapped comparing apples and oranges. We explored the possibility of creating sub-categories. For example, some CD's appear oriented toward students, others toward travelers. However, to do so would create other problems; the lines of division are not clear. Suffice it to say that our approach is one possible method; there could be others, but we argue that our representation has a degree of validity that is useful in making decisions about the usefulness or value of a given CD that contains data on Korea.
The CD-ROM reference works have a unique capability of combining both visual and textual features. Because test is able to provide the most information, we gave the most emphasis in our rating system to text. Therefore, on a 100 point scale, we allotted 50 points for text and divided the remaining 50 points between other important features.
In assigning points for text, the 50 points were based on the following breakdown; 20 points for the quantity of text that the CD contained, 20 points for the quality of test, and 10 points for the accuracy of the text.
As for the quantity of text, we initially determined which of the CD's contained the most text that was devoted to Korea. The CD was given the full 20 points. We then gave points based on a subjective analysis of the other CD's in comparison to the CD with the most text.
Regarding the quality of text, we found it necessary to bread down the possible 20 points into two separate categories; breadth and depth, worth ten points each. For breadth, the CD that covered the most topics: history, economy, government, politics, geography, culture, society, education, bibliography, and tourism of Korea were given a score to ten. Then, the other CD's were given scores in comparison to the range of topics covered by the CD with the most breadth. There were also ten points given to the CD according to the amount of depth and detail given to the topics that were covered. The CD that devoted the most detail to the topics covered was given a score of ten. The other CD's were given scores in comparison to the depth of the topic covered by the CD with the full ten points. Totals of depth and breadth were combined to give a score out of 20 possible points. Therefore, a CD that covered a large amount of topic, and yet gave those topics only minimal attention, could receive comparable points to a CD that covered only a few topics, but gave extensive detail to them. An optimal CD would contain text that had both a wide range of topics and detailed coverage of those topics.
Finally, we awarded up to ten points for the accuracy of the text. In reading over the text, we looked for errors and inconsistencies, and took away points from CD's that contained errors. The number of errors was compared to the quantity of text. Points were deducted according to the number of errors in relation to the amount of text. The points given were again based on a subjective analysis of each CD.
Another 50 points were given to each CD for including certain important features. Specifically, these features were broken down into ten categories: pictures, movies, up-to-date, maps, statistics, ease of use, coverage of North Korea, language, sounds and graphics. We gave each feature a score out of five possible points, after comparing how a given feature was represented on other CD's.