What are the sources of the reading passages in your test for a reading course?

[KiChan Park]

In my home country (Korea), when language teachers make an achievement test for the courses they teach, they only use the reading passages from the textbooks or the materials which were read or addressed in the class. Thus, all the reading passages in a reading test are not new texts but the texts with which the students were already engaged before. For that reason, what is assessed in a test is how much the students remember well the teacher's comments on the reading passages in the textbooks and the teaching materials rather than their actual reading skill which is needed when they read a new text.


This phenomenon seems awkward, but I think that there are some reasons for doing this.
First of all, if a teacher lets his students read a new text in an achievement test for a reading course, the students' scores would probably be from their underlying reading competence rather than the teacher's instruction. (For example, even though a teacher teaches about skimming strategy in the class, students who have good command of English can find the main points of a text more easily than others who are not able to use skimming strategy skillfully due to their limited English proficiency.) Thus, the students might feel that they do not need to listen to the teachers' instruction carefully in the class because they can think that the instruction may not give significant influence to the scores they would get in the test.
Secondly, presenting new reading passages seems unfair for the students whose original proficiency is lower than the others. Actually, language learning is a really slow process, so it is hard to see learners' improvement in a short period of time (e.g., during a semester) in many cases. Therefore, if a teacher presents new reading passages in an achievement test, there is great possibility that the students whose original competence was good would get a high score in the test. In other words, it means that the students who did not have sufficient competence before the starting of the semester would have lower possibility to get a good grade in the course no matter how much hard they make an effort. (This is more problematic because many educational institutions in Korea are forcing English teachers to use norm-referenced test.)


Therefore, I think that the test scores do not reflect well the students' actual reading skill, and it is also not possible to check the improvement of the students' English reading skill during a semester. Also, I think that the achievement tests do not provide positive washbacks for developing the students' reading skill. (Usually, it makes the students memorize or focus on what was told by the teacher in the class rather than motivates them to improve their overall L2 reading skill.)

Nevertheless, it is not easy to use new passages in an achievement test in Korea due to the reasons mentioned above.

Thus, I wonder whether the contexts of other countries are similar to that of Korea or not. What are the sources of the reading passages of your reading test?
Also, I hope to hear your opinions about which one you think more desirable between using familiar texts and using new texts in an achievement test for L2 reading course.

[Sebastian_Rocheleau]

What an interesting phenomena. Just to clarify when you say achievement test I assume you mean a low stakes in class summative tests?
 
I think it really depends on the construct you are assessing with the reading in the test. If your construct is writing then I don't see it being problematic. It might actually be beneficial as I would make the task more focused. For example, if you give students a text they have already read but they have to write something new based off the text. If the purpose of the text is to assess reading then using a previously covered text would cause issues with various aspects of test validity (predictive, convergent, face, and content). I'm not really sure what can be done to fix this situation in Korea. It really depends on what the instructors wish to asses and their awareness of the limitations of this form of testing.
 
When I was teaching in the Japanese junior high school system I saw some instructors generating test that did use previously covered texts as part of the item for reading, grammar and speaking assessment. Maybe a reason for this in EFL environments is instructor confidence in item design? Some teachers might doubt their ability to create a text item that covers the same aspects of the reading construct that were covered in the in-class text. I know that here at the University of Illinois we generate all texts for assessment. We normally base the text on an authentic source (newspaper article, research articles etc...) and then modify it to fit our needs but like any type of item design there are drawback to this approach. It can be difficult to maintain a consistent level of difficulty between semesters if the items are always changing. We use several different techniques to maintain the same level of difficulty over semesters. For example, to maintain a consistent level of vocabulary difficulty we set word frequency guidelines for text items (eg. From total amount of words in the text add up to the following percentages:  60% from the top 1k words, 15% 2k words, 15% Academic Word List, 10% above 2k) in our English Placement Test. This allows us to increase the test validity.
[/size]