In my home country (Korea), when language teachers make an achievement test for the courses they teach, they only use the reading passages from the textbooks or the materials which were read or addressed in the class. Thus, all the reading passages in a reading test are not new texts but the texts with which the students were already engaged before. For that reason, what is assessed in a test is how much the students remember well the teacher's comments on the reading passages in the textbooks and the teaching materials rather than their actual reading skill which is needed when they read a new text.
This phenomenon seems awkward, but I think that there are some reasons for doing this.
First of all, if a teacher lets his students read a new text in an achievement test for a reading course, the students' scores would probably be from their underlying reading competence rather than the teacher's instruction. (For example, even though a teacher teaches about skimming strategy in the class, students who have good command of English can find the main points of a text more easily than others who are not able to use skimming strategy skillfully due to their limited English proficiency.) Thus, the students might feel that they do not need to listen to the teachers' instruction carefully in the class because they can think that the instruction may not give significant influence to the scores they would get in the test.
Secondly, presenting new reading passages seems unfair for the students whose original proficiency is lower than the others. Actually, language learning is a really slow process, so it is hard to see learners' improvement in a short period of time (e.g., during a semester) in many cases. Therefore, if a teacher presents new reading passages in an achievement test, there is great possibility that the students whose original competence was good would get a high score in the test. In other words, it means that the students who did not have sufficient competence before the starting of the semester would have lower possibility to get a good grade in the course no matter how much hard they make an effort. (This is more problematic because many educational institutions in Korea are forcing English teachers to use norm-referenced test.)
Therefore, I think that the test scores do not reflect well the students' actual reading skill, and it is also not possible to check the improvement of the students' English reading skill during a semester. Also, I think that the achievement tests do not provide positive washbacks for developing the students' reading skill. (Usually, it makes the students memorize or focus on what was told by the teacher in the class rather than motivates them to improve their overall L2 reading skill.)
Nevertheless, it is not easy to use new passages in an achievement test in Korea due to the reasons mentioned above.
Thus, I wonder whether the contexts of other countries are similar to that of Korea or not. What are the sources of the reading passages of your reading test?
Also, I hope to hear your opinions about which one you think more desirable between using familiar texts and using new texts in an achievement test for L2 reading course.