Personal response systems through the prism of students' experiences

Personal response systems (PRSs) today offer an opportunity to the field of education in terms of improving teaching and learning outcomes through active engagement in classrooms. The present paper investigates students' attitudes to different types of PRSs, namely, Socrative and Clickers. Both qualitative and quantitative data are gathered and classified. The performed thematic analysis reveals major categories within the framework of this study, namely educational efficacy, psychological aspects, technology‐related issues, and administrative issues. It has been found that Socrative fares better in the “educational efficacy” and “administrative issues,” whereas Clickers outperforms Socrative in the “technological‐related issues.” It is worth pointing out that both Socrative and Clickers are tantamount in “psychological aspects” yielding no negative experiences. The results of this study reveal that two main factors, cost and technological infrastructure, are determinative in the incorporation and appreciation of such systems in an educational setting.


| INTRODUCTION
The benefits of active learning in higher education are widely known. Active learning is an approach in which the teacher creates a learning environment involving practices that encourage students to be active and engaged during the learning process [14]. Student response systems (SRSs) or personal response systems (PRSs) have been successfully used to foster such learning environments due to certain benefits, mainly that they: • Increase student participation and engagement during lectures, even in large classes [18,21]; • Provide anonymity, thus helping to engage students who would not otherwise do so [11,21]; • Make room for faculty to spot-check students' understanding of topics [21]; • Enable student to evaluate their own learning during lectures [18]; • Collect and analyze responses live [21]; and • Create a more interactive, collaborative, dynamic, and enjoyable environment in the class [18].
Students' perception of the study environment has been shown to have a significant impact on their behavior, academic progress, and sense of well-being [26]. Learning depends on several factors, but a crucial step is the engagement of the learner [25], which can be challenging, especially in large classes. PRS can be a valuable tool in this direction. When it comes to the learning process, it is also acknowledged by educators that a thorough understanding of a given subject by students is usually a slow and step-by-step process. Today, technology makes it easier for the faculty to spot-check students' understanding of a given topic in real-time by posing questions during the class and receiving immediate feedback. It helps students in self-assessment [20]. Moreover, teachers no longer have to rely on just asking the students if they understand the material or rely on their instincts or gut feeling before they move on to the next topic.
There are various tools available for this purpose. Socrative 1 is a web-based tool that allows students and teachers to use their own personal devices such as mobile phones, tablets, or laptops for assessments. Another tool that can be used is Clicker. 2 The classical Clicker technology is a transmitter-receiver system comprising a hand-held device, Clicker, and a computer software program that receives signals from the handheld device [9]. In both these tools, the instructor poses questions at different points during the lecture and students answer by pressing one of the buttons on their device/Clicker. The lecturer can see the results, for example, the percentage of students against different options for a particular question, on his/her computer and share it with students.
Socrative is a web-based PRS and therefore requires sound Internet connection. On the other hand, Clicker does not require the Internet. These two categories of PRS can be used interchangeably provided the learning environment possesses the necessary infrastructure. However, there may be some inherent differences between these two from the students' perspective. Socrative requires a sound Internet connection; moreover, students should either have smartphones, tablets, laptops, or other devices to connect to the Socrative Website through their browsers. On the other hand, Clicker is an expensive device, and therefore, lecturers may only be provided with a limited number of units, forcing them to share with other colleagues and for different courses. Additionally, lecturers will have to spend extra time to distribute Clickers at the start of the lecture, and even more time to collect and count at the end, causing waste of time. One option is to assign Clickers to students for the entire semester or study period, or ask them to buy their own Clickers, which means additional cost for the institute or the students. Further, the receiver attached to the lecturer's computer has to be situated in such a way so that it can receive the infrared signals from different Clickers across the auditorium otherwise the response will not be registered. The question is how these issues are perceived by the students. Specifically, do these challenges affect students in the learning process?
With these motivations, this study investigates students' attitudes to different types of PRSs, namely, Socrative and Clickers, and compares these PRSs in terms of educational efficacy, psychological aspects, technologyrelated issues, and administrative issues, all of which are major categories revealed within the framework of this study. There has been a study conducted in the United States to evaluate students' perception of the Socrative against Clickers, probing predominantly in psychological aspects and to some extent in educational efficacy category, in the pharmacy education [12]. In distinction from the preceding research, the present one fuses together the following features: • It provides a comparative analysis of two PRSs via students' attitudes; • It has been conducted in a developing country where university environments, including Internet connection and availability of Clickers are believed to be different and assumed either be an asset or a liability for any educational institutions. Therefore, the attitudes both of teachers and students can be different concerning the application of such devices in the learning environment; • It is related to engineering education, where students' perspectives toward the use of the Socrative against Clickers may vary from those in nontechnical education.
This paper is organized as follows: in the next section, literature review is provided as to the importance of using PRS for enhancing the learning process focusing on the Socrative and Clickers. The following section illustrates research methods, data gathering, and analysis. Section 4 presents the discussions and the final section infers the conclusions in this area.

| LITERATURE REVIEW
Brown et al. [4] showed that students are capable and willing to use the SRSs, reassuring faculty who are concerned about adapting them in their classroom. PRSs increase students' engagement, lead to a better understanding and retention of concepts, and also enhance the learning process in the classroom. Moreover, they make the class more entertaining, engaging, and interactive. They encourage and motivate students to participate in the lesson by providing immediate feedback and anonymity, if desired.

| Use of PRSs for educational efficacy
In a large group, it is difficult to control input from many students, to track who said what, and to assess whether everyone actually understands the concepts [22]. Clickers enhance the effectiveness of learning process, evaluate students' understanding, and engage them [22]. Apart from that, this technology improves students' understanding of complex subjects, ensures individual progress and comprehension [5,15]. Knottenbelt and Bukanga [16] found that Clickers promote the deep learning activity mainly through peer discussion and focused attention, along with maintaining excitement in the class through the use of technology to stimulate two-way communications. Clickers increase students' interactions with peers and with the instructor, which, in turn leads to a better understanding of the course materials [3]. On the other hand, it makes a teacher's awareness of learning problems easier [5,15]. Clickers allow students to get rapid feedback about their responses, giving a real-time selfevaluation of their own learning during lectures and contribute to a better knowledge of theoretical concepts [18]. What is more, it allows students to think critically about the subject and the alternative answers, thus achieving a deeper acquisition of knowledge [3].

| Usage of PRS for increased interactivity and motivation
López-Quintero et al. [18] discovered during their study involving first-year engineering undergraduates that the usage of Clickers increased students' participation during the lectures and enhanced the motivation they felt toward the course contents. Increased students' involvement and engagement with the material is widely recognized by the students as a positive feature of Clickers [2]. Masikunas et al. [19] showed how the use of Clickers in large classrooms motivates students, as well as develops their cognitive and social learning skills. Caldwell [5] acknowledged that many students in large classrooms often are hesitant to volunteer answers because they cannot face the chance of making mistakes in public and possible public disapproval. This situation can be overcome when using PRS due to the anonymity it provides. Clickers motivate such students to engage, to stay interested in class and make class times more enjoyable [11]. With PRS, students are also able to gauge their response against those of their peers, which may improve their self-confidence and perhaps spur them to diligence [15]. By fostering students' communication with their peers and teachers as well as promoting social and collaborative exchanges among them, Clickers stipulate developing students' communication abilities and a collaborative spirit [3].

| Web-based PRSs
Web-based PRSs such as Socrative, can serve as an alternative to Clickers by reducing the cost. Richardson et al. [23] describe the development and validation of an instrument for evaluating PRSs. Guarascio et al. [12] studied students' perception of the Socrative against Clickers in pharmacy education, concluding that students felt Socrative helped them to more actively participate in class and set a better environment in comparison to Clickers. To the best of our knowledge, there is no available study to report on students' perceptions as to the use of Clickers against web-based PRSs in the area of engineering education in a developing country, which may involve challenges different from those in developed countries.

| Research questions (RQs)
In distinction to the preceding research overviewed above, the present study aims not only to investigate an impact of PRSs on the education process, but also to provide a comparison of the two different PRSs. To achieve this goal, the RQs below have been formulated and distributed among the students: RQ1: What are the benefits of using Socrative and Clickers in teaching and learning?
RQ2: What is the best positive learning experience when using Socrative/Clickers? RQ2a: What is the best positive learning experience of using Socrative?
RQ2b: What is the best positive learning experience of using Clickers?
RQ3: What are the drawbacks of using Socrative/ Clickers? RQ4: What is the most negative learning experience with the use of Socrative/Clickers? RQ4a: What is the most negative learning experience of using Socrative?
RQ4b: What is the most negative learning experience of using Clickers? RQ5: Which technology, Socrative or Clickers, is preferred by the students and why?
RQ6: What can be done to enhance the use of these PRSs (Socrative and Clickers) in a university environment similar to the current one?
RQ7: Should PRSs (Socrative and/or Clickers) be more widely used in day-to-day teaching and learning at the university?
RQ8: How can one better manage Clickers' initial cost?
Mostly, these RQs are of exploratory type, while RQs 5 and 7 are yes/no questions and RQ8 is a question based on the Likert scale. This study provides a detailed analysis of the answers to these RQs.

| Participants
The data are collected from multiple courses such as engineering mobile application, introduction to computer science, mathematics for engineering, software engineering: architecture and design, offered at Monash University, Malaysia where both Socrative and Clickers were used. Most of these courses were taught to large groups, each comprising more than 150 freshman engineering students. The learners used both devices interchangeably, thus having a chance for comparison. The three course lecturers devised the questionnaire and interview topics. In total, 199 students completed the questionnaire.

| Procedure
Both Socrative and Clickers were used for 6 weeks each so that the students would have the opportunity to familiarize themselves with both tools and they would be able to compare the negative and positive features of them. In general, PRS aims to engage all students in a class and, since the target classes were large, it would be difficult to receive feedback otherwise. As there were limited number of Clickers available in the department, they had to be shared between different courses. The lecturers were required to distribute and collect devices before and after the class. Though, in the case of Socrative technique, students used their own equipment (laptops, phones, etc.).
During the lectures, questions were posed either with multiple potential answers (using Clickers/Socrative) or as open-ended questions (using Socrative). Students' responses were registered, obtained results were displayed on the teachers' computer screen, and their summary projected on large screen in the form of a histogram demonstrating how many students (in absolute numbers and also percentagewise) selected different potential answers. This feedback then dictated the focus of follow-up discussions in the lecture. Instructors first discussed each potential answer and why it could be correct/incorrect, then showed the correct answer. It helped students to think further to identify a misconception in their response. Sometimes, a majority of students chose incorrect answer. In such cases, lecturers tried to clarify the topic by approaching it from different perspectives, emphasizing key moments, and providing additional examples.
In the final week of the courses, a web-based questionnaire was distributed among the students covering both qualitative and quantitative data. In some of the questions, they were allowed to choose more than one option. Later, to collect further qualitative details about the usage of these PRSs, structured video recorded interviews were conducted to get detailed responses from the volunteer participants. The questionnaire was restricted because the participants were required to choose one or more of the options provided in each question with little flexibility although they could choose the "other" option and add their own views, but very few students did so. For this reason, the interviews proved to be a vital source of in-depth information about the advantages and disadvantages of both PRSs as well as the participants' preference of one over another along with the underlying reasons.

| Research methods
At the starting point of this study, questionnaires were designed for initial data collection and later, interviews were performed to obtain in-depth notions and ideas. Questionnaires are commonly known as an excellent tool to start descriptive research when data need to be gathered. According to Frechtling-Westat [10], the advantages of using questionnaires are that they are ideal for collecting basic descriptive data, are inexpensive, and the data generated from them can be easily imported or transcribed into statistical software. However, they usually tend to provide only broad, and not deep knowledge of the subject. Therefore, if the goal is to come up with a full and comprehensive description, then other data collection methods, such as interviews, may be more appropriate.
The RQs stated above require an application of a variety of methods, among which there are hypothesis testing, parameter estimation, and counting methods of the voting theory. These methods have been selected as they allow us to formulate the findings in the most appropriate way.
In the RQs 1-4 and RQ6, the emphasis is on identifying the general features in the students' attitude toward PRSs. In this part of the work, the methods of the qualitative thematic analysis prevail following the mains steps outlined by Aronson [1]. In addition, the thematic analysis was in part synthesized with the procedure of the qualitative content analysis as it is given, for example, in Schreier [24]. Conventionally, this procedure includes: selecting relevant data, structuring the gathered information, defining the main categories and subcategories, and revising the information after preliminary examination. This study was designed as an inductive thematic analysis-see Chapter 1 in Guest et al. [13]where the main categories also known as "dimensions" like in Schreier [24], were initially defined and later updated based on the data. A detailed description of all these categories, along with their subcategories, is provided in Section 3.4.
RQ5 requires to determine whether the number of students who prefer Clickers form a majority, which has to be confirmed rigorously. Such a precise statement requires mathematically accurate analysis, and, for this reason, statistical hypotheses testing is employed. As for RQ7, the majority for such a large sample can be claimed easily with the sample proportion of 83.9%. In this case, the confidence interval provides a more detailed information. Finally, the conclusion concerning RQ8 is reached with the help of the voting theory, whose methods allow to rank different options of managing the initial cost, the information precisely needed by this study. Since the preferences of students vary within four choices, the analysis based on the voting theory is an adequate one. See, for example, Farrell [8] and CO-MAP [7].
In essence, this study is a case study focused on an exhaustive analysis of a specific case rather than a comparison of different cases. According to Randolph [22], the goals of any given case study are to develop an in-depth understanding of a case, or multiple cases, and to gain insight into the interaction between the phenomenon and the case. On the whole, the research methods applied here include: qualitative data analysis, statistical methods, and elements of the voting theory. Such a variety of approaches allows us to shed light on different aspects of the responses possessing a rather nonhomogeneous structure owing to the diverse nature of the questionnaires.

| Data analysis
RQ1. In answering RQ1, the students could choose from the benefits suggested in the questionnaire and also add some other ideas of their own if any. Table 1 shows the percentages for the outcomes related to the proposed advantages of PRS.
RQ2. Thematic analysis is employed to analyze the qualitative data from the questionnaires as well as the interviews and the main categories that emerged from students' responses are: education efficacy, psychological aspects, and technology-related issues. The responses from the questionnaires are related with only the first two categories, while the last occurred to the research during the interviews. In this part of the RQ2, the students' responses yield further classification of the main categories into various subcategories as described below.
The outcomes concerning "educational efficacy" revealed the following breakdown.
(1) PRS helps to learn/understand the subject; (2) PRS helps to test knowledge/understanding;  As for the responses related to "psychological aspects," they lead to the subgroups stated below.
(1) The lesson becomes more entertaining and engaging; (2) The lesson becomes more interactive; (3) Immediate feedback is possible; (4) PRS encourages and motivates the students to participate and face challenges/competition; (5) There is anonymity; (6) Others.
The obtained information is summarized in Table 2, where along with the number and percentage of responses to each item, representative responses appear on the right side as "most popular/typical responses." Some participants had no opinion. More information on the subject appears in the interviews discussing Socrative and Clickers separately.
RQ2a. During the interviews, a distinction was made between Socrative and Clickers while extracting information about the most positive learning experience when using these technologies. When comparing the two, a new category appeared-"technology-related issues"-which consists of following items: (1) Availability in multiple devices; (2) Performance consistency; (3) Initial setup and use; (4) Dependence on external factors such as the Internet, battery, smartphones, and so forth. Tables 3 shows the summary of the interview responses related with the most positive learning experience of Socrative. Since the interviews were conducted with a limited number of students, some of the items mentioned previously in different categories did not come up during the interviews, hence their omission from the tables.
RQ2b. Table 4 shows the summary of the interview responses related with the most positive learning experience of Clickers.
RQ3. To provide a more balanced view of the situation concerning PRS's incorporation into educational processes, we inquired about the disadvantages, difficulties, and drawbacks when using these technologies. In answering RQ3, the students were asked to choose from the options supplied in the questionnaire and/or provide their own. Table 5 demonstrates the percentages of the outcomes related to the proposed question.
RQ4. While doing a thematic analysis of the responses related with the most negative experiences with PRS, one more category, namely "administrative issues," emerged which can be subcategorized as follows: (1) Initial costs; (2) Distribution and collection; (3) Others. Table 6 shows the questionnaire results with respect to most negative experience when using PRS. It has to be T A B L E 2 Top positive learning experience of personal response system

Option
No of responses % Most popular/typical responses

Educational efficacy
(1) Helps to learn/understand the subject 40 20 "Understand the lecture more clearly" (2) Helps to test knowledge 18 9 "Test my understanding" Psychological aspects (1) Lesson becomes more entertaining and engaging 53 27 "It's fun" (2) Lesson becomes more interactive 24 12 "Makes class interactive" (3) Immediate Feedback is possible 10 5 "The immediate feedback I get after I answer the question" (4) Encourages and motivates the students to participate and face challenges/competition 10 5 "Create a competition and competition makes us improve" (5) There is anonymity 2 1 "It's anonymous" To see what are the choices people make" stated that 16 respondents claimed that they had no negative impressions when using PRS whereas few participants did not answer this question. To gain a more detailed knowledge, the interviews were further organized separately as to negative learning experiences of Socrative and Clickers. RQ4a. Table 7 presents the summarized data related to negative experiences of Socrative.
RQ4b. Table 8 shows some of the negative experiences of Clickers.
RQ5. The data presented in Table 1 demonstrate close resemblance of students' opinions concerning the two PRSs. However, when asked directly of their preferences, a majority seems to prefer Clickers over Socrative. More specifically, 51.8% go for Clickers, 21.1% choose Socrative, and 18.1% like both. The others are either against both (5.5%) or have only used one of them (3.5%) and, therefore, are in no position to compare.
To derive a conclusion related to the preferences between Clickers and Socrative, it stands to reason to conduct T A B L E 3 Qualitative data from interviews (positive learning experience of Socrative) Educational efficacy (3) Offers flexibility in posing different type of questions "I think one of the main advantage Socrative has is the ability to ask open-ended questions to the student" "So if the lecturer wants you to write it in words, it really makes you think a lot for the question" Psychological aspects (2) The lesson becomes more interactive "It also increases learning opportunity as students and lecturer are able to interact through mobiles and it is easier and fun" (3) Immediate feedback is possible "Socrative is a platform where it can get a lot of students' responses instantly (immediate feedback) where it can create analysis report for the lecturer" "Engaging and there is instant feedback from the lecturer whether we are learning in the right direction or not" (5) There is anonymity "You can answer them on the spot and the lecturer would give feedback on the spot as well and for a person like me who don't like to talk in class, this is an easy way to get participation marks" Technology-related issues (1) Availability in multiple devices "The top learning experience for using Socrative for me would be the fact that it works with all my devices" "To me it is easy to access because as long as you have internet or your device has 3G or LTE, you will be able to access it" "Since like most of the students would have those devices, this comes at no cost to them or the institution" (6) Distant learning "Socrative can be used in distance learning" "So we can connect it anywhere, in the class or at your home itself you can answer the question online" T A B L E 4 Qualitative data from interviews (positive learning experience of Clickers)

Educational efficacy
None reported Psychological aspects (3) Immediate feedback is possible "Since the students only have to select from certain number of inputs, the students can react to the question in a much faster fashion thus providing the students' input almost instantly" "Students are able to submit their answers even faster than as opposed to be using Socrative" (5) There is anonymity "About anonymity, you can answer the question without scare of your answer, whether the answer is correct or false and it is very easy to use because it is just one click only" Technology-related issues (2) Performance consistency "The top positive learning experience using Clickers for me would be the fact that it works unlike Socrative. The main reason I say this is because our university does not have a good Wi-Fi so setting up Socrative take a lot of time and effort and sometimes after signing in and setting it all up, it just disconnects randomly. Clickers on the other hand just work, you just press the button and it works probably 90% of the time" "Best thing about Clickers would be they work more consistently" (3) Initial setup and use "Students will not be disturbed by the configuration and it is already configured by the lecturer" "You do not need a login functionality, all you need is just clicking the buttons and you can get your answers to be input to the server" and also yields the sample percentage p̅ = 71% with the sample size n = 145. We denote ƍ the actual population percentage of students who prefer Clickers and conduct the test to check whether its value is above 50%. More precisely, we test the null hypothesis H 0 : ƍ ≤ 50% against the alternative H 1 : ƍ > 50% at the α = .05 level of significance. Since the sample size is large, the application of Z test is justified. The observed value of the test statistic is: z obs = 5.057, which yields the p-value less than .0001 < α. The test shows convincingly that among the students who are inclined to just one of the PRSs, the majority is on the side of Clickers. Table 9 shows some typical arguments made by students concerning the advantages of the preferred system according to the interviews.
RQ6. In this question, learners provided suggestions to improve the use of Socrative and Clickers as shown in Table 10. Although less than half of the participants supplied their opinion, the data presented here are consistent with the preceding information: the main problems and, therefore, the most needed improvements occur in the area of technology and organization.
RQ7. Here, the students were supposed to provide answers to the aforementioned alternative, that is, whether PRS should be applied in the education process. The distribution of the positive and negative answers is: 83.9% "Yes" and 16.1% "No." Here, the test can be conducted similarly to the one performed in RQ6. However, the result for such a large sample (n = 199) does not leave room for any hesitation that the majority of students support PRS as a useful element in the education process. Therefore, it seems more illustrative to provide a 99% confidence interval for the percentage of those who approve of PRS use. Due to the sample size being large enough, using Z values in the construction of an interval is justified. The procedure described in Levin and Rubin [17], section 7.6 leads to a lower confidence limit of 77.18% and an upper confidence limit at 90.62%. Table 11 displays some typical responses from the interviews.
RQ8. Within the framework of this question, we analyze the students' opinions on one of the administrative issues, namely "initial costs." More details on the defined subcategories related to the administrative issues are given in RQ4. To be specific, the students were asked to rank their preferences about the different ways to obtain their own Clickers for subsequent use in the classroom. They were asked to rank with numbers, from one as the most preferred option to four as the least preferred one, their views regarding the four suggested methods to loan Clickers, denoted by A, B, C, and D as indicated in the Table 12. Note that no "ties" were allowed during this voting, that is, the outcome of each student's response is a permutation on four symbols and, on the whole, 24 different rankings are possible. This study question, in distinction to the previous ones, aims not only to determine a certain most popular method but also to reveal the ordering of the options A-D that reflect T A B L E 6 Most negative experiences when using personal response system (questionnaire)

Category
No. of responses % Most popular/typical responses the preferences of all students in the best way. One common tool appropriate for this type of problems is a rank method known as the Borda count. We refer to COMAP [7], section 11 and Taylor and Pacelli [27], chapter 4. The Borda count is used in a preferential voting, when each voter ranks all of the available alternatives, and submits the entire preference list, and not only her/his top choice. Although the Borda count has a variety of modifications, its common idea is to assign points to each alternative so that a higher ranking brings more points to an alternative. Here, we applied the Borda count in the most standard way, starting at 0, that is, each placement as the first brings 3 points (maximal possible), as the second brings 2, the third brings 1, while each placement as the last one brings 0. The greater number of gathered points secures better position of the candidate. Applying this method, one has according to Table 12: Borda (A) = 3 × 26 + 2 × 17 + 1 × 57 = 169. Likewise, we obtain that Borda (B) = 388, Borda (C) = 376, Borda (D) = 264.
As a result, the performed analysis leads to the following ordering of the students' preferences of the discussed options from best to worst: Qualitative data from interviews (negative learning experiences with Socrative)

None reported
Technology-related issues (2) Performance consistency "On an older device, it was also pretty slow to load Socrative. I found webpage is very heavy to load and it was very frustrating at a time to use the system" "The website was often unstable like you couldn't login to it" (4) Dependence on external factors (Internet) "Negative part of Socrative is that I would say that the main thing is the Wi-Fi issue" "Our university Wi-Fi is very slow and sometimes it can't even connect to my device" "Sometimes you just can't get access to the internet" "The problem is our universities' wi-fi is really bad sometimes" (4) Dependence on external factors (battery) "Other than the fact that Wi-Fi doesn't work, I had an old device, my phone was about dated and the battery did not last so I had the risk of going to the lecture with the low battery phone and my phone dying after the lecture. That wasn't really nice" "I do have an experience of a friend of mine being in class, her phone has no more battery and she did not bring her laptop or anything so she couldn't actually complete the Socrative activities" (4) Dependence on external factors (smartphone) "Maybe in this day and age, all students might have a device but there might be students that doesn't have one" "You need to have a mobile or any device and you need to bring it to the class. Some people like to use paper to do the notes and they might forget to bring their device. This would stop them to interact with the lecture"

Administrative issues
None reported Psychological aspects

None reported
Technology-related issues None reported

Administrative issues
(1) Initial cost "You need to purchase or to have a Clicker, not everyone could afford to buy a Clicker" "The bad part about Clickers is students will have to own a Clicker device which I would say is very costly. Is either student own it or the school will have to provide it every time there is a Clicker session" "I have heard buying a Clicker is pretty expensive and if students are forced to buy their own Clickers, they may not actually use the system at all" (2) Distribution and collection "The top disconfirming learning experience with Clickers on the other hand would be the fact that lecturer spends 5 or 10 minutes distributing Clickers to all the students that takes a lot of pressure time. This is a problem because some units we do not have enough time to complete all the materials in the given time and the use of Clickers actually hampers the delivery of the lecture" "It wastes a lot of time. Every time a class starts if your class has more than hundred students, we need to line up and get our own Clickers" (3) Others "It takes time for the students who does not own the Clicker in our university to collect the Clickers from the lecturer and also returning back. And if for some reasons the student comes late and have to collect the Clicker, it would create some disruption to the class"

| RESULTS AND DISCUSSION
PRS helps faculty to discover the students' level of understanding immediately on the spot. It can help in probing a topic further with students instead of merely one-way teaching (see Section 2.1 and Table 2). According to the results of the present paper, students agree that both Socrative and Clickers bear similar benefits. In their opinion, there is no major difference between the two when the top advantages of both are assessed. The class becomes more interactive and engaging as well as more interesting and lively. Nevertheless, a majority acknowledged that Clickers provide immediate feedback from the whole class and eliminate the waiting time, while the responses for this option are not as strong (51% vs. 37.9%) in favor of Socrative (Table 1).
Students almost equally support the benefits of PRS in terms of education efficacy and psychological aspects ( Table 2). The results of the analysis in Section 3.4 concerning RQ2 emphasize that PRSs motivate students to T A B L E 9 Qualitative data from interviews (students' preference between Socrative and Clickers)

Responses supporting Socrative Responses supporting Clickers No preference
Educational efficacy Technology-related issues "Personally I would prefer Socrative because Socrative gives us much more flexibility to answer the questions" "Between Socrative and Clickers, I would say I prefer Clickers because since it is multiple choices, we can answer things very straight forward and fast. It does not delay the class session" "I have no particular preference for these technologies. I believe that they have their own pros and cons. For units which due with more open-ended questions, Socrative would be the better option and for units which due with more objective questions, Clickers would be the suitable option" "I would prefer to use Socrative because you can get access to answer questions with subjective and also objective" "I would choose Clickers because first of all I dislike the set-up time takes to start using Socrative and the fact that I need to do it for almost every lecture I enter. I feel that the Clicker distribution does not take that long and therefore I prefer to use Clickers. Another reason is that since students can answer these questions in a quicker manner, we can go through more questions and learn more things" Technology-related issues "If me, I would choose Socrative because you do not need to buy any devices. Compare to Clicker you need to buy device. We already have a device that we use daily such as laptop, mobile phones or computer. We already have it as a student so we just need an internet connection" "I prefer Clicker, it doesn't require login and all you need is just clicking the buttons. Some people do not have smartphone. They cannot login to Socrative, limited access which is why I really prefer Clickers" "Personally I would prefer to use Socrative because it is much easier to use and more convenient. You can use it on any phone, any laptop" "Once our university solves the distribution of Clickers' problem, it would be a beautiful system to use. It works, it is reliable. I would prefer to use Clicker compare to Socrative because Socrative has its drawbacks such as your phone dying, also the wi-fi is bad. Clickers just work so I prefer to use that" Administrative issues "Also Socrative is cheaper, you do not need to buy a Clicker" find quick answers, which require to follow the lecture material permanently and to be focused on the study during the class and also make the education process interesting. Some of the sporadic responses, in our opinion, deserve attention. For example, "I learnt how to engage an audience for when I do presentations." This can be potentially taken into account for further development. Apart from that, the response "participation is not forced" is one of the psychological aspects identified by the students. This implies that PRS not only involves the learners in the process of understanding and selftesting the subject matter but, at the same time, it helps them by making the lecture more engaging, interactive, and vibrant. When probed separately about Socrative and Clickers, both fare at a par in "psychological aspects" (Tables 3  and 4) contrary to the result of an earlier study [12], which revealed that students preferred Socrative as it helped them to participate in class more effectively and facilitated answers to classroom questions. Socrative does not require students to login under their real names, and thanks to this anonymity, many who would not participate otherwise could be engaged in the class. As mentioned by students, Clickers provide faster feedback and they are easy to use. In addition, anonymity is guaranteed if the devices are not assigned to the students as in our case (Table 4). Moreover, another positive learning experience was instant feedback from the students and, as a result, feedback from lecturers to show why a particular answer is correct and why other options are not. However, in the present study students acknowledged Socrative's ability to assist in "educational efficacy" category due to its property of posing different types of questions (open-ended and objective type; Table 3). Students showed appreciation for Socrative as they could be asked not only objective-type questions, but also open-ended ones, since responding to the latter type requires a better understanding. Guarascio et al. [12] also reported that students felt that Socrative helped them to apply clinical case concepts more effectively than Clickers. In the technological category, Socrative is favored because it supports different types of devices and platforms (laptops, android-based phones, iPhones) as long as there is an Internet connection and allows distant learning implying that there is no need to be physically present in the class (Table 3). On the other hand, students appreciate Clicker in this category because of its consistent performance and since they are ready-to-use devices with no need for the Internet and requirements such as login, set up, and so forth (Table 4).
WIFI was mentioned as the main drawback of Socrative; whereas distribution/collection of Clickers followed by infrared connection are the main hassles in using Clickers (Table 5). A few stated that they simply select any random option while answering, perhaps implying that PRS does not help in educational efficacy (Table 6). This can be handled by using open-ended questions, which is only possible with Socrative. It is worth pointing out that neither the questionnaires (Table 6) nor the interviews (Tables 7 and 8) yielded any negative experience related to psychological issues while using PRS-which, in fact, should be considered as an important outcome of this study. The main concerns are, instead, related to the technological and administrative affairs. Although, such a situation cannot be viewed as positive, it can lead to ways toward overcoming these difficulties by, for example, improving the conditions and reviewing the regulations related to technology and organization. (2) Helps to test knowledge/understanding 15 7.5 "Give more questions and answers" (4) Others 9 4.5 "Use more frequently" Psychological aspects (1) Lesson becomes more entertaining and engaging 1 0.5 "Put background music while waiting to answer to make it fun" (4) Encourages and motivates the students to participate and face challenges/competition 2 1 "Have reward system"

Technology-related issues
(2) Performance consistency 9 4.5 "Use different channel for Clickers" (4) Dependence on external factors 41 21 "Better WIFI" Administrative issues (2) Distribution and collection 18 9 "Provide Clickers at the start of the semester" When inquired in detail about Socrative weaknesses, all of the students' responses are technology-related such as poor WIFI connection, battery problems, nonavailability of smartphones, and performance inconsistency ( Table 7). Socrative tends to be difficult and slow to load. Moreover, it takes a long time for everyone to join the activity. If the Internet connection is not solid, the experience is not pleasant, reducing the chances of any contribution by students. Also, if the students do not have a suitable device, such as smartphone or laptop, to access Socrative during the class, then they will not be able to participate at all. Yet, the Clickers' drawbacks lie in the "educational efficacy" category, including the inability to pose different types of questions, and administrative problems, like distribution/collection and extra equipment to keep track of plus the initial costs of Clickers (Table 8). As per the Table 8, one can say that Clickers is used to ask only objective-type questions and not open-ended ones. Students frequently click an option without thinking. What is more, the data collected imply that the administrative procedure tend to bring hurdles only for the usage of Clickers, which are more attractive for students (see Section 3.4,RQ5]. Obviously, such a controversy needs to be analyzed by educators at different levels, both faculty and administration. Furthermore, there is a waste of time before and after the lecture to distribute and collect the devices. There may be disruptions in the class and the need to keep track of an additional thing during the lecture. A way to avoid waste of time in distribution and collection of Clickers is that students purchase them in advance; yet they find it to be expensive. More than half of the students preferred to use Clickers, whereas a little over one-fifth of the students chose Socrative with another one-fifth favoring both (Section 3.4, RQ5 analysis). The result demonstrates convincingly that Clickers are more attractive to the students under the present circumstances of the use at the University. This result is inconsistent with the study by Guarascio et al. [12] where most of the students preferred Socrative. This preference for Clickers may be due to the technological problems, such as weak WIFI signals, encountered during the use of Socrative, which impacts the learning process. When inquired further regarding the reasons, students expressed preference of Clickers due to technological reasons, while the reasons for choosing Socrative falls under the "educational efficacy" (such as different types of questions can be posed), technology-related (such as works with various existing T A B L E 11 Qualitative data from interviews (should personal response system be used in day-to-day learning at our university?) Educational efficacy (1) Helps to learn/understand the subject. "Yes, I think both Socrative and Clickers should be used in all the class in our university because it really makes the class get into thinking rather than lecturer just stands in front teaching" (2) Helps to test knowledge/understanding. "It is easier to actually know which student is actually gaining progress in their studies and it actually brings more fun to the class" "Teacher can know what student understand for the unit"

Psychological aspects
(2) The lesson becomes more interactive. "I think all these personal response systems should be used more frequently in learning and teaching in our university because I find that it actually encourages interaction between students and teachers. "My answer would be yes, it definitely is useful in the class.
It does make the class more engaging. It just twists to the normal classroom style of teaching where only the teacher giving input to the students. The students can now give back inputs to the teacher and thus, the whole learning thing is improved" (4) Encourages and motivates the students to participate. "Yes, I believe that this personal response system should have a place in our university because it encourages participation from the students who would not participate otherwise" (5) There is anonymity. "So, with this personal response system, it encourages people to participate without fear of being mocked by other students if they are wrong since it is anonymous" Normally students will like to be more anonymous when they answer questions because they do not want to be judged and they are not very sure with their answers so this personal response system actually helps the students to answer questions anonymously. devices) and administrative (no need to buy anything else; Table 9). Students were also asked to provide suggestions regarding enhancing the use of these PRSs. There are general suggestions yielding more use of PRS and also providing more questions (Table 10). It can be observed that most suggestions address technology implying that if the related infrastructure (Internet connection, infrared signals, etc.) is upgraded, students' learning experiences can improve. Also, the organization process (mainly for large classes) has to be facilitated in terms of the distribution and collection of these devices. The suggestions specific to Socrative include improving WIFI, whereas Clickers-related suggestions are to "improve distribution/ collection process by providing Clickers at the start of the semester for the whole semester" (Table 10).
More than eighty percent students supported the use of PRS in day-to-day learning (Section 3.4, RQ7 analysis) with varied reasons ranging from educational efficacy (probing the topic, self-assessment) to psychological (increased participation and interactivity, anonymity). Some students suggested to use PRS in selected courses such as in difficult/theoretical courses or only when class size is big (Table 11). There are also other suggestions provided by the students regarding the use of PRS in Table 11. A few students stated that it should be used in certain units, which are either more theory-based or difficult. Quotes regarding this topic include: "It depends on the subject. If the subject is based on lot of theories, the Clickers or Socrative should be implemented in the study." "For some units it might make sense where you need a lot of interactions or may be with more difficult units to make sure that students are following, it might be a good idea to use Socrative or Clickers. But for some other units you don't really need it, I think it is just an extra hassle." On the other hand, a student opined that PRS should only be used when the class is large. One mentioned that PRS can be beneficial if only subject to certain modifications as different courses have different expectations and every lecturer has their own style of teaching.
Apart from the educational perspective, the specific organizational procedure cannot be left out of scrutiny when dealing with PRSs. It is very difficult to distribute Clickers to a large class of more than 150 students and, then, collect them at the end of the class as it involves counting them, too. Many times, a few Clickers are lost, never to be found again. To pinpoint students' visualization of the problem, RQ8 has been included to expose the ordering of the options A-D. This task, in turn, requires not only exclusively statistical, but also some combinatorial methods. To summarize, it can be accepted conclusively that in the environment of a university similar to the one in this study, option B is the most adoptable one. That is, to manage the problem related with distribution/collection of Clickers before/after every lecture, most of the students preferred to loan the Clicker with some deposit for the whole semester (Section 3.4, RQ8 analysis).

| CONCLUSION, LIMITATIONS, AND FUTURE WORK
This study reveals that the practice of incorporating PRSs into the process of education can be two-sided: helping and hindering. While such systems offer to provide certain positive outcomes in students' learning, there are also handicaps involved, mostly related to the technological and administrative aspects. Although a majority of students supported the use of PRSs in day-to-day learning, a large degree of preparation is required, namely with classroom equipment and technological infrastructure to accommodate PRSs into the learning process. On the bright side, however, there are positive effects such as educational efficacy, convenience, positive psychological influence, and encouraging atmosphere. As such, it is highly recommended that a departmental or institutional educational strategy be established to maximize the positive academic outcomes while embedding these tools, as also supported by Chien et al. [6].
Among other things, the research was intended to probe whether the students perceive both PRSs in a similarly positive way, or whether each system is different from the other in this respect. If the same, then both can be used interchangeably. In such a case, it may be better to use Socrative as it does not require any upfront costs except for the Internet and smartphones/laptops; whereas Clickers are expensive devices, which can be used in the absence of the Internet with seamless teaching without swapping screens. On the other hand, if the Internet connection is not good in the campus, it is not advisable to use Socrative.
Socrative fared better in the "educational efficacy" and "administrative issues" categories. On the other hand, Clickers outperformed Socrative in the "technologicalrelated issues." It is worth pointing out that both Socrative and Clickers were at par in the "psychological aspects" category and yielded no negative experiences. The data collected imply that the administrative issues cause serious obstacles only for Clickers, which are more attractive for students (see the outcomes of RQ5 and RQ6). Certainly, such a controversial situation has to be analyzed by educators at different levels: both academically and administratively. In the meantime, most of the students preferred to borrow the Clicker with some deposit for the whole semester.
As any case study, this study has its limitations. For example, this study was conducted in a developing country, where the costs of the device are considered high by students and the technological environment to use such devices in not as much developed. As a result, some of the outcomes are subject to location (in this case, a developing country) and, hence, they may be significantly different in more developed countries, where such infrastructure is more advanced. For deriving more grounded conclusions, it is necessary to add serious comparative analyses, both by longitudinal research and by comparison with the results of similar experiments designed at other universities and within different courses and disciplines.
Last but not least, similar studies have to be conducted from the instructors' perspective, and if, for example, the load of the instructors' work increases substantially due to the use of PRSs, this needs to be taken into account when assigning teaching duties. However, in our views, the benefits of PRSs should not be neglected and the research on the topic has to be continued.