Assignment Submission

Details

Category: Level 3, User Experience Design

Published: Thursday, 06 September 2018 08:57

Hits: 4333

1.0 Card Sort Process

The Card Sorts methods main purpose is the organisation of information into structures. This method in accepted for checking a websites structure. With this method, the difference between the users and the developers view. It gives an opportunity to visualise the mental model of the user and to fix any navigational problems (Wentzel, Müller, Beerlage-de Jong & van Gemert-Pijnen 2016).

1.1 User profile questionnaire

The most important part of the Card Sorts method is selecting representative users. For this purpose a User Profile Questionnaire is used. To create the right questionnaire is the first step is creating a summarisation of the products potential users and their behaviour. PACT analysis was made for this purpose (Reinius 2011). Therefore the first step of this study is to produce a PACT analysis, which can be found in appendix A1 section.

The questionnaire was created based on the PACT analysis. The questionnaire has focused on 3 main categories.

· User characteristic

· Usage characteristic

• Technology

Based on the categories a 17 question questionnaire was made, which can be found in appendix A2. The questions include demographic data, user ability and behaviour related question and it was ensured thta the questions closely relate to telecommunication websites.

The questionnaire was shared online for easier use. First Survey Monkey was going to be used, however it only allows 10 questions to be created with a free account. Instead Typeform was used.

The completed versions link https://fpisti35.typeform.com/to/ejaViJ , was shared on Facebook, while potential participants have received an e-mail.

The questionnaires questions, the images of the online versions administration and other related documents can be found in appendix A2 and C1.

1.2 Sort Development and Administration

The Card Sort Method requires serious preparations and administration during performing besides tasks with participants. The detailed descriptions of these are presented in this chapter. The preparations, tasks with participants, pilot test and the task performing can be viewed in the subchapters.

1.2.1 Card Preparation

The cards were used from Three.co.uk Discover page on 9th of April 2016. On the original site only the images and the title of the cards and only when the cursor is moved over one of these cards, the written description appears. When the contents of the webpage were saved, all the data was written on the cards, so these were used.

Examples of the cards they were printed out in colour.

All cards can be found in appendix B

1.2.2 User recruitment and participant descriptions

Persons were selected from the ones who filled the Typeform online form to perform Card Sort. The most important aspects of selecting were the answers for question number 7, which is about the user has mobile telecommunications or not, and number 11 which is about visiting the similar content websites. The screenshots proving the results of online forms are presented in C1 section of appendix, while the table about the contents can be viewed under section C2.

A Consent Form was made for participants to prove the voluntary participation and acknowledging the terms and conditions. (See in appendix C3.)

The detailed description of study is presented as an appendix for Consent Form and specified in Study Information Sheet. (It can be viewed in appendix C4.)

1.2.3 Piloting

To test the effectiveness of the cards and to complete the study quickly, one of our family members in our home has helped. (According to the module leader this was an option.) This family member has been subscribed to the company involved in the study for 5 years. He filled out the questionnaire, which has shown that he was an applicable representative user. The participant has received the Study Information Sheet and the Card Sort Task List and after understanding these, the participant has signed the Consent Form. The participant number is 001 (see appendix C3)

The pilot test results are shown in the following, which shows the structure created in a diagram.

The example diagram shows, that the cards fulfil their purpose to complete the tasks, so no changes were necessary. The original cards content was not modified, only 11 new cards were added to create the architecture that the user found to be the best possible.

Added cards

• 31 Discover
• 32 Services and devices
• 33 About us
• 34 Customer support
• 35 Guidance
• 36 Advice
• 37 Broadband
• 38 Dongles
• 39 Tablets
• 40 Our offers
• 41 Other offers

1.2.4 Procedure

The many requests for voluntary survey disheartened the participants and made finding volunteers harder for similar purposes (Hager & Brudney 2011). There are so many reasons behind this, but accommodating is necessary in this environment for the claims of the few participants. This affected the selection of locations, so the participants must be visited in the time and location what they chose.

1.2.4.1 Card Sort One

The first test took place in Luke Lane Community Centre (Heeburn NE31 2BA) at 11am, 17th of April 2016. The participant number is 002. The details of the participant can be found in appendix C2, 4th row in the table. According to the questionnaire that was filled out online, the participant has met the requirements to become a candidate.

The participant first read through the Study Information Sheet, then by signing the Consent form, has agreed to take part in the project. The subject only agreed to written recording of data, as the participant did not like to hear or see herself in audio or video recordings.

Next she was given the useable cards and the sheet including the task descriptions, the Task List (appendix D1) First the participant has read through each card, then organised the cards into simple groups. More complicated cards were put to one side, then later reading through them again to identify the functions. With a couple of new technological advancement the subject was not aware of, such as 4G, so she has asked some questions regarding them. After organising the cards into their final groups, blank cards, which were provided, have been filled out and adding them to create a structure that was easier to use. All cards have been used and there were no further amendments made. During the pilot test the participant did not find the order of the cards being important within a category, however this subject thought the opposite. She claimed that if the first couple of information did not provide interesting or important data, then the rest of the data in the same group would not be read.

The diagram based on the photos created about the structure made by this participant can be found in Appendix D2.

Added cards

• 31 Usage / Network Coverage
• 32 PR
• 33 Payment Methods
• 34 Support
• 35 Other Services
• 36 Question / Help
• 37 Our products
• 38 New Phones
• 39 4G
• 40 Tablets
• 41 Home internet
• 42 Mobile Broadband

1.2.4.2 Card Sort Two

The second card sort took place at the house of the participant (Sunderland) at 17:00, 17th of April 2016. The Participant Number is 003. The details of the participant can be found in appendix C2, 6th row in the table. The information gathered online has shown that he was an applicable candidate.

He was given the Study Information Sheet, then by signing the Consent Form has agreed to aid the study. He has also only agreed to the written recordings.

The participant has received the cards and the task description sheet, Task List (appendix D1). He attempted to organise them into groups. The finalised groups and has given names for each of the appropriately. He has found it important that within the groups the questions should be on the top, then the general information and finally, special content was placed at the bottom. The he has organised each group in order, as he found that to be important too. At the end every card was used without any modifications.

The finalised structure's diagram created, based on the photos taken from the structure, can be found in appendix D3.

Added cards

• 31 New devices
• 32 Tablets
• 33 Home & Mobile Broadband
• 34 Services and Supports
• 35 PR

1.2.4.3 Card Sort Three

The third Cart Sort test was also implemented inside a private house because this complied with the participant’s needs. The test was performed at 5 PM, 23 ^rd April 2016 and the location was Sunderland. The participant’s number is 004. Related documents were given to the participant before test, and all of them were verified and understood by the volunteer with signature. Selection of the participant was made available by filling the online form, which complied with the group of representative users. The data given by the participant can be viewed in row 12 of the table in appendix C2.

The participant read the cards’ contents first and then tried to interpret them. In this case, there were some minor questions about technology as well because the participant was not up to date with the most recent developments and services. Aside from these, all of the cards were interpretable and usable for the participant. The participant sorted the cards into groups first. A few cards were moved to the more matching groups for their subjects. Neither duplicated cards nor content changing was done in the final designation. The groups got their names after this. Inserting many cards under five simple groups was successful. Few minor modifications were done in the sequence of cards after naming groups, during the final examination. The participant could not draft exactly what aspects led to sorting them under a group, only said that these looked logical in this order to the participant.

Photos of final sorting and a graph based on map notes were put into appendix D4.

Added cards

• 31 Mobile Phones
• 32 Tablets
• 33 Broadband
• 34 Services
• 35 About Us

1.3 Results

The conducted Pilot Test and the structures were created by three volunteers and the conclusions from experiences during the tests were included into this chapter. Also, a summarized structure was created besides the test analyser, which can be viewed in this chapter. In the end, the differences between the current structure of website and the summarized structure were examined.

The finished structure diagrams were printed, but as they only contain the numbers of the first thirty cards, the short titles of cards were written next to them for the better transparency. The in-group sequence is important for most users, but they mean different things under sequence, and other things seem logic to them, which came to light during the personal provisions. Structure made by the first participant was different from the following two ones, and a more difficult menu system was created. The last two participants’ structures were nearly identical, only the group names contained smaller differences, and few cards were put in a different group. The sequences in groups were different because the second participant put the question card titles on the top. This was not considered important by the third person because only the card subject is important for grouping, neither declaratory nor questioning type of the title, according to the participant.

Only the Tablet and Broadband categories were identical in all participants’ structure. In all cases 5-5 cards were put into Tablet category, and only the third participant put the card number 26 (connect iPad to TV) to this category, while the card number 23 was rather put into Services category. Same cards were put into the Broadband category, while only the first participant split to Home and Mobile categories, but the other two participants did not see any meaning of making two categories for 2-2 elements. Also the third participant put the card number 25 (Top Up Mobile Broadband) into this category, while the first participant considered it rather into Payment Methods and the second participant into Services.

Both first and second users had very similar PR categories. The card numbers 11, 12, 13 were classified here by both users. All participants had a New Phone or New Devices category and they put the same cards here. These cards were number 2, 3, 4, 5, 8.

Another difference was in the first test participant’s structure the three-level directory structure. The other two participants only made a simple one-level category from five elements. Also the category names were similar, but the second participant saw the PR title well like the first one, while the third one saw the About Us better.

The structure of the third participant was considered as the most usable by abiding the principle of simplicity as above described. Starting from this, the similarities and differences were compared against the other two participants’ structures in those cases, when the other two uniformly placed the cards elsewhere, so these were moved according to their decision.

These were:

· card number 7 (No signal, no problem) was moved to Services from About Us category

· card number 15 (Google hangouts Hub) was moved to Services from About Us category

· card number 17 (Live devices in stock) was moved to Mobile Phones from Services Category

· card number 19 (Free WiFi, London Underground) ) was moved to Services from About Us category

· card number 23 (Should get phone, phablet, tablet) was moved to Tablets from Services category

· card number 26 (Connect iPad to TV) was moved to Services from Tablet category.

Thus, the Services category became too long, 10 elements. This is why paying attention to the first participant’s decision; some elements were removed that originally were put to the Questions/Help subcategory according to the participant’s decision. The size of the Services is reduced, and the subcategory was used, which was suggested by the participants.

Added cards

• 31 Mobile Phones
• 32 Tablets
• 33 Broadband
• 34 Services
• 35 About Us
• 36 Questions / Help

The structure created this way can be viewed in the diagram below.

Copy of original site structure can be found in Appendix E. As it can be seen on the pictures, the first 21 cards (1-21) were designed into Phones section of Discover page, where the contents of these 21 cards are represented on same page. The remaining 9 cards (22-30) were moved from Mobile Broadband section of Discover page. It is difficult to compare these two structures, the 21+9 cards with the final structure made by us because many elements can be found in the Mobile Broadband, the Apps and the Sharing stuff sections. One thing should be stated: the 21 picture links in one section is too much, so this is why it is expedient to sort them into smaller categories. The conclusion of the study and final category structure would be a good starting point for this. A similar study should be done with all cards of all sections, and a more usable system should be built. The usage of exactly 30 cards heavily limited the boundaries of this study.

2.0 Heuristic Evaluation

2.1 Process

The base of evaluation is provided by the 10 aspects of Nielsen’s Inspection Method (1994). Another starting point was the listing section of PACT activities besides abiding the principles.

System Scanning method was used during the first few visits, and only the contents and structural properties were analysed. During the further visits, System Searching and Goal Playing methods merged together, the most general functions were tested, and the experienced problems were noted. Similar content or structured websites were checked for repeated error types against the error list. About 8-10 hours were used for the recurring examinations, but it contains the time of making proof screenshots. Several discovered problems were sorted by seriousness, which were affected by the frequency of repeated errors. The sorted error list can be found in the next chapter.

2.2 Inspection Results

The errors located during the evaluation are listed below. The simple errors, where only a screenshot was made were not added to the appendix due to repeated scrolls, so instead these can be viewed after the mark and description of the specific error. The errors where more pictures were required for demonstration can be viewed in the appendix and only the exact location was defined by the specific error. Lighter errors were added to the beginning and heavier ones added to the end of the list.

During the websites review and the checking of ‘Email to Us’ function the system redirected to http://www.three.co.uk/terms-conditions/www.three.co.uk/emailus link and gave an error message. After few days, the error was not experienced during the recheck, so possibly it was fixed because of the feedbacks.

The next problem was experienced with ‘Sitemap’, which can be reached via the bottom menu. Only the two upper levels of menu system can be found there. This really limits the usability of the site, and also it is used by the search engines, so the site hit rates are being degraded.

Similarly to the previous error, the search result of site is degraded, which means the site hits will decreased, if the site’s ‘Meta Tags’ are highly incomplete. However, this error is hidden for most of the users, but it affects 10^th directive of the guidelines, which would like to provide easy search ability. Only the ‘About Us’ section contains sufficient amount and quality of metadata. The most important data, the key words are missing on the general pages. ‘Careers Centre’ and ‘Wholesale’ pages almost completely lack any metadata. The screenshots of websites’ informational windows can be viewed in the F1 chapter of appendix.

The next error violates the 4^th principle of Heuristic Evaluation named Consistency and Standards. The header of ‘About Three’ section is completely different from the other sections, which came to light during the check of pages continuity. More complicated navigation bars can be found there, and the working of few ones is not clear. For example, after the umpteen clicking attempt to the ‘Three. | Web.’ link can be seen an appearing small and not really visible “Powered by Google” banner, which marks the place of search. Further sections with different looking headers, layout systems and navigation menu are the ‘Careers Centre’ and ‘Wholesale’. Looks like these sections were made by different developers, who did not care about the work of other developers. The screenshots of headers can be found in F2 chapter of appendix.

.

The first serious unpleasantness of usage can be found at the download links of documents or files. None of continuity can be experienced with the links, which came to light during the examination. Three different download methods were experienced. During the first case, the link opened a blank page, which left open after the download start, and it had be closed by the user. A link like this can be found on the page below:

· Terms & Conditions / Returns & Exchanges / Three’s Returns & Exchanges Policy page’s ‘ Three's Returns & Exchange Policy PDF ’ link.

During the second case, a new window was not opened, just the current window became blank, so coming back to the previous site is possible only with the browser’s navigation button. Example links for this error:

· Three price guide from button menu

· Terms & Conditions / Three Products & Services / London Underground Wifi page’s ‘ Acceptable Use Policy.’ link.

During the third case the system worked properly, and only a popup window opened for securing the download, but the browser window was not changed. Links like these can be found on the page below:

· Terms & Conditions / Use of Three Services / Pay monthly & Pay as you go page’s links.

· Terms & Conditions / Price Guides page’s ‘ Pay Monthly Voice Packages ’ and other links.

Texts of the pages contain words that are completely the same, identically coloured and simply underlined, which does not open download links, just redirects to a new page. This can be experienced with the link below:

· Terms & Conditions / Three Products & Services / Home Signal from Three page’s ‘ www.three.co.uk/homesignal. ’ link.

So the user will not know what will happen after clicking on similar simple links. Picture proofs of experienced errors can be viewed in appendix F3 .

The next two serious rated errors and suggestions for fixing will be described by details in the next subchapters.

2.2.1 Problem Prediction One

First of all, heavy violations of easy usage rules can be experienced on “Check coverage & network stats” page. However, the link that directs to this page is not presented on a central place but can be found in the bottom menu system that is presented on every page, so it is placed on an important place. It can be reached directly from the first row in the middle of the site directly next to the store search box.

The first problematic function can be experienced with a specific device search box. A long drop-down menu will appear after typing the device type, where the exact type can be selected. There are too many choices, which means that more significant developing effort is needed. Whatever type will be selected, it will not cause any effect on the site. So this makes it useless. Otherwise, the working of it can be questioned by conception aspect because a device’s signal strength depends not only on the type, but this also affected by momentary properties like the used cover or the position of user’s hand (see iPhone 5 anomalies).

The next problem is the layout structure of the page. Only a little part of the map with search functions can be seen due to wrong structure. If the map will be placed on the middle of the screen, then the menus cannot be seen. The screenshots about this error can be viewed in the chapter F4 of appendix.

The further part that causes usability difficulties is the map itself. Searching on the map and resolution adjusting can be done only with plus and minus buttons. It only provides switching between four different resolution map parts. The change was too big for few steps, which means that resolution from the whole Europe map to streets level can be reached in four steps. The significant scrolling operations are unavoidable due to this high amount of scale changes. The system relatively works good if the zip code is known, but then an error is experienced that was described in the previous chapter, which means that the zip code search box and the map cannot be displayed on same screen.

Further usability degrading attribution is that only three different colours mark show the coverage of the defined area. Like ‘Indoor & Outdoor’, ‘Outdoor’ and ‘No Coverage’. But the exact value of these was nowhere specified. While checking the own post code, it shows the best signal coverage in both indoor and outdoor areas, but the mobile phones are useless in few premises of apartment because it lacks sufficient signal strength.

A button named ‘network problem’can be found on the bottom of the page, by clicking on this button, three new buttons open. Types of services can be selected with these buttons like ‘Calls, Internet or Everything’, but there is no opportunity to mark the exact error. Like the other errors of the site, this also looks so unthinking and heavily simplified, which questions the real usability.

Completed Problem Report can be viewed in appendix F5.1.

Suggestions for fixing the listed errors can be found in chapter 2.2.3.

2.2.2 Problem Prediction Two

The last and most critical problems are experienced with navigation system of ‘Breadcrumb’. Total un-organization is experienced here. Examples are found for almost every error, from incompleteness to the bad structuring. For example, causeless repetitions or missing chain links were found here. So, this system is useless in many cases, and there is no way to step back to previous levels, or it is not revealed how the user reached the specified page. The examples below are proving the experienced errors.

On Support. / Internet & apps. / Internet on your phone. page the second last section will lost from Breadcrumb after clicking on the ‘Payforit’ link.

Also Support. / Bills & contracts. / Understanding your bill. page contains this ‘Payforit’ link, but using from here causes the change of Breadcrumb structure except the first parts. So, the user will lose the last two steps and is placed to a new road.

After reaching the ‘Ways to pay’ page, a causeless repetition can be expected, which does not match the path done by the user.

Selecting a device from the ‘Popular phones’ section of bottom menu like Samsung Galaxy or iPhone will cause a page lacking Breadcrumb. A totally wrong path will be displayed while selecting other available devices from here like Samsung Galaxy S7.

On the Support / Bills page, the ‘Detailed Guides’ section by clicking on ‘Managing your account’ menu’s ‘About your My3 account’ link will also show a completely wrong Breadcrumb path on the next page.

Due to large size of whole system, there was no opportunity to perform a full examination of many pages and links, but the listed examples prove that the site still needs something to be fixed.

Completed Problem Report can be found in appendix F5.2.

2.2.3 Eliminated Problems

Advice is given in this chapter for trying to repair the errors listed above.

The very first error of ‘Email to us’ is already fixed.

Adding a further level or levels to ‘Sitemap’ might help its usability.

The different looking and structured layouts should be changed to a same working one, which means the users will not have to learn different management methods.

Avoiding several experienced errors on ‘Check coverage’ page is a more difficult task. The role of device selection possibilities must be considered, and if there is a meaning and usefulness of this thing, it should be informed with the users and make the changes visible. The usability would be increased by dissolving this page. The whole page wide map is not justified by anything. A smaller map would and share the search functions near it would be enough, so those will be available without scrolling. Fine tuning of map scale would make the search on it easier.

No new developments are necessary to repair the anomalies occurred with Breadcrumb; only good working solutions should be used in faulty places. This would improve the informing of users about where are they and how got there.

3.0 Usability Study

The third large chapter of the portfolio studies the user usability of the whole website. The finding of this study can be found in this chapter. At the start of the chapter the methods used to create the documents are described. Then the test tasks are defined. Followed by how the tests were carried out. To compare the tests, measurable tasks have to be implemented. These methods of measurement and its results are described in the next subchapter. Finally the Pilot Test's impact and changes made based on the experiences gained from it closes the unit.

3.1 Documentation

As this test was different to the previously used Card Sort test, therefore the documents related to it could only be used as a basis, but changes had to make to them.

The Consent Form, gives a short description on the purpose of the test and the time required to complete it, therefore this required the least amount of changes. The test's goals have been given a better description and the required time was increased from 40 minutes to 60 minutes. The two versions of the Consent Form can be viewed in appendix C3 and C7.

The Study Information Sheet needed much larger changes. The complete given test was described in this document, almost the whole document had to be changed. Only the Data Storage and Anonymity section was untouched, as the same rules apply to both tests. The differences between the documents can be seen in appendix C4 and C8.

The user profile questionnaire also required changes. This was not changed due to the change of the tests, but because during the feedback session the module leader's suggestions. As we have not previously dealt with a similar kind of questionnaire, the first version (Card Sort) obviously needed some improvements. The changes include allocating the questions into 3 different groups and these categories are marked. At the questions there were not any notes, to clarify what we want from the participant, so these had to be added. For example, at a multiple choice question, information had to be included to clarify that more than one answer may be given for those specific question. The Technology chapter only had one question, so it had to be extended by additional questions. The addition included a list of the most frequently used web browsers, which was based on an online list (The TOP Tens 2016). The edited questionnaire, similarly to the previous one, can be found on Typeform.

Link: https://fpisti35.typeform.com/to/v58hWk

The screen shot from online Typeform can be viewed in appendix C5.

It was shared on Facebook between friends. The two different questionnaire's original structure and questions can be found in appendix A2 and A3

3.2 Task Set Design

The basis of all the tests, just like for everything else, was a PACT analysis. The tasks had to be produced defined by the PACT analysis’s Activities part. However, it was not possible to fully implement the tasks, because there was no guarantee that the selected participants have an account. Even if they had accounts, due to ethical and data protection reasons, it was not possible to conduct it on personal accounts. So only tasks that did not require logging into an account, could be used.

The previous Card sort test results did not aid in finding the tasks, as only a small portion of the system was studied using this method. This test allowed to find tests that have a wider reach. A task however could be related to the previous test, because learning the service options provided by the company was a part of the activities.

Keeping the above in mind, apart from the PACT analysis other support for the study were the related lectures and exercises. Therefore, those have defined the specific criteria, which were taken into consideration.

3.2.1 The tasks

One and one groups were created for the tasks from PACT analysis based on methods described in the previous chapter. 2-3 tasks were created for each group. These tasks did not show just the most general user activities, but they showed rarely used but important goals too.

It’s a principle during tests like this not to make participants feel uncomfortable by finding the already known errors, but nevertheless a few tasks were defined, where no solution was found during using the site. These errors were not really malfunctions; rather the necessary information could not be found there. The supposition that participants will be able to find them during the test was proven right, but the detailed description is contained in the execution chapter.

The questions below were drawn up and sorted into groups based on the already written things.

Top Up

1. Find the page with Web Top Up options and define the required information to top up the account.

2. Attempt to find the Top Up limits and possible amounts.

3. Please find the Top Up minute rates in UK and compare it with other competitors.

Device Search

4. Find your most preferred manufacturer's most recent device.

5. Once you found the device, find the most important technical information about the device.

6. Attempt to trade in your current device for the device preferred by you.

Support

7. Look for the settings required to connect your current device to the internet.

8. Find the minute rate at your next holiday destination.

9. Attempt to cancel a made-up, expired subscription.

Job search

10. Look for a job in a selected central office appropriate to your education.

11. Attempt to find available retail store work in Sunderland.

Company information

12. Find the year the company was founded in the UK on the Three webpages.

13. Attempt to find the closes Three office to your home and prepare a route to that location to drive there.

14. Find the Wholesale brochure and download it.

As far as the questions revealed, those were mostly created to find the information. Tasks like this belong to the group of ‘Fact Tasks’. The tasks contain some easier and some more difficult ones, so performing them will take different durations. The whole list of tasks can be found in appendix G.

3.3 Test Procedure

The documents finished until this point already made available to perform these tests. Sufficient volunteers are available to choose from for the most suitable ones, and all of the technical devices are available for further analysis in optimal cases during performing the study. While staying on the ground of reality, it’s good if only a few persons can be found who are ready to fill the form, and it is also very good if one can be found who meets the requirements. If such a person can be found, it causes further difficulties to find the right time and place, where it can be performed. Very few people are ready to travel to a place, which is properly designed, and spend an hour there in an uncomfortable situation without any honouring.

It is not expedient to prepare serious test due to reality. A Consent Form (appendix C7) was specialised for this task, an information sheet containing the detailed description (appendix C8) and a list that exactly describes the tasks (appendix G) were made. The test tasks can be repeatedly performed in any environment, where a personal computer with Internet connection is available to browse three.co.uk site. Only the ‘Field Testing’ method made available these conditions.

The available devices only gave opportunity for audio recording. This is why the participants were asked to comment their activities and pronounce their thoughts. By using the Think-aloud) method were the later analyses possible.

Because the purpose of the study is making a website more usable than it was, it should be started from a rumour that sites have opportunities for performing further changes on them. Working out of a ‘Formative’ system was necessary with extensive tests and measurements.

3.4 Usability Measures

Due to plans and possibilities, only voice of participants being recorded, it heavily limits the possibilities of task surveying. Only ‘Performance Measures’ are possible based on the recorded audio. The ‘Cognitive Measures’ needs advanced technical background, while the ‘Satisfaction Measures’ needs precisely worked out forms and scaled answers.

The ‘Time’ proceeding gives the most information from the ‘Performance Measures’ methods. The received durations give better analysis possibilities than ‘Task Success’ proceeding about the successfulness of defined task. Knowledge of every movement would be important for ‘Errors’ method to find out what has caused the error, but it would be necessary to record the screen with every single click. The essence of ‘Navigation’ method is the precise path tracking, which is also not guaranteed with the Think-aloud method voice recording. By taking care of these, only requirements for ‘Time’ method are available, so this it must be used for measurements.

3.5 Recommendations from Pilot test

Due to plans and possibilities, only voice of participants being recorded, it heavily limits the possibilities of task surveying. Only ‘Performance Measures’ are possible based on the recorded audio. The ‘Cognitive Measures’ needs advanced technical background, while the ‘Satisfaction Measures’ needs precisely worked out forms and scaled answers.

The ‘Time’ proceeding gives the most information from the ‘Performance Measures’ methods. The received durations give better analysis possibilities than ‘Task Success’ proceeding about the successfulness of defined task. Knowledge of every movement would be important for ‘Errors’ method to find out what has caused the error, but it would be necessary to record the screen with every single click. The essence of ‘Navigation’ method is the precise path tracking, which is also not guaranteed with the Think-aloud method voice recording. By taking care of these, only requirements for ‘Time’ method are available, so this it must be used for measurements.

3.5 Working with volunteers is proven as written before by filling the form about this only by two persons. Their data can be viewed in appendix C6. Only one person of two complied the requirements of representative users. Because only one volunteer answered the question with yes about frequently visiting the websites about mobile telecommunication.

After the already described preparations, the ‘Pilot Test’ was done 27^th of April 2016, 7 PM at Sunderland, the house of participant number 005. The participant’s laptop used for test and the voice was recorded with mobile phone during the test. The participant have read the given information sheet and verified the Consent form with signature about accepting the terms and conditions and the participation in the test.

After receiving the task list, the participant read the introduction and the voice recording was started before the start of the task. The whole test was recorded in three parts because some stops were made during the progress to check the working of the recording to avoid the restarts of lacking these. The sound recordings can be found on the DVD attached to documentation.

The following remarks were made during the tests’ performance. The Think-aloud method can cause difficulties for persons who never performed it before. Some longer silent parts were recorded where the participant thought longer or forgot share the thoughts, so the resulted recording is highly incomplete. The sound recording is just a fragment of what happened during the test, so it will not give answers for any questions. More recording methods will give more help for performing similar tests. This is why it is important to record video and the screen of the used computer, but the best method that will be tracking the participant’s eye movements is the method named ‘Eye Tracking’ system. Recording made during pilots test allowed only the ‘Time’ method processing.

Another conclusion is that the test conductor gave too much help during the process, and it influenced the result. During the early stage of the test, the test participant’s attention should be attracted for a major structure error of the site. The user believed that on the bottom of the site is nothing due to the endless same coloured background, so the user only checked the upper parts of the pages and did not made any attempt for scrolling down. In those cases when the participant went into wrong direction first, later headed into the right direction.

Some of the selected tasks seemed too complicated and other ones seemed too easy. Also noticeable differences came to light during assembling the tasks and between the experiences during Pilot Test. For example, the participant could not finish the task number 2, even if the tasks that should be found are presented on the page, but not on a logical, available place for the user. The opponent of this is the question number 12, where the editor could not find the answer, but the test participant found it in only 40 seconds.

Also the draw up of tasks could be developed because the participant had to read it multiple times or go back to this during a test performing to remember the exact task description.

The table below summarizes the results of Pilot Test.

The total duration of recorded voice was about 50 minutes.

References

Hager M., Brudney J. (2011) Problems Recruiting Volunteers: Nature versus Nurture, NONPROFIT MANAGEMENT & LEADERSHIP, vol. 22, no. 2, Winter 2011 Published online in Wiley Online Library (wileyonlinelibrary.com) [ONLINE] Available from: http://www.academia.edu/1511750/Problems_Recruiting_Volunteers_Nature_versus_Nurture [ Accessed 19 April 2016].

Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods. John Wiley & Sons, New York, NY.

Reinius, J. (2011) The PACT Analysis Framework, A case study of 1177.se [ONLINE] Available from: http://www.diva-portal.org/smash/get/diva2:830667/FULLTEXT01.pdf [Accessed 15 April 2016].

The TOP Tens (2016), Best Web Browsers, [ONLINE] Available from: http://www.thetoptens.com/best-web-browsers/ [Accessed 18 April 2016].

Wentzel, J, Müller, F, Beerlage-de Jong, N, & van Gemert-Pijnen, J (2016), 'Card sorting to evaluate the robustness of the information architecture of a protocol website', International Journal Of Medical Informatics, 86, pp. 71-81, ScienceDirect, EBSCOhost, viewed 15 April 2016.

Appendices

Appendix A (PACT, User profile questionnaires)

A1 PACT (People, Activities, Contexts, Technologies)

Product Name: Three.co.uk

(Telecommunications and Internet Service Provider)

A2 User Profile Questionnaire for Card Sort

A3 User Profile Questionnaire for Usability Test

Appendix B (Card Preparation)

Appendix C (User recruitment and participant descriptions)

C1 Typeform Online Form for Card Sort

C2 Card Sort User Details

C3 Card Sort Participant Consent Forms

Evaluation of Three Telecommunication Company’s web site

Card Sort Participant Consent Form

The purpose of this study is to improve and develop the ease of use for Three Telecommunication Company’s web site. By completing different tests the study to evaluate the effictivity of the current wesite and to find possible problems. The completion of the test takes about 40 minutes in total.

Only volunteers can participate in the exercises. Volunteers will not recieve any payment. Volunteers can ask to stop the exercise or delete their data any time.

The performance of the exercises can be recorded either by digital or manual way, which are the part of the study. In these cases the personal details of volunteers are kept confidentially, and individual identifier numbers are used instead of personal details for identification. Personal details are not used for any other purposes, those are not available for third person.

If you have any questions, please do not hesitate to ask.

Please indicate that you have read and understood the study information sheet

I have read and understood the study information sheet

If you agree to participate in this study please tick the boxes below, print your name, sign and date the form.

Thank you.

Please tick the boxes below if you consent to:

The screen can be recorded

Your voice can be recorded

Your activity can be recorded in other ways (notices etc.)

Printed Name: ___________________________

Signature: ___________________________

Date: ___________________________

Participant Number: ___________________________

C4 Card Sort Study Information Sheet

Evaluation of Three Telecommunication Company’s web site

Card Sort Study Information Sheet

Evaluation of Three.co.uk

This information sheet will help a person decide whether he or she wants to participate in the study to evaluate the website's ease of use. Please read the following carefully.

Who is Included

Anyone who has filled out the 'User Profile Questionnaire' can participate in the study, based on the information given they will be added to the group of potential users. So people who frequently visit mobile communication related website and do most of the management of their mobile subscription online. Also the participant has to be willing to get involved in the study.

Aim of the Study

An important tool of our modern lives is mobile communication devices. To use this either a subscription or prepayment is required. As mobile communication is rapidly developing, continuous changes are made and more background information is attached to these changes. The contract creation and management, picking and purchasing a device, the use of other services, also the reporting of errors and information on troubleshooting have to be available to be used by returning users on the relevant pages. Gathering all this information and being able to manage them safely and securely is the main goal of these websites. This study is trying improve on the ease of use and by making a more intuitive and easier menu system for three.co.uk.

What You Will be Asked to Do

A participant will be asked to rearrange the current three.co.uk menu by cards, modify them or even add new cards. These adjustments will be used to make the webpage structure easier to use. The process of the modifications and it's final state will be recorded. The recording may be done by using digital audio, image archives or manually written notes. The study will take place at a time arranged with the participant at a location where the study can be carried out uninterrupted. It will take approximately 40 minutes.

Data Storage and Anonymity

Recorded personal information during the study will be handled abiding by related rules and regulations. The names of the participants will be changed to identification numbers and these numbers will be used for the evaluation. The personal information will only be available to the conductors of the study and the people who evaluate the results. This information will not be shared with a third party. Any recorded sound or footage will not be shared electronically.

Benefit / Risk of Participation

The result of the study will rework the structure of three.co.uk, thus making it easier to use. Taking part in the study does not risk the participant in any way. The participants are not forced to complete the study, they can request to stop the study and the destruction of their data.

Approval of the Study

This study has been approved by the University of Sunderland Research Ethics Committee.

Contact Details for Further Information

If you have any questions about the study or issues you want to discuss, please contact the Lead investigator Dr Sharon McDonald.

Telephone: +44(0) 191 515 3278

Email: sharon,This email address is being protected from spambots. You need JavaScript enabled to view it.

You can also contact:

Dr. E. Evans

Chairperson of Research Ethics Committee

Faculty of Applied Science

University of Sunderland

Sunderland

Email: This email address is being protected from spambots. You need JavaScript enabled to view it.

C5 Typeform Online Form for User Experience Test

C6 User Experience Test User Details

C7 User Experience Test Participant Consent Forms

Evaluation of Three Telecommunication Company’s web site

User Experience Test Participant Consent Form

Purpose of the study is developing the Three Telecommunication Company’s website. An easy-to-use website is a necessary cirteria for examining the aspects of users. So this is why this study wants to test most general functions by represantative users. Participants must perform defined tasks and test the site’s usability. The duration of the whole test is about 60 minutes.

Only volunteers can participate in the exercises. Volunteers will not recieve any payment. Volunteers can ask to stop the exercise or delete their data any time.

The performance of the exercises can be recorded either by digital or manual way, which are the part of the study. In these cases the personal details of volunteers are kept confidentially, and individual identifier numbers are used instead of personal details for identification. Personal details are not used for any other purposes, those are not available for third person.

If you have any questions, please do not hesitate to ask.

Please indicate that you have read and understood the study information sheet

I have read and understood the study information sheet

If you agree to participate in this study please tick the boxes below, print your name, sign and date the form.

Thank you.

Please tick the boxes below if you consent to:

The screen can be recorded

Your voice can be recorded

Your activity can be recorded in other ways (notices etc.)

Printed Name: ___________________________

Signature: ___________________________

Date: ___________________________

Participant Number: ___________________________

C8 User Experience Test Study Information Sheet

Evaluation of Three Telecommunication Company’s web site

User Experience Test Information Sheet

Participant Information Card Sort Evaluation of Three.co.uk

This information page wants to help the user to decide whether to participate in the website usability test. This is why it informs about the study, participants, goals, tests to perform and the foreseeable advantages and risks. Please read below carefully.

Who is Included

Anybody can be subject of the study who voluntarily participates in the study. Further condition is filling the related ‘User Profile Questionnaire’ and complying with its requirements. These requirements are serving the selection of potential users. That person can participate in the test, who frequently visits sites about mobile communication and performs various tasks there to achieve different goals.

Aim of the Study

Nowadays mobile phones became integral part of our lives. These days they are more than simple phones. The extra functions require extra services. The clients must be informed about this and support about usage must be given to them. This is mostly done by online user profiles, where the clients can manage the services. The online systems that provides these services became more complex, which made their usage more difficult. The purpose of this study is providing help for building a system, which is big enough to contain all information and services, but it will not go to the expense of usability.

What You Will be Asked to Do

The participant will be asked to try performing the defined tasks in the task list while browsing the three.co.uk website. The tasks want to simulate the ordinary user behaviour and test the usage of site. Only the reachable goals are specified, but the path leading to them must be found by the users. Recording of the whole test is necessary for evaluating the whole test, which can be done by picture or audio recording. The knowledge of the person’s thoughts is important, so the user will be asked to speak out the questions, motivations and chooses loudly.

Data Storage and Anonymity

Recorded personal information during the study will be handled abiding by related rules and regulations. The names of the participants will be changed to identification numbers and these numbers will be used for the evaluation. The personal information will only be available to the conductors of the study and the people who evaluate the results. This information will not be shared with a third party. Any recorded sound or footage will not be shared electronically.

Benefit / Risk of Participation

Purpose of the study is mapping and testing the functioning of an already existing system. Exploring the experienced errors and insufficiencies will provide ability to repair the system. The main aspects are the safety of participants, so it will not risk them or make them uncomfortable. If something like these will happen, the test can be aborted anytime.

Approval of the Study

This study has been approved by the University of Sunderland Research Ethics Committee.

Contact Details for Further Information

If you have any questions about the study or issues you want to discuss, please contact the Lead investigator Dr Sharon McDonald.

Telephone: +44(0) 191 515 3278

Email: sharon,This email address is being protected from spambots. You need JavaScript enabled to view it.

You can also contact:

Dr. E. Evans

Chairperson of Research Ethics Committee

Faculty of Applied Science

University of Sunderland

Sunderland

Email: This email address is being protected from spambots. You need JavaScript enabled to view it.

Appendix D (Card Sort Procedure)

D1 Card Sort Tasks List

The purpose of the study is to make the Three.co.uk website easier to use. The task of the participant is to place 30 cards made from the current website in a logical structure.

During the rearranging process the participant can add additional cards or remove existing ones. The title and the contents of the cards can be freely adjusted as well. The main point is for the participant to create an easy to use structure.

During the rearrangement it is advised to sort the cards into different groups and to put them into order and then titles to the main groups and then naming the sub-groups. However this is only an advice, the participant can freely decide to do it differently in a way that fits the participant best.

Thank you for the participation in the study.

D2 Card Sort One pictures and diagram

D3 Card Sort Two pictures and diagram

D4 Card Sort Three pictures and diagram

Appendix E (Current Architecture)

Appendix F (Product Inspections)

F1 Meta Tags

Main Pages

Careers Centre

About Us

Wholesale

F2 Consistency and Standards

About Us

Main pages

Careers Centre

Wholesale

F3 Links for downloads

F4 Layout problems

F5 Usability Problem Reports

F5.1 Problem ID: 01

A. Problem Description

F5.2 Problem ID: 02

B. Problem Description

Appendix G (User Experience Test Tasks List)

Evaluation of Three Telecommunication Company’s web site

User Experience Test Tasks List

The goal of the test is to assess the usability of the Three.co.uk website. Therefore this has to be carried out by simulating the general tasks completed by the visitors. During the objectives it is important to record the thoughts and experiences, they have to be spoken out and each action needs to be commented on. For measurability and repeatability it is essential to record sound or video.

Please keep all of the above in mind, while completing the following tasks.

Top Up

1. Find the page with Web Top Up options and define the required information to top up the account.

2. Attempt to find the Top Up limits and possible amounts.

3. Please find the Top Up minute rates in UK and compare it with other competitors.

Device Search

4. Find your most preferred manufacturer's most recent device.

5. Once you found the device, find the most important technical information about the device.

6. Attempt to trade in your current device for the device preferred by you.

Support

7. Look for the settings required to connect your current device to the internet.

8. Find the minute rate at your next holiday destination.

9. Attempt to cancel a made-up, expired subscription.

Job search

10. Look for a job in a selected central office appropriate to your education.

11. Attempt to find available retail store work in Sunderland.

Company information

12. Find the year the company was founded in the UK on the Three webpages.

13. Attempt to find the closes Three office to your home and prepare a route to that location to drive there.

14. Find the Wholesale brochure and download it.

We hope every tasks was completed, within a short period of time and we are thankful for your participation!

Appendix H (Printed Cards)

The original cards used for the Card Sort test can be found in the envelope on the back.

Appendix I (DVD)

The following structure is used to store documents and files created during the project on the provided DVD