TAK+Usability+Test

=**Usability Test:**=

**Tasks:**

 * Note**: User starts out on the "Search" page.


 * 1) Login**: At the moment, login will take any username or password, so this task involved users locating the 'Sign In' link, typing in an arbitrary username and password, and pressing 'enter'. They have succeeded once their selected username is showing in the top right corner of the header bar.


 * 2) Add A Command**: This step involved the user adding a command to the database. They will start out on the search page, so they have to find the link to the add a command page, then fill out the appropriate information on it, then submit the command to the database. They have succeeded when the "Command Added" confirmation message appears on the screen.


 * 3) Search for the Command**: This step involves the user searching for the command they've just added to the database. As this step follows the Add A Command step, they will have to find the link to the search page, then enter the appropriate information, then submit the search and find their command in the results. They have succeeded when they have found the listing for the command they added previously.


 * 4) Modify the Search**: This step involves the user modifying the parameters of the search they just entered. They may either add additional terms (e.g. program, OS), or search for a different command. They have succeeded once the new search results are showing.

Survey Data:
We collected the gender, age, time spent on the internet each day, and self-ascribed computer literacy of each user tested (1-7 scale). Link to Survey

Survey Questions:
1) What do you like most about The Any Key? 2) What do you dislike most about The Any Key? 3) How easy is it to use The Any Key (1-5 scale)? 4) How professional is the look and feel of The Any Key (1-5 scale)? 5) What changes or additional features would most improve The Any Key? 6) Any other comments?

First User:

 * Details:** Female, 18-20, 4-6 hours of internet use per week, computer literacy of 4


 * Test:**

Task 1: This user found the 'Sign In' button immediately, paused after entering username and password to look for a submit button before pressing enter.

Task 2: User found the add a command button within a few seconds. Had to inform the user that the 'Command' box was for the name of the command rather than the shortcut. User asked what to put for the name for the command they had chosen (the shortcut to get to the Windows Task Manager program), which suggests we may want to have specific standards for shortcuts that don't have an obvious name (e.g. copy), or perhaps a dropdown to select if the command opens a program, performs an action, etc. User at first only filled in a couple of pieces of information before deciding to submit, interviewer advised the user to fill in as many of the categories as they knew. This suggests we may want to have a pop-up if all forms are not filled to remind the user to fill in as many fields as they can before submitting a command to the database. User also was uncertain what to input for program, since the program was the same as the operating system. This suggests that we should have some particular method for specifying OS-level commands, perhaps a check-box, or an instruction to enter the OS as the program as well as the OS.

Task 3: User search went perfectly. User entered "Taks Manager" into the 'command' text box and searched. Returned their added shortcut information.

Task 4: Modified search went perfectly. User specified the program for their command and performed their search again, returning the same result.

Overall the test went well, with a few hiccups.

Issues raised:
 * Need a sign in button on entering username and password
 * Need to specify that "Command" means "The name of the command"
 * Need to specify how to format the command name if it does something without a usual name, like open the task manager
 * Need to prompt user to fill in as much information as possible before submitting a new command
 * Need a way to specify what to do (if anything) for 'program' box if command is an operating system-level command

Second User:

 * Details:** Male, 18-20, 6-8 hours internet use/week, computer literacy of 4


 * Test:**

Task 1: Same as last user, found the sign-in button quickly, paused looking for sign-in button and then pressed enter.

Task 2: User was instructed to add a command to the database, and at first tried to figure out how to do that on the Search page. This suggests we should make it more obvious that the search page is for search, so user knows to go to a different page for the add a command interface. Once there, as with first user, had to specify that command was the name of the command and not the shortcut. Also had to specify that user should put the OS for the program as well as the OS segment since it is on OS-level command. Also had to specify that the site does not do a keyboard capture of the command for the Shortcut box.Other than that, went smoothly. User entered information and submitted it successfully.

Task 3: User found the search page link quickly. Searched originally for their command by the name "Screencap", though they had entered "Screencap (screenshot)" as the name. No results were produced, because the search only finds exact matches at the moment. This suggests we should modify the search to allow partially completed names to match, or enforce a strict standard for naming the commands.

Task 4: Modified search went perfectly. User specified the program and operating system for their command and performed their search again, returning the same result.

Issues raised:
 * Need a sign in button on entering username and password
 * Need to specify that "Command" means "The name of the command"
 * Need to specify how to deal with program for OS-level commands
 * Need to specify how to format command name, or implement a search that matches with partial completions
 * Though the user searched correctly for program and operating system, they input Mac OS X, where others have used Mac or OSX, etc. May need to standardize this to avoid missing matches due to different formatting of OS names

Third User:

 * Details:** Female, 18-20, 4-6 hours internet use/week, computer literacy of 3


 * Test:**

Task 1: Had to look around a little for the login before finding it in the top bar. Entered username and password, short pause, pressed enter.

Task 2: Command adding went smoothly (again with specifying that command meant "command name") except for the shortcut box, where the user first tried to input with keyboard capture, and then had to inquire about what format to use for the command they were adding. This suggests we should add some sort of template or example shortcut entry to demonstrate what the user ought to do format-wise. User also was unsure as to what operating system to input, since their command ("paste") is frequently used in both Windows and Mac operating systems. Since shortcut is different for each, interviewer instructed her to enter the one that corresponded to the shortcut she was putting in the shortcut box. This suggests we should specify what to do if a command is common for many operating systems.

Task 3: User found the search page and executed the search for their command without a hitch.

Task 4: Modified search went perfectly. User specified the program in additino to the name for their command and performed their search again, returning the same result.

Issues raised:
 * Need a sign in button on entering username and password
 * Need to specify that "Command" means "The name of the command"
 * Need to specify how to deal with program for OS-level commands, including those that are shared between many OSes
 * Specify that shortcut input does not to key capture, or implement key capture
 * Need to specify how to format the shortcut input

Fourth User:

 * Details:** Female, 18-20, 4-6 hours internet use/week, computer literacy of 5


 * Test:**

Task 1: Found login quickly. entered username and password, pressed enter.

Task 2: Command adding again went smoothly with the clarification on Command meaning Command Name. Same specifying the OS issue as before, because the user entered an OS-level command. Also same issues on specifying that the program does not to keyboard capture. Other than that, all entry went smoothly, all info filled out, including description, and submission was successful.

Task 3: User found the search page and executed the search for their command without a hitch.

Task 4: User had already specified all details of the search for the previous step, and so was instructed to search for a different command rather than modify their previous search. This search was originally produced with an inexact name types, and so returned nothing, but upon correction, went correctly.

Issues raised:
 * Need to specify that "Command" means "The name of the command"
 * Need to specify how to deal with program for OS-level commands
 * Specify that shortcut input does not to key capture, or implement key capture

Fifth User:

 * Details:** Female, 18-20, 2-4 hours internet use/week, computer literacy of 4


 * Test:**

Task 1: This user needed help finding the Sign In link, originally started entering information into the search text boxes. Once located, entered username and password appropriately, paused before pressing enter, and then sign in was successful.

Task 2: Command adding went smoothly for the most part (again with specifying that command meant "command name"). This user, however, chose to add a command ("copy") that already existed in the database. User was advised to pick a different command, but this illustrated the necessity of adding a check for previously existing commands matching the parameters of the command to be added to make sure duplicates are not added to the database. User also wasn't sure what to put for program for the command they chose, which was OS-level, so that had to be specified again.

Task 3: User found the search page and executed the search for their command without a hitch.

Task 4: Modified search went perfectly. User specified the program in addition to the name for their command and performed their search again, returning the same result.

Issues raised:
 * Need a sign in button on entering username and password
 * Need to specify that "Command" means "The name of the command"
 * Need to specify how to deal with program for OS-level commands
 * Need to have a way of notifying user if they are adding a command that has already been added to the database before

Sixth User:
**Details:** Female, 18-20, 2-4 hours internet use/week, computer literacy of 3

**Test:**

Task 1 (Login): The user found the “Sign in” button fairly quickly and logged in without issue.

Task 2 (Add A Command): The user had trouble finding the link to the “add command” page. They eventually found it but commented that it should be “darker”. The user was unsure of what command they wanted to enter, and ended up entering a non-existent “joke” command. When it came time to enter the shortcut, they came up with an arbitrary shortcut but tried to enter it by typing the actual shortcut keys instead of writing their names. They soon realized and corrected this mistake and submitted the command soon after.

Task 3 (Search for the Command): The user was able to quickly navigate back to the search page, enter the command name, and hit the search button. There were no issues with the search.

Task 4 (Modify the Search): The user initially was unsure of what they were meant to search for, but eventually settled for searching for “copy”. I did notice that they instinctively hit “enter” after typing the command, which our implementation does not support as submitting the search, but they quickly clicked on the submit button when that didn't work.

Issues raised:
 * “Add a Command” link needs to be clearer and easier to notice
 * Specify that shortcut input does not accept key capture, or implement key capture
 * Expected input not clear for “add a command” fields

Seventh User:
**Details:** Female, 21-29, 4-6 hours internet use/week, computer literacy of 3

**Test:**

Task 1 (Login): The user found the “Sign in” button fairly quickly and logged in without issue.

Task 2 (Add A Command): The user had trouble getting to the “add command” page, they were not initially prompted to look for a link to get there, but after being told to look for a link they were able to get to the “add command” page. They were initially unsure about what to put in each of the fields, but after a description of what the fields represented as it relates to keyboard shortcuts, they came up with a command to enter. When it came to entering the shortcut, they initially tried to enter the shortcut by hitting the shortcut keys themselves, and had to be told to write out the words normally. Afterward they submitted the command without issue.

Task 3 (Search for the Command): The user was able to quickly navigate back to the search page, enter the command name, and hit the search button. There were no issues with the search.

<span style="font-family: Arial,Helvetica,sans-serif;">Task 4 (Modify the Search): The user had a little trouble coming up with another command to search for, but eventually completed the search without issue.

<span style="font-family: Arial,Helvetica,sans-serif;">Issues raised:
 * <span style="font-family: Arial,Helvetica,sans-serif;">“Add a Command” link needs to be clearer and easier to notice
 * <span style="color: #000000; font-family: Arial,Helvetica,sans-serif;">Specify that shortcut input does not accept key capture, or implement key capture
 * <span style="color: #000000; font-family: Arial,Helvetica,sans-serif;">Expected input not clear for “add a command” fields

<span style="font-family: Arial,Helvetica,sans-serif;">Eighth User:
<span style="font-family: Arial,Helvetica,sans-serif;">**Details:** Female, 50-59, 2-4 hours internet use/week, computer literacy of 4

<span style="font-family: Arial,Helvetica,sans-serif;">**Test:**

<span style="font-family: Arial,Helvetica,sans-serif;">Task 1 (Login): The user took a little while to find the “sign in” button, but eventually found it. After entering a user name and password, they looked for a button to submit their sign in, and had to be prompted to hit enter.

<span style="font-family: Arial,Helvetica,sans-serif;">Task 2 (Add A Command): The user had trouble getting to the “add command” page, and had to be directed to the link. On the “add command” page, the user didn't have any problems entering the command until the shortcut, for which they initially tried to press the keys for the shortcut, before being informed to write out the shortcut. They were also unsure about the way to word the shortcut, but eventually submitted the command.

<span style="font-family: Arial,Helvetica,sans-serif;">Task 3 (Search for the Command): The user was able to navigate back to the search page, enter the command name, and hit the search button. There were no issues with the search.

<span style="font-family: Arial,Helvetica,sans-serif;">Task 4 (Modify the Search): The user had to think a bit to come up with another command to search for, but eventually completed the search without issue.

<span style="font-family: Arial,Helvetica,sans-serif;">Issues raised:
 * <span style="font-family: Arial,Helvetica,sans-serif;">Sign in mechanism not immediately apparent
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need a sign in button on entering username and password
 * <span style="font-family: Arial,Helvetica,sans-serif;">“Add a Command” link needs to be clearer and easier to notice
 * <span style="font-family: Arial,Helvetica,sans-serif;">Specify that shortcut input does not accept key capture, or implement key capture
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need to specify how to format the shortcut input

<span style="font-family: Arial,Helvetica,sans-serif;">Ninth User:
<span style="font-family: Arial,Helvetica,sans-serif;">**Details:** Male, 50-59, 0-2 hours internet use/week, computer literacy of 3

<span style="font-family: Arial,Helvetica,sans-serif;">**Test:**

<span style="font-family: Arial,Helvetica,sans-serif;">Task 1 (Login): The user couldn't figure out how to log in, and eventually had to be directed to the sign in button. They also had to be told to press enter to sign in.

<span style="font-family: Arial,Helvetica,sans-serif;">Task 2 (Add A Command): The user had to be directed to the “add a command” link. The user needed to be informed that “Command” meant the name of the command, and that “OS” meant operating system. Once there, the user didn't have any problems entering a command.

<span style="font-family: Arial,Helvetica,sans-serif;">Task 3 (Search for the Command): The user quickly navigated back to the search page, entered the command name, and hit the search button. There were no issues with the search.

<span style="font-family: Arial,Helvetica,sans-serif;">Task 4 (Modify the Search): The user had to think for a little while about how to modify the search, but eventually entered another command and completed the search without issue.

<span style="font-family: Arial,Helvetica,sans-serif;">Issues raised:
 * <span style="font-family: Arial,Helvetica,sans-serif;">Sign in mechanism not immediately apparent
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need a sign in button on entering username and password
 * <span style="font-family: Arial,Helvetica,sans-serif;">“Add a Command” link needs to be clearer and easier to notice
 * <span style="color: #000000; font-family: Arial,Helvetica,sans-serif;">Expected input not clear for “add a command” fields

**Summary:**
(Note: statistics are not practical with such a small sample size, but specific numbers are given)

There were many issues raised during the usability tests, and none of the nine users went through the process without some issues. Six users had an issue with sign in, generally suggesting that there was a need for a clickable button. Four users expressed that the link from the home/search page to the "add a command" page was not easy enough to see initially. They generally implied that it should be "darker" and more prominent. Interestingly, while the link from the add page to the search page was identical, there were no issues or complaints about it, as the user already had experience from the initial link.

When it came to adding a command, all of the users had some issues related to their not knowing exactly what to type in the input fields. A common issue was that "command" was ambiguous and should clearly indicate that it is asking for the name of the command. Another was that it was unclear what to do with the "program" input box if the command was at the OS level. Multiple users also had issues with how to format their input. The exact wording for certain command names, operating systems, or even programs can vary, and the fact that the prototype didn't account for that was an issue. It was also an issue that there was no standard format for specifying the shortcut itself. Five of the users instinctively tried to enter the shortcut by hitting the shortcut keys themselves, implying that a key capture system would be useful but that even without it it should be clear that our implementation doesn't support it.

There were a few other issues raised as well, such as what to do about a user attempting to add a command that already exists in the database, and what to do if users try to submit a command without filling out all the fields. There were also a couple comments on the "bland' colors and "light" text. A couple users expressed an overall desire for more description and help options, saying for example that there was a "lack of description", that there should be "a roll-over for each search box/command, to further explain" or "a help/info page for certain jargon that others might not know".

Despite the large number of issues, many users expressed an overall positive opinion of The Any Key. Five users said that it was "moderately easy" to use, and four said that it was "very easy", with only "extremely easy" as a better option. Four users said that the look and feel was "moderately professional", three said that it was "very professional", and two said that it was "extremely professional". No user selected an option below "moderate" for either question. There were comments that the design was "simple", "creative" and "useful", with one user commenting that they "would definitely use it".

**<span style="color: #000000; font-family: 'Times New Roman',serif;">I temized list of problems and fixes (priority):**

 * <span style="font-family: Arial,Helvetica,sans-serif;">Need a sign in button on entering username and password (4)
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need to make what the input labels are asking for as clear and unambiguous as possible (5)
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need to clearly specify the format for all input, particularly for the shortcut (4)
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need to do more than simple string matching for searching, to allow for unpredictable input from users with regard to things like command names and operating system names (4)
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need to prevent users from submitting a new command without filling out all required fields (3)
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need to determine and specify what to do for “program” input for OS commands (3)
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need to specify that shortcut input does not accept key capture, or implement key capture (5)
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need a way to deal with users attempting to add commands that already exist in the database (4)
 * <span style="font-family: Arial,Helvetica,sans-serif;">Need to make buttons and links more visible (3)