Horizon-usability-test-findings

Horizon Usability Testing Prioritized Findings
The following is a summary of the findings from the usability testing for Horizon (~Icehouse release) that began the week of February 24 2014. Based on the study, we identified several areas of improvement for Horizon that were then prioritized by the OpenStack UX Team in order to create actionable items.

High Priority
 * Improve error messages and error message catalog - 100% - Blueprint (guidelines) and Blueprint (fix current)
 * Fix Launch Instance workflow for end user and power user - 83% - Blueprint
 * Improve informational help information about form fields - 83% Blueprint
 * Fix terminology (e.g. launch instance, boot, shutoff, shutdown, etc.) - 67% - Blueprint
 * Show details for key pair and network in Launch Instance workflow - 50%
 * Recommend a new Information Architecture - 50%

High / Medium Priority (Tie)
 * Create UI guidelines (of best practices) for Developers to use - 33% + 33% - Blueprint

Medium
 * Improve Online Help (?) - 67%
 * Provide clearer indication the application is working after clicking a button and the application doesn’t respond immediately. - 50%
 * Ensure consistency of network selection (Drag and drop of networks very inconsistent from the other pieces of the launch instance modal) - 50%
 * Instance table - action buttons recommendations (inconsistent visualization/selection of actions) - 50%
 * Suggest defaults for the forms entry fields - 50%

Medium / Low Priority (Tie)
 * Solution to allow users to edit the network an instance after launching instance - 33% + 33%

Low
 * Resolve confusion around the split inline actions button - 83%
 * Solution to explain what is the Instance Boot Source field in Create Instance modal - 83%
 * Provide description/high level information about flavors for flavor selection - 67%
 * Make sorting clearer visually - 50%
 * Provide solution for subnet checkbox to improve usability - 33%

High / Medium / Low (Tie)
 * Provide Image information details during image selection - 33% + 33% + 33%

High / Nice to Have (Tie)
 * Provide "Save as Draft" in the wizard - 33% + 33%

Medium / Nice to Have (Tie)
 * Change security group default name to "Default security group" - 33% + 33%

Horizon Usability Testing Observations and Details
[1] Workflow of creating a first instance is confusing. Two major stumbling blocks… a) Choosing a Keypair…         -A valid keypair is needed to successfully create an instance.          -There is no way to actually add a keypair straight from the launch instance modal. Only import.          -The user needs to back out of launching an instance and go to a completely different section to create a keypair.     b) Choosing a network. -No network created by default. -Even if there is a network, nothing selected by default. -The user needs to back out of launching an instance and go to a completely different section to create a network.

[2] New users looking for informational help information about form fields that will them understand certain fields. -Availability Zone. -Flavors (Specifically the m1 meaning) -Boot From -Admin state in Network creation. -Create subnet. -Disable gateway. -Power State (Field on Instance table is confusing considering VMs don’t technically have power. User suggested just “State”) -Ingress/Egress on Security Rules -other ones not from the test: -Security groups -Pause vs. suspend -Injected files -Gigabytes (for storage, instead of using stoiage)
 * When selecting the keypair and network, there was no information high-level details to help the user pick the keypair and network other than the name given to those 2 objects.

[3] Certain features/functionality that are available to “Self-Service” Users are concerning to Administrators we talked with. -Managing Networks.

[4] Create Subnet checkbox confused most users. Not sure why this was checked by default if it isn’t required to create network. This flow could be drastically improved to make creating a Network easier.

[5] No clear indication that the application is working after clicking a button and the application doesn’t respond immediately.

[6] Users expects to be able to edit the network an instance is on after launching the instance.

[7] Users unsure as to why they can’t go back in and edit the network configuration that they were able to enter upon creation.

[8] While selecting network in Launch Instance workflow, users would like to see more details about the Networks they can choose from.(e.g. dev network, qa network, prod network, private, public network, etc.)

[9] A bit of confusion around the split inline actions button. User missed the button called out on the right, was just looking in the drop down list for all actions.

[10] Some users were confused about the term “Launch” in the action launching an instance. They mentioned that they thought they would need to create the instance before launching it when in fact OpenStack creates and launches all in one step.

[11] Sorting functionality on tables wasn’t obvious to some users.

[12] For the default security group, rather than called it “default” user would suggest “Default security group 1” or even just “security group 1” to make it more clear what is being shown in this list.

[13] Based on the image selected, user would like to know how much memory, root disk would be used -- to verify that it would “fit.” Image information details would be helpful in image selection.

[14] Drag and drop of networks very inconsistent from the other pieces of the launch instance modal.

[15] Some of the Spanish translations didn't seem quite right. Sabor for Flavor, user said didn’t make sense. Not sure if this means theres a problem throughout.

[16] Some of the Spanish localization was inconsistent. Sometimes description / infotips were in English, and sometimes Spanish. Sometimes the action buttons were in Spanish, but the dropdown list for the actions were in English.

[17] Other terminology - e.g. boot vs launch, shutoff vs shutdown, status vs power state, launch instance (vs. create and launch instance) - was confusing, we could do a better job with terminology

[18] For the instance table, the action button should have the most common actions at the top of the list in the dropdown listbox, and there are some other actions shown next to the action button that the user expected to be with the other actions. User described the current actions (“Create snapshot”, “Associate Floating IP”) listed as peers with the action button as inconsistent.

High Level Concepts
 * Create instance workflow for a first-time end-user is confusing and also relies on the end-user having to know an awful lot to do so.
 * Piet - Content/information challenges: Not enough information provided to users during wizards, error codes, etc.  Particular relevant for Self-Service End Users rather than Admins
 * Many self-service users only use Horizon for minutes at time and very infrequently, so ease of use and minimal hurdles is very important
 * Making snapshots of an instance is a common task
 * Users don’t typically take the time to type in description when creating an instance.
 * Flavor sizes is not obvious to the self-service cloud end user from the dropdown listbox (until you look on the right side).
 * Need some information to help explain what is the Instance Boot Source field in Create Instance modal.
 * Vague error message (when creating dup rules) with little indication to help user troubleshoot and resolve the errors (error messaging is weak throughout Horizon)
 * Currently every component owns their own error messages. There’s an effort underway to address and improve error messages.
 * Some questions / concerns about the IA, e.g. why is volumes, access and security under compute?
 * End user being asked more information than needed (e.g. creating network)
 * Insufficient defaults used (e.g. keypairs, networks)
 * Terminology is inconsistent and confusing
 * There’s currently an assumption that the Self-service Cloud End User / Consumer has to have a previously created key pair and has to create a network. It was cited in a couple of sessions that the end user typically does not create either and that it’s provided / known by the Cloud Administrator.
 * Note: For self-service users, it should be fine for the user to create key pairs.
 * Need to reduce what the self-service end-user gets prompted for for creating a VM
 * Fixed network flavors for Administrators?
 * IBM uses a different UI based on OpenStack, does not use Horizon
 * Jeff: In the IBM private cloud implementation, user puts in subnet and gateway and the System assigns the rest of the network values
 * It would be good to think about different network scenarios and then let an administrator set these up for easy selct ion  .For example, many users might not need single networks or admins might just want to expose a set of networks to choose from.
 * Lack of OLH, a lot of abstract concepts, e.g. security groups -- lack of explanation in the panel
 * Lack of consistency in presence of OLH throughout the UI
 * More monitoring and troubleshooting needed for Overview pages
 * Lack of automated discovery of network makes the network configuration more an issue as it forces a lot of manual entries.
 * more opportunities for errors
 * Lack “Save as Draft” button for the wizards when user has to exit the “Launch Instance” to create key pairs and networks

Return to Horizon Usability Testing