Usability testing of Firewall

Method

• Test item: Cockpit - Firewall
• Method: Moderated remote usability test via BlueJeans ~ 1 hour
• 6 Participants: 4 internals and 2 externals (5 novices, 1 expert)
• Researcher: Sara Chizari
• Notes: Andreas Nilsson

Research Goal

The overarching research goal was to identify usability issues with the Firewall section and understand participants’ expectations as well as their pain points.

Recommendations

• Consider adding more columns on the Allowed Services table to show more information such whether the service is live/active and also if the rule is for the sending traffic or receiving traffic. issue #9047
• Considering allowing user to sort and filter the Allowed Services table.
• Consider allowing user to create custom firewall rules. issue #9048
• Consider allowing deactivating/disabling firewall rules. issue #9053
• Consider asking user if they are sure they want to add and/or delete a service.
• Consider showing a “loading state” for the pages that need some time to load. issue #9055

Summary of the observed usability issues and their frequencies

• Not clear if the allowed services are unidirectional or bidirectional. Not clear if the firewall is allowing sending traffic or receiving traffic from the rules listed on the page. (3x)
• The system doesn't allow deactivating/ disabling the firewall rules. (3x)
• The Add Services function doesn't allow adding custom services (i.e. a service that doesn't exist already). (3x)
• Not clear if adding a service will also activates the service. Participants assumed adding a service using Add Services dialog doesn't start the service, it only lets the service through the firewall. (3x)
• Wanted the system to ask for reassurance before allowing a service. (3x)
• Wanted a clear feedback from the system that the service is added and whether it is active/live. (3x) issue #9054
• Wanted the system to ask for reassurance before deleting a service. (3x)
• (Observation) Wanted to validate/confirm firewall changes/requests in Logs. (3x)
• (Observation - Firewall page) There is a mismatch between the terminology used and the function of 'Add Services' on Firewall page. The 'Add service' dialog allows adding 'ports' without providing any feedback on whether the associated service is live or active. (3x)
• Wanted to sort the list of Allowed Services. (2x)
• Wanted to see a list of possible/blocked services as well as allowed services. (2x)
• Wanted a clear feedback (but not a pop-up) from the system about what service is deleted. (2x)
• There is no way to undo or see history or what services used to be allowed. (2x)
• (Observation- Firewall page) Checked Logs first to see firewall requests. (2x)
• (Observation- Firewall page) Checked Services page to see if the added firewall service is active. (2x)
• (Observation - Cockpit) By navigating to a subsection of a page the state of the page remains the same as the sub-section. Even logging out/in doesn't help. (2x)
• The '5 Active Rules' is not very informative. Wanted to see more information (allow expansion for more information). (2x)
• The port numbers may not have clear meaning to all sysadmins. (1x)
• Wanted to be able to export a report of firewalls. (1x)
• Wanted to be able to set a template for a certain set of templates. (1x)
• Not clear why the name of services are bold. (1x)
• Wanted to see the source of the services on the list. (1x)
• Expected to see two tables. One for allowed ports, and one for allowed services. (1x)
• The title of the list says Allowed Services, but it actually is the list of Allowed ports. (1x)
• Wanted to see additional information such as ports and protocol about services in the Add Services dialog [watch]. (1x)
• “As a sysadmin, this is why I don't use these kind of tools, because I don't get enough information. I end up guessing. I want to see more in detail what the changes are going to be to the system.” (1x)
• (Observation) Looked under Services > Sockets for a firewall behaviour. (1x)
• Wanted to see information about processes that are taking up memory and CPU usage. (1x)
• Didn’t expect to see 'Firewall' as a link. (1x)
• Didn’t expect Firewall to be part of Networking. (1x)
• It is not obvious the graphs at the top are clickable. (1x)
• Wanted more information about the logs. Not clear if the logs is only showing info about networkmanager and dhcp services. (1x)
• Wanted to see error messages in a different color in logs. (1x)
• Didn’t expect to see the receiving scale goes up very high. (1x)
• (Networking and Firewall pages) Repetitive Firewall On/Off toggle on both pages. (1x)
• (Observation - Networking page) The distinction between Firewall link and 5 Active Rules is not clear. (1x)
• Network logs doesn't reflect the changes to the firewall rules. (1x)

Task overview

Legend:

• Participant had difficulty (based on observation)

• Participant had no difficulty (based on observation)

(+-++-+) Task 1: Navigate to where you can see the status of the firewall. (66% passed, ease: 4.8/5, 13 observed issues)

(++++++) Task 2: Identify allowed services (100% passed, ease: 4.7/ 5, 8 observed issues)

(+-++++) Task 3: Allow the httpd service to send traffic through the firewall (83% passed, ease: 4.8/ 5, 11 observed issues)

(++++++) Task 4: Stop allowing the IMAP service to send traffic through the firewall (100% passed, ease: 4.7/ 5, 4 observed issues)

Set up:

• Check out PR 8032
• ./bots/image-customize -s firewall-usabilitytest.sh fedora-27
• vm-run fedora-27 --network