User research and testing of cybersecurity solutions for an American company
Note: Due to the signed NDA, there are no print screens, specific data and no company name in this study.
Introduction
The Brno branch of the American company recently launched an innovative cyber security solution that aims to increase the efficiency and security of user interactions through automated and manual ticketing systems. This case study focuses on user research that was conducted to optimize the usability and effectiveness of this solution.
Problem identification
Although it was assumed that users would prefer automated ticketing that could handle tickets in minutes, it turned out that most users used "Other" tickets. This trend led to the need for manual ticket processing by the international OPS team, which caused delays and impaired efficiency. The challenge was clear: understand why users avoid the automated system and gain qualitative insights for improvement.
Primary research
Due to limitations, it was not possible to deploy any user behavior tracking tool on the system. The only statistical data available was the number of tickets marked as "Other". It was necessary to contact foreign employees who were in daily contact with the users and get qualitative information from them about who the end users are and why they prefer this category. Due to the ban on questionnaire surveys, it was only possible to find out more information through interviews and testing.
Establishing a hypothesis
Based on primary research, we identified two basic personas – technical and non-technical. For each, we've described their typical work day, level of system familiarity, and the most common problems when using a cybersecurity solution.
The hypothesis was that the technical persona does not enter automatic tickets because they do not know the system and do not know how to use it effectively. For non-technical users, we tried to find out whether they had sufficient training available regarding automated tickets, or whether they prefer manual ticketing due to time pressure or excess work. The testing aimed to check how intuitive and user-friendly the system is.
User testing
User testing took place with twelve users from OPS teams and from customer cyber security teams. We filmed more than 12 hours of testing, focused on the reasons for the preference of the "Other" category and the overall process of entering tickets. Testing revealed that users were not properly trained on automatic tickets and therefore leaned towards the manual ticket that was presented to them as the primary solution. In addition, they did not realize that its processing takes longer than with the automatic one, where the processing takes place in the order of minutes.
It also emerged that with the rise of AI, some users are concerned about the term "automated", as they believe it involves the use of artificial intelligence that they might find complex or impersonal.
Conclusion
The result of the testing was a recommendation to create training videos on the use of automatic tickets and to increase the emphasis on onboard training for existing and new users. During the testing, users positively evaluated the possibility of using automatic tickets and showed an interest in using them more actively.
The interviews showed that both identified persons have a technical education and that their preference for manual ticketing was only due to insufficient training. So there was no need to solve the problem of time pressure or the number of tickets.
Another interesting thing was the concern of some users from the OPS team that automation may gradually replace them.
During testing, minor errors in the system were detected, so we proposed several modifications, including changes in the layout of buttons, adding tooltips and adding explanatory labels to individual types of tickets.
