As with most things, there are automated security tools available to scan and assess your Office 365 tenant. Earlier this month, RapidFire Tools (part of Kaseya) released their new Network Detective Microsoft Cloud Assessment Module. Calling it “the fastest and most comprehensive Microsoft Cloud assessment solution on the market” and “the first and only automated system that gathers this critical information” it seems an especially great fit for Managed Service Providers (MSPs) who are charged with Office 365 maintenance for multiple customers. Certainly, the reputation of RapidFire Tools is well known and well deserved among MSPs. 



How does it work? Being the first and only automated system, does it do the “most comprehensive” job promised? Let’s take a look at an example Microsoft Cloud Security Assessment Report it generates, one of several reports and the one which RapidFire Tools says “brings together all of the security aspects of Microsoft Cloud under one umbrella.” 



The PDF document cover page has the MSP’s logo, company name, and the name of the customer assessed is automatically populated. It looks and feels like a form letter, because it is. The first section is the current Microsoft Security Score for the Office 365 tenant (which is a free tool available to everyone within Office 365 itself) and a red/yellow/green graphic showing the customer’s current score. While the theoretical maximum score changes, the RapidFire Tools report centers the chart on the average score of similar organizations using Office 365. In their example, the customer achieved a security score of 81, and it’s well in the green on the right, and indicates that this customer’s score is awesome, and they don’t need to worry about this security stuff. 

One problem with this – it only means the customer is doing better when compared with similar sized Office 365 organizations. The actual score based on the customer’s specific maximum is 81 out of 238 or just 34% - hardly an end zone achievement. 



The next section of the report is the Control Scores section, where specific security controls are assessed, described, and the results are scored. This is where a fully automated assessment falls short – the API scan only knows if you have a given control enabled or not, there’s no way for a computer to understand the context of that security control, or if certain best practices are implemented. 

Example 1: The report (screenshot above) shows that control MFARegistrationV2, or Multi-factor Authentication is enabled, but the customer only has 10 out of 2056 users registered and protected, flags this finding for review and action, and recommends adding more MFA authentication methods to increase protection. 



Why it isn’t enough: It’s a best practice that MFA be enforced for accounts with administrative access, at minimum, and best if it’s enforced for all users. With this report, we don’t know what admin roles the 10 users who require MFA for logon might or might not have. It is too broad to evaluate MFA enforcement status without correlating it to account roles. Suggesting additional MFA methods isn’t wrong, but the report doesn’t enumerate which methods are currently enabled, or account for best practices known within the security community – that you should disable the Microsoft Authenticator app as a method for Self Service Password Reset (under certain conditions, Microsoft Authenticator allows approval of a login from a locked phone) or that using SMS text messages or automated phone calls as MFA methods are sub-optimal choices. 

In summary, I find the claims of RapidFire Tools’ Network Detective Cloud Assessment providing the “most comprehensive” job and that it “brings together all of the security aspects of Microsoft Cloud under one umbrella” to be specious bordering on disingenuous. That said, it’s the very first automated tool by its own admission, so perhaps future iterations will improve. As things currently stand it reinforces that there’s simply no substitute for having knowledge and expertise behind your security assessments. When you work with experts you fully understand the context for or against certain controls when applied specifically to your business and what functionality you need, as well as the “why” and “how” for best practices. 



The human expert will always do better at understanding the infinite shades of gray where humans interact with cybersecurity, and in this case, the just released automatic tool falls short and provides only a false sense of security, rather than real, actionable security findings. 



To connect with a Microsoft Gold Partner and Security Expert click here.