There are many plugins out there that protect a WordPress website from Brute Force attacks. The problem is, they are very resource intensive since they run through PHP. Not to mention, having to load a plugin for every single WordPress site on a server causes even more strain on the server and is not an efficient way to protect you and your clients from Brute Force attacks
There are two popular WordPress brute force attacks. One aimed at hitting the wp-admin and the other aimed at hitting xmlrpc.php. Both of these seem very small, but can cause your CPU loads to rise, your websites to load slow, and in a worst case scenario, a hacked website.
Add the following rules to your mod_security ruleset and they will help block the two attacks