There are two options available to you, when restricting users from accessing Yammer.
Remove the licence from their Office 365 Account
Within Office 365 Admin, when editing the account for a specific user, you can remove the licence allocation to Yammer, which will prevent that person from logging in to any Yammer network with their credentials.
Block users from your Corporate Yammer Network, or specific External Networks
As a Verified Admin, when editing the Network Admin settings, go to the "Block Users" section (from the left-hand navigation), where you can then enter the email addresses of those who you wish to block. Please note that these can be internal or external email address.
When blocking someone in this manner, it will only prevent them from accessing the current network. They will still be able to access other Networks in which they are a member on Yammer.