"Insufficient privileges to complete the operation" error for only some users

I am trying to get the Azure AD Groups a user in member of:

https://graph.microsoft.com/v1.0/users/<user_id>/memberOf

For some users I am getting the requested information but for others I am getting this error:

{
    "error": {
        "code": "Authorization_RequestDenied",
        "message": "Insufficient privileges to complete the operation.",
        "innerError": {
            "request-id": "fd9345ee-ee2f-4dfb-b7e8-ca53d0c804d9",
            "date": "2019-07-03T15:50:25"
        }
    }
}

I'm getting the token using Client ID + Client Secret.

The app has these privileges:

(Application) Groups.Read.All
(Application) Users.Read.All

As I am getting the groups for some members it seems this is not a lack of privileges at Azure app level.

Can you please help me on this?

Thanks in advance!

PS: Fix by Caiyi Ju: "Since you are using client credential flow, you need to grant Directory.Read.All permission (application)."