REST API External user access

I’m currently trying to access a sharepoint site through the REST api using an external user from another organization.

The user has been added to the Site Members group with the Edit rights successfully. Tho, after retrieving an access token, I’m constantly getting this response:

GET https://<my-site>/_api/web # And all the other routes
  "error_description": "Exception of type 'Microsoft.IdentityModel.Tokens.AudienceUriValidationFailedException' was thrown."

I allowed sharing on this site through the office 365 admin center but it seems like it’s impossible to access it anyway.

I’ve been searching for this on google for a few hours now and I still can’t get any results. I tried tweaking the settings in the Sharepoint admin center but nothing worked. Also, after going through the Sharepoint REST API documentation I couldn’t find anything about this particular use case.

Is it possible to access the REST API of a site using a user that is external to the organization itself?