Problem with authenticating Azure AD tokens at Web API

I have 2 web applications.

• Site A is both front end (Angular) and a web API written in .NET core.
• Site B is a web API written in .NET core.

The web APIs in both sites are the same. The authentication is the same.

Test 1: When I ask Site A to use its own API to get the data (with Site A's authentication turned on) it works great.

Test 2: When I ask Site A to use Site B to get the data (with Site B's authentication turned OFF) it works great.

Test 3: When I ask Site A to use Site B to get the data (with Site B's authentication turned ON) it fails with unauthorised error 401.

Here is the code I am using to authenticate...

services.AddAuthentication(options =>
{
    options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
    options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;
})
.AddJwtBearer(o =>
{
    o.MetadataAddress = String.Format("https://login.microsoftonline.com/{0}/v2.0/.well-known/openid-configuration", Configuration["AzureAd:TenantId"]);
    o.Audience = Configuration["AzureAd:ClientId"];
});

So...

I know the tokens are getting passed correctly (because of Test A).

I know the Site B API being called is accessible and correct (because of Test B).

Therefore, I assume, the issue is with the values being passed to Azure AD. I assume Azure doesn't think the tokens being obtained in Site A and validated in Site B (that have different URIs) are not "the same" and therefore not authorised.

What do I need to pass to Azure to get it to work?

Thanks