How do I restore Nuget Packages from a private NugetFeed in a devcontainer on Windows?

I am using Windows, installed Docker (using lunix containers), installed VS Code and the devcontainer extension. I loaded a project which is using Nugets from a private NugetFeed and as i am trying to restore (interactive) the NugetPackages I am getting a 401 error from my NugetFeed.

/usr/share/dotnet/sdk/2.2.207/NuGet.targets(119,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/[SomeFeedUrl]/nuget/v3/index.json. [/workspaces/[SomeSolution].sln]
/usr/share/dotnet/sdk/2.2.207/NuGet.targets(119,5): error :   Response status code does not indicate success: 401 (Unauthorized). [/workspaces/[SomeSolution].sln]

I tried the same thing locally (credential provider installed) and everything is working fine. I also tried mounting the Nugetpackages path to the devcontainer, restore locally and then start the project (since this is my fix to get the restore working properly on my linux maschine).

Here's what my devcontainer.json looks like:

// For format details, see https://aka.ms/vscode-remote/devcontainer.json or the definition README at
// https://github.com/microsoft/vscode-dev-containers/tree/master/containers/azure-functions-dotnetcore-2.2
{
    "name": "Azure Functions & C# (.NET Core 2.2)",
    "dockerFile": "Dockerfile",

    // Use 'settings' to set *default* container specific settings.json values on container create. 
    // You can edit these settings after create using File > Preferences > Settings > Remote.
    "settings": { 
        "terminal.integrated.shell.linux": "/bin/bash"
    },

    "mounts": [
        "source=C:/Users/[user]/.nuget,target=/home/vscode/.nuget,type=bind"
        // "source=${env:HOME}${env:USERPROFILE}/.aspnet/https,target=/home/vscode/.aspnet/https,type=bind"
    ],

    "remoteEnv": {
        // [Optional] Override the default HTTP endpoints - need to listen to '*' for appPort to work
        "ASPNETCORE_Kestrel__Endpoints__Http__Url": "http://*:5000"
        // "ASPNETCORE_Kestrel__Endpoints__Https__Url": "https://*:5001",
        // "ASPNETCORE_Kestrel__Certificates__Default__Password": "SecurePwdGoesHere",
        // "ASPNETCORE_Kestrel__Certificates__Default__Path": "/home/vscode/.aspnet/https/aspnetapp.pfx",
    },

    // Uncomment the next line to run commands after the container is created.
    // "postCreateCommand": "dotnet restore",

    // Uncomment the next line to have VS Code connect as an existing non-root user in the container. 
    // On Linux, by default, the container user's UID/GID will be updated to match your local user. See
    // https://aka.ms/vscode-remote/containers/non-root for details on adding a non-root user if none exist.
    // "remoteUser": "vscode",

    // Add the IDs of extensions you want installed when the container is created in the array below.
    "extensions": [
        "ms-azuretools.vscode-azurefunctions",
        "ms-vscode.csharp"
    ]
}

I did not touch the dockerfile, the dockerfile is still a stock dockerfile.

Does anyone have the same issue/know how to get it working properly?