Google Analytics Report API v4 returns 503 too many times without obvious reasons

Every day morning we would get the previous day's / week's e-commerce product data from the Google Analytics Reporting API v4. We have been doing the same thing for 3 years and the report format hasn't changed. There can be multiple different time ranges that we want to get.

Here is a sample request. There can be up to 500k rows in total that we page through using the "pageToken" parameter. We do this sequentially.

{
   "reportRequests":[
      {
         "metrics":[
            {
               "expression":"ga:productListViews"
            },
            {
               "expression":"ga:productListClicks"
            },
            {
               "expression":"ga:productDetailViews"
            },
            {
               "expression":"ga:productAddsToCart"
            },
            {
               "expression":"ga:quantityAddedToCart"
            },
            {
               "expression":"ga:productRemovesFromCart"
            },
            {
               "expression":"ga:quantityRemovedFromCart"
            },
            {
               "expression":"ga:productCheckouts"
            },
            {
               "expression":"ga:quantityCheckedOut"
            },
            {
               "expression":"ga:uniquePurchases"
            }
         ],
         "hideTotals":true,
         "dimensions":[
            {
               "name":"ga:productSku"
            },
            {
               "name":"ga:productBrand"
            },
            {
               "name":"ga:dimension1"
            }
         ],
         "pageSize":3000,
         "includeEmptyRows":true,
         "viewId":"169501676",
         "dateRanges":[
            {
               "startDate":"2020-08-17",
               "endDate":"2020-08-17"
            }
         ],
         "pageToken":"210000",
         "hideValueRanges":true
      },
      {
         "metrics":[
            {
               "expression":"ga:itemQuantity"
            },
            {
               "expression":"ga:itemRevenue"
            }
         ],
         "hideTotals":true,
         "dimensions":[
            {
               "name":"ga:productSku"
            },
            {
               "name":"ga:productBrand"
            },
            {
               "name":"ga:dimension1"
            }
         ],
         "pageSize":3000,
         "includeEmptyRows":true,
         "viewId":"169501676",
         "dateRanges":[
            {
               "startDate":"2020-08-17",
               "endDate":"2020-08-17"
            }
         ],
         "pageToken":"210000",
         "hideValueRanges":true
      }
   ]
}

Recently, the API randomly returns a lot of 503 errors. We checked this page which suggests us to implement exponential backoff for 1s, 2s, 4s, 8s, 16s. Instead of the suggested intervals, we are doing 2h, 2h, 2h, 2h, 2h, but the error still persisted. https://developers.google.com/analytics/devguides/reporting/core/v3/errors#handling_500_or_503_responses

We have checked the quotas and limits and pretty sure we are not exceeding them. https://developers.google.com/analytics/devguides/reporting/core/v4/limits-quotas (can't post images without 10 reputations)

https://i.stack.imgur.com/jlZzV.png

https://i.stack.imgur.com/4q605.png

https://i.stack.imgur.com/2qTBI.png

If it matters, all requests run on Google App Engine python2 standard environment.