Web Push works because there is a persistent connection between the browser (e.g. Chrome) and the browser push service (e.g. FCM).
When your application server needs to send a notification to a browser, it cannot reach the browser directly with a connection, instead it contacts the browser push service (e.g. FCM for Chrome) and then it's the browser push service that delivers the notification to the user browser.
This is possible because the browser constantly tries to keep an open connection with the server (e.g. FCM for Chrome). This means that there isn't any problem for NAT, since it's the clients that starts the connection. Also consider that any TCP connection is bi-directional: so any side of the connection can start sending data at any time. Don't confuse higher level protocols like HTTP with a normal TCP connection.
If you want more details I have written this article that explains in simple words how Web Push works. You can also read the standards: Push API and IETF Web Push in particular.
Note: Firebase (FCM) is two different things, even if that is not clear from the documentation. It is both the browser push service required to deliver notifications to Chrome (like Mozilla autopush for Firefox, Windows Push Notification Services for Edge and Apple Push Notification service for Safari), but it is also a proprietary service with additional features to send notifications to any browser (like Pushpad, Onesignal and many others).
so the client/user agent/browser (ex: chrome) keeps open TCP connection to the push service (ex: FCM); and the connection is kept alive forever? and browser does this without being executed explicitly (ex: user click on chrome icon)?
Yes, there is usually a persistent connection between the browser and its own push service. If that cannot be done on a specific OS (for example, in background) then an alternative method is used: for example the push message is forwarded from the browser push service to the normal push service of the OS (e.g. this is done by Firefox on Android). If you need the exact implementation details you need to read the standards that I linked above.