For such case #2, then you should not use Web Push Notification.
Since you want system to send the message to all users (assume only for the logged-in users), then you need one or some additional tables to store all those messages, including the sender and the recipient.
Each time a user send the message to another user, then you need to handle from the web application to display the notification to the destination user when the recipient has already been logged in, or when the recipient has just been reloaded any page.
If necessary, you may add a field to distinguish whether a message has been read or not. For example, you need to add update functionality to flag a message has been read or not.