Queue delay , sending an email at a certain time

What want to achieve is to send and email , at a chosen time (by the user) so my approach was to calculate the time difference then make that the delay time on a queue.

but it seems it's not working as i expect all emails get delivered together and not at the right time

Controller

    public function notifyme($add)
{

    $created = Carbon::now()->addMinutes($add);
    $now = Carbon::now();

    $days = 1440*$created->diff($now)->days;
    $hours = 60*$created->diff($now)->h;
    $minutes = $days + $hours + $created->diff($now)->i;

    $user = Auth::user();


    $user->notify((new notifyme($user))->delay($minutes));

     return redirect('/notif');
}

route

Route::get('notifyme/{add}', 'HomeController@notifyme');

.env

QUEUE_DRIVER="database"

im using laravel 5.3 so the notifyme controller implements the ShouldQueue contract.

now when i run php artisan queue:work or php artisan queue:listen and test this the email sending is delayed, but when i do it again, (while the first job is still delayed) both emails get sent together immediately or after sometime (not at the right time $add)

Any idea what is wrong here? is there a better approach? like using a schedule? or what?