How to perform RxSwift in the background?

The first thing to understand with this code is that subscribeOn determines which thread the event generator of the observable will be called on, not which thread the the values will be emitted on. If the source Observable emits on the same thread it's subscribed on, then your code would have worked (for example, if you were using Observable.just(_:). But modelSelected emits on the main thread, regardless on what thread it was subscribed on. The flatMap operator calls its closure on the thread that its source emitted on so that also is on the main thread.

The upshot of all of this is that it's rarely useful to call subscribeOn(_:). The only time it actually does what you would expect is if the ultimate source (the one most upstream) Observable's generator is a synchronous, blocking, function. (Which isn't the case with tableView.rx.modelSelected(String.self))

What you want in this case is something more like:

tableView.rx.modelSelected(String.self)
    .observeOn(ConcurrentDispatchQueueScheduler(qos: .background))
    .flatMap { item -> Observable<String> in
        sleep(3)
        return Observable.just("Hello world!")
    }
    .retry()
    .observeOn(MainScheduler.instance)
    .subscribe(onNext: { value in
        print(value)
    })
    .disposed(by: bag)