Javax Websocket closing due to Illegal UTF-8 Sequence

The following is just a guess.

(1) On the server side, the large string is split into one text frame and one or more following continuation frames. Technically, the original large string is converted into a byte array and then the byte array is split into multiple sub byte arrays. The sub arrays are set to frames one by one (= Each frame contains one sub byte array).

(2) Although there is no guarantee that each sub byte array is a valid UTF-8 sequence, validity check is performed either on the server side or on the client side. If so, it's a bug of Tyrus.

WebSocketListener of nv-websocket-client has callback methods in frame granularity such as onFrame, onTextFrame, onContinuationFrame and others (note that onTextMessage and onTextFrame are different), so you can examine the byte array of each frame there.

WebSocket websocket = new WebSocketFactory()
    .createSocket("ws://...")
    .addListener(new WebSocketAdapter() {
        @Override
        public void onFrame(WebSocket ws, WebSocketFrame frame) {
            // If the frame is a text frame with FIN bit cleared, or
            // if the frame is a continuation frame.
            if ((frame.isTextFrame() && frame.getFin() == false) ||
                frame.isContinuationFrame()) {
                // The payload of the frame. There is no guarantee
                // that this byte array is a valid UTF-8 sequence.
                byte[] payload = frame.getPayload();

                // Check whether the payload is a valid UTF-8 sequence
                // if you want to.
                checkPayload(payload);
            }
        }
    })
    .connect();

Why don't you use nv-websocket-client to examine what is happening in your WebSocket connection?