RTMP handshake failing on s2 processing

[GeorgeBowers]

HI, I discovered a bug, and a solution, for the outbound RTMP handshake of the red5 client:

The code in question is below:

    if (c2 != null) {
                        // set state to indicate we're waiting for S2
                        conn.getState().setState(RTMP.STATE_HANDSHAKE);
                        //log.trace("C2 byte order: {}", c2.order());
                        session.write(c2);
                        // if we got S0S1+S2 continue processing
                        if (handshake.getBufferSize() >= Constants.HANDSHAKE_SIZE) {
                            // clear
                            buf.clear();
                            // re-set
                            buf = handshake.getBufferAsIoBuffer();
                            if (handshake.decodeServerResponse2(buf)) {
                                log.debug("S2 decoding successful");
                            } else {
                                log.warn("Handshake failed on S2 processing");
                            }
                            completeConnection(session, conn, rtmp, handshake);
                        }

In my use of red5, whenever the code reached this point, and tired to decode the S2 handshake here, it always fails, I believe it is due to the reason I have described in the below sequence diagram:

I believe the fix would be this:

int remaining = buf.remaining();
                if (remaining > 0) {
                    // store the remaining bytes in a thread local for use by S2 decoding
                    handshake.addBuffer(buf);
                    log.warn("Stored {} bytes for later decoding", remaining);
                }
                IoBuffer c2 = handshake.decodeServerResponse1(IoBuffer.wrap(dst));
                if (c2 != null) {
                    // set state to indicate we're waiting for S2
                    conn.getState().setState(RTMP.STATE_HANDSHAKE);
                    //log.trace("C2 byte order: {}", c2.order());
                    session.write(c2);
                    // if we got S0S1+S2 continue processing
                    if (handshake.getBufferSize() >= Constants.HANDSHAKE_SIZE && remaining > 0) {
                        // clear
                        buf.clear();
                        // re-set
                        buf = handshake.getBufferAsIoBuffer();
                        if (handshake.decodeServerResponse2(buf)) {
                            log.debug("S2 decoding successful");
                        } else {
                            log.warn("Handshake failed on S2 processing");
                        }
                        completeConnection(session, conn, rtmp, handshake);
                    }