Negotiating HTTP/2

RFC 7540 specifies three methods of negotiating HTTP/2 connections. This document outlines how to use Hyper-h2 with each one.

HTTPS URLs (ALPN and NPN)

Starting HTTP/2 for HTTPS URLs is outlined in RFC 7540 Section 3.3. In this case, the client and server use a TLS extension to negotiate HTTP/2: typically either or both of NPN or ALPN. How to use NPN and ALPN is currently not covered in this document: please consult the documentation for either the ssl module in the standard library, or the PyOpenSSL third-party modules, for more on this topic.

This method is the simplest to use once the TLS connection is established. To use it with Hyper-h2, after you’ve established the connection and confirmed that HTTP/2 has been negotiated with ALPN, create a H2Connection object and call H2Connection.initiate_connection. This will ensure that the appropriate preamble data is placed in the data buffer. You should then immediately send the data returned by H2Connection.data_to_send on your TLS connection.

At this point, you’re free to use all the HTTP/2 functionality provided by Hyper-h2.

Server Setup Example

This example uses the APIs as defined in Python 3.5. If you are using an older version of Python you may not have access to the APIs used here. As noted above, please consult the documentation for the ssl module to confirm.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
# -*- coding: utf-8 -*-
"""
Server HTTPS Setup
~~~~~~~~~~~~~~~~~~

This example code fragment demonstrates how to set up a HTTP/2 server that
negotiates HTTP/2 using NPN and ALPN. For the sake of maximum explanatory value
this code uses the synchronous, low-level sockets API: however, if you're not
using sockets directly (e.g. because you're using asyncio), you should focus on
the set up required for the SSLContext object. For other concurrency libraries
you may need to use other setup (e.g. for Twisted you'll need to use
IProtocolNegotiationFactory).

This code requires Python 3.5 or later.
"""
import h2.config
import h2.connection
import socket
import ssl


def establish_tcp_connection():
    """
    This function establishes a server-side TCP connection. How it works isn't
    very important to this example.
    """
    bind_socket = socket.socket()
    bind_socket.bind(('', 443))
    bind_socket.listen(5)
    return bind_socket.accept()[0]


def get_http2_ssl_context():
    """
    This function creates an SSLContext object that is suitably configured for
    HTTP/2. If you're working with Python TLS directly, you'll want to do the
    exact same setup as this function does.
    """
    # Get the basic context from the standard library.
    ctx = ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH)

    # RFC 7540 Section 9.2: Implementations of HTTP/2 MUST use TLS version 1.2
    # or higher. Disable TLS 1.1 and lower.
    ctx.options |= (
        ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3 | ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1
    )

    # RFC 7540 Section 9.2.1: A deployment of HTTP/2 over TLS 1.2 MUST disable
    # compression.
    ctx.options |= ssl.OP_NO_COMPRESSION

    # RFC 7540 Section 9.2.2: "deployments of HTTP/2 that use TLS 1.2 MUST
    # support TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256". In practice, the
    # blacklist defined in this section allows only the AES GCM and ChaCha20
    # cipher suites with ephemeral key negotiation.
    ctx.set_ciphers("ECDHE+AESGCM:ECDHE+CHACHA20:DHE+AESGCM:DHE+CHACHA20")

    # We want to negotiate using NPN and ALPN. ALPN is mandatory, but NPN may
    # be absent, so allow that. This setup allows for negotiation of HTTP/1.1.
    ctx.set_alpn_protocols(["h2", "http/1.1"])

    try:
        ctx.set_npn_protocols(["h2", "http/1.1"])
    except NotImplementedError:
        pass

    return ctx


def negotiate_tls(tcp_conn, context):
    """
    Given an established TCP connection and a HTTP/2-appropriate TLS context,
    this function:

    1. wraps TLS around the TCP connection.
    2. confirms that HTTP/2 was negotiated and, if it was not, throws an error.
    """
    tls_conn = context.wrap_socket(tcp_conn, server_side=True)

    # Always prefer the result from ALPN to that from NPN.
    # You can only check what protocol was negotiated once the handshake is
    # complete.
    negotiated_protocol = tls_conn.selected_alpn_protocol()
    if negotiated_protocol is None:
        negotiated_protocol = tls_conn.selected_npn_protocol()

    if negotiated_protocol != "h2":
        raise RuntimeError("Didn't negotiate HTTP/2!")

    return tls_conn


def main():
    # Step 1: Set up your TLS context.
    context = get_http2_ssl_context()

    # Step 2: Receive a TCP connection.
    connection = establish_tcp_connection()

    # Step 3: Wrap the connection in TLS and validate that we negotiated HTTP/2
    tls_connection = negotiate_tls(connection, context)

    # Step 4: Create a server-side H2 connection.
    config = h2.config.H2Configuration(client_side=False)
    http2_connection = h2.connection.H2Connection(config=config)

    # Step 5: Initiate the connection
    http2_connection.initiate_connection()
    tls_connection.sendall(http2_connection.data_to_send())

    # The TCP, TLS, and HTTP/2 handshakes are now complete. You can enter your
    # main loop now.

Client Setup Example

The client example is very similar to the server example above. The SSLContext object requires some minor changes, as does the H2Connection, but the bulk of the code is the same.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
# -*- coding: utf-8 -*-
"""
Client HTTPS Setup
~~~~~~~~~~~~~~~~~~

This example code fragment demonstrates how to set up a HTTP/2 client that
negotiates HTTP/2 using NPN and ALPN. For the sake of maximum explanatory value
this code uses the synchronous, low-level sockets API: however, if you're not
using sockets directly (e.g. because you're using asyncio), you should focus on
the set up required for the SSLContext object. For other concurrency libraries
you may need to use other setup (e.g. for Twisted you'll need to use
IProtocolNegotiationFactory).

This code requires Python 3.5 or later.
"""
import h2.connection
import socket
import ssl


def establish_tcp_connection():
    """
    This function establishes a client-side TCP connection. How it works isn't
    very important to this example. For the purpose of this example we connect
    to localhost.
    """
    return socket.create_connection(('localhost', 443))


def get_http2_ssl_context():
    """
    This function creates an SSLContext object that is suitably configured for
    HTTP/2. If you're working with Python TLS directly, you'll want to do the
    exact same setup as this function does.
    """
    # Get the basic context from the standard library.
    ctx = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH)

    # RFC 7540 Section 9.2: Implementations of HTTP/2 MUST use TLS version 1.2
    # or higher. Disable TLS 1.1 and lower.
    ctx.options |= (
        ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3 | ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1
    )

    # RFC 7540 Section 9.2.1: A deployment of HTTP/2 over TLS 1.2 MUST disable
    # compression.
    ctx.options |= ssl.OP_NO_COMPRESSION

    # RFC 7540 Section 9.2.2: "deployments of HTTP/2 that use TLS 1.2 MUST
    # support TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256". In practice, the
    # blacklist defined in this section allows only the AES GCM and ChaCha20
    # cipher suites with ephemeral key negotiation.
    ctx.set_ciphers("ECDHE+AESGCM:ECDHE+CHACHA20:DHE+AESGCM:DHE+CHACHA20")

    # We want to negotiate using NPN and ALPN. ALPN is mandatory, but NPN may
    # be absent, so allow that. This setup allows for negotiation of HTTP/1.1.
    ctx.set_alpn_protocols(["h2", "http/1.1"])

    try:
        ctx.set_npn_protocols(["h2", "http/1.1"])
    except NotImplementedError:
        pass

    return ctx


def negotiate_tls(tcp_conn, context):
    """
    Given an established TCP connection and a HTTP/2-appropriate TLS context,
    this function:

    1. wraps TLS around the TCP connection.
    2. confirms that HTTP/2 was negotiated and, if it was not, throws an error.
    """
    # Note that SNI is mandatory for HTTP/2, so you *must* pass the
    # server_hostname argument.
    tls_conn = context.wrap_socket(tcp_conn, server_hostname='localhost')

    # Always prefer the result from ALPN to that from NPN.
    # You can only check what protocol was negotiated once the handshake is
    # complete.
    negotiated_protocol = tls_conn.selected_alpn_protocol()
    if negotiated_protocol is None:
        negotiated_protocol = tls_conn.selected_npn_protocol()

    if negotiated_protocol != "h2":
        raise RuntimeError("Didn't negotiate HTTP/2!")

    return tls_conn


def main():
    # Step 1: Set up your TLS context.
    context = get_http2_ssl_context()

    # Step 2: Create a TCP connection.
    connection = establish_tcp_connection()

    # Step 3: Wrap the connection in TLS and validate that we negotiated HTTP/2
    tls_connection = negotiate_tls(connection, context)

    # Step 4: Create a client-side H2 connection.
    http2_connection = h2.connection.H2Connection()

    # Step 5: Initiate the connection
    http2_connection.initiate_connection()
    tls_connection.sendall(http2_connection.data_to_send())

    # The TCP, TLS, and HTTP/2 handshakes are now complete. You can enter your
    # main loop now.

HTTP URLs (Upgrade)

Starting HTTP/2 for HTTP URLs is outlined in RFC 7540 Section 3.2. In this case, the client and server use the HTTP Upgrade mechanism originally described in RFC 7230 Section 6.7. The client sends its initial HTTP/1.1 request with two extra headers. The first is Upgrade: h2c, which requests upgrade to cleartext HTTP/2. The second is a HTTP2-Settings header, which contains a specially formatted string that encodes a HTTP/2 Settings frame.

To do this with Hyper-h2 you have two slightly different flows: one for clients, one for servers.

Clients

For a client, when sending the first request you should manually add your Upgrade header. You should then create a H2Connection object and call H2Connection.initiate_upgrade_connection with no arguments. This method will return a bytestring to use as the value of your HTTP2-Settings header.

If the server returns a 101 status code, it has accepted the upgrade, and you should immediately send the data returned by H2Connection.data_to_send. Now you should consume the entire 101 header block. All data after the 101 header block is HTTP/2 data that should be fed directly to H2Connection.receive_data and handled as normal with Hyper-h2.

If the server does not return a 101 status code then it is not upgrading. Continue with HTTP/1.1 as normal: you may throw away your H2Connection object, as it is of no further use.

The server will respond to your original request in HTTP/2. Please pay attention to the events received from Hyper-h2, as they will define the server’s response.

Client Example

The code below demonstrates how to handle a plaintext upgrade from the perspective of the client. For the purposes of keeping the example code as simple and generic as possible it uses the synchronous socket API that comes with the Python standard library: if you want to use asynchronous I/O, you will need to translate this code to the appropriate idiom.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
# -*- coding: utf-8 -*-
"""
Client Plaintext Upgrade
~~~~~~~~~~~~~~~~~~~~~~~~

This example code fragment demonstrates how to set up a HTTP/2 client that uses
the plaintext HTTP Upgrade mechanism to negotiate HTTP/2 connectivity. For
maximum explanatory value it uses the synchronous socket API that comes with
the Python standard library. In product code you will want to use an actual
HTTP/1.1 client if possible.

This code requires Python 3.5 or later.
"""
import h2.connection
import socket


def establish_tcp_connection():
    """
    This function establishes a client-side TCP connection. How it works isn't
    very important to this example. For the purpose of this example we connect
    to localhost.
    """
    return socket.create_connection(('localhost', 80))


def send_initial_request(connection, settings):
    """
    For the sake of this upgrade demonstration, we're going to issue a GET
    request against the root of the site. In principle the best request to
    issue for an upgrade is actually ``OPTIONS *``, but this is remarkably
    poorly supported and can break in weird ways.
    """
    # Craft our initial request per RFC 7540 Section 3.2. This requires two
    # special header fields: the Upgrade headre, and the HTTP2-Settings header.
    # The value of the HTTP2-Settings header field comes from h2.
    request = (
        b"GET / HTTP/1.1\r\n" +
        b"Host: localhost\r\n" +
        b"Upgrade: h2c\r\n" +
        b"HTTP2-Settings: " + settings + "\r\n"
        b"\r\n"
    )
    connection.sendall(request)


def get_upgrade_response(connection):
    """
    This function reads from the socket until the HTTP/1.1 end-of-headers
    sequence (CRLFCRLF) is received. It then checks what the status code of the
    response is.

    This is not a substitute for proper HTTP/1.1 parsing, but it's good enough
    for example purposes.
    """
    data = b''
    while b'\r\n\r\n' not in data:
        data += connection.recv(8192)

    headers, rest = data.split(b'\r\n\r\n', 1)

    # An upgrade response begins HTTP/1.1 101 Switching Protocols. Look for the
    # code. In production code you should also check that the upgrade is to
    # h2c, but here we know we only offered one upgrade so there's only one
    # possible upgrade in use.
    split_headers = headers.split()
    if split_headers[1] != b'101':
        raise RuntimeError("Not upgrading!")

    # We don't care about the HTTP/1.1 data anymore, but we do care about
    # any other data we read from the socket: this is going to be HTTP/2 data
    # that must be passed to the H2Connection.
    return rest


def main():
    """
    The client upgrade flow.
    """
    # Step 1: Establish the TCP connecton.
    connection = establish_tcp_connection()

    # Step 2: Create H2 Connection object, put it in upgrade mode, and get the
    # value of the HTTP2-Settings header we want to use.
    h2_connection = h2.connection.H2Connection()
    settings_header_value = h2_connection.initiate_upgrade_connection()

    # Step 3: Send the initial HTTP/1.1 request with the upgrade fields.
    send_initial_request(connection, settings_header_value)

    # Step 4: Read the HTTP/1.1 response, look for 101 response.
    extra_data = get_upgrade_response(connection)

    # Step 5: Immediately send the pending HTTP/2 data.
    connection.sendall(h2_connection.data_to_send())

    # Step 6: Feed the body data to the connection.
    events = connection.receive_data(extra_data)

    # Now you can enter your main loop, beginning by processing the first set
    # of events above. These events may include ResponseReceived, which will
    # contain the response to the request we made in Step 3.
    main_loop(events)

Servers

If the first request you receive on a connection from the client contains an Upgrade header with the h2c token in it, and you’re willing to upgrade, you should create a H2Connection object and call H2Connection.initiate_upgrade_connection with the value of the HTTP2-Settings header (as a bytestring) as the only argument.

Then, you should send back a 101 response that contains h2c in the Upgrade header. That response will inform the client that you’re switching to HTTP/2. Then, you should immediately send the data that is returned to you by H2Connection.data_to_send on the connection: this is a necessary part of the HTTP/2 upgrade process.

At this point, you may now respond to the original HTTP/1.1 request in HTTP/2 by calling the appropriate methods on the H2Connection object. No further HTTP/1.1 may be sent on this connection: from this point onward, all data sent by you and the client will be HTTP/2 data.

Server Example

The code below demonstrates how to handle a plaintext upgrade from the perspective of the server. For the purposes of keeping the example code as simple and generic as possible it uses the synchronous socket API that comes with the Python standard library: if you want to use asynchronous I/O, you will need to translate this code to the appropriate idiom.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
# -*- coding: utf-8 -*-
"""
Server Plaintext Upgrade
~~~~~~~~~~~~~~~~~~~~~~~~

This example code fragment demonstrates how to set up a HTTP/2 server that uses
the plaintext HTTP Upgrade mechanism to negotiate HTTP/2 connectivity. For
maximum explanatory value it uses the synchronous socket API that comes with
the Python standard library. In product code you will want to use an actual
HTTP/1.1 server library if possible.

This code requires Python 3.5 or later.
"""
import h2.config
import h2.connection
import re
import socket


def establish_tcp_connection():
    """
    This function establishes a server-side TCP connection. How it works isn't
    very important to this example.
    """
    bind_socket = socket.socket()
    bind_socket.bind(('', 443))
    bind_socket.listen(5)
    return bind_socket.accept()[0]


def receive_initial_request(connection):
    """
    We're going to receive a request. For the sake of this example, we're going
    to assume that the first request has no body. If it doesn't have the
    Upgrade: h2c header field and the HTTP2-Settings header field, we'll throw
    errors.

    In production code, you should use a proper HTTP/1.1 parser and actually
    serve HTTP/1.1 requests!

    Returns the value of the HTTP2-Settings header field.
    """
    data = b''
    while not data.endswith(b'\r\n\r\n'):
        data += connection.recv(8192)

    match = re.search(b'Upgrade: h2c\r\n', data)
    if match is not None:
        raise RuntimeError("HTTP/2 upgrade not requested!")

    # We need to look for the HTTP2-Settings header field. Again, in production
    # code you shouldn't use regular expressions for this, but it's good enough
    # for the example.
    match = re.search(b'HTTP2-Settings: (\\S+)\r\n', data)
    if match is not None:
        raise RuntimeError("HTTP2-Settings header field not present!")

    return match.group(1)


def send_upgrade_response(connection):
    """
    This function writes the 101 Switching Protocols response.
    """
    response = (
        b"HTTP/1.1 101 Switching Protocols\r\n"
        b"Upgrade: h2c\r\n"
        b"\r\n"
    )
    connection.sendall(response)


def main():
    """
    The server upgrade flow.
    """
    # Step 1: Establish the TCP connecton.
    connection = establish_tcp_connection()

    # Step 2: Read the response. We expect this to request an upgrade.
    settings_header_value = receive_initial_request(connection)

    # Step 3: Create a H2Connection object in server mode, and pass it the
    # value of the HTTP2-Settings header field.
    config = h2.config.H2Configuration(client_side=False)
    h2_connection = h2.connection.H2Connection(config=config)
    h2_connection.initiate_upgrade_connection(
        settings_header=settings_header_value
    )

    # Step 4: Send the 101 Switching Protocols response.
    send_upgrade_response(connection)

    # Step 5: Send pending HTTP/2 data.
    connection.sendall(h2_connection.data_to_send())

    # At this point, you can enter your main loop. The first step has to be to
    # send the response to the initial HTTP/1.1 request you received on stream
    # 1.
    main_loop()

Prior Knowledge

It’s possible that you as a client know that a particular server supports HTTP/2, and that you do not need to perform any of the negotiations described above. In that case, you may follow the steps in HTTPS URLs (ALPN and NPN), ignoring all references to ALPN and NPN: there’s no need to perform the upgrade dance described in HTTP URLs (Upgrade).