proxy-authentication header missing with https

[maddo7]

I want to create a mitm proxy that can only be access by providing correct credentials, e.g. I check if proxy-authentication has the correct value:

(async () => {
    const mockttp = require('mockttp');

    // Create a proxy server with a self-signed HTTPS CA certificate:
    const https = await mockttp.generateCACertificate();
    const server = mockttp.getLocal({ https });

server.forAnyRequest().thenCallback((request) => {
    return {
        status: 200,
        // Return a JSON response with an incrementing counter:
        json: request
    };
});
    await server.start(8080);

    // Print out the server details:
    const caFingerprint = mockttp.generateSPKIFingerprint(https.cert)
    console.log(`Server running on port ${server.port}`);
    console.log(`CA cert fingerprint ${caFingerprint}`);
})(); // (Run in an async wrapper so we can use top-level await everywhere)

With http it works flawlessly, the proxy-authorization header is present:

curl -k -v --proxy "user:[email protected]:8080" http://www.google.com

{
   "id":"8978f1a3-8a4f-4395-b0dc-0cf8929e760a",
   "matchedRuleId":"5a1bc167-7e34-4b0d-9f51-f8e49015b349",
   "protocol":"http",
   "httpVersion":"1.1",
   "method":"GET",
   "url":"http://www.google.com/",
   "path":"/",
   "remoteIpAddress":"::ffff:127.0.0.1",
   "remotePort":32932,
   "headers":{
      "host":"www.google.com",
      "proxy-authorization":"Basic dXNlcjpwYXNz",
      "user-agent":"curl/7.83.1",
      "accept":"*/*",
      "proxy-connection":"Keep-Alive"
   },
   "rawHeaders":[
      [
         "Host",
         "www.google.com"
      ],
      [
         "Proxy-Authorization",
         "Basic dXNlcjpwYXNz"
      ],
      [
         "User-Agent",
         "curl/7.83.1"
      ],
      [
         "Accept",
         "*/*"
      ],
      [
         "Proxy-Connection",
         "Keep-Alive"
      ]
   ],
   "tags":[
      
   ],
   "timingEvents":{
      "startTime":1663860475270,
      "startTimestamp":7655.8840999901295,
      "bodyReceivedTimestamp":7656.588100001216
   },
   "body":{
      "buffer":{
         "type":"Buffer",
         "data":[
            
         ]
      }
   }
}

Now the problem is that if it runs through https, the proxy-authorization disappears:

curl -k -v --proxy "user:[email protected]:8080" https://www.google.com
{
   "id":"dd9f61c9-8ecb-4f94-87aa-095fd2f40da6",
   "matchedRuleId":"5a1bc167-7e34-4b0d-9f51-f8e49015b349",
   "protocol":"https",
   "httpVersion":"1.1",
   "method":"GET",
   "url":"https://www.google.com/",
   "path":"/",
   "remoteIpAddress":"::ffff:127.0.0.1",
   "remotePort":34557,
   "headers":{
      "host":"www.google.com",
      "user-agent":"curl/7.83.1",
      "accept":"*/*"
   },
   "rawHeaders":[
      [
         "Host",
         "www.google.com"
      ],
      [
         "User-Agent",
         "curl/7.83.1"
      ],
      [
         "Accept",
         "*/*"
      ]
   ],
   "tags":[
      
   ],
   "timingEvents":{
      "startTime":1663860737403,
      "startTimestamp":269786.7910999954,
      "bodyReceivedTimestamp":269787.29159998894
   },
   "body":{
      "buffer":{
         "type":"Buffer",
         "data":[
            
         ]
      }
   }
}

Is there anything I'm unaware of that causes this behaviour?

[pimterry]

Hi @maddo7, this is a good question. This happens because there's two different ways of doing HTTP proxying.

The first way is that the client sends a request to the proxy like GET http://example.com/abc - i.e. it sends the entire request to the proxy, and the proxy parses that, extracts the target server (example.com) and forwards the request there. In this case, all headers are included in that one request. This is probably what's happening with your HTTP proxying.

The second way is that the client sends a CONNECT request to the proxy like CONNECT example.com:80, asking for a tunnel a remote server, and then it sends a separate request like GET /abc inside that tunnel, talking directly to the remote server. In this case, proxy-specific headers only appear on the first request. Wikipedia has some more details: https://en.wikipedia.org/wiki/HTTP_tunnel.

Currently, in that second case, you can't interact with the outer tunneling request or see that data via Mockttp at all. Mockttp simply unwraps and discards all layers of CONNECT tunnelling, and only considers the final request to the end server. Extending this is a bit more complicated than it sounds, because requests aren't even 1-1, for example a client can CONNECT through the proxy once, and then send many independent requests inside the resulting tunnel to the remote server (or even none at all, if it changes its mind). You can even CONNECT to create a tunnel to a different proxy server, and then CONNECT there too, at unlimited depth, using different authentication headers for each step.

For this specific use case of authenticated tunneling though, I think it's probably possible to handle this though - we could add a proxyAuth option like getLocal({ https, proxyAuth: { username: '...', password: '...' } }), and then enforce that on incoming requests before we start normal final-request processing. PRs very welcome if that's something you'd be interested in.

[skilbjo]

hi @pimterry i am also interested in this, but with password authentication dynamic (a lookup of api keys from supabase).

since this issue is ~2.5yrs old, are you still interested in having this be implemented? do you have any suggestions on how it should be implemented?

ty, john

[pimterry]

Hi @skilbjo, sure I'd be open to this. I think the proxyAuth API I suggested above would work fine.

Actual implementation may be a bit tricky - as noted above, Mockttp just unwraps CONNECT tunnels, potentially recursively (if you try to connect to a proxy then to a proxy then to a proxy then to a server, Mockttp strips off all the proxying steps and just handles the server request directly) but authentication should really just apply to the outer connection.

It's also unclear how to handle unauthenticated direct connections - where a client sends GET / + Host: example.com directly as if Mockttp as a normal server, instead of CONNECT example.com then GET / inside the tunnel. We currently treat them exactly the same. I guess you'd want to just reject these entirely if auth is required, but we should test & document that.

Have a look and see what you can come up with. All the relevant code lives in https://github.com/httptoolkit/mockttp/blob/main/src/server/http-combo-server.ts and httpolyglot.