Re: filtering HTTPS/CONNECT (summary and continuation of discussion)

On 03/16/2012 03:05 PM, Marcus Kool wrote:
> There were 4 threads about 'filtering HTTPS' and I will try to
> summarise here.
>
> Current situation with Squid 3.1.19:
> What happens inside a CONNECT is practically not filterable because
> 1) sslBump is not used, or
> 2) sslBump is used and SSL+HTTP can be filtered, but it breaks the
> other data streams for Skype et al. Using the unsafe options
> 'sslproxy_cert_error allow all' and 'sslproxy_flags DONT_VERIFY_PEER'
> to circumvent the latter problem are far from desirable.
>
> The wiki features pages say that Alex Rousskov is working on
> BumpSslServerFirst
> and MimicSslServerCert but unfortunately Alex has not (yet) participated
> in the discussion.

Sorry, I was on a business trip when the discussion started and could
not respond until now (I tried!).

> To filter HTTP is trivial. To filter HTTPS there are two options:
> 1) to filter without sslBump and then the filter only receives
> "CONNECT <endpoint>:443" on which it has to make a decision to block
> or not. This cripples the filter since it does not has access to the
> content and in many cases can not detect which application sends
> what (type of) data.
> An additional drawback is that connection can be blocked but an
> understandable error message cannot be presented to the end user.

I believe this is already supported.

> 2) use sslBump. The filter will receive "CONNECT <endpoint>:443" as well as
> "https://endpoint/path" (and content for RESPMOD) for SSL+HTTP based
> connections so this is optimal for filtering SSL+HTTP connections.
> The discussion was much around what to do with data streams that are not
> SSL+HTTP. This can be any protocol encapsulated by SSL or simply any
> protocol.
>
> To be able to filter all data, Squid needs a modification to present raw
> data
> about the non-SSL+HTTP data streams to a filter (URL redirector or ICAP).

or eCAP.

> To keep the discussion focussed on one type of filter I will assume that
> an ICAP server is used as the filter.
>
> The ICAP protocol has a considerable overhead (CPU processing) and
> extending
> the ICAP protocol for data stream filtering is not the first choice.
> Amos and Henrik were "optimistic" about implementing a new pipe filter.
>
> The data streams for a bidirectional pipe have a different behavior than
> HTTP and SSL+HTTP. Both client and server can send data at any time. And
> for some, the server initiates the protocol and for others, the client
> initiates. OpenVPN is a chameleon and can pretend to be an SSL+HTTP server
> but is also a VPN server.
>
> In all cases that Squid sends a request to a filter, it would be
> a *big* plus if it informs the filter what it already knows about the
> CONNECT endpoint. E.g. If it has SSL/TLS or not.
>
> Since sslBump is being rewritten for 3.3 it is a good opportunity
> to make Squid suitable for filtering *all* data streams.

Sure, although please keep in mind that the bump-server-first and
certificate mimicking code is pretty much complete. We are going through
beta testing and code polishing cycles now. I hope I would not have to
rewrite a lot of stuff that already works!

> The new sslBump flow could be something like this:
>
> A) open socket to server. If error, close socket to client.

If there is an error, bump-ssl-server-first returns an error to the
client, after establishing a secure connection with it. Closing the
connection can sometimes be a good option as well, of course.

> B) do the logic for ICAP REQMOD CONNECT endpoint:443

Bump-ssl-server-first does not change the order of ICAP processing and
server connection establishment. And it would be wrong to change it,
IMO. In other words, your (B) should come before (A) because (B) may
change where we are connecting or even prohibit the CONNECT request
(among other things):

  1. Receive CONNECT.
  2. Authenticate/etc.
  3. Adapt/redirect/etc.
  4. Bump.

> C) start SSL handshake to server and take care of all certificate issues.

Bump-ssl-server-first does that.

> If the SSL handshake fails with a PROTOCOL error, the socket must be
> closed,
> a new socket must be opened, and Squid will assume that the endpoint
> uses an other protocol than SSL. Squid goes into tunnel mode and all
> filtering will be done by the new pipe filter.
> Squid may get a new option to define its behaviour in case the SSL
> handshake
> fails. The options could be called sslBumpForNoneSSL with values
> prohibitNoneSSL (terminate connection), passNoneSSL (always allow),
> filterNoneSSL (default value - let new pipe filter decide).

s/None/Not/ or s/None/Non/

I suspect we should not allow any default here because the right
decision is impossible to guess correctly as it depends on why SslBump
was enabled in the first place. We could

 - serve a secure error (current bump-server-first code);
 - tunnel things through (not yet supported),
 - terminate the client connection (not yet supported);
 - ask a 3rd party filter (supported via external ACLs?).

> D) Squid now knows that the connection has a SSL/TLS wrapper but does
> not know yet if inside the wrapper HTTP is used.
> Squid monitors what the client *and* the server send on the pipe. If the
> client sends first and sends a valid HTTP command, Squid assumes that
> the connection has SSL+HTTP.
> If there is no SSL+HTTP Squid goes into tunnel mode

Yes, provided Squid was configured to do that.

> and all filtering will be done with the new pipe filter.

Yes, provided Squid was configured to do that.

> E) do the "normal processing" and ICAP REQMOD/RESPMOD for
> https://endpoint/path
>
> The total work of Squid+filter can be reduced if B) is done after C) since
> Squid can inform the filter about the SSL handshake and the filter does
> not have to do its own probe.

If the filter needs handshake information, it can get it when it is
available, but please do not change the order of ICAP/eCAP CONNECT
adaptation -- adaptation must happen before we do anything with the
CONNECT request (unless you add support for a post-cache REQMOD
vectoring point).

> There was a suggestion for a connection cache which allows it to skip
> checks and make assumptions about a new CONNECT to an endpoint that was
> CONNECTed before.

Sure, but that is just a secondary optimization.

> The new pipe filter requires a new protocol yet to be defined.

And/or a new API like eCAP.

> Squid initially tells the filter what it already knows about the endpoint.
> I.e. uses SSL or not, time to CONNECT, endpoint address, cached
> information. The Squid pipe sends copies of all data to the filter and the filter can
> reply with one of the following: OK (proceed with this data), REPLACE-CONTENT
> (content and a flag to optionally also terminate the connection), TERMINATE (just
> close sockets), OK-FOR-ALL (proceed and do not consult me any more for this
> connection). Squid also informs the filter when the connection is terminated by the
> client or the server.

The above is very similar to ICAP except every I/O (instead of every
HTTP message) becomes a transaction. I do not think you can specify all
ICAP-like interactions in a short paragraph. Even ICAP folks had to add
a few extensions to make the data pipeline more efficient, and here the
efficiency would be more critical because we would be dealing with
bytes, not messages.

It would be nice not to repeat ICAP errors for this new protocol/API. I
suggest that you look at relatively recent ICAP extension to incorporate
it in your design:
http://www.icap-forum.org/documents/specification/draft-icap-ext-partial-content-07.txt

> How do we go on from here?

I will respond to that separately.

Thank you,

Alex.
Received on Fri Mar 16 2012 - 22:45:23 MDT