Re: Squid 3.2 performance question

Sorry for disinformation, BIND requests are present in both RSBAC logs.
IOCTL's were removed by adding --disable-eui to Squid configuration
command, but that did not give any performance increase.

On Tue, 2012-03-20 at 12:37 +0400, Alexander Komyagin wrote:
> On Mon, 2012-03-19 at 23:26 -0600, Alex Rousskov wrote:
> > On 03/18/2012 11:07 PM, Amos Jeffries wrote:
> > > On 13/03/2012 10:14 p.m., Alexander Komyagin wrote:
> > >> Hello. We're now trying to give a chance to the new Squid 3.2 on our
> > >> server, mainly because of it's SMP feature. But our tests are showing
> > >> that 3.2 (3.2.0.14 and 3.2.0.16 were tested) performance is noticeably
> > >> lower than 3.1 (3.1.15).
> > >>
> > >> We're using "httperf --client=0/1 --hog --server x.x.x.x --rate=100
> > >> --num-conns=1000 --timeout=5 --num-calls=10" for testing. And for 3.2
> > >> it's showing about 140 client timeouts (from 1000), while for 3.1 there
> > >> are no errors at all.
> > >>
> > >> Different workers numbers were checked (1,2,4), but results are still
> > >> the same -- completely unchanged -- which is rather _strange_, since as
> > >> far as I know (by squid website and source browsing), in our
> > >> configuration workers shall NOT share anything but one listening socket
> > >> (y.y.y.y:3128).
> > >> More than that, CPU use is _only_ about 20% per worker (2 CPU's - 2
> > >> workers), vmstat reports no high memory consumption and iostat reports
> > >> 0% on iowait.
> > >>
> > >> Also according to logs, that clients timeouts are caused by some of new
> > >> connections not being spotted and accepted as well (not gone through
> > >> doAccept() routine from TcpAcceptor.cc).
> > >
> > > That is sounding very much like a kernel issue, or TCP accept rate
> > > limiting issue.
> >
> > Why would Squid v3.1 results differ from single-worker Squid v3.2
> > results then? I assume both v3.1 and v3.2 use the same kernel and the
> > same OS configuration (including ulibc).
>
> Actually, I don't know for sure. That's why I'm asking for help ;) I
> have even tried running squid 3.2 in non-daemon mode (pure single
> thread) - still no luck.
>
> >
> >
> > > Once a TCP connection is picked up by oldAccept() in the doAccept()
> > > sequence the results can be attributed to Squid, but if they never
> > > actually arrive there something is wrong at a deeper level down around
> > > the TCP stack or sockets libraries.
> >
> > > So from your results I conclude that one worker grabbed almost all the
> > > traffic and responded OK. But there is insufficient data about the
> > > interesting part of the traffic. What was going on there? which kid
> > > serviced it?
>
> Nope. Both workers are doing their job. Just not very well.
>
> >
> > I agree that making one of the workers super fast essentially
> > invalidates the test (unless you do the same to v3.1 too, but then you
> > just removed or scaled up the problem so it may not be the best test
> > direction anyway).
> >
> >
> > My recommendation is to use a single v3.2 worker for now and figure out
> > why a single v3.2 worker is dropping or ignoring connections when v3.1
> > does not. There could be bugs in the new accept code that we need to
> > fix. Use no-daemon mode for both versions.
> >
> > I would start by trying to understand whether those connection errors
> > result from connections never seen by Squid or from connections accepted
> > but later ignored/forgotten by Squid. I do not know much about httperf,
> > but with just 1000 transactions, that should be relatively easy to
> > determine because you can record and match each transaction on both
> > sides of the test.
> >
> >
> > HTH,
> >
> > Alex.
>
> Alex, I have performed some more tests (including oprofile profiling,
> no-daemon mode, 1 worker, 2 workers, etc.). For now, it seems that the
> problem is highly related to RSBAC Networking which is enabled in our
> kernel. When I disabled it, the performance issue _has gone_. According
> to RSBAC logs, no single operation is denied.
>
> With RSBAC-Net enabled, 3.2 with 1, 2 workers and in no-daemon mode
> produces the problem. However, 3.1 works fine.
>
> Without RSBAC-Net everything is fine.
>
> By comparing oprofile results for 3.2 with and w/o RSBAC-Net, I can
> assume that RSBAC-Net subsystem performs some internal operations on
> list structures, which are indeed protected by locks - and this, in my
> point of view, may block simultaneous squid socket operations and affect
> performance.
>
> Also when I enable RSBAC full logging for squid process, 3.2 and 3.1
> logs are different in two points:
> - 3.2 has some mystical IOCTL operations on TCP sockets, right after
> create, while 3.1 hasn't;
> - 3.1 produces BIND requests, while 3.2 doesn't.
>
> So far I agree that the problem probably resides in a socket level, but
> still wonder what the significant difference between 3.1 socket ops and
> 3.2?
>
> I'll check squid sources again in hope to find the answers.
>

-- 
Best wishes,
Alexander Komyagin