🚀 go-pugleaf

On 04/11/2012 08:02:41 AM, Andy Lemin wrote:
> Hello,
> I know this has been discussed before, 

You will want to see this thread:
Working example of bi-directional asymmetric ALTQ + NAT ruleset?
http://marc.info/?t=129472965800001&r=1&w=2

It talks about being able to have a single queue
on more than one interface, so you'd use a single
outbound queue on all your internal interfaces to
effectively rate-limit your inbound wan traffic.
You'll want to use the hfsc scheduler because
you're trading bandwidth for latency.  And hfsc has
sub-queues too so that might help allocate the
traffic per internal interface.  I haven't thought 
about this in quite some time but I think that this approach
will work.  But because I've not thought about
it I could be all wrong.  :-)

It's not a perfect solution; it won't work in the
general case where you've more than one interface
you want to limit inbound traffic on.

If sharing a queue on your internal 
interfaces does not do it you could get ugly and
use an extra 2 real interfaces (instead of the loopback interface
as you suggest) and a separate routing table and physically
loop the traffic back.  This is less ugly than having another box.

I suspect the loopback interface approach won't work, but that's
a total guess.  If it does work I'm not sure I'd want
to count on such a kludge continuing to work long-term.

I'm very interested in what works and what doesn't so
it would be good to hear back from you.

> We have to use inbound queuing, without it our WAN link saturates 
> with
> low priority traffic, and we need to maintain headroom for high
> priority VoIP traffic etc.

Don't forget the "empty" ack packets.

> If we had to bounty this, how much? I might be able to get =A3100 for
> a
> bounty?

I heard the number $20,000 (US) thrown around.  I have no idea if
that's a realistic number.




Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

Hi, thanks very much for your suggestion.

> You will want to see this thread:
> Working example of bi-directional asymmetric ALTQ + NAT ruleset?
> http://marc.info/?t=129472965800001&r=1&w=2
> 
> It talks about being able to have a single queue
> on more than one interface, so you'd use a single
> outbound queue on all your internal interfaces to
> effectively rate-limit your inbound wan traffic.

I read through that archive but the problem it seems to be discussing seems pretty trivial as its discussing semantics of queuing downstream traffic which you just do by queuing on the internal interface (which works perfectly if you only have one internal interface).

It does suggest creating a queue structure without assigning a specific interface as you suggest, however when you do this I believe PF simply creates copy's of the queue structure and creates instances per interfaces? I.e. the different interfaces are not sharing the same physical queue in memory. Is this true?

I need to be able to perform downstream queuing pre-routing effectively so that all the downstream traffic (regardless of the internal zone) is all queued equally as one, allowing all internal interfaces to share the total downstream bandwidth.


> If sharing a queue on your internal 
> interfaces does not do it you could get ugly and
> use an extra 2 real interfaces (instead of the loopback interface
> as you suggest) and a separate routing table and physically
> loop the traffic back.  This is less ugly than having another box.

Sadly we don't have enough ports/slots on our firewall to add a physical loop back and so it has to be a virtual interface :(


> I suspect the loopback interface approach won't work, but that's
> a total guess.  If it does work I'm not sure I'd want
> to count on such a kludge continuing to work long-term.

Can you tell me why you feel it's a hack to do it using a virtual interface (assuming it works)?
It seems like a legitimate solution for queuing downstream traffic pre-routing (altq processing is done during the last stage of egress processing) as this would allow you to define unique downstream queues on any/multiple sources (WAN or VPN) etc.

Or do you think it would be better to allow multiple internal interfaces to share the same physical queue in memory?

Whilst its an ALTQ thing, I would assume both solutions would be best implemented in PF?

I'm just looking for ideas to try as I'm genuinely surprised BSD can't do this :(


> I'm very interested in what works and what doesn't so
> it would be good to hear back from you.

Yes I'll keep my progress posted, as I know their are a lot of other people out there who also want this functionality considering the number of threads I have come across on this, some dating back years.


> Don't forget the "empty" ack packets.

Indeed ;)


> > If we had to bounty this, how much? I might be able to get =A3100 for
> > a
> > bounty?
> 
> I heard the number $20,000 (US) thrown around.  I have no idea if
> that's a realistic number.
> 

That's a little out of my budget of a few hundred dollars! :(

Cheers for your time, Andy.

On 2012/04/12 02:48, Andy Lemin wrote:
> Hi, thanks very much for your suggestion.
> 
> > You will want to see this thread:
> > Working example of bi-directional asymmetric ALTQ + NAT ruleset?
> > http://marc.info/?t=3D129472965800001&r=3D1&w=3D2
> >=20
> > It talks about being able to have a single queue
> > on more than one interface, so you'd use a single
> > outbound queue on all your internal interfaces to
> > effectively rate-limit your inbound wan traffic.
> 
> I read through that archive but the problem it seems to be discussing seems=
>  pretty trivial as its discussing semantics of queuing downstream traffic w=
> hich you just do by queuing on the internal interface (which works perfectl=
> y if you only have one internal interface).
>
> It does suggest creating a queue structure without assigning a specific int=
> erface as you suggest, however when you do this I believe PF simply creates=
>  copy's of the queue structure and creates instances per interfaces? I.e. t=
> he different interfaces are not sharing the same physical queue in memory. =
> Is this true?

Yes, that thread is talking about creating multiple queues with the
same name, this is good for traffic classification but it is not a
single queue shared between multiple interfaces.

> I need to be able to perform downstream queuing pre-routing effectively so =
> that all the downstream traffic (regardless of the internal zone) is all qu=
> eued equally as one, allowing all internal interfaces to share the total do=
> wnstream bandwidth.

Currently two machines is the cleanest (and maybe only) way to do this,
I have a feeling you'll get better results making these route rather than
bridge.

altq's queues are all per-interface. Anything to change this would
be major work. Plan is for the queue code to be completely replaced
so working on this before that happens would get in the way.

> > If sharing a queue on your internal=20
> > interfaces does not do it you could get ugly and
> > use an extra 2 real interfaces (instead of the loopback interface
> > as you suggest) and a separate routing table and physically
> > loop the traffic back.  This is less ugly than having another box.
> 
> Sadly we don't have enough ports/slots on our firewall to add a physical lo=
> op back and so it has to be a virtual interface :(

lo(4) is a bit of a specialist interface to be trying this with.

You might get somewhere with vether(4), the uncommitted diff "virtual
crossover link with vether(4)" on tech@openbsd.org could be of interest.

Another option might be to route things to a userland daemon via tun(4)
that reinjects the packets then you can queue on the tun interface
instead. I have some old code from kjc sitting in my inbox that might
be useful here, I'll try and find time to dust it off. Higher overhead
but somewhere that is going to need to 'borrow' downstream bandwidth
between interfaces is likely to not be seeing super-high-speed
traffic anyway so this might not be a problem.

On 04/12/2012 07:40:47 AM, Stuart Henderson wrote:
> Yes, that thread is talking about creating multiple queues with the
> same name, this is good for traffic classification but it is not a
> single queue shared between multiple interfaces.

I am all wet then.  Isn't the first time.  Sorry to distract you.



Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein