Discussion:
[nznog] Hello Hurricane Electric, doubling the size of our domestic table?
Michael Fincham
2018-09-18 22:41:45 UTC
Permalink
Hi all,

So my domestic tables now have some 76,000 prefixes in them, as of ~ 6:30 last night.

I guess hello HE and welcome to AKL-IX?

Domestic table for NZ was more like ~ 10,000 prefixes not that long ago!

--
Michael
Liam Farr
2018-09-18 22:46:46 UTC
Permalink
They moved into 220q a few weeks back, routes went live on Megaport
Auckland yesterday.

They’re also selling transit now in Auckland, (great to see more options
arriving locally).

It does however seem that Auckland is just an extension of their Sydney POP
with traffic going AKL>SYD>USA.


On Wed, 19 Sep 2018 at 10:42 AM, Michael Fincham <***@hotplate.co.nz>
wrote:

> Hi all,
>
> So my domestic tables now have some 76,000 prefixes in them, as of ~ 6:30
> last night.
>
> I guess hello HE and welcome to AKL-IX?
>
> Domestic table for NZ was more like ~ 10,000 prefixes not that long ago!
>
> --
> Michael
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
--
Kind Regards


Liam Farr

Maxum Data
+64-9-950-5302
Michael Fincham
2018-09-18 22:47:32 UTC
Permalink
On Wed, 19 Sep 2018 10:41:45 +1200
Michael Fincham <***@hotplate.co.nz> wrote:

> I guess hello HE and welcome to AKL-IX?

Sorry, Megaport.

--
Michael
Liam Farr
2018-09-18 22:56:51 UTC
Permalink
One would assume however that being based at 220q they will also pickup
AKL-IX in due course as the other main Auckland IX.

On Wed, 19 Sep 2018 at 10:47 AM, Michael Fincham <***@hotplate.co.nz>
wrote:

> On Wed, 19 Sep 2018 10:41:45 +1200
> Michael Fincham <***@hotplate.co.nz> wrote:
>
> > I guess hello HE and welcome to AKL-IX?
>
> Sorry, Megaport.
>
> --
> Michael
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
--
Kind Regards


Liam Farr

Maxum Data
+64-9-950-5302
Joe Wooller
2018-09-18 23:28:03 UTC
Permalink
In progress
So please adjust your filters accordingly in preparation! :)

We will send out an email to AKL-IX participants shortly.

Thanks
Joe

> On 19 Sep 2018, at 6:56 am, Liam Farr <***@maxumdata.com> wrote:
>
> One would assume however that being based at 220q they will also pickup AKL-IX in due course as the other main Auckland IX.
>
> On Wed, 19 Sep 2018 at 10:47 AM, Michael Fincham <***@hotplate.co.nz <mailto:***@hotplate.co.nz>> wrote:
> On Wed, 19 Sep 2018 10:41:45 +1200
> Michael Fincham <***@hotplate.co.nz <mailto:***@hotplate.co.nz>> wrote:
>
> > I guess hello HE and welcome to AKL-IX?
>
> Sorry, Megaport.
>
> --
> Michael
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz <mailto:***@list.waikato.ac.nz>
> https://list.waikato.ac.nz/mailman/listinfo/nznog <https://list.waikato.ac.nz/mailman/listinfo/nznog>
> --
> Kind Regards
>
>
> Liam Farr
>
> Maxum Data
> +64-9-950-5302
Gavin Tweedie
2018-09-18 23:33:11 UTC
Permalink
HE peers everywhere they can, so yes I assume you'll see them on AKL-IX
too... APE might depend if they see value, last I checked between MegaIX
and AKL-IX you collected most of the routes on APE direct or indirect.

To be honest none of the IX's in AKL have been truely domestic for a long
time, there's been chunks of AU and other parts of the world, CDNs, global
OTTs (Microsoft etc) presenting non NZ routes. This has then flowed into
the concept of domestic transit, I've certainly seen bits of international
in there a long time.. as did AU when we used to talk about domestic table
too (and NZ was always in it!)

Ultimately it's probably a good thing, new names in a market is new
competition and even if you don't shop with them you'll probably gain from
the market getting a shake.

As a reminder, HE will give you even MORE routes if you bring up a
bilateral with them email the peering email @ https://peeringdb.com/net/291
or talk to Walt from Hurricane.. I'll throw him on the CC as I'm not sure
if he's on the list. Maybe he'll visit NZNOG next year.

Gav



On Wed, 19 Sep 2018 at 06:57, Liam Farr <***@maxumdata.com> wrote:

> One would assume however that being based at 220q they will also pickup
> AKL-IX in due course as the other main Auckland IX.
>
> On Wed, 19 Sep 2018 at 10:47 AM, Michael Fincham <***@hotplate.co.nz>
> wrote:
>
>> On Wed, 19 Sep 2018 10:41:45 +1200
>> Michael Fincham <***@hotplate.co.nz> wrote:
>>
>> > I guess hello HE and welcome to AKL-IX?
>>
>> Sorry, Megaport.
>>
>> --
>> Michael
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
> --
> Kind Regards
>
>
> Liam Farr
>
> Maxum Data
> +64-9-950-5302
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>


--
Global Interconnection Director
Megaport <https://www.megaport.com>
+61 498 498 458
Mike Taylor
2018-09-19 00:05:37 UTC
Permalink
Eek, I wonder if this is why my NZ domestic table size had a blow out at
around 0100 this morning and tipped one of my (smaller) boxes over the
edge...

I've spent all morning trying to triage the damn thing, knowing that the
BGP table had blown out and taken all of the TCAM memory...

We're still on the old style internet ISP plan from FX days;

1x International feed with a default route only

1x Domestic feed with All NZ Domestic routes, (and now a shedload more
in this table)


Cheers,

Mike



On 19/09/18 11:33, Gavin Tweedie wrote:
> HE peers everywhere they can, so yes I assume you'll see them on
> AKL-IX too... APE might depend if they see value, last I checked
> between MegaIX and AKL-IX you collected most of the routes on APE
> direct or indirect.
>
> To be honest none of the IX's in AKL have been truely domestic for a
> long time, there's been chunks of AU and other parts of the world,
> CDNs, global OTTs (Microsoft etc) presenting non NZ routes. This has
> then flowed into the concept of domestic transit, I've certainly seen
> bits of international in there a long time.. as did AU when we used to
> talk about domestic table too (and NZ was always in it!)
>
> Ultimately it's probably a good thing, new names in a market is new
> competition and even if you don't shop with them you'll probably gain
> from the market getting a shake.
>
> As a reminder, HE will give you even MORE routes if you bring up a
> bilateral with them email the peering email
> @ https://peeringdb.com/net/291 or talk to Walt from Hurricane.. I'll
> throw him on the CC as I'm not sure if he's on the list. Maybe he'll
> visit NZNOG next year.
>
> Gav
>
>
>
> On Wed, 19 Sep 2018 at 06:57, Liam Farr <***@maxumdata.com
> <mailto:***@maxumdata.com>> wrote:
>
> One would assume however that being based at 220q they will also
> pickup AKL-IX in due course as the other main Auckland IX. 
>
> On Wed, 19 Sep 2018 at 10:47 AM, Michael Fincham
> <***@hotplate.co.nz <mailto:***@hotplate.co.nz>> wrote:
>
> On Wed, 19 Sep 2018 10:41:45 +1200
> Michael Fincham <***@hotplate.co.nz
> <mailto:***@hotplate.co.nz>> wrote:
>
> > I guess hello HE and welcome to AKL-IX?
>
> Sorry, Megaport.
>
> --
> Michael
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz <mailto:***@list.waikato.ac.nz>
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
> --
> Kind Regards
>
>
> Liam Farr
>
> Maxum Data
> +64-9-950-5302
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz <mailto:***@list.waikato.ac.nz>
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
>
>
> --
> Global Interconnection Director
> Megaport <https://www.megaport.com>
> +61 498 498 458
>
>
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
Cameron Bradley
2018-09-19 00:26:33 UTC
Permalink
Yes,

This is exactly why this happened. Vocus made a change around 0050 this morning. We dropped some peerings when the prefix limits breached.

If your box isn't coping you could consider rejecting routes received from Megaport with HE in the path.

Cam

-----Original Message-----
From: nznog-***@list.waikato.ac.nz <nznog-***@list.waikato.ac.nz> On Behalf Of Mike Taylor
Sent: Wednesday, 19 September 2018 12:06 PM
To: ***@list.waikato.ac.nz
Subject: Re: [nznog] Hello Hurricane Electric, doubling the size of our domestic table?

Eek, I wonder if this is why my NZ domestic table size had a blow out at around 0100 this morning and tipped one of my (smaller) boxes over the edge...

I've spent all morning trying to triage the damn thing, knowing that the BGP table had blown out and taken all of the TCAM memory...

We're still on the old style internet ISP plan from FX days;

1x International feed with a default route only

1x Domestic feed with All NZ Domestic routes, (and now a shedload more in this table)


Cheers,

Mike



On 19/09/18 11:33, Gavin Tweedie wrote:
> HE peers everywhere they can, so yes I assume you'll see them on
> AKL-IX too... APE might depend if they see value, last I checked
> between MegaIX and AKL-IX you collected most of the routes on APE
> direct or indirect.
>
> To be honest none of the IX's in AKL have been truely domestic for a
> long time, there's been chunks of AU and other parts of the world,
> CDNs, global OTTs (Microsoft etc) presenting non NZ routes. This has
> then flowed into the concept of domestic transit, I've certainly seen
> bits of international in there a long time.. as did AU when we used to
> talk about domestic table too (and NZ was always in it!)
>
> Ultimately it's probably a good thing, new names in a market is new
> competition and even if you don't shop with them you'll probably gain
> from the market getting a shake.
>
> As a reminder, HE will give you even MORE routes if you bring up a
> bilateral with them email the peering email @ 
> https://peeringdb.com/net/291 or talk to Walt from Hurricane.. I'll
> throw him on the CC as I'm not sure if he's on the list. Maybe he'll
> visit NZNOG next year.
>
> Gav
>
>
>
> On Wed, 19 Sep 2018 at 06:57, Liam Farr <***@maxumdata.com
> <mailto:***@maxumdata.com>> wrote:
>
> One would assume however that being based at 220q they will also
> pickup AKL-IX in due course as the other main Auckland IX.
>
> On Wed, 19 Sep 2018 at 10:47 AM, Michael Fincham
> <***@hotplate.co.nz <mailto:***@hotplate.co.nz>> wrote:
>
> On Wed, 19 Sep 2018 10:41:45 +1200
> Michael Fincham <***@hotplate.co.nz
> <mailto:***@hotplate.co.nz>> wrote:
>
> > I guess hello HE and welcome to AKL-IX?
>
> Sorry, Megaport.
>
> --
> Michael
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz <mailto:***@list.waikato.ac.nz>
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
> --
> Kind Regards
>
>
> Liam Farr
>
> Maxum Data
> +64-9-950-5302
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz <mailto:***@list.waikato.ac.nz>
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
>
>
> --
> Global Interconnection Director
> Megaport <https://www.megaport.com>
> +61 498 498 458
>
>
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________
NZNOG mailing list
***@list.waikato.ac.nz
https://list.waikato.ac.nz/mailman/listinfo/nznog
Mike Taylor
2018-09-19 01:15:20 UTC
Permalink
Thanks Cam, good idea

Turns out that the node in question only has one exit point to the
internet anyway (now), so it only really needs a default route for IPv4
and the same for IPv6

Problem solved!


Regards,

Mike


Mike Taylor
The Total Team

XMPP: ***@gmail.com
0800 888 326 / +64 3 3779050

On 19/09/18 12:26, Cameron Bradley wrote:
> Yes,
>
> This is exactly why this happened. Vocus made a change around 0050 this morning. We dropped some peerings when the prefix limits breached.
>
> If your box isn't coping you could consider rejecting routes received from Megaport with HE in the path.
>
> Cam
>
> -----Original Message-----
> From: nznog-***@list.waikato.ac.nz <nznog-***@list.waikato.ac.nz> On Behalf Of Mike Taylor
> Sent: Wednesday, 19 September 2018 12:06 PM
> To: ***@list.waikato.ac.nz
> Subject: Re: [nznog] Hello Hurricane Electric, doubling the size of our domestic table?
>
> Eek, I wonder if this is why my NZ domestic table size had a blow out at around 0100 this morning and tipped one of my (smaller) boxes over the edge...
>
> I've spent all morning trying to triage the damn thing, knowing that the BGP table had blown out and taken all of the TCAM memory...
>
> We're still on the old style internet ISP plan from FX days;
>
> 1x International feed with a default route only
>
> 1x Domestic feed with All NZ Domestic routes, (and now a shedload more in this table)
>
>
> Cheers,
>
> Mike
>
>
>
> On 19/09/18 11:33, Gavin Tweedie wrote:
>> HE peers everywhere they can, so yes I assume you'll see them on
>> AKL-IX too... APE might depend if they see value, last I checked
>> between MegaIX and AKL-IX you collected most of the routes on APE
>> direct or indirect.
>>
>> To be honest none of the IX's in AKL have been truely domestic for a
>> long time, there's been chunks of AU and other parts of the world,
>> CDNs, global OTTs (Microsoft etc) presenting non NZ routes. This has
>> then flowed into the concept of domestic transit, I've certainly seen
>> bits of international in there a long time.. as did AU when we used to
>> talk about domestic table too (and NZ was always in it!)
>>
>> Ultimately it's probably a good thing, new names in a market is new
>> competition and even if you don't shop with them you'll probably gain
>> from the market getting a shake.
>>
>> As a reminder, HE will give you even MORE routes if you bring up a
>> bilateral with them email the peering email @ 
>> https://peeringdb.com/net/291 or talk to Walt from Hurricane.. I'll
>> throw him on the CC as I'm not sure if he's on the list. Maybe he'll
>> visit NZNOG next year.
>>
>> Gav
>>
>>
>>
>> On Wed, 19 Sep 2018 at 06:57, Liam Farr <***@maxumdata.com
>> <mailto:***@maxumdata.com>> wrote:
>>
>> One would assume however that being based at 220q they will also
>> pickup AKL-IX in due course as the other main Auckland IX.
>>
>> On Wed, 19 Sep 2018 at 10:47 AM, Michael Fincham
>> <***@hotplate.co.nz <mailto:***@hotplate.co.nz>> wrote:
>>
>> On Wed, 19 Sep 2018 10:41:45 +1200
>> Michael Fincham <***@hotplate.co.nz
>> <mailto:***@hotplate.co.nz>> wrote:
>>
>> > I guess hello HE and welcome to AKL-IX?
>>
>> Sorry, Megaport.
>>
>> --
>> Michael
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz <mailto:***@list.waikato.ac.nz>
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
>> --
>> Kind Regards
>>
>>
>> Liam Farr
>>
>> Maxum Data
>> +64-9-950-5302
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz <mailto:***@list.waikato.ac.nz>
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
>>
>>
>> --
>> Global Interconnection Director
>> Megaport <https://www.megaport.com>
>> +61 498 498 458
>>
>>
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
Michael Fincham
2018-09-19 04:17:35 UTC
Permalink
OK, so the routes are on AKL-IX now [cue a bunch more peoples import limits breaking again].

Did we not learn anything from the last time this broke a whole bunch of stuff?

HE needs to be giving us some heads up before they turn up here and increase the size of NZ's domestic route table by an order of magnitude...

Hello?

--
Michael
Dave Mill
2018-09-19 04:23:56 UTC
Permalink
Megaport AKL emailed their customers multiple times about HE turning up
their services.

NZIX also emailed.

So I assume its just 'domestic' transit providers that maybe aren't
notifying us?

Dave

On Wed, Sep 19, 2018 at 4:17 PM, Michael Fincham <***@hotplate.co.nz>
wrote:

> OK, so the routes are on AKL-IX now [cue a bunch more peoples import
> limits breaking again].
>
> Did we not learn anything from the last time this broke a whole bunch of
> stuff?
>
> HE needs to be giving us some heads up before they turn up here and
> increase the size of NZ's domestic route table by an order of magnitude...
>
> Hello?
>
> --
> Michael
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
Michael Fincham
2018-09-19 04:25:35 UTC
Permalink
On Wed, 19 Sep 2018 16:23:56 +1200
Dave Mill <***@mill.net.nz> wrote:

> Megaport AKL emailed their customers multiple times about HE turning up
> their services.
>
> NZIX also emailed.
>
> So I assume its just 'domestic' transit providers that maybe aren't
> notifying us?

I don't think this is the right way to look at it. Those routes are going to end up in everyone's "domestic" feeds. HE should have been here on NZNOG months ago making sure everyone was OK with this and ready.

Nobody is running a 10x safety margin on their BGP limits...

--
Michael
Michael Fincham
2018-09-19 04:27:05 UTC
Permalink
On Wed, 19 Sep 2018 16:25:35 +1200
Michael Fincham <***@hotplate.co.nz> wrote:

> I don't think this is the right way to look at it. Those routes are going to end up in everyone's "domestic" feeds. HE should have been here on NZNOG months ago making sure everyone was OK with this and ready.

And to elaborate on this: the Internet is a community resource and right now HE are being bad neighbors by not participating appropriately.

--
Michael
Liam Farr
2018-09-19 04:27:16 UTC
Permalink
They did notify, "months ago" back in June... (as below) and again
yesterday.

-------------------------------
Date: Tue, 18 Sep 2018 at 11:30
Subject: Re: [Megaport Auckland IX] Upcoming large increase in prefixes
from MegaIX Auckland Route Servers
To: <auckland-ix-***@lists.megaport.com>


Hi AKL peers,

Just a reminder regarding the increase of prefixes from the MegaIX Auckland
Route Servers mentioned below. We are expecting the increase to occur in
the next few days depending on peer turn-up schedule.

Gavin


On Fri, 29 Jun 2018 at 07:54, Gavin Tweedie <***@megaport.com>
wrote:

> Hi Auckland Peers,
>
> Please be aware that soon(ish) we are expecting an increase in prefixes on
> the Auckland MegaIX Route Servers driven by Hurricane Electric (AS6939)
> <https://peeringdb.com/net/291> connecting to the IX within the next 60
> days.
>
>
> If you set a maximum prefix (MaxPFX) limit on the BGP neighbours with our
> route servers or if you have other limitations in terms of prefixes you can
> receive from the route server you should start preparing.
>
> Our new recommended MaxPFX setting is always maintained in route-server
> peeringdb entries (Auckland @ https://peeringdb.com/net/12814) in case
> you automate this.
>
>
>
> For reference our Auckland Route servers are AS63839,
>
> 43.243.22.1 & 43.243.22.2
>
> 2001:dea:0:40::1 & 2001:dea:0:40::2
>
>
>
> The new recommended minimum MaxPFX settings are,
>
> IPv4 65,000
>
> IPv6 35,000
>
>
>
> If you are not sure if your router is able to handle such a large table I
> would recommend filtering by AS-Path routes received from AS6939 from the
> route server while you investigate it so that you can experiment in a
> window that suits you, not just when the routes show up.
>
>
> If you require any assistance please feel free to reach out to myself or
> our NOC team via our portal live chat or via email at ***@megaport.com
>
>
>
>
> Regards
>
> Gavin
>
> --
>
> Global Interconnection Director
>
> Megaport <https://www.megaport.com/>
>
> +61 498 498 458
>


On Wed, 19 Sep 2018 at 16:25, Michael Fincham <***@hotplate.co.nz>
wrote:

> On Wed, 19 Sep 2018 16:23:56 +1200
> Dave Mill <***@mill.net.nz> wrote:
>
> > Megaport AKL emailed their customers multiple times about HE turning up
> > their services.
> >
> > NZIX also emailed.
> >
> > So I assume its just 'domestic' transit providers that maybe aren't
> > notifying us?
>
> I don't think this is the right way to look at it. Those routes are going
> to end up in everyone's "domestic" feeds. HE should have been here on NZNOG
> months ago making sure everyone was OK with this and ready.
>
> Nobody is running a 10x safety margin on their BGP limits...
>
> --
> Michael
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>


--
Kind Regards


Liam Farr

Maxum Data
+64-9-950-5302
Michael Fincham
2018-09-19 04:28:36 UTC
Permalink
On Wed, 19 Sep 2018 16:27:16 +1200
Liam Farr <***@maxumdata.com> wrote:

> They did notify, "months ago" back in June... (as below) and again
> yesterday.

Yes, but how is this meant to filter out to all the other operators not on that exchange exactly, who have spent much of the day fixing things and helping other people fix things?

--
Michael
Liam Farr
2018-09-19 04:31:45 UTC
Permalink
If your not on the exchange directly, you only buy (domestic) transit from
an upstream, and your upstream didn't notify you... perhaps its time to
find a new upstream or have a long hard chat with your account manager /
their ops team?

On Wed, 19 Sep 2018 at 16:29, Michael Fincham <***@hotplate.co.nz>
wrote:

> On Wed, 19 Sep 2018 16:27:16 +1200
> Liam Farr <***@maxumdata.com> wrote:
>
> > They did notify, "months ago" back in June... (as below) and again
> > yesterday.
>
> Yes, but how is this meant to filter out to all the other operators not on
> that exchange exactly, who have spent much of the day fixing things and
> helping other people fix things?
>
> --
> Michael
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>


--
Kind Regards


Liam Farr

Maxum Data
+64-9-950-5302
Michael Fincham
2018-09-19 04:35:34 UTC
Permalink
On Wed, 19 Sep 2018 16:32:09 +1200
Nathan Ward <***@daork.net> wrote:

> Through whoever you buy domestic transit from - communication for these sorts of changes, and if required protecting customers who want to run such a service on c2610s, is a core part of such a service. If your transit providers are unable to do this I would suggest looking towards other operators.

This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of stuff, doesn't even bother to post to NZNOG letting us know they're going to break things, and you're just blaming my transit provider?

Also, none of the IXes posted on NZNOG?

--
Michael
Nathan Brookfield
2018-09-19 04:38:17 UTC
Permalink
The IX’s mailed out to their customers, I guess they assumed it would be unwelcome noise and that members of the exchanges would get notified that way

Nathan Brookfield
Chief Executive Officer

Simtronic Technologies Pty Ltd
http://www.simtronic.com.au

On 19 Sep 2018, at 14:36, Michael Fincham <***@hotplate.co.nz> wrote:

On Wed, 19 Sep 2018 16:32:09 +1200
Nathan Ward <***@daork.net> wrote:

> Through whoever you buy domestic transit from - communication for these sorts of changes, and if required protecting customers who want to run such a service on c2610s, is a core part of such a service. If your transit providers are unable to do this I would suggest looking towards other operators.

This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of stuff, doesn't even bother to post to NZNOG letting us know they're going to break things, and you're just blaming my transit provider?

Also, none of the IXes posted on NZNOG?

--
Michael
_______________________________________________
NZNOG mailing list
***@list.waikato.ac.nz
https://list.waikato.ac.nz/mailman/listinfo/nznog
Dave Mill
2018-09-19 04:39:29 UTC
Permalink
On Wed, Sep 19, 2018 at 4:35 PM, Michael Fincham <***@hotplate.co.nz>
wrote:

> On Wed, 19 Sep 2018 16:32:09 +1200
> Nathan Ward <***@daork.net> wrote:
>
> > Through whoever you buy domestic transit from - communication for these
> sorts of changes, and if required protecting customers who want to run such
> a service on c2610s, is a core part of such a service. If your transit
> providers are unable to do this I would suggest looking towards other
> operators.
>
> This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of
> stuff, doesn't even bother to post to NZNOG letting us know they're going
> to break things, and you're just blaming my transit provider?
>
> Also, none of the IXes posted on NZNOG?
>
>
Joe emailed NZNOG at 11.28am today. Sorry that there wasn't more notice.

Dave
Joe Wooller
2018-09-19 04:39:53 UTC
Permalink
Hey All,

We have only opened the filters for HE up on one of our route servers (RS2). We chose RS2 because unfortunately only 50% of our peers have actually bothered to configure their session to this route server.
The reason we chose to do it this way is to reduce the impact on participants. If you hit the limit the session drops, but RS1 will still hum along. We plan to open up RS1 when we see all the peers that did drop resume normal operation on RS2.

We only just sent out an email about this today. So sorry for the lack of notice on this one, my bad.

I want to also use this opportunity to remind people to bring up their sessions with our second route server if they have not done so already! Feel free to ping me off list if you need help and/or the details!

Cheers
Joe


> On 19 Sep 2018, at 12:35 pm, Michael Fincham <***@hotplate.co.nz> wrote:
>
> On Wed, 19 Sep 2018 16:32:09 +1200
> Nathan Ward <***@daork.net> wrote:
>
>> Through whoever you buy domestic transit from - communication for these sorts of changes, and if required protecting customers who want to run such a service on c2610s, is a core part of such a service. If your transit providers are unable to do this I would suggest looking towards other operators.
>
> This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of stuff, doesn't even bother to post to NZNOG letting us know they're going to break things, and you're just blaming my transit provider?
>
> Also, none of the IXes posted on NZNOG?
>
> --
> Michael
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
Liam Farr
2018-09-19 04:39:50 UTC
Permalink
IMO I think its great that a large carrier has come to our neck of the
woods, and they are openly peering on both our major IX's. This is a good
thing for internet in NZ.

Notifications were sent to the affected parties who peer, months in advance.

I don't believe HE had any malicious intent to break your internet.

If your stuff broke because you have a shit upstream, fix it and move on.

On Wed, 19 Sep 2018 at 16:35, Michael Fincham <***@hotplate.co.nz>
wrote:

> On Wed, 19 Sep 2018 16:32:09 +1200
> Nathan Ward <***@daork.net> wrote:
>
> > Through whoever you buy domestic transit from - communication for these
> sorts of changes, and if required protecting customers who want to run such
> a service on c2610s, is a core part of such a service. If your transit
> providers are unable to do this I would suggest looking towards other
> operators.
>
> This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of
> stuff, doesn't even bother to post to NZNOG letting us know they're going
> to break things, and you're just blaming my transit provider?
>
> Also, none of the IXes posted on NZNOG?
>
> --
> Michael
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>


--
Kind Regards


Liam Farr

Maxum Data
+64-9-950-5302
Michael Fincham
2018-09-19 04:44:11 UTC
Permalink
On Wed, 19 Sep 2018 16:39:50 +1200
Liam Farr <***@maxumdata.com> wrote:

> IMO I think its great that a large carrier has come to our neck of the
> woods, and they are openly peering on both our major IX's. This is a good
> thing for internet in NZ.

No contest. And I'm sure HE isn't here intentionally breaking the Internet.

It is great they're here. I am just unhappy that nobody thought that maybe increasing the size of the table we all carry as "national" by this much wasn't worth mentioning to the operator community.

I really don't think this is a "your transit provider" is bad moment. Who actually got a message about this from their transit provider?

--
Michael
Bill
2018-09-19 04:46:57 UTC
Permalink
Perhaps those that had issues should use such tools as prefix limits on their inbound feeds. We get notified by our edge routers quite regularly that the number of prefixes has or is about to max out.

Sent from my iPhone

> On 19/09/2018, at 4:39 PM, Liam Farr <***@maxumdata.com> wrote:
>
> IMO I think its great that a large carrier has come to our neck of the woods, and they are openly peering on both our major IX's. This is a good thing for internet in NZ.
>
> Notifications were sent to the affected parties who peer, months in advance.
>
> I don't believe HE had any malicious intent to break your internet.
>
> If your stuff broke because you have a shit upstream, fix it and move on.
>
>> On Wed, 19 Sep 2018 at 16:35, Michael Fincham <***@hotplate.co.nz> wrote:
>> On Wed, 19 Sep 2018 16:32:09 +1200
>> Nathan Ward <***@daork.net> wrote:
>>
>> > Through whoever you buy domestic transit from - communication for these sorts of changes, and if required protecting customers who want to run such a service on c2610s, is a core part of such a service. If your transit providers are unable to do this I would suggest looking towards other operators.
>>
>> This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of stuff, doesn't even bother to post to NZNOG letting us know they're going to break things, and you're just blaming my transit provider?
>>
>> Also, none of the IXes posted on NZNOG?
>>
>> --
>> Michael
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
>
> --
> Kind Regards
>
>
> Liam Farr
>
> Maxum Data
> +64-9-950-5302
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
Tim Hoffman
2018-09-19 04:50:06 UTC
Permalink
Don't forget that the NZ/AU concept of 'domestic table' is a fairly unique
concept that isn't seen in much of the rest of the world- generally any
feed is going to be default, a peering feed (customer routes), or a full
table.... Doubt anyone thought of it ;).

Congrats to Mike & Walt on extending to NZ... pretty cool to see a
worldwide carrier finally making it there - fantastic to see how far the NZ
internet has progressed!

On Tue, Sep 18, 2018 at 9:47 PM Bill <***@wjw.nz> wrote:

>
> Perhaps those that had issues should use such tools as prefix limits on
> their inbound feeds. We get notified by our edge routers quite regularly
> that the number of prefixes has or is about to max out.
>
> Sent from my iPhone
>
> On 19/09/2018, at 4:39 PM, Liam Farr <***@maxumdata.com> wrote:
>
> IMO I think its great that a large carrier has come to our neck of the
> woods, and they are openly peering on both our major IX's. This is a good
> thing for internet in NZ.
>
> Notifications were sent to the affected parties who peer, months in
> advance.
>
> I don't believe HE had any malicious intent to break your internet.
>
> If your stuff broke because you have a shit upstream, fix it and move on.
>
> On Wed, 19 Sep 2018 at 16:35, Michael Fincham <***@hotplate.co.nz>
> wrote:
>
>> On Wed, 19 Sep 2018 16:32:09 +1200
>> Nathan Ward <***@daork.net> wrote:
>>
>> > Through whoever you buy domestic transit from - communication for these
>> sorts of changes, and if required protecting customers who want to run such
>> a service on c2610s, is a core part of such a service. If your transit
>> providers are unable to do this I would suggest looking towards other
>> operators.
>>
>> This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of
>> stuff, doesn't even bother to post to NZNOG letting us know they're going
>> to break things, and you're just blaming my transit provider?
>>
>> Also, none of the IXes posted on NZNOG?
>>
>> --
>> Michael
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
>
>
> --
> Kind Regards
>
>
> Liam Farr
>
> Maxum Data
> +64-9-950-5302
>
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
Tom Paseka
2018-09-19 05:02:35 UTC
Permalink
If I can ask, what does a "domestic" table genuinely bring you. If you
carry routes to your edge(s) that have both peering and transit, do you
need to extend the DFZ all the way through your network? If you've got a
single exit path (or a few) do you need to extend the domestic routes any
further?

Another thing to note: BGP Sanity. Apply AS_PATH filters to all your peers.
This doubly so if you're extending the routes throughout your network. Much
of HE's routes might not be relevant to your network and could have easily
been sanitized with some nice filters.

Also congrats to HE/Team!
-Tom

On Tue, Sep 18, 2018 at 9:50 PM Tim Hoffman <***@hoffman.net.nz> wrote:

> Don't forget that the NZ/AU concept of 'domestic table' is a fairly unique
> concept that isn't seen in much of the rest of the world- generally any
> feed is going to be default, a peering feed (customer routes), or a full
> table.... Doubt anyone thought of it ;).
>
> Congrats to Mike & Walt on extending to NZ... pretty cool to see a
> worldwide carrier finally making it there - fantastic to see how far the NZ
> internet has progressed!
>
> On Tue, Sep 18, 2018 at 9:47 PM Bill <***@wjw.nz> wrote:
>
>>
>> Perhaps those that had issues should use such tools as prefix limits on
>> their inbound feeds. We get notified by our edge routers quite regularly
>> that the number of prefixes has or is about to max out.
>>
>> Sent from my iPhone
>>
>> On 19/09/2018, at 4:39 PM, Liam Farr <***@maxumdata.com> wrote:
>>
>> IMO I think its great that a large carrier has come to our neck of the
>> woods, and they are openly peering on both our major IX's. This is a good
>> thing for internet in NZ.
>>
>> Notifications were sent to the affected parties who peer, months in
>> advance.
>>
>> I don't believe HE had any malicious intent to break your internet.
>>
>> If your stuff broke because you have a shit upstream, fix it and move on.
>>
>> On Wed, 19 Sep 2018 at 16:35, Michael Fincham <***@hotplate.co.nz>
>> wrote:
>>
>>> On Wed, 19 Sep 2018 16:32:09 +1200
>>> Nathan Ward <***@daork.net> wrote:
>>>
>>> > Through whoever you buy domestic transit from - communication for
>>> these sorts of changes, and if required protecting customers who want to
>>> run such a service on c2610s, is a core part of such a service. If your
>>> transit providers are unable to do this I would suggest looking towards
>>> other operators.
>>>
>>> This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch
>>> of stuff, doesn't even bother to post to NZNOG letting us know they're
>>> going to break things, and you're just blaming my transit provider?
>>>
>>> Also, none of the IXes posted on NZNOG?
>>>
>>> --
>>> Michael
>>> _______________________________________________
>>> NZNOG mailing list
>>> ***@list.waikato.ac.nz
>>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>>
>>
>>
>> --
>> Kind Regards
>>
>>
>> Liam Farr
>>
>> Maxum Data
>> +64-9-950-5302
>>
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
Dave Mill
2018-09-19 05:05:44 UTC
Permalink
Buying 'domestic' is a way of paying for the privilege of sending/receiving
traffic to Spark :)

On Wed, Sep 19, 2018 at 5:02 PM, Tom Paseka <tom+***@cloudflare.com>
wrote:

> If I can ask, what does a "domestic" table genuinely bring you. If you
> carry routes to your edge(s) that have both peering and transit, do you
> need to extend the DFZ all the way through your network? If you've got a
> single exit path (or a few) do you need to extend the domestic routes any
> further?
>
> Another thing to note: BGP Sanity. Apply AS_PATH filters to all your
> peers. This doubly so if you're extending the routes throughout your
> network. Much of HE's routes might not be relevant to your network and
> could have easily been sanitized with some nice filters.
>
> Also congrats to HE/Team!
> -Tom
>
> On Tue, Sep 18, 2018 at 9:50 PM Tim Hoffman <***@hoffman.net.nz> wrote:
>
>> Don't forget that the NZ/AU concept of 'domestic table' is a fairly
>> unique concept that isn't seen in much of the rest of the world- generally
>> any feed is going to be default, a peering feed (customer routes), or a
>> full table.... Doubt anyone thought of it ;).
>>
>> Congrats to Mike & Walt on extending to NZ... pretty cool to see a
>> worldwide carrier finally making it there - fantastic to see how far the NZ
>> internet has progressed!
>>
>> On Tue, Sep 18, 2018 at 9:47 PM Bill <***@wjw.nz> wrote:
>>
>>>
>>> Perhaps those that had issues should use such tools as prefix limits on
>>> their inbound feeds. We get notified by our edge routers quite regularly
>>> that the number of prefixes has or is about to max out.
>>>
>>> Sent from my iPhone
>>>
>>> On 19/09/2018, at 4:39 PM, Liam Farr <***@maxumdata.com> wrote:
>>>
>>> IMO I think its great that a large carrier has come to our neck of the
>>> woods, and they are openly peering on both our major IX's. This is a good
>>> thing for internet in NZ.
>>>
>>> Notifications were sent to the affected parties who peer, months in
>>> advance.
>>>
>>> I don't believe HE had any malicious intent to break your internet.
>>>
>>> If your stuff broke because you have a shit upstream, fix it and move on.
>>>
>>> On Wed, 19 Sep 2018 at 16:35, Michael Fincham <***@hotplate.co.nz>
>>> wrote:
>>>
>>>> On Wed, 19 Sep 2018 16:32:09 +1200
>>>> Nathan Ward <***@daork.net> wrote:
>>>>
>>>> > Through whoever you buy domestic transit from - communication for
>>>> these sorts of changes, and if required protecting customers who want to
>>>> run such a service on c2610s, is a core part of such a service. If your
>>>> transit providers are unable to do this I would suggest looking towards
>>>> other operators.
>>>>
>>>> This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch
>>>> of stuff, doesn't even bother to post to NZNOG letting us know they're
>>>> going to break things, and you're just blaming my transit provider?
>>>>
>>>> Also, none of the IXes posted on NZNOG?
>>>>
>>>> --
>>>> Michael
>>>> _______________________________________________
>>>> NZNOG mailing list
>>>> ***@list.waikato.ac.nz
>>>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>>>
>>>
>>>
>>> --
>>> Kind Regards
>>>
>>>
>>> Liam Farr
>>>
>>> Maxum Data
>>> +64-9-950-5302
>>>
>>> _______________________________________________
>>> NZNOG mailing list
>>> ***@list.waikato.ac.nz
>>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>>
>>> _______________________________________________
>>> NZNOG mailing list
>>> ***@list.waikato.ac.nz
>>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>>
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
>
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
>
Tom Paseka
2018-09-19 05:13:24 UTC
Permalink
I'd call that partial transit ;-)

(BGP sanity questions remain).

On Tue, Sep 18, 2018 at 10:05 PM Dave Mill <***@mill.net.nz> wrote:

> Buying 'domestic' is a way of paying for the privilege of
> sending/receiving traffic to Spark :)
>
> On Wed, Sep 19, 2018 at 5:02 PM, Tom Paseka <tom+***@cloudflare.com>
> wrote:
>
>> If I can ask, what does a "domestic" table genuinely bring you. If you
>> carry routes to your edge(s) that have both peering and transit, do you
>> need to extend the DFZ all the way through your network? If you've got a
>> single exit path (or a few) do you need to extend the domestic routes any
>> further?
>>
>> Another thing to note: BGP Sanity. Apply AS_PATH filters to all your
>> peers. This doubly so if you're extending the routes throughout your
>> network. Much of HE's routes might not be relevant to your network and
>> could have easily been sanitized with some nice filters.
>>
>> Also congrats to HE/Team!
>> -Tom
>>
>> On Tue, Sep 18, 2018 at 9:50 PM Tim Hoffman <***@hoffman.net.nz> wrote:
>>
>>> Don't forget that the NZ/AU concept of 'domestic table' is a fairly
>>> unique concept that isn't seen in much of the rest of the world- generally
>>> any feed is going to be default, a peering feed (customer routes), or a
>>> full table.... Doubt anyone thought of it ;).
>>>
>>> Congrats to Mike & Walt on extending to NZ... pretty cool to see a
>>> worldwide carrier finally making it there - fantastic to see how far the NZ
>>> internet has progressed!
>>>
>>> On Tue, Sep 18, 2018 at 9:47 PM Bill <***@wjw.nz> wrote:
>>>
>>>>
>>>> Perhaps those that had issues should use such tools as prefix limits on
>>>> their inbound feeds. We get notified by our edge routers quite regularly
>>>> that the number of prefixes has or is about to max out.
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On 19/09/2018, at 4:39 PM, Liam Farr <***@maxumdata.com> wrote:
>>>>
>>>> IMO I think its great that a large carrier has come to our neck of the
>>>> woods, and they are openly peering on both our major IX's. This is a good
>>>> thing for internet in NZ.
>>>>
>>>> Notifications were sent to the affected parties who peer, months in
>>>> advance.
>>>>
>>>> I don't believe HE had any malicious intent to break your internet.
>>>>
>>>> If your stuff broke because you have a shit upstream, fix it and move
>>>> on.
>>>>
>>>> On Wed, 19 Sep 2018 at 16:35, Michael Fincham <***@hotplate.co.nz>
>>>> wrote:
>>>>
>>>>> On Wed, 19 Sep 2018 16:32:09 +1200
>>>>> Nathan Ward <***@daork.net> wrote:
>>>>>
>>>>> > Through whoever you buy domestic transit from - communication for
>>>>> these sorts of changes, and if required protecting customers who want to
>>>>> run such a service on c2610s, is a core part of such a service. If your
>>>>> transit providers are unable to do this I would suggest looking towards
>>>>> other operators.
>>>>>
>>>>> This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch
>>>>> of stuff, doesn't even bother to post to NZNOG letting us know they're
>>>>> going to break things, and you're just blaming my transit provider?
>>>>>
>>>>> Also, none of the IXes posted on NZNOG?
>>>>>
>>>>> --
>>>>> Michael
>>>>> _______________________________________________
>>>>> NZNOG mailing list
>>>>> ***@list.waikato.ac.nz
>>>>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>>>>
>>>>
>>>>
>>>> --
>>>> Kind Regards
>>>>
>>>>
>>>> Liam Farr
>>>>
>>>> Maxum Data
>>>> +64-9-950-5302
>>>>
>>>> _______________________________________________
>>>> NZNOG mailing list
>>>> ***@list.waikato.ac.nz
>>>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>>>
>>>> _______________________________________________
>>>> NZNOG mailing list
>>>> ***@list.waikato.ac.nz
>>>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>>>
>>> _______________________________________________
>>> NZNOG mailing list
>>> ***@list.waikato.ac.nz
>>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>>
>>
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
>>
>
Dylan Hall
2018-09-19 04:27:40 UTC
Permalink
Did the AKL-IX people send an email ahead of this?

I ask because we didn't see anything so I'm wondering if we just lost it in
a spam folder somewhere or if others got missed also.

Thanks,

Dylan


On 19 September 2018 at 16:23, Dave Mill <***@mill.net.nz> wrote:

> Megaport AKL emailed their customers multiple times about HE turning up
> their services.
>
> NZIX also emailed.
>
> So I assume its just 'domestic' transit providers that maybe aren't
> notifying us?
>
> Dave
>
> On Wed, Sep 19, 2018 at 4:17 PM, Michael Fincham <***@hotplate.co.nz>
> wrote:
>
>> OK, so the routes are on AKL-IX now [cue a bunch more peoples import
>> limits breaking again].
>>
>> Did we not learn anything from the last time this broke a whole bunch of
>> stuff?
>>
>> HE needs to be giving us some heads up before they turn up here and
>> increase the size of NZ's domestic route table by an order of magnitude...
>>
>> Hello?
>>
>> --
>> Michael
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
>
>
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
>
Jesse Archer
2018-09-19 04:28:23 UTC
Permalink
We didn’t get an email from akl IX either

On Wed, 19 Sep 2018 at 4:27 PM, Dylan Hall <***@eth0.net.nz> wrote:

> Did the AKL-IX people send an email ahead of this?
>
> I ask because we didn't see anything so I'm wondering if we just lost it
> in a spam folder somewhere or if others got missed also.
>
> Thanks,
>
> Dylan
>
>
> On 19 September 2018 at 16:23, Dave Mill <***@mill.net.nz> wrote:
>
>> Megaport AKL emailed their customers multiple times about HE turning up
>> their services.
>>
>> NZIX also emailed.
>>
>> So I assume its just 'domestic' transit providers that maybe aren't
>> notifying us?
>>
>> Dave
>>
>> On Wed, Sep 19, 2018 at 4:17 PM, Michael Fincham <***@hotplate.co.nz>
>> wrote:
>>
>>> OK, so the routes are on AKL-IX now [cue a bunch more peoples import
>>> limits breaking again].
>>>
>>> Did we not learn anything from the last time this broke a whole bunch of
>>> stuff?
>>>
>>> HE needs to be giving us some heads up before they turn up here and
>>> increase the size of NZ's domestic route table by an order of magnitude...
>>>
>>> Hello?
>>>
>>> --
>>> Michael
>>> _______________________________________________
>>> NZNOG mailing list
>>> ***@list.waikato.ac.nz
>>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>>
>>
>>
>> _______________________________________________
>> NZNOG mailing list
>> ***@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
>>
> _______________________________________________
> NZNOG mailing list
> ***@list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
--





*Jesse Archer*
*General Manager*Full Flavour


*p. *07 577 0099 *ddi*. 07 281 1391
*s*. Skype "myfullflavour"
*e*. ***@fullflavour.nz <***@fullflavourmedia.co.nz>
*w*. fullflavour.nz <http://www.fullflavourmedia.co.nz/>
*a. *Basestation, 148 Durham Street, Tauranga
*a*. PO Box 13403, Tauranga Central, Tauranga 3141, New Zealand
Loading...