NIC to NIC connection

[Kent Anderson]

We are upgrading our servers and have run into an interesting situation. It
has been proposed that we have a direct connection from the web servers to
the postgres server via extra NICs. Has anyone done this before and how big
a project would it be to alter ASP and Java applications to make use of such
a connection?

Before we even waste time installing the NIC's I would like a sense of how
hard it is to get postgres to use that kind of a connection vs over the
Internet. We are looking to increase communication speed between the web
servers and database server as much as possible.

Thanks
Kent Anderson

[Andrew Sullivan]

Post by Kent Anderson
We are upgrading our servers and have run into an interesting situation. It
has been proposed that we have a direct connection from the web servers to
the postgres server via extra NICs.
Before we even waste time installing the NIC's I would like a sense of how
hard it is to get postgres to use that kind of a connection vs over the
Internet.

I don't understand. What interface other than a NIC are you now
using to go "over the Internet"? Or do you mean that you want an
_extra_ IP address so that the web servers don't have to compete for
the bandwidth? In that case, sure, you'll probably get a minor
performance increase, depending on your current network traffic.

A
--
Andrew Sullivan | ***@crankycanuck.ca
In the future this spectacle of the middle classes shocking the avant-
garde will probably become the textbook definition of Postmodernism.
--Brad Holland

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

[Matt Clark]

You would assign a different subnet to the connection, and then tell the
servers to connect to the PG server's address on that subnet. No other
changes required. Very odd setup though. If you want a 'private'
connection then use a switch, rather than needing umpty NICs in the PG
server.

-----Original Message-----
From: pgsql-admin-***@postgresql.org
[mailto:pgsql-admin-***@postgresql.org] On Behalf Of Kent Anderson
Sent: 19 October 2004 17:05
To: Pgsql-***@Postgresql. Org
Subject: [ADMIN] NIC to NIC connection


We are upgrading our servers and have run into an interesting situation. It
has been proposed that we have a direct connection from the web servers to
the postgres server via extra NICs. Has anyone done this before and how big
a project would it be to alter ASP and Java applications to make use of such
a connection?

Before we even waste time installing the NIC's I would like a sense of how
hard it is to get postgres to use that kind of a connection vs over the
Internet. We are looking to increase communication speed between the web
servers and database server as much as possible.

Thanks
Kent Anderson

[Bruno Wolff III]

On Tue, Oct 19, 2004 at 17:37:43 +0100,

Post by Matt Clark
You would assign a different subnet to the connection, and then tell the
servers to connect to the PG server's address on that subnet. No other
changes required. Very odd setup though. If you want a 'private'
connection then use a switch, rather than needing umpty NICs in the PG
server.

Switches are not security devices. While it is harder to sniff packets on
switches, you can't count on them to prevent hostile machines on the
switch from playing games with the arp protocol. Also I believe that if
a switch doesn't remember where a particular mac address is it will send
the packet to all of the attached ports.

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Matt Clark]

Post by Bruno Wolff III
Switches are not security devices. While it is harder to sniff packets on
switches, you can't count on them to prevent hostile machines on the
switch from playing games with the arp protocol. Also I believe that if
a switch doesn't remember where a particular mac address is it will send
the packet to all of the attached ports.

If you have 6 app servers it's just daft to stick 6 NICs in your DB
server. If absolute privacy is a concern (not mentioned by the OP),
then use a dedicated switch (or switches) for the 'private' subnet.
Even better, use SSH. But all this is over the top for 99.9% of uses
anyway. A VLAN is as private as anything else, so you can just create a
VLAN on your current switch fabric and use that. No kind of traffic on
a VLAN will hit any other VLAN. Unless of course someone has hacked
your switch, set up a mirror port, attached a sniffer or other hacked
machine to it, and is assiduously reading your traffic, in which case
you have bigger problems....


M

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)

[Bruno Wolff III]

On Tue, Oct 19, 2004 at 23:13:44 +0100,

Post by Matt Clark
If you have 6 app servers it's just daft to stick 6 NICs in your DB
server.

While there might be some cases where that makes sense most likely it
isn't something you would want to do. I believe the original motivation
was to solve bandwidth congestion rather than security.
Most likely the systems were already connected to switches.
If the nic to the web server(s) were really swamped (which seems unlikely,
but could be the case), then adding a second nic and connecting to another
port on the switch they are already using would probably work as well
as running cables directly from the web server(s) to the database.
They would need to adjust the routing tables so that some traffic went
over each nic. An easy change would be to specify that traffic destined
for the database server IP address is to go over the new nic and all other
traffic is to go over the old nic.

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[Matt Clark]

Post by Bruno Wolff III

Post by Matt Clark
If you have 6 app servers it's just daft to stick 6 NICs in your DB
server.

While there might be some cases where that makes sense most likely it
isn't something you would want to do. I believe the original motivation
was to solve bandwidth congestion rather than security.

If the NICs are saturated then channel bonding is the answer. Any
modern switch and OS will let you bind together multiple NICs so they
appear as one. You can get an 4Gbps link this way.

[Gaetano Mendola]

Post by Bruno Wolff III
Also I believe that if
a switch doesn't remember where a particular mac address is it will send
the packet to all of the attached ports.

I don't think so, I guess the switch perform a sort of arpping in order to
detect who have a macaddress assigned, even the multicast is not sent
to all ports but only to that ports where "someone" sent an arp packet saying
the he was registered to a multicast address.

However I don't think exist a standard.


Regards
Gaetano Mendola

[Bruno Wolff III]

On Thu, Oct 21, 2004 at 10:07:33 +0200,

Post by Gaetano Mendola

Post by Bruno Wolff III
Also I believe that if
a switch doesn't remember where a particular mac address is it will send
the packet to all of the attached ports.

I don't think so, I guess the switch perform a sort of arpping in order to
detect who have a macaddress assigned, even the multicast is not sent
to all ports but only to that ports where "someone" sent an arp packet saying
the he was registered to a multicast address.

Switches don't do arps; they are a layer 2 device (and hence don't know
about IP addresses. They cache mac address port bindings. If their cache
expires before the cached IP mac address bindings in some attached device
then they can get requests to send to mac addresses for which they don't
have a cached port. In that case they are supposed to do a port flood of
the packet.

I actually had a problem with a switch that was misconfigured and didn't
do a port flood in these circumstances. That combined with a linux box
that wasn't chatty enough to stay in the switch's cache and that the
router cached IP address - mac address bindings for hours while the switch
kept mac address - port bindings for minutes caused my machine to be
unreachable to other hosts for periods of time.

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[Jay A. Kreibich]

Post by Gaetano Mendola

Post by Bruno Wolff III
Also I believe that if
a switch doesn't remember where a particular mac address is it will send
the packet to all of the attached ports.

I don't think so, I guess the switch perform a sort of arpping in order to
detect who have a macaddress assigned,

No, he's right. If the MAC to port mapping has not been learned by
the switch, the packet is flooded to all ports or (for really bad
switches) dropped. A switch is a pure layer-two device and ARP
involves layer-three addresses and concepts.

The only part of a switch that will ARP is the management system,
assuming it has an IP address of its own.

This is actually a standard test we run when evaluating switches for
deployment on campus. We hook about ten devices up to a switch and
send them tons of traffic. The systems count incoming packets but
they do not generate any traffic, so the switch is forced to flood
all the traffic. The vast majority of low-cost switches roll over and
die at that point.

Post by Gaetano Mendola
even the multicast is not sent
to all ports but only to that ports where "someone" sent an arp packet
saying the he was registered to a multicast address.

Not ARP, IGMP (Internet Group Management Protocol). It's part of the
IP suite, and is much higher up in the protocol stack.

And again, this isn't true unless the switch does IGMP snooping or
has multicast assist from the router. IGMP snooping is becoming more
common on higher-end switches, but there are some companies that
prefer the router assist design. The problem with that is that the
switches and routers need to be from the same company.

IGMP snooping is also really tricky to do right, and there are still
some situations where you are forced to flood traffic.

-j

[Jay A. Kreibich]

We have some switches that are able to do ip routing too... :-(

So called "layer-three switches" are a whole different game.

Essentualy, they are just routers where the "routing" interfaces and
the physical interfaces are abstracted from each other with a layer-two
switching engine. The term "layer-three swtich" is really more of a
marketing term, however, and is not a very accurate description of
what the device is or how it works.

I have to admint that I'm not a switch specialist but given the ability
to do routing I was imagine the arpping trich...

Yes, but such systems are generally routers first and switches second.

-j

[Gaetano Mendola]

Post by Jay A. Kreibich

We have some switches that are able to do ip routing too... :-(

So called "layer-three switches" are a whole different game.

Ok that's explain all, I was able to create two different VLAN's with a
cable between two ports ( in order to simulate two different switches), each port was
appartaining to a different VLAN and I was seeing ARPREQUEST passing by the cable in
order to detect the address in the other side, just for fun I replaced the cable with
a traffic shaper and all was working very fine...
However I wasn't curious enough to see who was the arp request source, may be was not
the "router/switcher" but a client that was doing it.


Regards
Gaetano Mendola





---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly

[Gaetano Mendola]

Post by Jay A. Kreibich

Post by Gaetano Mendola

Post by Bruno Wolff III
Also I believe that if
a switch doesn't remember where a particular mac address is it will send
the packet to all of the attached ports.

I don't think so, I guess the switch perform a sort of arpping in order to
detect who have a macaddress assigned,

No, he's right. If the MAC to port mapping has not been learned by
the switch, the packet is flooded to all ports or (for really bad
switches) dropped. A switch is a pure layer-two device and ARP
involves layer-three addresses and concepts.

We have some switches that are able to do ip routing too... :-(
I have to admint that I'm not a switch specialist but given the ability to do
routing I was imagine the arpping trich...


Regards
Gaetano Mendola





---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

[Scott Marlowe]

This is more a networking issue than a PostgreSQL one. What you'll like
want to do is to set up a nic in each box and use a "rolled" cable
directly between them. Assign an IP to each machine like 10.0.0.1 and
10.0.0.2 and edit your route tables to make each machine use those IPs
respectively to talk to each other. I'm a unix guy, so explaining
exactly how to do this in windows is a mystery to me. I'm guessing
windows because of the ASP you mention. The basic theory stays the same
though.

Post by Kent Anderson
We are upgrading our servers and have run into an interesting
situation. It has been proposed that we have a direct connection from
the web servers to the postgres server via extra NICs. Has anyone done
this before and how big a project would it be to alter ASP and Java
applications to make use of such a connection?
Before we even waste time installing the NIC's I would like a sense of
how hard it is to get postgres to use that kind of a connection vs
over the Internet. We are looking to increase communication speed
between the web servers and database server as much as possible.
Thanks
Kent Anderson

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

[Jay A. Kreibich]

Post by Scott Marlowe
This is more a networking issue than a PostgreSQL one. What you'll like
want to do is to set up a nic in each box and use a "rolled" cable
directly between them

Actually, you want a "cross-over" cable, not a "roll" cable. There is
a difference.

With many newer, high quality NICs, you don't even need that. Many
modern NICs do auto MDI/MDI-X detection, so any standard cable will do.


-j

[R***@raytheon.com]

Depends on the NIC. If your running 100MBS Ether and put in 1GBS Ether or
fiber, it would make a difference.

Rick



Andrew Sullivan
<***@crankycanuck.ca> To: "Pgsql-***@Postgresql. Org" <pgsql-***@postgresql.org>
Sent by: cc:
pgsql-admin-***@pos Subject: Re: [ADMIN] NIC to NIC connection
tgresql.org


10/19/2004 11:28 AM

Post by Kent Anderson
We are upgrading our servers and have run into an interesting situation.

It

Post by Kent Anderson
has been proposed that we have a direct connection from the web servers

to

Post by Kent Anderson
the postgres server via extra NICs.
Before we even waste time installing the NIC's I would like a sense of

how

Post by Kent Anderson
hard it is to get postgres to use that kind of a connection vs over the
Internet.

I don't understand. What interface other than a NIC are you now
using to go "over the Internet"? Or do you mean that you want an
_extra_ IP address so that the web servers don't have to compete for
the bandwidth? In that case, sure, you'll probably get a minor
performance increase, depending on your current network traffic.

A

--
Andrew Sullivan | ***@crankycanuck.ca
In the future this spectacle of the middle classes shocking the avant-
garde will probably become the textbook definition of Postmodernism.
--Brad Holland

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html






---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)

[Andrew Sullivan]

Post by R***@raytheon.com
Depends on the NIC. If your running 100MBS Ether and put in 1GBS Ether or
fiber, it would make a difference.

Not if you're not using 100 megs in the first place :)

A

[Ben Kim]

My 2 pence.

I think it's not a big task. If the servers are located nearby, all you
need is.

Connect the two machines:
===========================

- get a crossover patch cable (Should be crossover - you can buy or make
one by googling instructions.)

- install one extra NIC on each machine and connect the two NICs using the
crossover cable

- give a private address to each NIC so that they share the same internal
network, like 192.168.0.3 and 192.168.0.4.

Change connection string:
===========================

- change the database host in your ASP or JSP database connection file(s)
to this internal address. (For example, jdbc:postgresql://db.some.com/mydb
to jdbc:postgresql://192.168.0.4/mydb.)

If you are using dreamweaver, etc., if you will need to change the
connection setting there too.

In my network a bulk transfer speed through the internal link is not much
better than that of the external link. But the former is significantly
better during a busy day. (But then, the database sessions are more likely
to be interactive type of traffic...)

You also get security as a side benefit. (Data can't be snooped.)


Regards,

Ben Kim
Database Developer/Systems Administrator
College of Education
Texas A&M University

Post by Kent Anderson
We are upgrading our servers and have run into an interesting situation. It
has been proposed that we have a direct connection from the web servers to
the postgres server via extra NICs. Has anyone done this before and how big
a project would it be to alter ASP and Java applications to make use of such
a connection?
Before we even waste time installing the NIC's I would like a sense of how
hard it is to get postgres to use that kind of a connection vs over the
Internet. We are looking to increase communication speed between the web
servers and database server as much as possible.
Thanks
Kent Anderson

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

[R***@Raytheon.com]

True. I can't imagine a modern server room, even running windows, that
doesn't have a gigabit or fiber backbone for the servers.

Rick



Andrew Sullivan
<***@crankycanuck.ca> To: "Pgsql-***@Postgresql. Org" <pgsql-***@postgresql.org>,
Sent by: pgsql-admin-***@postgresql.org
pgsql-admin-***@pos cc:
tgresql.org Subject: Re: [ADMIN] NIC to NIC connection


10/19/2004 12:58 PM

Post by R***@raytheon.com
Depends on the NIC. If your running 100MBS Ether and put in 1GBS Ether

or

Post by R***@raytheon.com
fiber, it would make a difference.

Not if you're not using 100 megs in the first place :)

A

--
Andrew Sullivan | ***@crankycanuck.ca
I remember when computers were frustrating because they *did* exactly what
you told them to. That actually seems sort of quaint now.
--J.D. Baldwin

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend






---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

[Christian Fowler]

There have been many comments about this already Kent. My $.02:

The most common practice I am aware of is to install 2 NIC's in each
appserver - one to your load balancer, and one to your private network
(192.168.*) where your database server sites. In fact, ideally your
database machine has no publically addressable nic at all. I have
personally dealt with such a setup in several installs, some doing of
millions of page views per *day*, and it has always been very reliable,
secure, and fast.

Use gigabit everywhere on your 192.168 "database" network. If you are
concernced about bandwidth, wire up http://www.mrtg.org and look at the
traffic for yourself.

getting postgres to use this setup should be a piece of cake. Just make
sure your settings in pg_hba.conf are setup right.

Good luck.

Post by Kent Anderson
We are upgrading our servers and have run into an interesting situation. It
has been proposed that we have a direct connection from the web servers to
the postgres server via extra NICs. Has anyone done this before and how big
a project would it be to alter ASP and Java applications to make use of such
a connection?
Before we even waste time installing the NIC's I would like a sense of how
hard it is to get postgres to use that kind of a connection vs over the
Internet. We are looking to increase communication speed between the web
servers and database server as much as possible.
Thanks
Kent Anderson

[ \ /
[ >X< Christian Fowler | ***@viovio.com
[ / \ http://www.viovio.com | http://www.tikipro.org

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html