Bridged Networks for OpenVZ in ALTLinux

Let's asume we have a computer with 2 network cards, one connected to internet router, has IP address 5.88.52.189/27 and it's default route should gateway via 5.88.52.161. And secord network connected to local area network 10.61.0.0/24. We want to create multiple OpenVZ Virtual Environments there with 2 virtual cards each, one with public IP from 5.88.52.0/27 range and other connected to local network.

OpenVZ has different network implementations for Virtual Environments. Default is venet, which is fastest, easiest and most secure.
Another option is veth, and that grants to VE right to have it's own routing table, and assign ip addresses to these interfaces.
And third option is granting access to hardware NIC (or virtual device, like VLAN, tun or tap) to VE, in that case it will be exclusively visible and manageable by this VE.

Creating bridges in HE

To use bridged networks in ALTLinux you should do the following:

In ALTLinux interfaces with names eth0, eth1... are deprecated. There is no way to ensure their names and order are preserved on reboots and on kernel upgrades. So they should be renames using /etc/iftab to logical names, like wan, lan... To use bridges we must ensure that this table defines interfaces not by macaddresses, but by businfo or combination of businfo and other characteristics, which can be obtained by running ethtool -i:
```
# ethtool -i eth0
driver: forcedeth
version: 0.60
firmware-version:
bus-info: 0000:00:08.0
# ethtool -i msgs
driver: forcedeth
version: 0.60
firmware-version:
bus-info: 0000:00:09.0
# cat <<EOF > /etc/iftab
wan    driver forcedeth businfo 0000:00:08.0 mac 00:30:48:7a:70:fe
lan    driver forcedeth businfo 0000:00:09.0 mac 00:30:48:7a:70:ff
EOF
```
Network in ALTLiunx is managed by etcnet package. Let's look at it's configuration:
```
# cd /etc/net/ifaces
# ls
default  lo  lan  wan  unknown  venet0
```
directories lan and wan are created to store configurations for these 2 NICs. They can be just copies of directories eth0 and eth1 created for these NICs by installer, but with line
```
TYPE=eth
```
added to their {lan,wan}/options files.

Now we create 2 directiries for bridges brwan and brlan:

# mkdir brwan brlan
# echo 'stp AUTO off' > brwan/brctl
# echo '5.88.52.189/27 broadcast 5.88.52.255 dev brwan' > brwan/ipv4address
# echo 'default via 5.88.52.161' > brwan/ipv4route
# cat <<EOF >brwan/options
BOOTPROTO=static
TYPE=bri
HOST='wan'
ONBOOT=yes
EOF

# echo 'stp AUTO off' > brlan/brctl
# echo '10.61.0.15 broadcast 10.61.0.255 dev brlan' > brlan/ipv4address
# echo '10.8.0.0/24 dev brlan' > brlan/ipv4route
# cat <<EOF >brlan/options
BOOTPROTO=static
TYPE=bri
HOST='wan'
ONBOOT=yes
EOF

Now we should remove ip addresses and routes from lan/wan interfaces:

# echo '0.0.0.0/0' > lan/ipv4address
# echo '0.0.0.0/0' > wan/ipv4address
# rm -f wan/ipv4route
# rm -f lan/ipv4route

Now it should be safe to /sbin/service network restart, but if you connected by ssh, better restart first networking for interface other then you connected by. Assume I'm connected by eth1:
```
# ifconfig eth0 0 down;
# ifrename -i eth0
wan
# ifup wan
# ifup brwan
```
If everything goes O.K. ifconfig will show interface eth0 renamed to wan, up but without ip address, and brwan interface holding it's previous address. Check availability of this address by ping from another host.
At that point you can restart network by service network restart knowing that it will be back at least to wan network.

Using bridges in VE

Now will work with bridges for VE. First we should totally disable venet by adding line DISABLED=yes to it's options file:
```
# echo 'DISABLED=yes' >> /etc/net/ifaces/venet0
```
Let's assume Virtual ID to be 610015. Ensure /etc/vz/conf/116211.conf does not contain lines starting with IP_ADDRESS

To create 2 VETH interfaces, eth0 and eth1, for our VE, do the following commands:

# vzctl set 160015 --netif_add eth0 --save
# vzctl set 160015 --netif_add eth1 --save

These 2 commands created in /etc/vz/conf/160015.conf the following line (wrapped here for easy reading):

NETIF="ifname=eth0,mac=00:18:51:92:67:32,\
host_ifname=veth610013.0,host_mac=00:18:51:3C:41:74;\
ifname=eth1,mac=00:18:51:01:68:5E,\
host_ifname=veth610013.1,\
host_mac=00:18:51:3D:E2:1B"

Add the following lines to this file:
```
cat <<EOF >>/etc/vz/conf/610013.conf
CONFIG_CUSTOMIZED="yes"
VETH_IP_ADDRESS="10.61.0.13/24;5.88.52.183/27"
BRIDGEDEV="brlan;brwan"
VE_DEFAULT_GATEWAY="5.88.52.161"
```
Here I've assigned IP addresses for my interfaces eth0 and eth1 of VEID and defined corresponding HE's interfaces veth610013.{0,1}.

Now let's create file for OpenVZ bridge configuration:

#echo <<EOF >/usr/sbin/vznetcfg.custom
#!/bin/bash
# /usr/sbin/vznetcfg.custom
# a script to bring up bridged network interfaces (veth's) in a VE

GLOBALCONFIGFILE=/etc/vz/vz.conf
VECONFIGFILE=/etc/vz/conf/$VEID.conf
vzctl=/usr/sbin/vzctl
brctl=/sbin/brctl
ip=/sbin/ip
ifconfig=/sbin/ifconfig
. $GLOBALCONFIGFILE
. $VECONFIGFILE

NETIFS=`echo $NETIF | sed 's/;/\n/g'`
for NETIFX in $NETIFS
do

  NETIF_OPTIONS=`echo $NETIFX | sed 's/,/\n/g'`
  for str in $NETIF_OPTIONS; do \
          # getting 'ifname' parameter value
          if [[ "$str" =~ "^ifname=" ]]; then
                  # remove the parameter name from the string (along with '=')
                  VEIFNAME=${str#*=};
          fi
          # getting 'host_ifname' parameter value
          if [[ "$str" =~ "^host_ifname=" ]]; then
                  # remove the parameter name from the string (along with '=')
                  VZHOSTIF=${str#*=};
          fi
  done

  BRIDGEX=${BRIDGEDEV%%;*}
  BRIDGEL=${BRIDGEDEV#*;}
  BRIDGEDEV=$BRIDGEL;

  VETH_IP_ADDRX=${VETH_IP_ADDRESS%%;*}
  VETH_IP_ADDRL=${VETH_IP_ADDRESS#*;}
  VETH_IP_ADDRESS=$VETH_IP_ADDRL;

  if [ ! -n "$VETH_IP_ADDRX" ]; then
     echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
     exit 1
  fi


  if [ ! -n "$VZHOSTIF" ]; then
     echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
     exit 1
  fi

  if [ ! -n "$VEIFNAME" ]; then
     echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
     exit 1
  fi

  echo "Initializing interface $VZHOSTIF for VE$VEID."
  $ifconfig $VZHOSTIF 0

  VEROUTEDEV=$VZHOSTIF

  if [ -n "$BRIDGEX" ]; then
     echo "Adding interface $VZHOSTIF to the bridge $BRIDGEX."
     VEROUTEDEV=$BRIDGEX
     $brctl addif $BRIDGEX $VZHOSTIF
  fi

  # Up the interface $VEIFNAME link in VE$VEID
  $vzctl exec $VEID $ip link set $VEIFNAME up

  for IP in $VETH_IP_ADDRX; do
     echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
     $vzctl exec $VEID $ip address add $IP dev $VEIFNAME

     # removing the netmask
     IP_STRIP=${IP%%/*};

     echo "Adding a route from VE0 to VE$VEID."
     $ip route add $IP_STRIP dev $VEROUTEDEV
  done

  if [ -n "$VE0_IP" ]; then
     echo "Adding a route from VE$VEID to VE0."
     $vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
  fi

  if [ -n "$VE_DEFAULT_GATEWAY" ]; then
     echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
     $vzctl exec $VEID \
          $ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
  fi

done
exit 0
EOF

It's based on This script from the Wiki on OpenVZ site, but modified to serve multiple bridges, networks and interfaces.

Now let's configure OpenVZ to use this script:
```
# chmod +x /usr/sbin/vznetcfg.custom
# echo 'EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"' >/etc/vz/vznet.conf
```
At this point VE is ready to be started/restarted and should appear after start/restart in these networks.

stuff

Friday, February 15, 2008

2 veth with 2 bridges on OpenVZ at ALTLinux

Bridged Networks for OpenVZ in ALTLinux

Creating bridges in HE

Using bridges in VE

1 comment:

Blog Archive

About Me