Friday, February 15, 2008

2 veth with 2 bridges on OpenVZ at ALTLinux

Bridged Networks for OpenVZ in ALTLinux



Let's asume we have a computer with 2 network cards, one connected to internet router, has IP address 5.88.52.189/27 and it's default route should gateway via 5.88.52.161. And secord network connected to local area network 10.61.0.0/24. We want to create multiple OpenVZ Virtual Environments there with 2 virtual cards each, one with public IP from 5.88.52.0/27 range and other connected to local network.


OpenVZ has different network implementations for Virtual Environments. Default is venet, which is fastest, easiest and most secure.
Another option is veth, and that grants to VE right to have it's own routing table, and assign ip addresses to these interfaces.
And third option is granting access to hardware NIC (or virtual device, like VLAN, tun or tap) to VE, in that case it will be exclusively visible and manageable by this VE.


Creating bridges in HE



To use bridged networks in ALTLinux you should do the following:


  1. In ALTLinux interfaces with names eth0, eth1... are deprecated. There is no way to ensure their names and order are preserved on reboots and on kernel upgrades. So they should be renames using /etc/iftab to logical names, like wan, lan... To use bridges we must ensure that this table defines interfaces not by macaddresses, but by businfo or combination of businfo and other characteristics, which can be obtained by running ethtool -i:
    # ethtool -i eth0
    driver: forcedeth
    version: 0.60
    firmware-version:
    bus-info: 0000:00:08.0
    # ethtool -i msgs
    driver: forcedeth
    version: 0.60
    firmware-version:
    bus-info: 0000:00:09.0
    # cat <<EOF > /etc/iftab
    wan driver forcedeth businfo 0000:00:08.0 mac 00:30:48:7a:70:fe
    lan driver forcedeth businfo 0000:00:09.0 mac 00:30:48:7a:70:ff
    EOF
  2. Network in ALTLiunx is managed by etcnet package. Let's look at it's configuration:
    # cd /etc/net/ifaces
    # ls
    default lo lan wan unknown venet0

    directories lan and wan are created to store configurations for these 2 NICs. They can be just copies of directories eth0 and eth1 created for these NICs by installer, but with line
    TYPE=eth
    added to their {lan,wan}/options files.
  3. Now we create 2 directiries for bridges brwan and brlan:
    # mkdir brwan brlan
    # echo 'stp AUTO off' > brwan/brctl
    # echo '5.88.52.189/27 broadcast 5.88.52.255 dev brwan' > brwan/ipv4address
    # echo 'default via 5.88.52.161' > brwan/ipv4route
    # cat <<EOF >brwan/options
    BOOTPROTO=static
    TYPE=bri
    HOST='wan'
    ONBOOT=yes
    EOF

    # echo 'stp AUTO off' > brlan/brctl
    # echo '10.61.0.15 broadcast 10.61.0.255 dev brlan' > brlan/ipv4address
    # echo '10.8.0.0/24 dev brlan' > brlan/ipv4route
    # cat <<EOF >brlan/options
    BOOTPROTO=static
    TYPE=bri
    HOST='wan'
    ONBOOT=yes
    EOF

  4. Now we should remove ip addresses and routes from lan/wan interfaces:
    # echo '0.0.0.0/0' > lan/ipv4address
    # echo '0.0.0.0/0' > wan/ipv4address
    # rm -f wan/ipv4route
    # rm -f lan/ipv4route
  5. Now it should be safe to /sbin/service network restart, but if you connected by ssh, better restart first networking for interface other then you connected by. Assume I'm connected by eth1:
    # ifconfig eth0 0 down;
    # ifrename -i eth0
    wan
    # ifup wan
    # ifup brwan
    If everything goes O.K. ifconfig will show interface eth0 renamed to wan, up but without ip address, and brwan interface holding it's previous address. Check availability of this address by ping from another host.
  6. At that point you can restart network by service network restart knowing that it will be back at least to wan network.





Using bridges in VE


  1. Now will work with bridges for VE. First we should totally disable venet by adding line DISABLED=yes to it's options file:
    # echo 'DISABLED=yes' >> /etc/net/ifaces/venet0
  2. Let's assume Virtual ID to be 610015. Ensure /etc/vz/conf/116211.conf does not contain lines starting with IP_ADDRESS
  3. To create 2 VETH interfaces, eth0 and eth1, for our VE, do the following commands:
    # vzctl set 160015 --netif_add eth0 --save
    # vzctl set 160015 --netif_add eth1 --save
    These 2 commands created in /etc/vz/conf/160015.conf the following line (wrapped here for easy reading):
    NETIF="ifname=eth0,mac=00:18:51:92:67:32,\
    host_ifname=veth610013.0,host_mac=00:18:51:3C:41:74;\
    ifname=eth1,mac=00:18:51:01:68:5E,\
    host_ifname=veth610013.1,\
    host_mac=00:18:51:3D:E2:1B"

  4. Add the following lines to this file:
    cat <<EOF >>/etc/vz/conf/610013.conf
    CONFIG_CUSTOMIZED="yes"
    VETH_IP_ADDRESS="10.61.0.13/24;5.88.52.183/27"
    BRIDGEDEV="brlan;brwan"
    VE_DEFAULT_GATEWAY="5.88.52.161"
    Here I've assigned IP addresses for my interfaces eth0 and eth1 of VEID and defined corresponding HE's interfaces veth610013.{0,1}.
  5. Now let's create file for OpenVZ bridge configuration:
    #echo <<EOF >/usr/sbin/vznetcfg.custom
    #!/bin/bash
    # /usr/sbin/vznetcfg.custom
    # a script to bring up bridged network interfaces (veth's) in a VE

    GLOBALCONFIGFILE=/etc/vz/vz.conf
    VECONFIGFILE=/etc/vz/conf/$VEID.conf
    vzctl=/usr/sbin/vzctl
    brctl=/sbin/brctl
    ip=/sbin/ip
    ifconfig=/sbin/ifconfig
    . $GLOBALCONFIGFILE
    . $VECONFIGFILE

    NETIFS=`echo $NETIF | sed 's/;/\n/g'`
    for NETIFX in $NETIFS
    do

    NETIF_OPTIONS=`echo $NETIFX | sed 's/,/\n/g'`
    for str in $NETIF_OPTIONS; do \
    # getting 'ifname' parameter value
    if [[ "$str" =~ "^ifname=" ]]; then
    # remove the parameter name from the string (along with '=')
    VEIFNAME=${str#*=};
    fi
    # getting 'host_ifname' parameter value
    if [[ "$str" =~ "^host_ifname=" ]]; then
    # remove the parameter name from the string (along with '=')
    VZHOSTIF=${str#*=};
    fi
    done

    BRIDGEX=${BRIDGEDEV%%;*}
    BRIDGEL=${BRIDGEDEV#*;}
    BRIDGEDEV=$BRIDGEL;

    VETH_IP_ADDRX=${VETH_IP_ADDRESS%%;*}
    VETH_IP_ADDRL=${VETH_IP_ADDRESS#*;}
    VETH_IP_ADDRESS=$VETH_IP_ADDRL;

    if [ ! -n "$VETH_IP_ADDRX" ]; then
    echo "According to $CONFIGFILE VE$VEID has no veth IPs configured."
    exit 1
    fi


    if [ ! -n "$VZHOSTIF" ]; then
    echo "According to $CONFIGFILE VE$VEID has no veth interface configured."
    exit 1
    fi

    if [ ! -n "$VEIFNAME" ]; then
    echo "Corrupted $CONFIGFILE: no 'ifname' defined for host_ifname $VZHOSTIF."
    exit 1
    fi

    echo "Initializing interface $VZHOSTIF for VE$VEID."
    $ifconfig $VZHOSTIF 0

    VEROUTEDEV=$VZHOSTIF

    if [ -n "$BRIDGEX" ]; then
    echo "Adding interface $VZHOSTIF to the bridge $BRIDGEX."
    VEROUTEDEV=$BRIDGEX
    $brctl addif $BRIDGEX $VZHOSTIF
    fi

    # Up the interface $VEIFNAME link in VE$VEID
    $vzctl exec $VEID $ip link set $VEIFNAME up

    for IP in $VETH_IP_ADDRX; do
    echo "Adding an IP $IP to the $VEIFNAME for VE$VEID."
    $vzctl exec $VEID $ip address add $IP dev $VEIFNAME

    # removing the netmask
    IP_STRIP=${IP%%/*};

    echo "Adding a route from VE0 to VE$VEID."
    $ip route add $IP_STRIP dev $VEROUTEDEV
    done

    if [ -n "$VE0_IP" ]; then
    echo "Adding a route from VE$VEID to VE0."
    $vzctl exec $VEID $ip route add $VE0_IP dev $VEIFNAME
    fi

    if [ -n "$VE_DEFAULT_GATEWAY" ]; then
    echo "Setting $VE_DEFAULT_GATEWAY as a default gateway for VE$VEID."
    $vzctl exec $VEID \
    $ip route add default via $VE_DEFAULT_GATEWAY dev $VEIFNAME
    fi

    done
    exit 0
    EOF
    It's based on This script from the Wiki on OpenVZ site, but modified to serve multiple bridges, networks and interfaces.
  6. Now let's configure OpenVZ to use this script:
    # chmod +x /usr/sbin/vznetcfg.custom
    # echo 'EXTERNAL_SCRIPT="/usr/sbin/vznetcfg.custom"' >/etc/vz/vznet.conf
    At this point VE is ready to be started/restarted and should appear after start/restart in these networks.