Secure LXC networking

Marian HackMan Marinov<mm@1h.com>

CEO of 1H Ltd.CTO of GetClouder.com

Who am I?➢ System Administrator since 1998➢ CEO of 1H Ltd.➢ CTO of GetClouder Ltd.➢ Head of DevOps for Siteground.com➢ Organizer of OpenFest, BG Perl workshops and others➢ This year I helped with the organization of YAPC europe and EuroBSDcon in Sofia➢ In my spare time I teach Linux System Administration and Network Security courses in Sofia University➢ For the past year I'm playing mainly with containers!

We don't really need networking...

MAC addresses➢ Keep a central DB with all MAC addresses to prevent collisions➢ Use a reliable way to generate MAC addresses

➢ Keep in mind:➢local or global➢unicast or multicast

generate MAC address in bash

function gen_mac() { mac_vars=(0 1 2 3 4 5 6 7 8 9 a b c d e f) mac_base='52:00:01:' ret='' for i in {1..6}; do n=$RANDOM let 'n %= 16' ret="${ret}${mac_vars[$n]}" if [ $i -eq 2 ] || [ $i -eq 4 ]; then ret="${ret}:" fi done echo "${mac_base}${ret}"}

generate mac address in PLPGSQLCREATE OR REPLACE FUNCTION generate_mac() RETURNS text LANGUAGE plpgsql AS $$DECLARE

mac TEXT;a CHAR;count INTEGER;

BEGINmac='52:00:01:';FOR count IN 1..6 LOOP

SELECT substring('0123456789abcdef' FROM (random()*16)::int + 1 FOR 1) INTO a;-- This fixes an issue, where the above SELECT returns NULL or empty string-- If for some reason we concatenate with a NULL string, the result will be NULL stringWHILE a IS NULL OR a = '' LOOP

SELECT substring('0123456789abcdef' FROM (random()*16)::int + 1 FOR 1) INTO a;

END LOOP;mac = mac || a;IF count = 2 OR count = 4 THEN

mac = mac || ':';END IF;

END LOOP;RETURN mac;

END;$$;

generate MAC address in Python

#/usr/bin/pythonimport randommac = [random.choice(range(256)) for i in range(6)]mac[0] |= 0x02mac[0] &= 0xfeprint ':'.join('%02x' % m for m in mac)

Types of LXC networking

➢none

➢empty

➢macvlan

➢macvtap (did not have time to test it)

➢veth (linux or ovs bridge)

➢vlan

➢physical

➢VPN device(haven't tried that either)

None

lxc.network.type = none

lxc.network.hwaddr = 00:16:3a:61:45:a6

lxc.network.flags = up

Empty

lxc.network.type = empty

lxc.network.hwaddr = 00:16:3a:61:45:a6

lxc.network.flags = up

VETH

lxc.network.type = veth

lxc.network.veth.pair = vethc3070

lxc.network.flags = up

lxc.network.name = eth0

lxc.network.ipv4 = X.X.X.X/24

lxc.network.ipv4.gateway = X.X.X.1

lxc.network.hwaddr = 00:16:3e:28:73:b3

VETH

lxc.network.veth.pair = vethc3070

11: vethD6YPJ1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP qlen 1000

link/ether f2:0:32:02:55:2f brd ff:ff:ff:ff:ff:ff

valid_lft forever preferred_lft forever

MACVLAN

lxc.network.type = macvlan

lxc.network.macvlan.mode = bridge

lxc.network.flags = up

lxc.network.link = lxcbr1

lxc.network.name = eth0

lxc.network.ipv4 = X.X.X.X/24

lxc.network.ipv4.gateway = X.X.X.1

lxc.network.hwaddr = 00:16:3e:28:73:b3

MACVLAN

➢ If you want to manually setup the networking

ip link add link PARENT [NAME] type macvlan [address MAC]

➢ Auto generated MAC adresses

# ip link add link eth0 lxc0 type macvlan

➢ Manually assigned

# ip link add link eth0 lxc1 type macvlan address f0:de:f1:81:0a:2a

➢ Additional parameter: mode

➢ macvlan mode { private | vepa | bridge | passthru }

MACVLAN

➢ private (filter all incoming packets)

➢ bridge (all packets on the same iface can be seen from all vlans)

➢ pasthru (requires enabled STP)

➢ VEPA (Virtual Ethernet Port Aggregator)

MACVLAN

➢ Edge Virtual Bridging EVB

➢ Top-of-Rack (ToR)

➢ End-of-Row (EoR)

➢ Virtual Ethernet Bridge (VEB)

➢ Linux bridge

➢ OpenVswitch

➢ Virtual Ethernet Port Aggregator (VEPA)

➢ used for EVB

➢ VEPA 802.1Qab - HP, IBM, Brocade, Juniper

➢ Standard mode➢ Multi-channel VEPA (Q-in-Q)

➢ VN-Tag 802.1Qbh - Cisco

VLAN

lxc.network.type = vlan

lxc.network.vlan.id = 10

lxc.network.flags = up

lxc.network.link = eth0

lxc.network.name = eth0

lxc.network.ipv4 = X.X.X.X/24

lxc.network.ipv4.gateway = X.X.X.1

lxc.network.hwaddr = 00:16:3e:28:73:b3

VLAN

# vconfig add eth0 10

# ip link add link eth0 vlan10 type vlan id 10

# ip link show vlan10

10: vlan10@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT

link/ether f0:de:f1:81:0a:2a brd ff:ff:ff:ff:ff:ff

Physical

lxc.network.type = phys

lxc.network.flags = up

lxc.network.link = eth2

lxc.network.name = eth0

lxc.network.ipv4 = X.X.X.X/24

lxc.network.ipv4.gateway = X.X.X.1

lxc.network.hwaddr = 00:16:3e:28:73:b3

Bridging :)

➢Linux Bridge➢ setup with brctl

➢ setup with ip route

➢OpenVswitch (OVS)➢ setup with its tools

Bridging :)

➢What is OpenVswitch➢ multilayer virtual switch

➢Why OpenVswitch➢ greater flexibility

➢ more control over the traffic

➢ native VXLAN support

Bridging :)

# brctl show

bridge name bridge id STP enabled interfaces

# brctl addbr br0

# brctl show

bridge name bridge id STP enabled interfaces

br0 8000.000000000000 no

Bridging :)

# brctl addif br0 eth0

# brctl show

bridge name bridge id STP interfaces

br0 8000.f0def1810a2a no eth0

adding a veth device

# brctl addif br0 vethc3070

adding a vlan

# brctl addif br0 eth0.4

Bridging :)

# ip link add name lxcbr0 type bridge

# ip link set dev lxcbr0 up

# ip link show lxcbr0

7: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT

link/ether fe:d8:b2:55:ce:5b brd ff:ff:ff:ff:ff:ff

# ip link set dev eth0 promisc on

# ip link set dev eth0 up

# ip link set dev eth0 master bridge_name

# ip link set dev eth0 nomaster

Securing all of these

➢Do not allow traffic out of the container with MAC address that was not assigned to the container

➢Do not allow traffic out of the container with IP address that was not assigned to the container

➢Do not allow multicast traffic to go to container which is not part of the multicast group

➢Actually if possible allow network traffic only to its gateway :)

Securing all of these

➢Do not use NAT for connecting your containers

➢NAT is susceptible to DoS. By spoofing many connections from one container can block the connectivity of the whole machine!

Broadcasts...

➢It depends on your network design➢ Generally limit the broadcast destinations that a

container can reach

➢ If possible use source routing to route the traffic directly to where it is supposed to go

OpenVswitch security

➢ Implement OpenFlow rules to enforce the previous rules

➢ For each containerhard_timeout=0,idle_timeout=0,cookie=$cookie,priority=150 dl_type=0x0800 in_port=$input_port nw_dst=$container_gw actions=NORMAL

hard_timeout=0,idle_timeout=0,cookie=$cookie,priority=100 dl_type=0x0800 in_port=$input_port nw_dst=$container_network actions=drop

hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_src=$container_mac nw_src=$container_ip dl_type=0x0806 priority=50 actions=NORMAL

hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_src=$container_mac dl_type=0x0800 nw_src=$container_ip priority=25 actions=NORMAL

hard_timeout=0,idle_timeout=0,cookie=$cookie,priority=20 dl_type=0x0800 dl_src=$container_mac nw_dst=$container_ip actions=output:$input_port

hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port priority=5 actions=drop

➢ For each additional IP on the containerhard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_type=0x0800 dl_src=$container_mac nw_src=$additional_ip priority=10 actions=NORMAL

hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_src=$container_mac nw_src=$additional_ip dl_type=0x0806 priority=50 actions=NORMAL

OpenVswitch security

➢ OpenVswitch networking DOES NOT go trough the normal linux networking so you CAN NOT use ipatables/ebtables to limit the traffic

➢ Even if you use net_cls it still DON'T WORK

Thank you!Thank you!

Questions?Questions?

Marian Marinov <mm@1h.com>http://getclouder.com

Jabber: hackman@jabber.orgIRC: irc.freenode.net HackMan

ICQ: 7556201GitHub: http://github.com/hackman