I recently helped install an IP camera deployment in a home and ran into some weird issues so I figured I’d post here in case anyone else had a similar problem. This will also be useful for anyone who has too many devices on their network for a typical home router to handle.

;tldr - You need to setup a computer to act as a router for a secondary network that will forward traffic between networks and from the secondary network to the outside world. Jump to Configuration Section

Background on IP Cameras

Anyone having performance issues with home networking can benefit from this post, even without being intimately familiar with IP cameras, but the terminology will make more sense if we review over IP cameras real quick.

In the world of IP Cameras, there are 3 main types of devices. A Power over Ethernet (PoE) switch that’s used to power each camera (so you just have to route an Ethernet cable to cameras and nothing else), the Cameras themselves, and a Network Video Recorder (NVR) for logging camera output.

The cameras typically run some form of embedded Linux and have web interfaces available on port 80 of each device in addition to serving up a video streaming protocol on port 37777 (TCP) and 37776 (UDP). The networking options on the camera are pretty standard - settings such as whether or not to self assign an IP or use DHCP are exposed to the end user and overall the devices seem like pretty decent little Linux boxes.

In my case, the NVR is a dedicated hardware device with 3TB of storage for continuous recording. In other cases it may be software that runs on a standard computer or an add-on package for a NAS system. For NVR’s the options vary widely as some have the ability to act as a DHCP server for assigning IPs to cameras while others simply record video. There have been issues associated with DHCP and NVR’s discussed before - these are the sorts of problems we’d like to avoid.

The complexity stems from the fact that most IP cameras are distributed by a reseller/installer who may charge additional fees for installation and support. Manufacturers ensure that consumers must go through local distributors to get firmware updates and manuals. This can make it difficult for consumers to work on the devices on their own.

My Hardware

For a full reference of the hardware I’m using:

  • Router: Cisco e4200 v2
  • NVR: Sapphire Model MSNVR3216
  • 4 Standard IP Cameras
  • Standard 24 Port PoE switch

And further down in the post:

The Issue

To install the cameras, we simply wired the home with Ethernet and plugged each camera into the PoE switch via Cat5. The switch was connected to the router and the NVR was also connected to the switch. The router assigned IPs to each of the cameras and to the NVR. Logging into the NVR interface and searching for the cameras allowed us to connect to each camera easily and recording was done for all 4 cameras. Everything seemed to be fine.

However, after having the system up for fours hours, we began to notice some problems. Internet performance started to suffer and any device connected to the router would not be assigned an IP address. This problem was particularly prominent over WiFi as it prevented phones and tablets from connecting. It seemed that the router’s DHCP server was being overloaded.

The first attempted fix required statically assigning IP addresses to all hardwire devices on the network. We went into each camera and the NVR and statically assigned an IP address outside of the range the router assigned, but on the same “192.168.1.XXX” subnet. This took the time to slow network performance to somewhere around 8 hours, but didn’t fix the problem.

We were a bit stumped at this point and thought that there might simply have been too many devices on the network, particularly when the whole family was home (up to 14 devices if no guests were over). In an effort to help mobile devices get assigned IP addresses, we offloaded the DHCP assignment to another machine and disabled it on the router (more on doing this yourself below). This gave us up to a week, but the network seemed to crash after a weeks time regardless.

Why was the network simply failing after a period of time? New devices couldn’t connect and when they did performance was at a crawl.

The Problem

I believe that consumer grade routers are simply ineffective at managing even a remotely large number of devices. That is compounded with the fact that the IP cameras and NVR seems to have a buggy networking stack makes tracking down issues difficult. A number of friends have suggested simply rebooting the router daily. Even manufacturers know the network stacks of devices are bugging - the IP cameras have an option to reboot at 2 AM every mornings simply to freshen up devices.

After pulling out Wireshark, it became apparent that these devices were quite noisy in terms of network traffic. Devices were continuously asking for another IP address (which is likely why we got more uptime after switching to static IPs).

The Solution

With firmware updates hidden behind layers of phone calls, the best option was to simply isolate the cameras on a separate network. The challenge was finding a way to allow access to the cameras and the NVR from the home’s main network.

Luckily, Linux has built in support for Network Address Translation through the use of IP Tables. This allows us to setup a machine that will act as a gateway between the networks. The home router will see only a single machine and that machine will have access to both networks. Ports can then be forwarded from one network to another, allowing tuned access from the home network to devices on the camera network.

Deployment of a Secondary Network

In order to do this, you’ll need a computer with two Ethernet ports, one will be plugged into the WiFi router and the other into a switch. We used an old HP laptop and added an extra Ethernet port using a USB Adapter. For our usage, we didn’t need anything faster than a 100mbps adapter, which worked out nicely because USB 2.0 won’t support a full 1gbps adapter. Unless you are going to be doing large file transfers, this should be fine. That Cable Matters adapter linked to above is also fully compatible with Linux, so you don’t have to worry about any driver issues!

So we’ll need to do the follow:

  1. Setup Linux - we used Xubuntu Long Term Support because it is both lightweight and will be supported for 3 years. Actually, the underlying operating system will get security updates for 5 years, so you should be able to set this up and forget about it.
    • If you aren’t familiar with doing this, I suggest downloading the ISO file from the Xubuntu site linked above and using a 1GB flash drive and UNetbootin to create an bootable flash drive. From there you can follow the on screen instructions.
  2. Configure a DHCP server to assign IPs to the secondary network - in our case this allows the computer to assign IPs to the cameras, NVR, and any other devices that plugged into the secondary switch. Using a Core 2 Duo laptop for this allowed for better scalability than a traditional home router.
  3. Use IP Tables to forward traffic from the secondary network to the home network so that devices on the secondary network can access the Internet. While this wasn’t particularly important for the IP Cameras, we may add devices for Netflix or general web browsing to the secondary network in the future. This allows those devices to access the Internet through Network Address Translation.
  4. Forward ports from the main network to devices on the secondary network. This is the same thing as opening ports on your router to allow remote access to services inside the home, but here, the computer is acting as the router for the secondary network.

Setup

There are a number of moving parts when setting up a software router, so look over each of the next sections closely. Again, I’m using Xubuntu 14.04 LTS, but the instructions will be similar for other Ubuntu 14.04 LTS releases and probably other versions numbers as well.

All file edits in this section need to be run as the root user. To do this, open up a terminal (by pressing CTRL-ALT-T in Ubuntu) and type:

  • sudo mousepad if on Xubuntu - or -
  • sudo gedit if on regular Ubuntu

That opens up the text editor as the root user. From there, you can file->open any of the files in the instructions below.

Configuring Network Interfaces

In /etc/network/interfaces:

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.1.253
    netmask 255.255.255.0
    network 192.168.1.0
    gateway 192.168.1.1
    broadcast 192.168.1.255
    dns-nameservers 208.67.220.220 208.67.222.222

auto eth1
iface eth1 inet static
    address 192.168.2.1
    netmask 255.255.255.0

Description:

  • Statically assign an IP address to the first Ethernet port eth0 - make sure this is the device connected to your home router. If your secondary port is a USB to Ethernet adapter, then the adapter port will always be eth1.
    • The only line you might need to change is address 192.168.1.253 - I picked this because the WiFi router being used only assigns addresses up to .200.
  • The part for eth1 - the USB to Ethernet device simply lets the laptop self assign an IP address to the interface. The value 192.168.2.1 will be used in the next section.

Configuring the DHCP Server

We need a way to assign IP addresses to each of the IP Cameras and the NVR. Those devices will be connected to a switch, which has no server to assign IP addresses. The DHCP server is what does this.

You shouldn’t have two DHCP servers on a single network, so we’ll have to make sure that the DHCP server is only serving IP addresses on the Ethernet port (an interface in Linux terminology) connected to the switch, and not on the one that’s connected to your primary router.

So let’s see what that looks like.

In /etc/dhcp/dhcpd.conf: - Comment out the lines at the top of the file with option domain-name and option domain-name-servers - we’ll be adding those back in at the bottom

# NEAR THE TOP ADD #'S IN FRONT OF THESE LINES:
#option domain-name "example.org";
#option domain-name-servers ns1.example.org, ns2.example.org;
...
# INSERT THESE LINES AT THE BOTTOM
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.2.255;
option routers 192.168.2.1;
option domain-name-servers 208.67.222.222, 208.67.220.220;
option domain-name "cameraBridge.org";

subnet 192.168.2.0 netmask 255.255.255.0 {
	range 192.168.2.2 192.168.2.200;
}

host CameraRecorderNVR {
	hardware ethernet 51:03:f9:25:13:b8;
	fixed-address 192.168.2.3;
}

Description of options:

  • Note that many of the options use 192.168.2.X as a setting - this is because for the secondary network, we want to use something besides the standard 192.168.1.X being used on the main network.
  • option domain-name-servers - this tells devices how to resolve domain names, the servers listed here are pointing to OpenDNS.
  • option domain-name - this doesn’t really affect anything, but you may want to pick a name specific to your situation.
  • option routers 192.168.2.1 - this sets the IP address of the machine that is acting as the router. In this case, it will be this machine. We’ll assigned 192.168.2.1 to ourselves in the previous section.
  • Configuring how many devices can be supported:
subnet 192.168.2.0 netmask 255.255.255.0 {
    range 192.168.2.2 192.168.2.200;
}

This block says that for this subnet, we can assign a range of IP addresses, from .2 to .200 for a total of 199 devices.

Reserving IP Addresses for particular machines:

host CameraRecorderNVR {
    hardware ethernet 90:02:a9:85:03:a8;
    fixed-address 192.168.2.3;
}
  • host NAME - here NAME can be whatever you like to call the machine which needs a static IP address.
  • hardware ethernet - must be set to the MAC address of the target machine.
  • fixed-address - the address you want the machine to have. It can be inside or outside of the range specified above.

To ensure it is only serving IP Addresses on eth1 we need to edit /etc/default/isc-dhcp-server:

# EDIT THIS LINE AT THE BOTTOM OF THE FILE
INTERFACES="eth1"

Setting up NAT & Port Forwarding

Now we have the ability to assign IP addresses to the devices on the secondary network, and our Linux device can communicate with devices on both networks. But now we need devices on either network to be able to communicate with each other. This is where Network Address Translation (NAT) comes in.

Create a file bridge_nat.sh in your home directory. For me this was /home/read/bridge_nat.sh. The file will need to have execute permissions, so open a terminal and type chmod +x bridge_nat.sh.

Parts of the file below come from the Mallory Project.

The first half of the file, up to echo 1 > /proc/sys/net/ipv4/ip_forward allows for devices on the secondary network to make calls to external networks by enabling the responses to be routed back through our new software router.

With that in place, we have to forward traffic to and from the NVR if we want to be able to access it from the primary network. There are quite a few ports that are open, and entries have to be created for each port and each protocol on that port (TCP or UDP). So let’s look at one entry:

# This line forwards traffic into the NVR from eth0 ($extiface) on port $external_port1
# to the destination specified by $NVR_ip:$NVR_port1
iptables -t nat -A PREROUTING -p tcp -i $extiface --dport $external_port1 -j DNAT --to-destination $NVR_ip:$NVR_port1

# With traffic being routed properly, we need to setup a rule that allows for the stream to be forwared back out to the primary network.
iptables -A FORWARD -p tcp -d $eth0_ip --dport $external_port1 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

The full file - bridge_nat.sh:

#!/bin/bash

extiface="eth0" # interface facing a router/the Internet
serveriface="eth1" # interface facing the switch
NVR_ip=192.168.2.3 # the ip address of the NVR (assigned by this machine)
NVR_port1=80 # web interface to on the NVR
NVR_port2=37777 # port that iphone connects to on the NVR
NVR_port_udp=37778 # port that iphone connects to on the NVR
NVR_port_rtsp=554

external_port1=80 # port that will be open on the router
external_port2=37777 # port that will be open on the router
external_udp=37778
external_rtsp=554

eth0_ip=192.168.1.253

echo "Killing network manager"
/etc/init.d/network-manager stop

echo "Setting up default iptables policy"
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

echo "Setting ip masquerading for $extiface"
iptables -t nat -A POSTROUTING -o $extiface -j MASQUERADE

echo "Setting ip masquerading for $serveriface"
iptables -t nat -A POSTROUTING -o $serveriface -j MASQUERADE

echo "Enabling IP forwarding"
echo 1 > /proc/sys/net/ipv4/ip_forward

# port 1
iptables -t nat -A PREROUTING -p tcp -i $extiface --dport $external_port1 -j DNAT --to-destination $NVR_ip:$NVR_port1
iptables -A FORWARD -p tcp -d $eth0_ip --dport $external_port1 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

# port 2
iptables -t nat -A PREROUTING -p tcp -i $extiface --dport $external_port2 -j DNAT --to-destination $NVR_ip:$NVR_port2
iptables -A FORWARD -p tcp -d $eth0_ip --dport $external_port2 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

# udp
iptables -t nat -A PREROUTING -p udp -i $extiface --dport $external_udp -j DNAT --to-destination $NVR_ip:$NVR_port_udp
iptables -A FORWARD -p udp -d $eth0_ip --dport $external_udp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

# rtsp
iptables -t nat -A PREROUTING -p udp -i $extiface --dport $external_rtsp -j DNAT --to-destination $NVR_ip:$NVR_port_rtsp
iptables -A FORWARD -p udp -d $eth0_ip --dport $external_rtsp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

iptables -t nat -A PREROUTING -p tcp -i $extiface --dport $external_rtsp -j DNAT --to-destination $NVR_ip:$NVR_port_rtsp
iptables -A FORWARD -p tcp -d $eth0_ip --dport $external_rtsp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

Making IP Tables Persist Across Reboot

There are a number of methods for doing this - you may have seen some across the web. I’ve found this method the easiest to use because it allows for easy modifications and is reliable in general. If this were an enterprise setting, other methods would likely be better.

When Linux boots, it executes rc.local. So if we put an entry in rc.local to enable our NAT rules, then they will get set every time the Linux computer boots.

So in /etc/rc.local add:

# REPLACE USER WITH YOUR USER NAME
./home/USER/bridge_nat.sh
exit 0

Summary

In this post, you have seen how to setup a software router on Linux. This is useful in a number of situations - in my case, it allowed me to separate out buggy network devices onto an isolated network while reducing load on a home router.

The camera network has not gone down in over a month! The NVR is accessible from iPhone applications and web browsers on the home’s primary network.

If you’ve found this post useful, or are having trouble with a particular step, I’d like to hear from you in the comments below!

How to Choose an Open Source Library and when to Roll Your Own

When starting a project, one is often faced with the choice between multiple open source libraries vs. the option of rolling a custom sol...… Continue reading