It may have been noticed, but my website was offline for most of the night yesterday. I have been reconfiguring my home theatre system, which runs partially on the same system. In my solution, I have a FreeNAS server, which holds a pair of Ubuntu virtual machines, each of which run pieces of the overall solution. In an attempt to improve performance, enable updates, and generally improve efficiency, I have been moving some of the pieces off of the FreeNAS base system and onto one of the two VMs. For the most part, everything was working fine, and actually improving the environment. And then yesterday, I ran into an issue. Basically, the VMs were running out of memory, due to large processing of files. So I made a swap from SABnzbd to NZBget (supposedly a much smaller memory footprint) – but this ended up actually making it worse.
Of course in all of this, I have to travel for work, so my administration is being done remotely. So when I needed to reboot the FreeNAS box itself, to resolve another lockup, the server ended up offline since it was a) past the wife’s bedtime, and b) unable to restart cleanly on its own. A short call in the morning with the wife to properly hard-reset the server, and everything was back online.
And with a bit more focused troubleshooting, I discovered that the NZBget configuration was wrong – I was confusing umask with chmod settings. This was causing all activity to be created with 000 permissions (no rights) instead of 777 permissions (full rights), which in turn was causing the various servers to thrash over the inaccessible files multiple times per second. And in turn, cause performance problems and lockups on the system.
With that issue corrected, suddenly memory and cpu usage dropped significantly, the servers became responsive, and my expectation is that the environment is fully operational again. For now…
