Tutorial #20: Home (Network) Improvement using a Pi


Update #1 (Aug 19, 2018):

With the passage of time, things change and in this case it is mostly for the better. I was setting up the Pi once again and was pleasantly surprised that some of the workarounds mentioned previously are no longer needed. Thus, I have edited the original tutorial to accommodate these changes while striking out the old text.

Also, after comparison, I found AB-Solution to be a better solution for network-wide ad blocking, if you happen to have a Merlin supported Asus router like I do. Since the router is on 24x7, having an old ext2 formatted pen drive plugged in to the router itself and running ab-solution is a better alternative as against running the Pi 24x7. It also has various preset ad blocking files that suit different needs while pixelserv-tls does a great job with HTTPS ads.

Lastly, I failed to mention the option of having a Samba server running on the Pi itself in order to access the files directly from the pen drive over the network. This can be accomplished by simply following the official tutorial on the Raspberry Pi website.

Original Article (May 3, 2018):

I had previously shared some tutorials in setting up the Pi and putting it to good use. However, the use cases I mentioned then have ceased to exist. The Fire TV has taken care of most of my multimedia needs and I have come to realise that I really don't have much time to go back in time for nostalgia. For the retro needs that remain, the lesser-used Core M 5Y10 equipped Windows tablet of mine does a much better job plugged in to the TV.

Hence, it is time to put the Pi to good use in a different sense. Thankfully, the versatility of the Pi means that it is not difficult to identify its next project. Ads can become a nuisance, especially for the more aged members of the family and hence my first intent was to set up an ad blocker across my home network. However, putting the Pi to such limited use and keeping it on 24/7 would be quite a waste, so I decided to also repurpose it as a download box with centralised storage.

Setting up the tools necessary to accomplish these tasks seemed straightforward. However, the relevancy of publicly available tutorials diminish over time due to changes in technology. In fact, I had to put quite some effort beyond the listed tutorials and hence I have decided to put the same to words for posterity.

Before starting out:

PINN is a great utility when multi-booting across different distributions on the same SD card. However, Raspbian alone fits the bill for the current use case. Hence, writing the raw Raspbian image directly on the card is preferable as it provides more usable space. As far as writing on the card goes, Etcher is the way to go.

1. Pi-hole®

As the website so prominently displays, all you need is a single command.

curl -sSL https://install.pi-hole.net | bash
However, I made a couple of settings that are worth mentioning:

a. The predefined list of upstream DNS providers does not yet include Cloudflare which I found to be the fastest of the lot. Hence, it would be worthwhile to use the custom option and enter the Cloudflare DNS Server IP addresses of 1.1.1.1 and 1.0.0.1.

b. The other part of the equation is setting up the home equipment to use the local DNS server. In case of Asus routers, this implies changing the DNS Server IP address to the Pi-hole one, not only on the WAN page (under WAN DNS Settings) but also under LAN > DHCP Server > DNS and WINS Server Setting. Make sure that there are no other IP addresses present in either of the pages. You could also run the DHCP server on Pi-hole, in which case the latter setting is not needed. However, since I use the router-assigned IP addresses for other functions (eg. VPN), I prefer to have the DHCP server running on the router itself.

2. qBittorrent

qBittorrent has been my preferred Bittorrent client for quite some time with it being open-source and having proper support for proxies as against Transmission. It can be installed easily using APT, though I prefer the headless route.

sudo apt-get install qbittorrent-nox
My primary endeavour was to have the downloaded files ready on the USB 3.0 hard disk connected to my router (and thus acting like a NAS) while minimising the read-write operations. Since it is not a great idea to write to the SD card running the client, I decided to plug in an old 32 GB pen drive to act as the "working folder" by adding it under Downloads > Hard Disk > Keep incomplete torrents in:

The next part was to add the network drive as the final resting place by entering its address under  Downloads > Hard Disk >  Save files to location: and also under the Copy .torrent files for finished downloads to field. The latter is sometimes necessary due to some quirks in different versions of qBittorrent. The external network drive needs to be mounted within Raspbian on boot which can be accomplished by editing /etc/fstab with these details:

mount -t cifs //xx.xx.xx.xx/folder /media/NAS -o rw,vers=2.0,username=abc,password=xyz
//xx.xx.xx.xx/folder /media/NAS cifs vers=1.0,username=abc,password=xyz,x-systemd.automount

where,
xx.xx.xx.xx -> LAN IP address as configured on the router running the Samba server
folder -> Path to the folder on the network drive that needs to be mounted
/media/NAS -> Path on the Pi where the network drive is to be mounted

In my case, I had to specifically mention the SMB version (2.0), without which the mounting would fail as well as the rw argument to be able to write to the device. Also. it is a good idea to update the cifs-util package from APT prior to editing the fstab file, as mentioned above.

Note: The earlier entry no longer worked with the June 2018 version of Raspbian due to which I had to use the alternative entry in /etc/fstab mentioned above. Also, I was only able to get v1.0 working this time despite the server supporting v2.0 as well.

Finally, to cover for unexpected reboots, it is preferable to have qBittorrent autostart which can be accomplished using systemd. First, create the startup script using:

sudo nano /etc/systemd/system/qbittorrent.service
Next, enter its contents as follows:

[Unit]
Description=qBittorrent Daemon Service
After=network.target
[Service]
User=pi
ExecStart=/usr/bin/qbittorrent-nox
ExecStop=/usr/bin/killall -w qbittorrent-nox
[Install]
WantedBy=multi-user.target 
Lastly, enable the script.

sudo systemctl enable qbittorrent
3. pyLoad

The final part of the exercise was to set up a download manager. I had briefly given thought to JDownloader but decided against running JRE just for it. Hence, I opted for pyLoad instead. The tutorial listed over here works fine for the most part but needed quite some tweaks along the way. For the sake of completion, I will list all the steps in brief.

1. Create system user for pyload

sudo adduser --system pyload
2. Edit /etc/apt/sources.list to be able to install the dependencies. For Raspbian Stretch, the source URLs are as follows:

deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi
deb-src http://archive.raspbian.org/raspbian/ stretch main contrib non-free rpi
3. Update package list and install dependencies.

sudo apt-get update
sudo apt-get -y install git liblept4 python python-crypto python-pycurl python-imaging tesseract-ocr zip unzip python-openssl libmozjs-24-bin
sudo apt-get -y build-dep rar unrar-nonfree
sudo apt-get source -b unrar-nonfree
sudo dpkg -i unrar_*_armhf.deb
sudo rm -rf unrar-*
4.  Create symlink to get "spidermonkey" working.
cd /usr/bin
sudo ln -s js24 js
5. Unlike in the linked tutorial, I had to first switch to the stable branch before I could initiate the pyLoad setup. This is done as follows:

cd /opt
sudo git clone https://github.com/pyload/pyload.git
cd pyload
git branch -r
git checkout stable 
 6. The next step, as per the linked tutorial, should have been the initiation of the pyLoad setup using:


sudo -u pyload python pyLoadCore.py
However, doing so produced the following error: "ImportError: No module named pycurl". Hence the next logical step was to install pycurl:


sudo pip -v install pycurl --upgrade
This in turn resulted in the error: "InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-IsWfyN/pycurl/". This was resolved by:


sudo apt-get install libcurl4-gnutls-devpip
 As you can now guess, this in turn resulted in yet another error: "Failed building wheel for pycurl" which was remedied as follows:


sudo apt-get install libgnutls28-dev

After all this effort, I was finally able to install pycurl using the command mentioned previously...


sudo pip -v install pycurl --upgrade 
...and execute the pyLoad setup using a slightly shorter command:


 python pyLoadCore.py
On the June 2018 release, I was able to get pyload wizard started by executing
sudo -u pyload python pyLoadCore.py 

The final tweak was to get pyLoad running on boot with the commands executed in a manner similar to what has been already covered for qBittorrent.

sudo nano /etc/systemd/system/pyload.service 
[Unit]
Description=Python Downloader
After=network.target
[Service]
User=pyload
ExecStart=/usr/bin/python /opt/pyload/pyLoadCore.py
[Install]
WantedBy=multi-user.target
 sudo systemctl enable pyload.service
Beyond Home

The benefits of this setup can be extended beyond the home network, though a lot depends on the vagaries of the network setup.

Pi-hole can be put to use on external networks by accessing the home network over OpenVPN, though speed and latency might be factors to consider. It can also be setup as a Public DNS but it is extremely risky and not at all recommended.

qBittorrent and pyLoad can be simply accessed using the IP address and port, provided they have been setup to be accessible from outside the LAN. For dynamic IP addresses, the Dynamic DNS (DDNS) option available on most routers can be put to use and my suggestion would be to pick up a cheap $1 domain from NameCheap for this purpose. However, if you happen to be under a multi-layered NAT network under ISP control like me, then there is no option other than to pay for a static IP for public access.

Thankfully, there is a last resort to access the Pi over a public network. The licensed copy of RealVNC that comes with Raspbian offers Cloud Connect that enables one to remotely control the OS and thereby all applications on it. It is quite cumbersome to use if your intent is to only load some links on qBittorrent or pyLoad, but it is better than nothing.

Thus, the Pi can be extremely useful even when used in a rather sedentary capacity and you grow to appreciate the efforts that everyone has put in to make this possible.

Tutorial #5: Efficient file sharing with remote BitTorrent + FTP Setup

Over the years, the word BitTorrent (or more popularly Torrent) has come to be associated with piracy in spite of it actually referring to a highly efficient peer to peer communication protocol. A typical complaint from a user is the speed of download which is highly dependent on the quality as well as quantity of peers available at a given point of time. Hence, it is usually much more efficient to have a BitTorrent based file download remotely, minimising the utilization of your own resources and then fetch the same using the standard FTP protocol. While there might be web services available that accomplish this, there is nothing like doing it yourself with full control. Note that the commands I have listed below are executed with root login. It is a better idea to operate as a normal user and escalate access using su or sudo whenever necessary. In your case, if you get an error when executing a command, then prefix ‘sudo’ to each.

Step 1: Get a cheap VPS
VPS refers to a Virtual Private Server. It is essentially a virtual machine running on a remote server while allowing you complete control over the machine. It serves as a great platform for learning without the fear of messing things up irreversibly. While costs and specifications vary, if your purpose is very specific and limited, it makes sense to purchase an annual subscription for a low end box which would cost a little more than $1/month. Of course, the specs will be downright modest with a single or dual core processor, 128 - 512 MB RAM, 100 GB or less of storage and around half a TB of bandwidth per month.

Step 2: Install an OS
When dealing with such a low end box, you have to purely operate a headless Linux server (through command line). Almost all VPS providers have a one-click OS install option. Make sure to choose a 32-bit minimal installation (my preference is for Debian) to maximize performance and minimize resource utilization.

Step 3: Log in to the server
Once installed, you will be able to login to your server using the SSH protocol. You only need the server IP address (available through Client Panel or your account initiation mail). On Linux, you would need nothing more than the terminal whereas on Windows, Putty is a really great client. If you would like to operate on the go, then JuiceSSH is an absolutely awesome SSH client on Android.

Step 4: Install a BitTorrent client
While there is quite an abundance of torrent clients available, what you really need is a client with a web access panel so that you can check the status and add torrents through any device. Transmission, rtorrent (with rutorrent), Deluge are the usual culprits here. However, I would recommend QBittorrent if you are not a particularly heavy user for it is quite light on the system, has a relatively meaty interface and is certainly reliable. Moreover, it has support for various proxy protocols that are lacking in the bellweather ones like Transmission and rtorrent. On Debian, you simply need to install it thorugh apt-get as follows. Before you do so, make sure you run the update and upgrade commands:
apt-get update && upgrade
apt-get install qbittorrent-nox
Step 5: Run the client 24x7
If you simply run the application on your terminal and then close the window, you will find that the client has stopped running. Hence, you need to ensure that it keeps running continuously. Doing so is possible by using the command:
nohup qbittorrent-nox &
disown -h %1
You can check the status using:
tail -f nohup.out
You can exit the status screen using Ctrl + C.

However, the better option is to use screen. If you are on a minimal install, you may need to install screen using:
apt-get install screen
You can then run a screen session by simply typing "screen" on the command prompt.

After accepting the initial disclaimer, you can run qbittorrent like you normally would.
qbittorrent-nox
You can now see the status of the client. You can detach yourself from this screen using:

Ctrl + A followed by D

The client will continue to run in to background and you may safely close the session.

To view all running screen sessions, you can use:
screen -ls
You can re-attach to a session using:
screen -r <session id>
You can quit a session using
screen -X -S <session id> quit
This covers all that you need to know to run the BitTorrent client in the background.

Step 6: Installing and accessing a FTP server

The BitTorrent client will get the file on your VPS and allow sharing of the same with others. But if you want to download the shared file efficiently, then it is best to install a FTP server so that you can download it using a FTP client or even from your browser. My personal preference is proftpd for I found it to be quite efficient and robust. The steps involved are as follows:

a. Install ProFTPd
apt-get install proftpd
b. Configure ProFTP for access control

For this, first open the config file
nano /etc/proftpd/proftpd.conf
Change the server name to your hostname by editing the following line:
ServerName "example.com"
Uncomment the ’DefaultRoot’ line so that access is limited to the user folder
# Use this to jail all users in their homes
DefaultRoot ~
You may now restart the FTP server
service proftpd restart
c. Accessing the FTP server

To access, either through the browser or a dedicated FTP client like FileZilla, you simply need to use the URL ‘ftp://example.com’ with the default port of 21.

Note that this instruction covers the insecure FTP protocol which transfers username and password using plain text. So it is not a great idea to use it on a website that is known to others. In that case, it a much better idea to use the SFTP protocol.

If you are sticking with insecure FTP, then it would be a good idea to use a separate login to do so. You can create a new user using the following command:
useradd username
Then, configure the password using:
passwd userpassword
Finally set the home directory using:
mkhomedir_helper username
As a final step, you should configure the torrent download directory through the GUI so that it is the same as this new user directory, in which case you will have direct access to the files each time you login using FTP.

With this you have a seamless 24x7 setup to file sharing and downloading at the maximum speed that your connection can afford. Going a step further, you can automate and schedule downloading of torrents on the remote machine and syncing them on your local machine but that may be a tutorial for some other day.