Tutorial #20: Home (Network) Improvement using a Pi


Update #1 (Aug 19, 2018):

With the passage of time, things change and in this case it is mostly for the better. I was setting up the Pi once again and was pleasantly surprised that some of the workarounds mentioned previously are no longer needed. Thus, I have edited the original tutorial to accommodate these changes while striking out the old text.

Also, after comparison, I found AB-Solution to be a better solution for network-wide ad blocking, if you happen to have a Merlin supported Asus router like I do. Since the router is on 24x7, having an old ext2 formatted pen drive plugged in to the router itself and running ab-solution is a better alternative as against running the Pi 24x7. It also has various preset ad blocking files that suit different needs while pixelserv-tls does a great job with HTTPS ads.

Lastly, I failed to mention the option of having a Samba server running on the Pi itself in order to access the files directly from the pen drive over the network. This can be accomplished by simply following the official tutorial on the Raspberry Pi website.

Original Article (May 3, 2018):

I had previously shared some tutorials in setting up the Pi and putting it to good use. However, the use cases I mentioned then have ceased to exist. The Fire TV has taken care of most of my multimedia needs and I have come to realise that I really don't have much time to go back in time for nostalgia. For the retro needs that remain, the lesser-used Core M 5Y10 equipped Windows tablet of mine does a much better job plugged in to the TV.

Hence, it is time to put the Pi to good use in a different sense. Thankfully, the versatility of the Pi means that it is not difficult to identify its next project. Ads can become a nuisance, especially for the more aged members of the family and hence my first intent was to set up an ad blocker across my home network. However, putting the Pi to such limited use and keeping it on 24/7 would be quite a waste, so I decided to also repurpose it as a download box with centralised storage.

Setting up the tools necessary to accomplish these tasks seemed straightforward. However, the relevancy of publicly available tutorials diminish over time due to changes in technology. In fact, I had to put quite some effort beyond the listed tutorials and hence I have decided to put the same to words for posterity.

Before starting out:

PINN is a great utility when multi-booting across different distributions on the same SD card. However, Raspbian alone fits the bill for the current use case. Hence, writing the raw Raspbian image directly on the card is preferable as it provides more usable space. As far as writing on the card goes, Etcher is the way to go.

1. Pi-hole®

As the website so prominently displays, all you need is a single command.

curl -sSL https://install.pi-hole.net | bash
However, I made a couple of settings that are worth mentioning:

a. The predefined list of upstream DNS providers does not yet include Cloudflare which I found to be the fastest of the lot. Hence, it would be worthwhile to use the custom option and enter the Cloudflare DNS Server IP addresses of 1.1.1.1 and 1.0.0.1.

b. The other part of the equation is setting up the home equipment to use the local DNS server. In case of Asus routers, this implies changing the DNS Server IP address to the Pi-hole one, not only on the WAN page (under WAN DNS Settings) but also under LAN > DHCP Server > DNS and WINS Server Setting. Make sure that there are no other IP addresses present in either of the pages. You could also run the DHCP server on Pi-hole, in which case the latter setting is not needed. However, since I use the router-assigned IP addresses for other functions (eg. VPN), I prefer to have the DHCP server running on the router itself.

2. qBittorrent

qBittorrent has been my preferred Bittorrent client for quite some time with it being open-source and having proper support for proxies as against Transmission. It can be installed easily using APT, though I prefer the headless route.

sudo apt-get install qbittorrent-nox
My primary endeavour was to have the downloaded files ready on the USB 3.0 hard disk connected to my router (and thus acting like a NAS) while minimising the read-write operations. Since it is not a great idea to write to the SD card running the client, I decided to plug in an old 32 GB pen drive to act as the "working folder" by adding it under Downloads > Hard Disk > Keep incomplete torrents in:

The next part was to add the network drive as the final resting place by entering its address under  Downloads > Hard Disk >  Save files to location: and also under the Copy .torrent files for finished downloads to field. The latter is sometimes necessary due to some quirks in different versions of qBittorrent. The external network drive needs to be mounted within Raspbian on boot which can be accomplished by editing /etc/fstab with these details:

mount -t cifs //xx.xx.xx.xx/folder /media/NAS -o rw,vers=2.0,username=abc,password=xyz
//xx.xx.xx.xx/folder /media/NAS cifs vers=1.0,username=abc,password=xyz,x-systemd.automount

where,
xx.xx.xx.xx -> LAN IP address as configured on the router running the Samba server
folder -> Path to the folder on the network drive that needs to be mounted
/media/NAS -> Path on the Pi where the network drive is to be mounted

In my case, I had to specifically mention the SMB version (2.0), without which the mounting would fail as well as the rw argument to be able to write to the device. Also. it is a good idea to update the cifs-util package from APT prior to editing the fstab file, as mentioned above.

Note: The earlier entry no longer worked with the June 2018 version of Raspbian due to which I had to use the alternative entry in /etc/fstab mentioned above. Also, I was only able to get v1.0 working this time despite the server supporting v2.0 as well.

Finally, to cover for unexpected reboots, it is preferable to have qBittorrent autostart which can be accomplished using systemd. First, create the startup script using:

sudo nano /etc/systemd/system/qbittorrent.service
Next, enter its contents as follows:

[Unit]
Description=qBittorrent Daemon Service
After=network.target
[Service]
User=pi
ExecStart=/usr/bin/qbittorrent-nox
ExecStop=/usr/bin/killall -w qbittorrent-nox
[Install]
WantedBy=multi-user.target 
Lastly, enable the script.

sudo systemctl enable qbittorrent
3. pyLoad

The final part of the exercise was to set up a download manager. I had briefly given thought to JDownloader but decided against running JRE just for it. Hence, I opted for pyLoad instead. The tutorial listed over here works fine for the most part but needed quite some tweaks along the way. For the sake of completion, I will list all the steps in brief.

1. Create system user for pyload

sudo adduser --system pyload
2. Edit /etc/apt/sources.list to be able to install the dependencies. For Raspbian Stretch, the source URLs are as follows:

deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi
deb-src http://archive.raspbian.org/raspbian/ stretch main contrib non-free rpi
3. Update package list and install dependencies.

sudo apt-get update
sudo apt-get -y install git liblept4 python python-crypto python-pycurl python-imaging tesseract-ocr zip unzip python-openssl libmozjs-24-bin
sudo apt-get -y build-dep rar unrar-nonfree
sudo apt-get source -b unrar-nonfree
sudo dpkg -i unrar_*_armhf.deb
sudo rm -rf unrar-*
4.  Create symlink to get "spidermonkey" working.
cd /usr/bin
sudo ln -s js24 js
5. Unlike in the linked tutorial, I had to first switch to the stable branch before I could initiate the pyLoad setup. This is done as follows:

cd /opt
sudo git clone https://github.com/pyload/pyload.git
cd pyload
git branch -r
git checkout stable 
 6. The next step, as per the linked tutorial, should have been the initiation of the pyLoad setup using:


sudo -u pyload python pyLoadCore.py
However, doing so produced the following error: "ImportError: No module named pycurl". Hence the next logical step was to install pycurl:


sudo pip -v install pycurl --upgrade
This in turn resulted in the error: "InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-IsWfyN/pycurl/". This was resolved by:


sudo apt-get install libcurl4-gnutls-devpip
 As you can now guess, this in turn resulted in yet another error: "Failed building wheel for pycurl" which was remedied as follows:


sudo apt-get install libgnutls28-dev

After all this effort, I was finally able to install pycurl using the command mentioned previously...


sudo pip -v install pycurl --upgrade 
...and execute the pyLoad setup using a slightly shorter command:


 python pyLoadCore.py
On the June 2018 release, I was able to get pyload wizard started by executing
sudo -u pyload python pyLoadCore.py 

The final tweak was to get pyLoad running on boot with the commands executed in a manner similar to what has been already covered for qBittorrent.

sudo nano /etc/systemd/system/pyload.service 
[Unit]
Description=Python Downloader
After=network.target
[Service]
User=pyload
ExecStart=/usr/bin/python /opt/pyload/pyLoadCore.py
[Install]
WantedBy=multi-user.target
 sudo systemctl enable pyload.service
Beyond Home

The benefits of this setup can be extended beyond the home network, though a lot depends on the vagaries of the network setup.

Pi-hole can be put to use on external networks by accessing the home network over OpenVPN, though speed and latency might be factors to consider. It can also be setup as a Public DNS but it is extremely risky and not at all recommended.

qBittorrent and pyLoad can be simply accessed using the IP address and port, provided they have been setup to be accessible from outside the LAN. For dynamic IP addresses, the Dynamic DNS (DDNS) option available on most routers can be put to use and my suggestion would be to pick up a cheap $1 domain from NameCheap for this purpose. However, if you happen to be under a multi-layered NAT network under ISP control like me, then there is no option other than to pay for a static IP for public access.

Thankfully, there is a last resort to access the Pi over a public network. The licensed copy of RealVNC that comes with Raspbian offers Cloud Connect that enables one to remotely control the OS and thereby all applications on it. It is quite cumbersome to use if your intent is to only load some links on qBittorrent or pyLoad, but it is better than nothing.

Thus, the Pi can be extremely useful even when used in a rather sedentary capacity and you grow to appreciate the efforts that everyone has put in to make this possible.

Review #29: TP-Link TL-WN823N USB WiFi Adapter (A Pi perspective)

The Raspberry Pi is always a moving target in terms of projects as you may have read previously. The dumb Ethernet is sometimes the best option for the Pi as it doesn't require any configuration at all. I preferred this option as well when the Pi was sitting close to my router and was interfaced with the PC monitor but its use as a media centre meant the proximity changed towards the living room TV. Samsung's TV WiFi adapter proved useful in this regard as it had dual band support and was based on the common Realtek RTL8188 chipset which has wide support across Linux distributions. However, the TV gained its smartness after a long time as the Netflix, Amazon Prime Video and iPlayer apps were finally put to good use. This meant the search for a new USB WiFi adapter for the Pi.

The intention was to purchase the fastest budget adapter and the TP-Link TL-WN823N popped up at the top since it was priced lower than comparable N150 adapters. The one available on Amazon is the v2 variant with the RTL8192EU chipset. Initially, I simply replaced the Samsung adapter with this one when using OSMC and it worked fine. However, later on when I booted to Rapbian, it didn't indicate the presence of a wireless interface. It was then that I realized that most of the information on the web about the compatibility of this adapter with Linux is based on the 3.x kernels and there is no out of the box support with any 4.x kernels.

It was easy enough to load the drivers on Raspbian using the information available on the Pi forums and it worked flawlessly. However, other distributions like LibreELEC or Lakka were a different ball game as they used different kernels. In case of LibreELEC, the 8192eu drivers are already present but don't work with this adapter even after adding the device ID. In case of Ubuntu, there seem to be driver packages available but I haven't tested the same. Even then, it will require patching with every kernel update and isn't the "just works" solution that most would be looking for.

Thus, it seems best to stick with the well proven chipsets from the past. A N150 adapter is still faster than the 100 Mbps Ethernet port and much faster than the reading/writing speeds of most SD cards, so you are not giving up much. Sometimes, it is better to just take a step back to move forwards.

Tutorial #9: The perfect (IMO) multi-boot option for the Raspberry Pi

I had previously stated the myriad of ways in which the Raspberry Pi can be put to use. While utilizing separate SD cards for each OS is a viable solution, it is a bit cumbersome and perhaps expensive if you don't have a lot of spare SD cards lying around. Hence, what's better than being able to run a whole bunch of OSes from a single card and being able to switch seamlessly between them. A point to note is that a LOT depends on the speed of the SD card and I have settled on using the Samsung Evo + which might be a bit of an overkill for the Pi but it ensures the fastest experience.

In my case, I had the following requirements of a multi-boot system.
1. First and foremost, Kodi. It is a powerful media player by itself but what makes it truly worthwhile are the countless add-ons that significantly boost its functionality. The Pi makes for a formidable media centre with its hardware x264 decoding and finds a lot of its utility online.
2. An OS to execute the various Pi projects. A CLI may do but a GUI makes things a lot easier.
3. A retro games emulator. Who can beat nostalgia?

While there are multiple ways to fulfill each of the requirements I mentioned above, my evaluation lead me to settle on the following:

1. Kodi - OSMC: OpenELEC, LibreELEC and OSMC are the standalone Kodi OSes available and while Kodi is Kodi, there are still some pros and cons to each. OpenELEC and LibreELEC operate on a "just enough OS" principle wherein the installation contains no extra fluff. This certainly speed things up but may limit the extent to which you can play around in the background with SSH. LibreELEC is a fork of OpenELEC and the situation is akin to what happened with Open Office wherein most developers have eloped to the "Libre" camp, so one might imagine that it would see more proactive development going ahead. However, after trying out LibreELEC, I settled with OSMC, mainly because I was having a hell of a time with Bluetooth on LibreELEC. Also, the OSMC skin looks much more refined and the vertical interface is much easier to use when you are left dangling with a TV remote. Moreover, while I have only used SSH to enable OpenVPN, I am pretty sure that OSMC will allow for a lot more tinkering that the "ELECs" if need be.

2. OS - Raspbian: Yup, I settled with the de-facto OS. The project support for this is unparalleled which certainly makes things a lot easier. Ubuntu Mate comes in at a close second if you are looking for a true "desktop" experience but it comes at the expense of speed even though it is compiled for ARM v7 as the MATE desktop environment is more resource hungry. Lubuntu is a good alternative but you lose out a lot of the Ubuntu look and easy customization. Having said that, the Pixel desktop environment has made Raspbian a lot more welcoming and easy to use. The built-in license for RealVNC as well as the GUI for Raspi config are like icing on the cake.

3. Retro gaming console - Recalbox: RetroPie is the more popular option here but in my testing, I found it to behave a bit irrationally. The configuration done for RetroPie didn't map properly to many of the emulators. Also, the bluetooth interface kept on throwing errors wherein I couldn't remove an existing device because it wasn't listed and neither could I add it since it was stated to be already present. In that sense, Recalbox is more user friendly. The interface does hide more of the advanced options for RetroArch, but it worked well for me out of the box and was more reliable in general.

With my options in place, the next thing was getting them to play ball on a single SD card. To that end, I considered the following options:

1. NOOBS: The option that the Raspberry Foundation themselves provide. It is quite easy to add offline OSes by adding them to the "os" folder on the SD card and then selecting them at the time of installation. The downside is that by default the OS selection requires the use of a keyboard since it doesn't support HDMI-CEC. Since I was looking at a wireless option, I had to leave this one out in its unadulterated form.

2. BerryBoot: It certainly solves the OS selection problem by supporting HDMI-CEC and allowing the use of the TV remote to select an OS. The OS images have to be modified to be compatible with BerryBoot but thankfully there is an updated repository of all prominent OSes maintained here. Unfortunately there are some limitations to this approach. It uses SquashFS, a compressed file system that limits performance and also prevents one from making automated system level updates.

3. Multi Boot Pi: This eponymous option is the first thing that pops up on Google and is indeed a great option built on NOOBS. The built-in OS switcher makes it easy to switch between OS and also set the default boot option. I had selected the latest available download at the time of this writing which offers quad boot support. I presumed that being based on NOOBS, I can avoid installing RasPlex and that doing so would adjust the menu entries in the respective OSes. However, that is not the case. RasPlex hangs around no matter what and requires manual editing to remove the references. Also, updating the various OSes led to a change in the look of the Kodi Program launcher for the other OSes and I presume future updates might break it altogether. Even otherwise, the RetroPie option that this goes with just wouldn't work for me.

4. Matt Huisman's Dual Boot: This is built on the same principle as Multi Boot Pi, but includes Recalbox by default which is what I prefer. Although the default installation is for dual boot, adding Raspbian was as easy as copying the folder that comes with the Raspbian NOOBS zip. It boots to OSMC by default which is again as per my preference, though that can be edited easily as well. The site also includes the necessary scripts including the one on adding Raspbian to Kodi, so any unexpected link breakages can be easily rectified. Having said that, I updated all the OSes without any adverse effects. It would also be pertinent to note that you can and should adjust the partition sizes for each OS as per your requirement. I have kept about 2 GB for OSMC, 8 GB for Recalbox and the rest for Raspbian.

So, that's it as far as my setup is concerned. It strikes the perfect balance between all that I require of the Pi with convenience to boot. (pun intended.)

Tutorial #3: Starting out with the Raspberry Pi

I have been playing around with the Raspberry Pi 2 for a year now and I can’t even remember the count for the number of times I have changed the OS. The beauty of the Pi is that you are never short of any options for there is always someone creating something new for it. The best use for the Pi will always be the maker projects should you partake in them for it certainly brings in a different form of satisfaction. 

Generally, there are 2 ways that an OS can be loaded on to the memory card. The ones aimed at new users (like the Raspbian NOOBS installer) have a simple extract to memory card approach. The slightly advanced way of installing the OS is to write the image to the SD card. On Linux and Mac OS X, you can do so directly using the dd command but doing so on Windows requires a few more tools. You would do well to keep the following handy:


While the first is used for SDCard specific formatting, the second allows you to write the image on the SD Card. One experience that I would like to share is that if you frequently have the SDCard ejecting itself when writing the image, then the fault solely lies with the fact that you are using a cheap card writer for there is nothing wrong with the Win32DiskImager utility.

Having covered the installation process, here are a few general purpose things that you can engage your Pi in:

1. A fully functional PC

The Pi can function as a desktop replacement, should you choose to use it as such. However, it requires considerable patience considering the hardware on offer and more specifically the speed of the SD Card in use. Having said that, it is a good platform to mess with and can be used to learn Linux, programming or to introduce someone to the world of computing.  The debate over an ARM v6 and v7 OS will always rage on and Raspbian admittedly offers a better starting experience. However, Ubuntu MATE for the Pi certainly offers a more mainstream experience akin to a normal Linux desktop. It comes at a cost though for it is significantly taxing on the Pi hardware and for this reason, you may choose to move to a lighter desktop like LXDE, thereby creating a Lubuntu OS. Previously, I used to install Ubuntu MATE and then switch to LXDE desktop using the following commands which should be familiar to anyone remotely conversant with Linux:
sudo apt-get update && sudo apt-get -y upgrade && sudo apt-get -y dist-upgrade
sudo apt-get -y install lubuntu-desktop lxde
sudo apt-get remove --purge ubuntu-mate-core ubuntu-mate-desktop
sudo apt-get remove mate-*
sudo apt-get autoremove
However, I recently came across a site which gives you the choice of Ubuntu flavour out of the box (https://ubuntu-pi-flavour-maker.org/). It is certainly a very convenient option, especially when dealing with Bluetooth and WiFi with support for some of the functions that come built-in with Raspbian. My recommendation would be to go with Lubuntu.

2. Media Centre

The Pi GPU has remained constant over the generations as we come to the Pi 3, but it is still a capable one for it supports hardware decoding for H.264. That makes it a very capable media centre or HTPC. My experience with high bit rate Full HD streams on the Pi 2 has been choppy to say the least so I won’t really consider this to be a very good HTPC option when you take in to consideration the fact that H.265 seems a better compression option when you are starving on space. The Pi 3 should be more capable on this front in dealing with high bit rate H.264 streams and should you use it a media centre, you won’t be short on options. Having said that, the only real option that you have is to use some derivative of Kodi or Plex if you are in to that ecosystem. 

For me OpenElec and OSMC offered the best stepping stone in to the Media Centre world. I say so because Kodi is immensely customizable and you can’t go wrong starting with either.

3. Retro Arcade 

Emulation has always been a grey area when it comes to legality but you can always re-live your childhood nostalgia using them. The most common option here is RetroPie that uses RetroArch as the back-end and Emulation Station as the front end. The same combination is put to use by some others like Recalbox which is significantly more user friendly. These are supposed to be the most optimised for the Pi but unfortunately and I couldn’t get my Xbox 360 controller to work reliably across all emulators, though in fairness I have been using the test versions. Other options include Lakka which does a different take by being based on Libretro thereby imparting a much more fluid UI and PiPlay with which I haven’t spent much time to comment on. I would recommend Recalbox as the first option, failing which you may try out Lakka.

This article just glosses over some prominent uses for the Pi but as you can see, you will never be short of things to do when dealing with any iteration of the Pi. You can easily swap OSes by using multiple SD cards (easy option) or using something like BerryBoot (a bit more difficult option with a slower SquashFS file system).  With the Pi, the force is always with you.