Tutorial #20: Home (Network) Improvement using a Pi


Update #1 (Aug 19, 2018):

With the passage of time, things change and in this case it is mostly for the better. I was setting up the Pi once again and was pleasantly surprised that some of the workarounds mentioned previously are no longer needed. Thus, I have edited the original tutorial to accommodate these changes while striking out the old text.

Also, after comparison, I found AB-Solution to be a better solution for network-wide ad blocking, if you happen to have a Merlin supported Asus router like I do. Since the router is on 24x7, having an old ext2 formatted pen drive plugged in to the router itself and running ab-solution is a better alternative as against running the Pi 24x7. It also has various preset ad blocking files that suit different needs while pixelserv-tls does a great job with HTTPS ads.

Lastly, I failed to mention the option of having a Samba server running on the Pi itself in order to access the files directly from the pen drive over the network. This can be accomplished by simply following the official tutorial on the Raspberry Pi website.

Original Article (May 3, 2018):

I had previously shared some tutorials in setting up the Pi and putting it to good use. However, the use cases I mentioned then have ceased to exist. The Fire TV has taken care of most of my multimedia needs and I have come to realise that I really don't have much time to go back in time for nostalgia. For the retro needs that remain, the lesser-used Core M 5Y10 equipped Windows tablet of mine does a much better job plugged in to the TV.

Hence, it is time to put the Pi to good use in a different sense. Thankfully, the versatility of the Pi means that it is not difficult to identify its next project. Ads can become a nuisance, especially for the more aged members of the family and hence my first intent was to set up an ad blocker across my home network. However, putting the Pi to such limited use and keeping it on 24/7 would be quite a waste, so I decided to also repurpose it as a download box with centralised storage.

Setting up the tools necessary to accomplish these tasks seemed straightforward. However, the relevancy of publicly available tutorials diminish over time due to changes in technology. In fact, I had to put quite some effort beyond the listed tutorials and hence I have decided to put the same to words for posterity.

Before starting out:

PINN is a great utility when multi-booting across different distributions on the same SD card. However, Raspbian alone fits the bill for the current use case. Hence, writing the raw Raspbian image directly on the card is preferable as it provides more usable space. As far as writing on the card goes, Etcher is the way to go.

1. Pi-hole®

As the website so prominently displays, all you need is a single command.

curl -sSL https://install.pi-hole.net | bash
However, I made a couple of settings that are worth mentioning:

a. The predefined list of upstream DNS providers does not yet include Cloudflare which I found to be the fastest of the lot. Hence, it would be worthwhile to use the custom option and enter the Cloudflare DNS Server IP addresses of 1.1.1.1 and 1.0.0.1.

b. The other part of the equation is setting up the home equipment to use the local DNS server. In case of Asus routers, this implies changing the DNS Server IP address to the Pi-hole one, not only on the WAN page (under WAN DNS Settings) but also under LAN > DHCP Server > DNS and WINS Server Setting. Make sure that there are no other IP addresses present in either of the pages. You could also run the DHCP server on Pi-hole, in which case the latter setting is not needed. However, since I use the router-assigned IP addresses for other functions (eg. VPN), I prefer to have the DHCP server running on the router itself.

2. qBittorrent

qBittorrent has been my preferred Bittorrent client for quite some time with it being open-source and having proper support for proxies as against Transmission. It can be installed easily using APT, though I prefer the headless route.

sudo apt-get install qbittorrent-nox
My primary endeavour was to have the downloaded files ready on the USB 3.0 hard disk connected to my router (and thus acting like a NAS) while minimising the read-write operations. Since it is not a great idea to write to the SD card running the client, I decided to plug in an old 32 GB pen drive to act as the "working folder" by adding it under Downloads > Hard Disk > Keep incomplete torrents in:

The next part was to add the network drive as the final resting place by entering its address under  Downloads > Hard Disk >  Save files to location: and also under the Copy .torrent files for finished downloads to field. The latter is sometimes necessary due to some quirks in different versions of qBittorrent. The external network drive needs to be mounted within Raspbian on boot which can be accomplished by editing /etc/fstab with these details:

mount -t cifs //xx.xx.xx.xx/folder /media/NAS -o rw,vers=2.0,username=abc,password=xyz
//xx.xx.xx.xx/folder /media/NAS cifs vers=1.0,username=abc,password=xyz,x-systemd.automount

where,
xx.xx.xx.xx -> LAN IP address as configured on the router running the Samba server
folder -> Path to the folder on the network drive that needs to be mounted
/media/NAS -> Path on the Pi where the network drive is to be mounted

In my case, I had to specifically mention the SMB version (2.0), without which the mounting would fail as well as the rw argument to be able to write to the device. Also. it is a good idea to update the cifs-util package from APT prior to editing the fstab file, as mentioned above.

Note: The earlier entry no longer worked with the June 2018 version of Raspbian due to which I had to use the alternative entry in /etc/fstab mentioned above. Also, I was only able to get v1.0 working this time despite the server supporting v2.0 as well.

Finally, to cover for unexpected reboots, it is preferable to have qBittorrent autostart which can be accomplished using systemd. First, create the startup script using:

sudo nano /etc/systemd/system/qbittorrent.service
Next, enter its contents as follows:

[Unit]
Description=qBittorrent Daemon Service
After=network.target
[Service]
User=pi
ExecStart=/usr/bin/qbittorrent-nox
ExecStop=/usr/bin/killall -w qbittorrent-nox
[Install]
WantedBy=multi-user.target 
Lastly, enable the script.

sudo systemctl enable qbittorrent
3. pyLoad

The final part of the exercise was to set up a download manager. I had briefly given thought to JDownloader but decided against running JRE just for it. Hence, I opted for pyLoad instead. The tutorial listed over here works fine for the most part but needed quite some tweaks along the way. For the sake of completion, I will list all the steps in brief.

1. Create system user for pyload

sudo adduser --system pyload
2. Edit /etc/apt/sources.list to be able to install the dependencies. For Raspbian Stretch, the source URLs are as follows:

deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi
deb-src http://archive.raspbian.org/raspbian/ stretch main contrib non-free rpi
3. Update package list and install dependencies.

sudo apt-get update
sudo apt-get -y install git liblept4 python python-crypto python-pycurl python-imaging tesseract-ocr zip unzip python-openssl libmozjs-24-bin
sudo apt-get -y build-dep rar unrar-nonfree
sudo apt-get source -b unrar-nonfree
sudo dpkg -i unrar_*_armhf.deb
sudo rm -rf unrar-*
4.  Create symlink to get "spidermonkey" working.
cd /usr/bin
sudo ln -s js24 js
5. Unlike in the linked tutorial, I had to first switch to the stable branch before I could initiate the pyLoad setup. This is done as follows:

cd /opt
sudo git clone https://github.com/pyload/pyload.git
cd pyload
git branch -r
git checkout stable 
 6. The next step, as per the linked tutorial, should have been the initiation of the pyLoad setup using:


sudo -u pyload python pyLoadCore.py
However, doing so produced the following error: "ImportError: No module named pycurl". Hence the next logical step was to install pycurl:


sudo pip -v install pycurl --upgrade
This in turn resulted in the error: "InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-IsWfyN/pycurl/". This was resolved by:


sudo apt-get install libcurl4-gnutls-devpip
 As you can now guess, this in turn resulted in yet another error: "Failed building wheel for pycurl" which was remedied as follows:


sudo apt-get install libgnutls28-dev

After all this effort, I was finally able to install pycurl using the command mentioned previously...


sudo pip -v install pycurl --upgrade 
...and execute the pyLoad setup using a slightly shorter command:


 python pyLoadCore.py
On the June 2018 release, I was able to get pyload wizard started by executing
sudo -u pyload python pyLoadCore.py 

The final tweak was to get pyLoad running on boot with the commands executed in a manner similar to what has been already covered for qBittorrent.

sudo nano /etc/systemd/system/pyload.service 
[Unit]
Description=Python Downloader
After=network.target
[Service]
User=pyload
ExecStart=/usr/bin/python /opt/pyload/pyLoadCore.py
[Install]
WantedBy=multi-user.target
 sudo systemctl enable pyload.service
Beyond Home

The benefits of this setup can be extended beyond the home network, though a lot depends on the vagaries of the network setup.

Pi-hole can be put to use on external networks by accessing the home network over OpenVPN, though speed and latency might be factors to consider. It can also be setup as a Public DNS but it is extremely risky and not at all recommended.

qBittorrent and pyLoad can be simply accessed using the IP address and port, provided they have been setup to be accessible from outside the LAN. For dynamic IP addresses, the Dynamic DNS (DDNS) option available on most routers can be put to use and my suggestion would be to pick up a cheap $1 domain from NameCheap for this purpose. However, if you happen to be under a multi-layered NAT network under ISP control like me, then there is no option other than to pay for a static IP for public access.

Thankfully, there is a last resort to access the Pi over a public network. The licensed copy of RealVNC that comes with Raspbian offers Cloud Connect that enables one to remotely control the OS and thereby all applications on it. It is quite cumbersome to use if your intent is to only load some links on qBittorrent or pyLoad, but it is better than nothing.

Thus, the Pi can be extremely useful even when used in a rather sedentary capacity and you grow to appreciate the efforts that everyone has put in to make this possible.

Tutorial #19: Optimally managing photos and videos on iOS


Apple’s anaemic storage options (specifically at the lowest tier) have been a running joke for a majority of the iPhone’s existence. I was at the receiving end of Apple’s largesse with the entry-level iPhone 7 switching to 32 GB which was further enhanced to 64 GB with the iPhone 8. Even with these storage options, one can easily fill it up with content other than captured photos and videos. Also, it is not a sound idea to have all your files stored locally on the device, irrespective of its storage capacity.

Apple provides a few options to mitigate the storage issues resulting from ever larger multimedia content. These are as follows:

1. The default option that most may take recourse to is the iCloud Photo Library. However, Apple provides a meagre 5 GB of storage for starters and as is typical of the company, you are expected to pay more to use this option practically. It only makes sense to go with Apple’s cloud if you live on it through other devices in the Apple ecosystem, like the Mac or iPad. The more important thing to note here is that by default Apple syncs your local photo library with the iCloud one, so you can end up permanently deleting your photos from the device as well as the cloud if you are not paying attention.

2. With the release of iOS 11, Apple introduced the high-efficiency formats, HEVC for videos and HEIF for images, that significantly reduce the file size on modern iDevices. The down side to this is that compatibility for this format is still not standard across platforms and devices. Most notably, the HEIF format is not yet natively supported by Windows. Even within Apple’s ecosystem, sharing the images or videos with older devices necessitates a conversion which takes up time as well as processing power on the mobile device.

3. Lastly, iOS also provides an ‘Optimize Storage’ option that keeps a lower quality version of the image on the phone for immediate viewing purpose while retrieving the full quality image from iCloud. This helps in dealing with storage issues but yet again results in the usage of additional time and data.

Luckily for iOS users, there are several third-party options available that allow one to back-up and retrieve photos and videos without having to pay or worry about running out of storage. After using quite a few options, I have shortlisted two well-known ones that together offer an unbeatable 1-2 combination. They are Flickr and Google Photos.

Before starting out, I would recommend that you go to Camera > Formats and select the ‘Most Compatible’ option which uses JPEG/H.264 instead of HEIF/HEVC. This ensures that the images are available for use without any conversion and accessible on all platforms. It will, of course, take up additional space but since we are offloading most of the stuff on to the cloud anyway, storage isn’t a constraint. On the other hand, data usage can be a constraint if you are limited to a cellular network, but the solution here ensures that even that eventuality is covered. As for the ‘Optimize Storage’ option, you can leave it enabled as iOS always provides the full quality image to any other app that requests it.

Our primary solution to the image storage problem is Flickr. One can argue that Flickr has seen better days and the Yahoo hacks might have left a few people dishevelled. Many photographers might have a preference for 500px as well, but that doesn’t take anything away from Flickr as far as our use case is concerned. Assuming that Oath (Verizon) wouldn’t bring about any drastic storage policy changes, Flickr offers the best value proposition for free users. The 1000 GB of storage space is unprecedented and the photography focus of the site is much better for image/photo management compared to a paid, storage-only option like OneDrive.

While Flickr has moved some of its tools like the desktop ‘Uploadr’ under the Pro banner, the iOS app is unaffected. It is capable of syncing with the iOS Photo Library and more importantly, uploading the original image to the cloud. It does not however support the HEIF format as is evident when you try to upload these images over the website. On iOS however, the images in the Photo Library are still uploaded after conversion to JPEG. Hence, I have previously recommended the usage of the ‘Most Compatible’ option to prevent unnecessary conversions. Unfortunately, Flickr doesn’t allow the segregation between photos and videos when uploading over a cellular connection and hence I would recommend syncing only over WiFi, unless you have an uncapped cellular connection.

The sidekick to our protagonist Flickr is Google Photos. On its own, Google Photos is an awesome product. However, ‘original quality’ images and videos are limited to the storage available on Google Drive for non-Pixel users, which in most cases is 15 GB. Luckily, Google offers an unlimited ‘High Quality’ option, which one should note, significantly compresses the image. However, thanks to clever use of algorithms and machine learning, the effects are not visible on most devices unless the image is blown up significantly.

As a secondary backup solution, Google Photos offers some distinct advantages. Firstly, it caches lower quality variants of all the images so that the entire library is accessible even when you are offline. Secondly, it offers smaller-sized files on account of the compression and resolution limitations of 16 MP/1080p, which is useful when accessing or sharing something over a cellular connection on social media. Thirdly, it allows photos and videos to be synced separately over WiFi and cellular connections, so that images can be synced immediately while larger videos can be uploaded later over WiFi. Fourthly, once images are backed up, they can be easily deleted from the device (and iCloud) using the ‘Free up space’ option. However, for this, you should ensure that the original images are first uploaded to Flickr. Lastly, the machine-learning powered search is really useful in unearthing hidden images and recreating past memories.

Thus, the combination of Flickr and Google Photos ensures that you have all your images and videos backed up regularly with redundancy and available on demand. While Flickr provides the long-term, original quality storage; Google Photos complements it with smaller-sized, optimized content for on-the-go consumption. It completely cuts off iCloud from the picture and ensures that you more storage available on your device for things that you need and use far more regularly.

Tutorial #18: Unlocking the bootloader on Redmi Note 3

As I had mentioned in my review of the Redmi Note 3, it was good value for money. However, MIUI proved to be a hindrance for the target user because of which I had switched the device to LineageOS while not rooting it and keeping the bootloader locked. However, with the phone now back in my hands, it was time to break the shackles for good.

To Xiaomi's credit, they have an official process in place for unlocking the bootloader. However, it has its quirks and more often than not following the official guide results in the process being stuck at 50% due to incompatibilities. This was the case with my first attempt and hence I decided to proceed with it unofficially.

As always, I headed to XDA to quickly gather the process for this device. However, the process seems to be a bit outdated and perhaps a bit difficult to follow for the uninitiated, so I have listed the steps undertaken by me. There could be other ROM versions or files you can use but I have mainly picked up the ones from the XDA thread linked above which you might visit in case you need visual reference.

Note: Prior to starting with the flashing process or even after flashing the MIUI ROM in Step 4, make sure you have the 'OEM unlocking' option selected under Developer Options, without which the fastboot unlocking will fail.

Step 1: Download and extract/install the following:
1. MIUI Global ROM v7.2.5.0 (You will have to extract the file twice to get the folder contents)
2. Mi Flash Tool (Official MI tool to flash ROMs - used v2017.7.20.0 at the time of writing)
3. Unlocked emmc_appsboot file (Primary file needed to unlock bootloader)
4. EDL Fastboot (To enter emergency download mode)
5. Minimal ADB and Fastboot (Installs necessary ADB and fastboot drivers)
5. TWRP Recovery (Gateway to flashing anything on the device)

Step 2:
Browse to the 'images' folder within the extract ROM folder (kenzo_global_images_V7.2.5.0.LHOMIDA_20160129.0000.14_5.1_global) and replace the emmc_appsboot.mbn file in that folder with the downloaded one.


Step 3:
After installing the Minimal ADB and Fastboot drivers, connect the phone and run 'edl.cmd' from within the extracted 'fastboot_edl' folder to boot the phone to the emergency download (EDL) mode. If you don't, then the Mi Flash tool may give a 'tz error'. This mode can be recognised by the flashing red LED on the device.


Step 4:
Run the Mi Flash tool using 'XiaoMiFlash.exe' and select the folder containing the extracted ROM files (kenzo_global_images_V7.2.5.0.LHOMIDA_20160129.0000.14_5.1_global). Clicking on 'Refresh' should list the device and then subsequently, click on 'Flash'. The process will take 4-5 minutes to complete after which you will be able to see the 'success' status.

Step 5:
Boot the phone to the normal fastboot mode using the Volume Down + Power button. Open a command prompt window and browse to the 'Minimal ADB and Fastboot' folder. Here, execute the following command:
fastboot oem device-info
It should indicate 'Device unlocked: false', following which execute the command:
fastboot oem unlock-go
Running the 'fastboot oem device-info' command once again should now indicate 'Device unlocked: true'. That's it, your device now has an unlocked bootloader.

Step 6:
While not part of the bootloader unlocking process, a follow-up step should be to flash the TWRP recovery which opens a whole window of opportunities. This can be done by copying the 'twrp-3.1.1-0-kenzo' (latest file at the time of writing) to the 'Minimal ADB and Fastboot' folder and running the following command from the cmd window:
fastboot flash recovery twrp-3.1.1-0-kenzo.img
Following this, you will have complete freedom to tinker with the device in any way you deem apt. Oreo, anyone?

Tutorial #17: A batter(!) understanding of dosa economics


Raghuram Rajan has been featuring a lot in the media recently in promotion of his book 'I Do What I Do'. While I am yet to read it, there has been no escaping it as select excerpts and anecdotes have been making their way to the news every day now. Earlier today it was the turn of 'Dosa Economics' on BBC.

On the face of it, it is a simple concept of understanding the real interest rate as against the nominal one. Most people tend to look at interest rate in absolute terms since it is the most visible one and inflation as the silent killer is rarely understood. It was a noble attempt by Rajan at explaining this concept, though how many pensioners received the message even after the simplification of numbers is debatable.

However, I see no reason for Raghuram Rajan to have a monopoly on dosas in economics. Moreover, one would be hard pressed to find a dosa for ₹50 in a city like Mumbai, let alone 1-year fixed deposits at 8% and real-life consumer inflation at 5.5%. So, now you get to create your own realistic dosa economics, provided you have the appetite for it.

Open in new tab

Tutorial #16: The Securities Trade Lifecycle


Trade is one of the basic tenets of investment banking. Yet, detailed information on its lifecycle is not easily forthcoming on the web. There are of course articles available describing the same at a high level, though it seems that few agree on the exact terminologies and sequence to be used. Books on the other hand, even the eponymous ones, divert to discussing trade strategies rather than the lifecycle itself.

They key then seems to be in garnering the details of the process. Having gone through a number of sources, I can recommend the tutorial available on Udemy. It covers the lifecycle starting from trade execution, though some would prefer to start from sale or trade initiation. What's important is that it covers all the steps in sequence and builds up on the details chronologically with illustrative examples. This facilitates a far better understanding than what most words would do.

It is certainly not free, but worth its price or indeed a trial. The tutorial spans over 22 videos (with accompanying PDF files) and may take up to 4 hours of your time, if you are running it an 1x speed. I found it convenient to follow even at 1.6x, so your mileage may vary. While this may sound more of a review than a tutorial, especially compared to my other ones, I found it best to classify it as one since it is aimed at promoting learning more than anything else.

Tutorial #15: How to get official licenses (Windows/Office) for cheap


There was a time when piracy was considered to be a necessity. The unavailability of the software locally along with dollar pricing made it impossible for anyone to even contemplate purchasing the software. However, things have changed a lot since then. Local availability along with local pricing has made these products far more accessible.

But, and the big but, is affordability. The pricing is certainly competitive from a commercial perspective but personal users would still find the price prohibitive, especially when the usage is limited to writing personal documents and filing income tax returns. Piracy can't be condoned, so what other valid options are available?

By valid, I refer to the ability to download and register the software using official sources. I remember getting a Windows 8 license for less than $10 during its launch due to a Microsoft promotion and I wish they were generally generous in their pricing in developing countries. However, I presume Internet anonymity has made it difficult to separate the wheat from the chaff.

Hence, it becomes necessary to take recourse to other options. One of the most prominent ones is the Microsoft Software Swap Marketplace on Reddit, though there are other forums available. The prices are certainly competitive compared to retail pricing but still on the expensive side for those not dealing with USD as local currency. Hence, the best option is to head to good old ebay.com. The price fluctuates from seller to seller and availability is entirely dependent on timing. However, if you are in luck, then the prices range in low single digits, as far as USD is concerned. Local credit cards might not work directly with ebay.com but PayPal comes to the rescue. Do keep in mind to use your bank conversion since the fees are usually much less than PayPal's, the premium mostly being less than 5% depending on the size of the transaction. The proclamation is that these codes have been salvaged from scrapped machines and hence it is legitimate to resell the same. What I can confirm is that the codes work fine with office.com and are instantly redeemed along with the download links for any Microsoft account. Similarly, Windows activates just fine with the supplied key, if used with a fresh installation.

If you prefer Office 365 instead, then there are Educational subscriptions available that offer multiple year access for about the same price. This one certainly feels a bit dodgier because you are restricted to an academic email address being governed by administrators. However, it offers multiple installations and 1TB of SkyDrive space, though it is difficult to trust an address you are not entirely in control of.

Whatever be the case, there are certainly legitimate options available that if nothing else help protect from options that are untrustworthy and laden with malware, at a significantly affordable price.

Tutorial #14: Tips on a fresh installation of Windows Creators Update


For all the cruft that had built up since the Anniversary Update, I decided to do a fresh installation of the Creators Update released earlier this month on my tablet (Dell Venue 11 Pro 7140). Re-installations are much easier now than a decade ago with cloud backups eliminating the worries of losing data. However, it still takes some effort to reduce installation time and to ensure that Microsoft's data acquisition and bloatware installation is limited. Following are my tips learnt from experience.

1. Direct ISO downloads are a thing of the past as far as Microsoft is concerned and it wants you to rely on the Media Creation Tool instead. That's fine for the most part, but it happens to download a device specific ISO. So, if you happen to have multiple Windows devices, especially with different editions of Windows 10, then it is best to download the international ISO.

2. USB installations are undoubtedly faster than disc based ones. However, SSDs are much faster than standard USB flash drives. In my case, I have converted a discarded 64GB M.2 SSD into a USB 3 flash drive which reduces the initial installation process to just 5 minutes.

3. Rufus is by far the best tool to write the ISO to the USB drive. In order to write to SSD flash drives, make sure you enable the 'List USB Hard Drives' option.

4. After installation, when you boot to the profile setup screen, I would recommend not connecting to the Internet. This causes the PC to reboot into the offline setup mode and you don't need to link your Outlook account. It is said that an offline account limits the telemetry sent to Microsoft, though I can't vouch for it personally.

5. For additional privacy, you can disable all the privacy options presented on the setup screen. If you need any of them, you can always enable them later.

6. Make sure you keep the drivers from your manufacturer handy before the installation. You can copy them to the USB drive that you use for the Windows setup. In my case, Dell provides a single CAB file containing all the drivers and I usually place the extracted CAB file on my USB SSD drive for easy access. This enables the manual installation of the correct drivers using the Device Manager.

7. A very important step is to disable installation of hardware drivers from Windows update in case you already have all the manufacturer drivers. In my case, I found that the drivers from Microsoft for my device caused a lot of issues, especially with the display and battery management. Hardware driver installation can be disabled from Advanced System Settings > Hardware > Device Installation Settings.

8. Microsoft also tends to install a lot of sponsored apps like Candy Crush Saga on the device as soon as you connect to the web. Hence, it is a great idea to open the Start Menu and remove all the icons for the apps that are awaiting download. Note that you can only do this in case you didn't connect to the Internet during the setup process.

9. Whenever you login to Microsoft Apps like Mail or OneDrive, make sure that you sign in only to the app and not associate it with the Windows account. This again ensures better privacy and account management.

This just about covers the most important things to keep in mind when undertaking a fresh installation of Windows. It maximises privacy and minimises the conflicts that you may encounter, thereby streamlining the installation process.